vanesamagar
New member
Hello I am running init_atmosphere with MPAS verison 8.1.0 for the 7.5 km quasi-uniform mesh and I get this error. This doen't seem to have been discussed before in the forum. Any suggestions?
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
ERROR: MPAS IO Error: PIO error -237: Size of I/O request exceeds INT_MAX
- Thread starter vanesamagar
- Start date
vanesamagar
New member
Hi, Vanesa,
Thank you for uploading the files. It seems that you are running the ideal case of jw_baroclinic_wave on the 7.5km mesh. Let me know if I am wrong.
Your namelist.init_atmopshere looks fine to me. However, I have one concern of your streams.init_atmosphere. Note that you are running on such a dense mesh, those 3-d fields can easily exceed the 4 GB limit imposed by the classic netCDF format. Therefore, you need to use an "io_type" that supports large variables, such as "pnetcdf,cdf5" or "netcdf4".
Please add io_type="pnetcdf,cdf5" to your output stream, then rerun this case in parallel mode.
I am hesitating to repeat this case because it requires quite a lot of computation resources. I would appreciate if you can let me know whether it works or not.
Thank you for uploading the files. It seems that you are running the ideal case of jw_baroclinic_wave on the 7.5km mesh. Let me know if I am wrong.
Your namelist.init_atmopshere looks fine to me. However, I have one concern of your streams.init_atmosphere. Note that you are running on such a dense mesh, those 3-d fields can easily exceed the 4 GB limit imposed by the classic netCDF format. Therefore, you need to use an "io_type" that supports large variables, such as "pnetcdf,cdf5" or "netcdf4".
Please add io_type="pnetcdf,cdf5" to your output stream, then rerun this case in parallel mode.
I am hesitating to repeat this case because it requires quite a lot of computation resources. I would appreciate if you can let me know whether it works or not.
vanesamagar
New member
Hi,
Yes you are right, I am running the idealized baroclinic wave test case with quasi-uniform 7.5km mesh. or trying to run, I have not managed to produce the *init.nc with any of the experiments I have run so far with the init_atmosphere core.
I have added io_type="pnetcdf,cdf5" in streams.init_atmosphere and streams.atmosphere files. I have also changed the configuration for a parallel run, and I undersubscribe the nodes. ARCHER2 nodes have ~190GB usable memory and 128 cores per node. I attach the mpas_par_init.slurm file, the namelist* and streams* files, and the output files. I dont get an *err file and the long.init_atmosphere.0000.out has no errors, but the slurm-* file prints errors we don't know how to solve. I have used some UCX commands to change from OFI to UCX network setup but this in fact only made the problem worse... Any suggestions?
Thanks,
vmagar
Yes you are right, I am running the idealized baroclinic wave test case with quasi-uniform 7.5km mesh. or trying to run, I have not managed to produce the *init.nc with any of the experiments I have run so far with the init_atmosphere core.
I have added io_type="pnetcdf,cdf5" in streams.init_atmosphere and streams.atmosphere files. I have also changed the configuration for a parallel run, and I undersubscribe the nodes. ARCHER2 nodes have ~190GB usable memory and 128 cores per node. I attach the mpas_par_init.slurm file, the namelist* and streams* files, and the output files. I dont get an *err file and the long.init_atmosphere.0000.out has no errors, but the slurm-* file prints errors we don't know how to solve. I have used some UCX commands to change from OFI to UCX network setup but this in fact only made the problem worse... Any suggestions?
Thanks,
vmagar
Attachments
vmagar,
This is still looks like a memory issue. With the command line in your job script,
It seems that you run this case with a single processor. Again, for such a dense mesh, single processor cannot work.
Please consult your computer manager how to run a job in parallel mode with multiple processors.
I would suggest you start from a smaller case, for example using the 240km mesh. A smaller case can be done quickly and give you general ideas how the model works. After that, you can move on to run big cases.
This is still looks like a memory issue. With the command line in your job script,
Code:
srun ./init_atmosphere_modelIt seems that you run this case with a single processor. Again, for such a dense mesh, single processor cannot work.
Please consult your computer manager how to run a job in parallel mode with multiple processors.
I would suggest you start from a smaller case, for example using the 240km mesh. A smaller case can be done quickly and give you general ideas how the model works. After that, you can move on to run big cases.
vanesamagar
New member
Hi Ming,
We have not been able to run the init_atmosphere code in parallel, only in serial. We are runnning the atmosphere code in several processors without issues, so I don't think the problem is with ARCHER2 (our computing system), but with running init_atmosphere in parallel. Has this been sorted in version 8.1.x? there was a post saying that some lines of a code had to be commented out in order to run init_atmosphere in parallel, but we have not been able to see where those lines are, maybe that has changed in the newer MPAS versions.
Here are tests performed with 120km quasi-uniform mesh. I attach two zip files, one has the files for a run of init_atmosphere in serial, which worked. The second one is a run of init_atmosphere in parallel, which did not work. The error is different to the errors I get with the 7.5km case, but it is also related to running init_atmosphere in parallel with undersubcribed nodes. Our nodes have 128 cores.
Shall I start a new thread where we debug the parallel init_atmosphere for the 120km quasi-uniform mesh?
Thank you for your help.
We have not been able to run the init_atmosphere code in parallel, only in serial. We are runnning the atmosphere code in several processors without issues, so I don't think the problem is with ARCHER2 (our computing system), but with running init_atmosphere in parallel. Has this been sorted in version 8.1.x? there was a post saying that some lines of a code had to be commented out in order to run init_atmosphere in parallel, but we have not been able to see where those lines are, maybe that has changed in the newer MPAS versions.
Here are tests performed with 120km quasi-uniform mesh. I attach two zip files, one has the files for a run of init_atmosphere in serial, which worked. The second one is a run of init_atmosphere in parallel, which did not work. The error is different to the errors I get with the 7.5km case, but it is also related to running init_atmosphere in parallel with undersubcribed nodes. Our nodes have 128 cores.
Shall I start a new thread where we debug the parallel init_atmosphere for the 120km quasi-uniform mesh?
Thank you for your help.
Attachments
Hi,
I just did a quick test running the idealized baroclinic wave with the 120-km mesh. I am trying to repeat your issue using the same version of MPAS, same mesh, and same namelist and steams input files. However, for my test, "init_atmosphere_model" and "atmosphere_model" both work fine.
I used 24 processors to run this case in parallel mode, and the command line is:
I am perplexed by the issue you posted .....
Also, you mentioned that "there was a post saying that some lines of a code had to be commented out in order to run init_atmosphere in parallel", ---- would you please tell me the link to this post?
I just did a quick test running the idealized baroclinic wave with the 120-km mesh. I am trying to repeat your issue using the same version of MPAS, same mesh, and same namelist and steams input files. However, for my test, "init_atmosphere_model" and "atmosphere_model" both work fine.
I used 24 processors to run this case in parallel mode, and the command line is:
Code:
mpiexec -n 24 ./init_atmosphere_modelI am perplexed by the issue you posted .....
Also, you mentioned that "there was a post saying that some lines of a code had to be commented out in order to run init_atmosphere in parallel", ---- would you please tell me the link to this post?
vanesamagar
New member
Hi Ming,
Yes, this post mentions that to run init_atmosphere in parallel we need to do something with lines 217-222 in src/core_init_atmosphere/mpas_init_atm_cases.F:
Problem generating static file for 15 km with 3 km refinement mesh
but I don't see
I don't understand either why my parallel test works for you and not for me... I will check tomorrow and ask our computing support as well, see if we get anywhere.
Also, could you post the *part.info files that are missing in MPAS-Atmosphere mesh downloads ? we are doing some performance tests for 120km, 60km, 30km, 15km, 7.5km and 3km quasi-uniform meshes but we are quite limited when we dont have the part.info file. Thanks!
Yes, this post mentions that to run init_atmosphere in parallel we need to do something with lines 217-222 in src/core_init_atmosphere/mpas_init_atm_cases.F:
Problem generating static file for 15 km with 3 km refinement mesh
but I don't see
I don't understand either why my parallel test works for you and not for me... I will check tomorrow and ask our computing support as well, see if we get anywhere.
Also, could you post the *part.info files that are missing in MPAS-Atmosphere mesh downloads ? we are doing some performance tests for 120km, 60km, 30km, 15km, 7.5km and 3km quasi-uniform meshes but we are quite limited when we dont have the part.info file. Thanks!
Not sure if this is 100% related, but I had a similar issue of running init_atmosphere_model in parallel for the jw_baroclinic_wave test case. In my case I received a complain from the Intel MPI library regarding some sort of assertion error (see attached file which was created running
`mpirun -np 6 ./init_atmosphere_model >& init_atmosphere_model.txt`). After some googling and trying different things, in the end I was able to solve this issue by setting the following environment variable as part of my slurm submission script: setenv I_MPI_HYDRA_TOPOLIB 'ipl' (inspired by the comment of
"Michael_Intel" on mpirun error running in a cpuset). After that I was able to create x1.40962.init.nc.
Unfortunately, my MPI or HPC knowledge is not deep enough to fully understand why this worked, but maybe there is an equivalent setting for MPICH or OpenMPI that could solve the issue? I_MPI_HYDRA_TOPOLIB seems to be specific to Intel MPI. I am using Intel(R) MPI Library for Linux* OS, Version 2019 Update 7 Build 20200312 (id: 5dc2dd3e9) on a compute node with 2 x AMD EPYC 7351, 2.9 GHz, 16 cores.
`mpirun -np 6 ./init_atmosphere_model >& init_atmosphere_model.txt`). After some googling and trying different things, in the end I was able to solve this issue by setting the following environment variable as part of my slurm submission script: setenv I_MPI_HYDRA_TOPOLIB 'ipl' (inspired by the comment of
"Michael_Intel" on mpirun error running in a cpuset). After that I was able to create x1.40962.init.nc.
Unfortunately, my MPI or HPC knowledge is not deep enough to fully understand why this worked, but maybe there is an equivalent setting for MPICH or OpenMPI that could solve the issue? I_MPI_HYDRA_TOPOLIB seems to be specific to Intel MPI. I am using Intel(R) MPI Library for Linux* OS, Version 2019 Update 7 Build 20200312 (id: 5dc2dd3e9) on a compute node with 2 x AMD EPYC 7351, 2.9 GHz, 16 cores.
Attachments
vanesamagar
New member
Hello Marc and Ming
As discussed offline with Marc yesterday, and following suggestions from Marc and from the EPCC's ARCHER2 support team, I managed to sort out the problem which appeared to come from a faulty build of the init_atmosphere core. I have now managed to run the 120km test case in parallel, and also the 7.5km test case in parallel, after at least a month of bumping our heads on a wall...
I attach the working files as well as the Makefile we use for the build, for the 7p5 test case. Note that we have used the latest MPAS version 8.2.0. I don't know if this has made a difference. Yesterday we exprimented with core pinning, I had a line in my slurm submission code that was commented out, I uncommented it:
export OMP_PLACES=cores
But this in my cases did not seem to make a difference, what did make a difference was a fresh build of init_atmosphere using version 8.2.0 of MPAS.
Thanks,
Vanesa
As discussed offline with Marc yesterday, and following suggestions from Marc and from the EPCC's ARCHER2 support team, I managed to sort out the problem which appeared to come from a faulty build of the init_atmosphere core. I have now managed to run the 120km test case in parallel, and also the 7.5km test case in parallel, after at least a month of bumping our heads on a wall...
I attach the working files as well as the Makefile we use for the build, for the 7p5 test case. Note that we have used the latest MPAS version 8.2.0. I don't know if this has made a difference. Yesterday we exprimented with core pinning, I had a line in my slurm submission code that was commented out, I uncommented it:
export OMP_PLACES=cores
But this in my cases did not seem to make a difference, what did make a difference was a fresh build of init_atmosphere using version 8.2.0 of MPAS.
Thanks,
Vanesa
Attachments
Last edited:
vanesamagar
New member
Sorry Ming, What I meant is that the tar file "x1.40962.tar.gz" that you download from MPAS-Atmosphere mesh downloads
doesn't include a "x1.40962.graph.info" file, and you need this file to generate new x1.40962.graph.info.part.* files. Am I making sense? So we are restricted to the available x1.40962.graph.info.part.* files when we experiment with the 120km test case. Thanks.
doesn't include a "x1.40962.graph.info" file, and you need this file to generate new x1.40962.graph.info.part.* files. Am I making sense? So we are restricted to the available x1.40962.graph.info.part.* files when we experiment with the 120km test case. Thanks.
Hi Vanesa
Thank you for the update and description how to fix the issue. I will send your post to our software engineers just in case they may want to take a look.
I will send you the file " x1.40962.graph.info" later, ---- our WRF tutorial will start in less than half hour and I need to take care of some issues.
Thank you for the update and description how to fix the issue. I will send your post to our software engineers just in case they may want to take a look.
I will send you the file " x1.40962.graph.info" later, ---- our WRF tutorial will start in less than half hour and I need to take care of some issues.