Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

ERROR: MPAS IO Error: PIO error -237: Size of I/O request exceeds INT_MAX

vanesamagar

New member
Hello I am running init_atmosphere with MPAS verison 8.1.0 for the 7.5 km quasi-uniform mesh and I get this error. This doen't seem to have been discussed before in the forum. Any suggestions?
 
What error did you see in your log file? Please upload your namelist, steams and log files for us to take a look.
 
Hello Ming,
Thanks. I attach the files requested. We are now attempting a run of init_atmosphere with undersubscribed nodes, we have not been successful with serial runs.
Regards,
Vanesa
 

Attachments

  • 20240703_Upload_In_MPAS_Forum.zip
    6.5 KB · Views: 1
Hi, Vanesa,

Thank you for uploading the files. It seems that you are running the ideal case of jw_baroclinic_wave on the 7.5km mesh. Let me know if I am wrong.
Your namelist.init_atmopshere looks fine to me. However, I have one concern of your streams.init_atmosphere. Note that you are running on such a dense mesh, those 3-d fields can easily exceed the 4 GB limit imposed by the classic netCDF format. Therefore, you need to use an "io_type" that supports large variables, such as "pnetcdf,cdf5" or "netcdf4".

Please add io_type="pnetcdf,cdf5" to your output stream, then rerun this case in parallel mode.

I am hesitating to repeat this case because it requires quite a lot of computation resources. I would appreciate if you can let me know whether it works or not.

 
Hi,
Yes you are right, I am running the idealized baroclinic wave test case with quasi-uniform 7.5km mesh. or trying to run, I have not managed to produce the *init.nc with any of the experiments I have run so far with the init_atmosphere core.
I have added io_type="pnetcdf,cdf5" in streams.init_atmosphere and streams.atmosphere files. I have also changed the configuration for a parallel run, and I undersubscribe the nodes. ARCHER2 nodes have ~190GB usable memory and 128 cores per node. I attach the mpas_par_init.slurm file, the namelist* and streams* files, and the output files. I dont get an *err file and the long.init_atmosphere.0000.out has no errors, but the slurm-* file prints errors we don't know how to solve. I have used some UCX commands to change from OFI to UCX network setup but this in fact only made the problem worse... Any suggestions?
Thanks,
vmagar
 

Attachments

  • 20240703_undersubscribed_test1.tar.gz
    4.7 KB · Views: 1
  • 20240704_undersubscribed_test2.tar.gz
    10.7 KB · Views: 1
vmagar,

This is still looks like a memory issue. With the command line in your job script,

Code:
 srun ./init_atmosphere_model

It seems that you run this case with a single processor. Again, for such a dense mesh, single processor cannot work.
Please consult your computer manager how to run a job in parallel mode with multiple processors.

I would suggest you start from a smaller case, for example using the 240km mesh. A smaller case can be done quickly and give you general ideas how the model works. After that, you can move on to run big cases.
 
Hi Ming,

We have not been able to run the init_atmosphere code in parallel, only in serial. We are runnning the atmosphere code in several processors without issues, so I don't think the problem is with ARCHER2 (our computing system), but with running init_atmosphere in parallel. Has this been sorted in version 8.1.x? there was a post saying that some lines of a code had to be commented out in order to run init_atmosphere in parallel, but we have not been able to see where those lines are, maybe that has changed in the newer MPAS versions.

Here are tests performed with 120km quasi-uniform mesh. I attach two zip files, one has the files for a run of init_atmosphere in serial, which worked. The second one is a run of init_atmosphere in parallel, which did not work. The error is different to the errors I get with the 7.5km case, but it is also related to running init_atmosphere in parallel with undersubcribed nodes. Our nodes have 128 cores.

Shall I start a new thread where we debug the parallel init_atmosphere for the 120km quasi-uniform mesh?

Thank you for your help.
 

Attachments

  • 20240708_parallel_init_atmosphere_test.tar.gz
    4.5 KB · Views: 1
  • working_files.tar.gz
    3 KB · Views: 0
Hi,
I just did a quick test running the idealized baroclinic wave with the 120-km mesh. I am trying to repeat your issue using the same version of MPAS, same mesh, and same namelist and steams input files. However, for my test, "init_atmosphere_model" and "atmosphere_model" both work fine.

I used 24 processors to run this case in parallel mode, and the command line is:
Code:
 mpiexec -n 24  ./init_atmosphere_model

I am perplexed by the issue you posted .....

Also, you mentioned that "there was a post saying that some lines of a code had to be commented out in order to run init_atmosphere in parallel", ---- would you please tell me the link to this post?
 
Hi Ming,

Yes, this post mentions that to run init_atmosphere in parallel we need to do something with lines 217-222 in src/core_init_atmosphere/mpas_init_atm_cases.F:
Problem generating static file for 15 km with 3 km refinement mesh
but I don't see

I don't understand either why my parallel test works for you and not for me... I will check tomorrow and ask our computing support as well, see if we get anywhere.

Also, could you post the *part.info files that are missing in MPAS-Atmosphere mesh downloads ? we are doing some performance tests for 120km, 60km, 30km, 15km, 7.5km and 3km quasi-uniform meshes but we are quite limited when we dont have the part.info file. Thanks!
 
Not sure if this is 100% related, but I had a similar issue of running init_atmosphere_model in parallel for the jw_baroclinic_wave test case. In my case I received a complain from the Intel MPI library regarding some sort of assertion error (see attached file which was created running
`mpirun -np 6 ./init_atmosphere_model >& init_atmosphere_model.txt`). After some googling and trying different things, in the end I was able to solve this issue by setting the following environment variable as part of my slurm submission script: setenv I_MPI_HYDRA_TOPOLIB 'ipl' (inspired by the comment of
"Michael_Intel" on mpirun error running in a cpuset). After that I was able to create x1.40962.init.nc.
Unfortunately, my MPI or HPC knowledge is not deep enough to fully understand why this worked, but maybe there is an equivalent setting for MPICH or OpenMPI that could solve the issue? I_MPI_HYDRA_TOPOLIB seems to be specific to Intel MPI. I am using Intel(R) MPI Library for Linux* OS, Version 2019 Update 7 Build 20200312 (id: 5dc2dd3e9) on a compute node with 2 x AMD EPYC 7351, 2.9 GHz, 16 cores.
 

Attachments

  • init_atmosphere_model.txt
    2.9 KB · Views: 0
Hello Marc and Ming
As discussed offline with Marc yesterday, and following suggestions from Marc and from the EPCC's ARCHER2 support team, I managed to sort out the problem which appeared to come from a faulty build of the init_atmosphere core. I have now managed to run the 120km test case in parallel, and also the 7.5km test case in parallel, after at least a month of bumping our heads on a wall...
I attach the working files as well as the Makefile we use for the build, for the 7p5 test case. Note that we have used the latest MPAS version 8.2.0. I don't know if this has made a difference. Yesterday we exprimented with core pinning, I had a line in my slurm submission code that was commented out, I uncommented it:
export OMP_PLACES=cores
But this in my cases did not seem to make a difference, what did make a difference was a fresh build of init_atmosphere using version 8.2.0 of MPAS.
Thanks,
Vanesa
 

Attachments

  • init_working_files.tar.gz
    8.2 KB · Views: 0
  • Makefile.zip
    8.9 KB · Views: 0
Last edited:
Sorry Ming, What I meant is that the tar file "x1.40962.tar.gz" that you download from MPAS-Atmosphere mesh downloads
doesn't include a "x1.40962.graph.info" file, and you need this file to generate new x1.40962.graph.info.part.* files. Am I making sense? So we are restricted to the available x1.40962.graph.info.part.* files when we experiment with the 120km test case. Thanks.
 
Hi Vanesa

Thank you for the update and description how to fix the issue. I will send your post to our software engineers just in case they may want to take a look.
I will send you the file " x1.40962.graph.info" later, ---- our WRF tutorial will start in less than half hour and I need to take care of some issues.
 
Top