Trouble running MPAS-Atmosphere with high resolution

[satya]

Hello, I am facing issues running MPAS-Atmosphere (atmosphere_model) with high resolution grid (nCells=999426). However, its running successfully with low resolution grid (nCells=40962). I have attached the log files and the namelist. Could you please help?

 

[Ming Chen]

I am out of office for a business trip. I will look at your issue after I am back after July 5. Thank you for your patience.

Just a quick question: how many processors do you use to run this case?

 

[satya]

Thanks for your response Ming.

> how many processors do you use to run this case?
I am using 512 processors (8 nodes, 64 processors per node).

 

[satya]

Dear Ming, hope you are checked the issue posted by me. Could you please help me in resolving it.

With thanks
Satya

 

[Ming Chen]

Hi Satya,

Thank you or reminding me of this issue, ---- I am terribly sorry that somehow your post is skipped. I was distracted by many issues after getting out of office for weeks.

I looked at your log file and TEST1.out.txt. The only error message I can find is 'segmentation fault', which can be attributed to various reasons.

In your namelist.input, you set config_len_disp = 10000.0. Would you please tell me what mesh did you use to run this case?

Did you modify any codes in MPAS-V8.2.2? What data did you ungrib to produce initial condition for this case?

Thanks.

 

[satya]

Hi Satya,

Thank you or reminding me of this issue, ---- I am terribly sorry that somehow your post is skipped. I was distracted by many issues after getting out of office for weeks.

I looked at your log file and TEST1.out.txt. The only error message I can find is 'segmentation fault', which can be attributed to various reasons.

In your namelist.input, you set config_len_disp = 10000.0. Would you please tell me what mesh did you use to run this case?

Did you modify any codes in MPAS-V8.2.2? What data did you ungrib to produce initial condition for this case?

Thanks.

Hi Ming,

Thanks for responding to my message.

>Would you please tell me what mesh did you use to run this case?
I am using x6.999426 mesh.

>Did you modify any codes in MPAS-V8.2.2?
No, I have not made any change in the code. I Have tried doing the same simulation with 8.3.1 also but got the same error.

>What data did you ungrib to produce initial condition for this case?
For initial conditions, I tried with both ERA5 and GFS data. No success.

With thanks
Satya

 

[Ming Chen]

Satya,

The x6.999426 mesh is global 60-10km mesh. Please tell me the following information:

(1) Did you run grid_rotate to relocate the 10-km mesh to some other area?

(2) please send me your namelist and streams files (for static and initial condition as well as for the model run itself)

(3) please stay with GFS data, and rerun your failed case using more than 1500 processors (this test will tell us whether the memory could be an issue for the model crash)

Thanks.

 

[satya]

Hi Ming, yes I am trying to do global simulation.

>(1) Did you run grid_rotate to relocate the 10-km mesh to some other area?
Yes,we have rotated the mess, and fixed over the Indian region. The issue persisted even after we ran the model without rotated grid.
We have also experimented using the uniform 30 km grid (x1.655362.grid.nc), but the issue persisted.

>(2) please send me your namelist and streams files (for static and initial condition as well as for the model run itself)
I have attached ns.tar file containing all namelist and streams files. These namelist and stream file sets are used for creating static, initial and boundary conditions. The namelist that contains “fct” was used to run the model in forecast mode.

>(3) please stay with GFS data, and rerun your failed case using more than 1500 processors (this test will tell us whether the memory could be an issue for the model crash)
Since we do not have 1500 processors, we are unable to run on that many processors. Maximum I can use 1024 processors.

With thanks
Satya

 

[Ming Chen]

Satya,

Thank you for sending the files. I will take a look and get back to you. It may take some time because this week is WRF tutorial week. Thanks for your patience.

Just a quick question: have you run the case with 1024 processors? How large is your run-time memory? I am suspicious that insufficient memory could be a reason for the model crash.

 

[satya]

Hello Ming, yes I have run the same case on 1024 processors. It also stops after giving the segmentation fault error. The log file (log.atmosphere.0000.txt) is attached with this thread. By "run-time memory" do you mean the total memory usage? I am firing job using qsub, don't know how to find the total memory usage when the job is run via qsub. However, the log file also contains something rearding memory allocation, as follows.

Allocating fields ...
97 MB allocated for fields on this task
98958 MB total allocated for fields across all tasks
----- done allocating fields -----

If this issue is because of insufficient memory, then will it be possible for you to help me in fixing it?

With thanks
Satya