Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

CAM radiation scheme running in MPAS-A, facing segmentation error

wrf

New member
In MPAS-A (v8.2.1) I have ran first simulation using rrtmg radiation scheme. After running this simulation I wanted to run the same simulation for CAM radiation. The even is in the same date just changing the radiation scheme. After executing the model we are facing the segmentation fault error. I have provided the all logfile and picture for your kind reference.
 

Attachments

  • namelist.input
    226 bytes · Views: 2
  • Error Log.txt
    19.3 KB · Views: 1
  • log atmospher out.txt
    21.2 KB · Views: 1
  • log init atmosphere out.txt
    12.3 KB · Views: 0
Is this a global or regional MPAS run? What is the mesh you used?

The namelist.input you posted is for grid rotation. Can you upload your namelist.atmosphere for me to take a look?

Also, please confirm that

(1) your run with RRTMG radiation was done successfully. But when you switched to CAM, MPAS failed immediately.
(2) you didn't change anything, except that the radiation scheme is changed.
(3) you run MPAS, not MPAS-JEDI
 
Variable mesh of 46-12km


1. Yes
2. Yes
3. Yes
 

Attachments

  • namelist.txt
    2.3 KB · Views: 1
Last edited:
In your namelist.txt, "config_apply_lbcs = false", implying this is a global MPAS run, is this correct?

What is your input data that is used to create initial condition for MPAS?
 
Can you compile MPAS in debug mode, then rerun this case? Please save all the log.out and log.error files for me to take a look. Thanks.
 
In your namelist.txt, "config_apply_lbcs = false", implying this is a global MPAS run, is this correct?

What is your input data that is used to create initial condition for MPAS?
Yes, this is global run. I have used GFS data.
 
Here is your debug file.
 

Attachments

  • log_error.txt
    30.6 KB · Views: 2
  • log_out.txt
    21.3 KB · Views: 2
Hi,
Thank you for the test and uploaded files. I can repeat your issue, and sorry that I don't have an immediate answer how to fix this problem. I will look into the codes and let you know after I figure out what is wrong. Debugging may take some time and thank you for your patience.
 
Hi,

Thank you for your quick response. We are actually working on a journal paper on heatwave. So, we choose this CAM radiation scheme for the research purpose. So, if you have enough time to do quickly fix this. It will be really helpful for us.
 
Hi,
I have fixed a few bugs in the CAM radiation scheme. And my test case with the bug fixes works just fine. Please see the attached file in this post. You need to replace your original file by this modified file, then recompile MPAS and run your case.

Let me know whether it works for you. Thanks.
 

Attachments

  • mpas_atmphys_driver_radiation_sw.F
    47.3 KB · Views: 3
We have compiled the MPAS using the file you have given. We successfully ran the model with CAM radiation scheme. But we face issue with ram allocation after getting into the time step it consumes the whole memory of our server pc then it stable after the first step which is sometimes not successful. So, what can we do in that case. Our server pc has 192 core and 187gb of ram.

Update: Its not working. Our model stopped after some time. It just create 3 files of history and two files of diag. After that it allocates full memory and stops. We don't know what's happening.
 
Last edited:
I modified the code based on MPAS-V8.3.1. Note that there are quite a few differences between V8.2.1 and V8.3.1. Can you run V8.3.1 with the modified code? Let me know whether the RAM issue still persists. Thanks.
 
Can you clarify "the issue with ram allocation after getting into the time step it consumes the whole memory of our server pc then it stable after the first step which is sometimes not successful"? How did you check ram allocation?
 
We saw the system monitor for checking the ram. Though it starts the model after some time then the model stops.
 
Summary of the post loop regarding your problem:

(1) The original version of MPAS-v8.3.1 failed when CAM radiation scheme is activated, --- I can repeat your problem

(2) We debugged and fixed a few problems in CAM

(3) Our tests indicate that with the modified code, MPAS-v8.3.1 works fine, ---- your problem no longer exists in our tests.

I am perplexed why the model doesn't work for you. My next question is, what HPC did you use to run this case? I am suspicious that the issue you have might be machine-related. Let me know where you are working, ---- NCAR HPC could be accessible to people in some universities/institutes in the United States.

Please inform me of any updates related to your case. Thanks in advance.
 
Thank you.

My HPC is lenovo x3950 X6 machine which has 192 core and 176gb of ram. Our mesh is 42-16km variable mesh. No, we don't have access the NCAR HPC. We are using our own lab HPC.

I have some question regarding the model rerun after compilation with your fixed file.

1. Do we need to create fresh init file, sfc_update file and static file using the new compiled model or we just delete the previous model component from the mesh folder and link the new model component in the mesh folder, then run the model?
 
Last edited:
Top