init_atmopshere stops

[zmenzo]

While running the inital_atmosphere model executable, the program runs fine for nearly 18mintues and then simply stops and no error messages are made. My submission script allows for it to run up to 10hours so time should not be an issue. I have retried several times and it stops at the same spot each time. Do you have any recommendations for how/where to start troubleshooting?

 

[mgduda]

It looks like your screenshot is from an execution of the atmosphere_model program (i.e., the model itself) rather than the init_atmosphere_model program. The log output indicates that NaNs were present in the model velocity fields, which is likely the reason why the model eventually failed. Can you attach your namelist.atmosphere file? On which mesh are you running your simulation?

 

[zmenzo]

Since the last post, I have been attempting to troubleshoot this problem. I have finally even gone to the extreme of rebuilding the entire model and rerunning the same steps. Unfortunately, I am getting the same issue, but without the wind error you mentioned in the previous comment. It simply stops a few moments into running the init step and does not produce and error file. Knowing that this may be an issue, I capture each step via screenshots and TextEdit files and have attached them to this message. There were no errors present after creating the static.nc file. If you have any suggestions, I would be very grateful.

 

[zmenzo]

Attachments1

 

[zmenzo]

Attachments2

 

[zmenzo]

The email for the failed batch script:

PBS Job Id: 1366142.chadmin1.ib0.cheyenne.ucar.edu
Job Name: init_1
Execution terminated
Exit_status=265
resources_used.cpupercent=2859
resources_used.cput=00:17:00
resources_used.mem=44449788kb
resources_used.ncpus=36
resources_used.vmem=2500381880kb
resources_used.walltime=00:00:30

 

[mgduda]

This could be an issue of memory usage. Could you try running the job to create the initial conditions on four nodes of Cheyenne?

 

[mgduda]

For a bit more information on memory usage, we've generally found that providing about 0.175 MB of memory per grid column (with 55 vertical layers) is adequate when running the model itself (i.e., the atmosphere_model program). The initialization program requires less memory because it allocates fewer fields; I don't have a good figure, but just guessing wildly, it may be that the init_atmosphere_model program requires only half the memory of the model. For a mesh with 535554 columns, then, you might want to ensure that you have at least 535554 * 0.175 MB = 93.7 GB of aggregate memory when running a model simulation, and the initialization of the model might require around 46 GB. Although most of the Cheyenne batch nodes have 64 GB of memory, not all of this is available to user processes. So, using at least a couple of nodes when creating the initial conditions, and using more nodes still when running the model is probably good.