CRITICAL ERROR: NaN detected in 'w' field

[jpen]

I am using variable (60-km - 3-km) mesh "x20.835586.grid.nc" and using grid_rotate changed the center over South America. However, atmosphere_model crashes with

CRITICAL ERROR: NaN detected in 'w' field

I have set

&nhyd_model
config_time_integration_order = 2
config_dt = 18.0 also with [15.0 / 10.0 / 24.0 / 50.0 ]
&printout
config_print_global_minmax_vel = true
config_print_detailed_minmax_vel = true

How to resolve this error?

 

[mgduda]

Can you attach your namelist.atmosphere and log.atmosphere.0000.out files? What dataset are you using for initial conditions?

 

[jpen]

Hi Michael, Thank you! I have accidentally deleted the log.atmosphere.out file but I am attaching the namelist file. I remember, the atmosphere_model run creates diag and history files for the initial timestep and then crashes. I will rerun again to obtain log.atmosphere.out.
For ICs. I used ds083.3 dataset available from rda.ucar.edu and processed it using WPS. For static fields, I used the database given in MPAS tutorial https://www2.mmm.ucar.edu/projects/mpas/tutorial/Boulder2019/input_data/mpas_tutorial.tar.gz.

 

[mgduda]

Thanks for attaching your namelist! I think you'll need to set

config_len_disp = 3000.0

in your namelist.atmosphere file, since config_len_disp is assumed to be the minimum horizontal grid distance in the mesh. Specifying a larger value for this parameter can lead to instability, which is what you might be encountering.

 

[jpen]

I changed it to 3000.0 yet the error remains. Please see the attached log and namelist files.

 

[mgduda]

From your namelist.atmosphere file, you're using a time step of 32 seconds. Typical guidance is to use a timestep around 5 or 6 times the minimum grid distance in km. Could you try a 15 s timestep?

 

[jpen]

I tried a 15 s timestep an it crashed after simulating one day. Attached are the atmosphere log and screenshot of error message with files written. It could not write the restart file. My guess is that io_type="pnetcdf,cdf5" should be included for restart as well in the streams.atmosphere!

 

[mgduda]

My guess is that io_type="pnetcdf,cdf5" should be included for restart as well in the streams.atmosphere!

That would be my guess as well.

 

[jpen]

It was right. Thanks again!
One more question - Since running global simulation is computationally expensive, is it appropriate to define a desired region for limited area simulation using the variable resolution input (x20.835586.static.nc)? The desired region lies in the refined region of the variable resolution mesh. Or do I need to use 3-km quasi uniform mesh? Also, is the database from mpas tutorial good enough for generating static fields and initial data from ds083.3 for generating ICs at 3 km?

 

[mgduda]

It was right. Thanks again!
One more question - Since running global simulation is computationally expensive, is it appropriate to define a desired region for limited area simulation using the variable resolution input (x20.835586.static.nc)? The desired region lies in the refined region of the variable resolution mesh. Or do I need to use 3-km quasi uniform mesh? Also, is the database from mpas tutorial good enough for generating static fields and initial data from ds083.3 for generating ICs at 3 km?

It's good to hear that setting io_type="pnetcdf,cdf5" for the "restart" stream resolved the I/O issues. Since your follow-up questions aren't related to the original topic of this discussion, and since we'd like to try to keep topics more focused so that they're hopefully easier to browse, would you mind creating a separate thread to discussion your questions about variable-resolution regional 3-km simulations?

 

[jpen]

Agreed. I posted it in a new thread

 

[bgrasic]

Hello all!

I have created similar domain as jpen where the centre of my domain is in Slovenia (Europe). Resolution of my mesh is 10km and the radius of circular domain is 300 km. I made domain using MPAS-A version 8.2.2 and I am getting the same error: "CRITICAL ERROR: NaN detected in 'w' field.". If I make simulation on the same domain with MPAS-A version 8.0.1 everything works fine.
I have attached output files from both simulations. I did some experimentation with namelist parameters but unsuccessful.
The simulation is also unstable in version 8.1.0.

Does anyone has some idea which parameter would made my simulation stable?

 

[mgduda]

@bgrasic It could be worth trying the simulation with MPAS v8.2.2, but with physics turned off. In your namelist, you can set

config_physics_suite = 'none'

. If the dynamics-only simulation is stable, we can look more into changes in the physics schemes between v8.0.1 and v8.1.0.

 

[bgrasic]

Thank you for suggestion. I repeated the simulation with physics turned off, but it is still unstable.
The logs are attached. Are there some other options that I can try?

 

[mgduda]

Looking through the release notes for MPAS v8.1.0, it doesn't seem that much has changed in the way of the MPAS-A dynamical core. However, we did make changes to how we interface with the MPI library (through the mpi_f08 module, if it is available). I recall that some older versions of MPICH had issues with certain routines in the mpi_f08 module, so perhaps there's some issue there.

Which MPI library are you using, and which version of that library do you have? In any case, it could be worth trying the dynamics-only (no-physics) simulation again after adding

&development
config_halo_exch_method = 'mpas_dmpar'
/

to your namelist.atmosphere file. This will cause MPAS-A to use an older infrastructure module for communicating halos among MPI tasks.

 

[bgrasic]

Thank your very much for your help.

I have upgraded the mpich library from version 4.0.2 to the latest version 4.2.3 and now the model works perfectly .

I have also tested the model vith mpich version 4.0.2 using your suggested setting in namelist.atmosphere file and it also worked.