30 km resolution mesh, model integration failed

[Sylvia]

Hello everyone,

I'm running 30 km resolution mesh and getting problems with the model integration. Since I submitted my job using nohup, followed are contents of the nohup log file.

-----------
forrtl: warning (406): fort: (1): In call to GET_PBLH, an array temporary was created for argument #6

Image PC Routine Line Source
atmosphere_model 0000000002F1243F Unknown Unknown Unknown
atmosphere_model 000000000119998F module_bl_ysu_mp_ 1329 module_bl_ysu.F
atmosphere_model 000000000111C4AE module_bl_ysu_mp_ 229 module_bl_ysu.F
atmosphere_model 0000000000E3EE1B mpas_atmphys_driv 618 mpas_atmphys_driver_pbl.F
atmosphere_model 0000000000DFEDCD mpas_atmphys_driv 288 mpas_atmphys_driver.F
atmosphere_model 0000000000BC19B8 atm_core_mp_atm_d 869 mpas_atm_core.F
atmosphere_model 0000000000BBCDB1 atm_core_mp_atm_c 664 mpas_atm_core.F
atmosphere_model 000000000041CA2E mpas_subdriver_mp 347 mpas_subdriver.F
atmosphere_model 0000000000417AEA MAIN__ 16 mpas.F
atmosphere_model 0000000000417A62 Unknown Unknown Unknown
libc-2.17.so 00002AD5C97A0C05 __libc_start_main Unknown Unknown
atmosphere_model 0000000000417969 Unknown Unknown Unknown
forrtl: warning (406): fort: (1): In call to GET_PBLH, an array temporary was created for argument #7

Image PC Routine Line Source
atmosphere_model 0000000002F1243F Unknown Unknown Unknown
atmosphere_model 0000000001199A84 module_bl_ysu_mp_ 1329 module_bl_ysu.F
atmosphere_model 000000000111C4AE module_bl_ysu_mp_ 229 module_bl_ysu.F
atmosphere_model 0000000000E3EE1B mpas_atmphys_driv 618 mpas_atmphys_driver_pbl.F
atmosphere_model 0000000000DFEDCD mpas_atmphys_driv 288 mpas_atmphys_driver.F
atmosphere_model 0000000000BC19B8 atm_core_mp_atm_d 869 mpas_atm_core.F
atmosphere_model 0000000000BBCDB1 atm_core_mp_atm_c 664 mpas_atm_core.F
atmosphere_model 000000000041CA2E mpas_subdriver_mp 347 mpas_subdriver.F
atmosphere_model 0000000000417AEA MAIN__ 16 mpas.F
atmosphere_model 0000000000417A62 Unknown Unknown Unknown
libc-2.17.so 00002AD5C97A0C05 __libc_start_main Unknown Unknown
atmosphere_model 0000000000417969 Unknown Unknown Unknown
forrtl: error (65): floating invalid
Image PC Routine Line Source
atmosphere_model 0000000002F1B29B Unknown Unknown Unknown
libpthread-2.17.s 00002B31C082A5E0 Unknown Unknown Unknown
atmosphere_model 0000000002FE3589 Unknown Unknown Unknown
atmosphere_model 0000000000CC1CFF atm_time_integrat 3027 mpas_atm_time_integration.F
atmosphere_model 0000000000CB9B2C atm_time_integrat 2896 mpas_atm_time_integration.F
atmosphere_model 0000000000C2C36C atm_time_integrat 917 mpas_atm_time_integration.F
atmosphere_model 0000000000C02CBC atm_time_integrat 121 mpas_atm_time_integration.F
atmosphere_model 0000000000BC19DC atm_core_mp_atm_d 873 mpas_atm_core.F
atmosphere_model 0000000000BBCDB1 atm_core_mp_atm_c 664 mpas_atm_core.F
atmosphere_model 000000000041CA2E mpas_subdriver_mp 347 mpas_subdriver.F
atmosphere_model 0000000000417AEA MAIN__ 16 mpas.F
atmosphere_model 0000000000417A62 Unknown Unknown Unknown
libc-2.17.so 00002B31C0C5CC05 __libc_start_main Unknown Unknown
atmosphere_model 0000000000417969 Unknown Unknown Unknown
forrtl: error (65): floating invalid
Image PC Routine Line Source
atmosphere_model 0000000002F1B29B Unknown Unknown Unknown
libpthread-2.17.s 00002AFD795AD5E0 Unknown Unknown Unknown
atmosphere_model 0000000002FE3589 Unknown Unknown Unknown
atmosphere_model 0000000000CC1CFF atm_time_integrat 3027 mpas_atm_time_integration.F
atmosphere_model 0000000000CB9B2C atm_time_integrat 2896 mpas_atm_time_integration.F
atmosphere_model 0000000000C2C36C atm_time_integrat 917 mpas_atm_time_integration.F
atmosphere_model 0000000000C02CBC atm_time_integrat 121 mpas_atm_time_integration.F
atmosphere_model 0000000000BC19DC atm_core_mp_atm_d 873 mpas_atm_core.F
atmosphere_model 0000000000BBCDB1 atm_core_mp_atm_c 664 mpas_atm_core.F
atmosphere_model 000000000041CA2E mpas_subdriver_mp 347 mpas_subdriver.F
atmosphere_model 0000000000417AEA MAIN__ 16 mpas.F
atmosphere_model 0000000000417A62 Unknown Unknown Unknown
libc-2.17.so 00002AFD799DFC05 __libc_start_main Unknown Unknown
atmosphere_model 0000000000417969 Unknown Unknown Unknown
Fatal error in PMPI_Wait: Unknown error class, error stack:
PMPI_Wait(216)........................: MPI_Wait(request=0x40a68f60, status=0x1) failed
MPIR_Wait(112)........................:
MPIDU_Complete_posted_with_error(1137): Process failed
Fatal error in PMPI_Wait: Unknown error class, error stack:
PMPI_Wait(216)........................: MPI_Wait(request=0x4085f100, status=0x1) failed
MPIR_Wait(112)........................:
MPIDU_Complete_posted_with_error(1137): Process failed
Fatal error in PMPI_Wait: Unknown error class, error stack:
PMPI_Wait(216)........................: MPI_Wait(request=0x40964380, status=0x1) failed
MPIR_Wait(112)........................:
MPIDU_Complete_posted_with_error(1137): Process failed
Fatal error in PMPI_Wait: Unknown error class, error stack:
PMPI_Wait(216)........................: MPI_Wait(request=0x403830c0, status=0x1) failed
MPIR_Wait(112)........................:
MPIDU_Complete_posted_with_error(1137): Process failed


Please find log.atmosphere.0000.out, streams.atmosphere and namelist.atmosphere files in the attached.

Just in case, I used Intel basic compiler (ifort, icc and icpc) and mvapich MPI compiler (mpif90, mpicc, mpicxx) to compile and install the model.

The input data I used is CFSR:2010-10-23_00, which is the Sample real-data input file provided by the MPAS website http://mpas-dev.github.io/

Thank you in advance!

 

[mcurry]

Thanks for the great post and providing your log file, namelist, the streams file and providing which compiler you used. That all greatly helps us diagnose your issue!

Checking your log file the minimum and maximum global values for w and u are starting to reach unrealistic values. I've copied the second and third timestep below and by the third time step you can see the min and max of w and u have dramatically increased:

Begin timestep 2010-10-23_00:01:30
 --- time to run the LW radiation scheme L_RADLW =F
 --- time to run the SW radiation scheme L_RADSW =F
 --- time to run the convection scheme L_CONV    =T
 --- time to apply limit to accumulated rainc and rainnc L_ACRAIN   =F
 --- time to apply limit to accumulated radiation diags. L_ACRADT   =F
 --- time to calculate additional physics_diagnostics               =F
  split dynamics-transport integration 3
 
 global min, max w -2.00301039986907 1.52931660670286
 global min, max u -89.6745810237067 95.5238699435222
  Timing for integration step: 242.885921716690 s
 
 Begin timestep 2010-10-23_00:03:00
 --- time to run the LW radiation scheme L_RADLW =F
 --- time to run the SW radiation scheme L_RADSW =F
 --- time to run the convection scheme L_CONV    =T
 --- time to apply limit to accumulated rainc and rainnc L_ACRAIN   =F
 --- time to apply limit to accumulated radiation diags. L_ACRADT   =F
 --- time to calculate additional physics_diagnostics               =F
  split dynamics-transport integration 3
 
 global min, max w -1060.09169941988 110.230844780067
 global min, max u -26585.8278944526 23900.8248794970
  Timing for integration step: 242.270318508148 s

This suggests to me that your simulation is unstable. The most common causes of this is incorrectly setting either the timestep (config_dt) or the horizontal length scale (config_len_disp).

I've see in your namelist that you have correct set config_dt to a value that is stable for the 30 km mesh (90 seconds). However, your config_len_disp is still set to the default of 120,000.0 meters. It will need to be set to 30,000.0 meters for the 30 km mesh. Give this a try and let us know how it goes.


As an aside, you should be able to use a larger timestep for running a simulation on the 30 km mesh. As a rule of thumb, we multiply the smallest grid distance of a mesh by 6 to get a stable timestep. So for the 30 km mesh that would be: 180 seconds (6 * 30).

 

[Sylvia]

Thank you very much! The run succeed after I changing config_len_disp from 120,000 to 30,000. By the way, should I always set the value of config_len_disp to the value of the mesh resolution?

 

[mcurry]

Yes, you will always need to set the config_len_disp to the smallest grid distance of your mesh. The default atmosphere namelist generated at compiled time (and in the default_inputs directory) has config_dt and config_len_disp set for the 120 km mesh (x1.40962.grid.nc), but its always good to double check.

As I stated above, you'll need to set config_len_disp (and config_dt) to the smallest grid distance of the mesh you are using. So, for instance, if you are using the 60-km - 15-km variable resolution mesh you will need to set config_len_disp to 15000.0 meters and config_dt to 90.0 seconds (15 * 6).


I also wanted to make a correct to my initial response. In my response I stated that a bad value for config_len_disp would violate the CFL; however, config_len_disp has to do with Smagorinksy horizontal diffusion formulation and 3-d divergence damping and not the CFL. While I wont go into detail here, a bad value for the config_len_disp field will still cause a simulation to become unstable. I'll correct my response above.

 

[Sylvia]

Got it! Thank you very much!