Regional model stops without error log

[Yue_Ma]

Hi,
While I run limited-area MPAS-A model it stops without error log，and screen prints as below

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x2B25CC81EB97
#1  0x2B25CC81DD90
#2  0x378AC3291F
#3  0x64283F in taumol16.6160 at module_ra_rrtmg_sw.F:0
#4  0x6516B3 in __rrtmg_sw_taumol_MOD_taumol_sw
#5  0x66A825 in __rrtmg_sw_spcvmc_MOD_spcvmc_sw
#6  0x676B89 in __rrtmg_sw_rad_MOD_rrtmg_sw
#7  0x67FC6D in __module_ra_rrtmg_sw_MOD_rrtmg_swrad
#8  0x52491C in __mpas_atmphys_driver_radiation_sw_MOD_driver_radiation_sw
#9  0x509982 in __mpas_atmphys_driver_MOD_physics_driver
#10  0x4A3581 in __atm_core_MOD_atm_do_timestep
#11  0x4A4458 in __atm_core_MOD_atm_core_run
#12  0x406DBD in __mpas_subdriver_MOD_mpas_run

Diagnostic file and history file at first time are generated, with output log stops at

 min/max of meshScalingDel2 = 1.00023401578361 3.99770867615113
 min/max of meshScalingDel4 = 1.00070221165381 63.8900794452041
 Troubles finding level 100.000000000000 above ground.
 Problems first occur at (76)
 Surface pressure = NaN hPa.
 *** MSLP field will not be computed
 Timing for diagnostic computation: 219.958779096603 s
 Timing for stream output: 240.639831066132 s
WARNING: Variable lbc_qi not in input file.
WARNING: Variable lbc_qs not in input file.
WARNING: Variable lbc_qg not in input file.
 
 Begin timestep 2013-10-05_00:00:00
WARNING: Variable lbc_qi not in input file.
WARNING: Variable lbc_qs not in input file.
WARNING: Variable lbc_qg not in input file.
 ----------------------------------------------------------------------
 Updated lateral boundary conditions. LBCs are now valid
 from 2013-10-05_00:00:00 to 2013-10-05_06:00:00
 ----------------------------------------------------------------------
 ----------------------------------------------------------------------
   Read 'surface' input stream valid at 2013-10-05_00:00:00
    Timing for stream input: 0.506101846694946 s
 ----------------------------------------------------------------------
 --- time to update background surface albedo, greeness fraction.
 --- time to run the LW radiation scheme L_RADLW =T
 --- time to run the SW radiation scheme L_RADSW =T
 --- time to run the convection scheme L_CONV    =T
 --- time to update the ozone climatology for RRTMG radiation codes
 --- time to apply limit to accumulated rainc and rainnc L_ACRAIN   =F
 --- time to apply limit to accumulated radiation diags. L_ACRADT   =F
 --- time to calculate additional physics_diagnostics               =F

Any ideas what may go wrong here?

 

[mcurry]

Hi Yue_ma, can you please provide your namelist.atmosphere and streams.atmosphere files? That will help us better asses where things might have gone wrong! Thank you!

 

[Yue_Ma]

Here is my View attachment namelist.atmosphere.txt and View attachment streams.atmosphere.txt. They work well in global model (with lbc option turning off).

 

[mcurry]

Hi Yue_ma, thanks for including the namelist and streams files. I'm not seeing any irregularities as of yet. Can you also include the entire log.atmosphere.0000.out? I believe you posted a portion of it above, but it would be helpful to see it in its entirety.

Can you also tell me which MPAS mesh you have created your region from? As well, can you tell me information about your region? (I.E. The method you used for creating your region (Circle, Ellipse, Custom, etc) and if you created your region from a MPAS grid.nc file or a MPAS static.nc file).

 

[mgduda]

That global model simulations run while a regional simulation fails suggests that the LBCs might be corrupt, introducing physically unrealistic values into the model prognostic state and causing an error in the RRTMG SW scheme. Have you tried plotting the fields in your LBC files valid at 2013-10-05_00:00:00 and 2013-10-05_06:00:00? If not the LBCs, it may be that there are problems with the ICs instead, so it may also be worth taking a look at the fields in the initial conditions file, too.

 

[Yue_Ma]

Thanks for your suggestions! That is my View attachment log.atmosphere.0000.out.txt. This region is a Square area as

Name: target_mesh
Type: custom
Point: 35, 135
35, 110
15, 110
15, 135

and is created from a rotated 46-12km mesh with namelist.input

&input
   config_original_latitude_degrees = 0
   config_original_longitude_degrees = 0

   config_new_latitude_degrees = 30
   config_new_longitude_degrees = 130
   config_birdseye_rotation_counter_clockwise_degrees = 90
/

.

LBC files and INIT file could be read correctly. Which variable may lead RRTMG SW scheme stop work？

 

[mgduda]

Although the ICs and LBCs may have been read correctly by the model, this doesn't guarantee that the values in the fields are reasonable. There is one hint in the log.atmosphere.0000.out file that something may be wrong with the ICs:

 Troubles finding level 100.000000000000 above ground.
 Problems first occur at (76)
 Surface pressure = NaN hPa.

Can you use the convert_mpas tool to interpolate your ICs to a regular latitude-longitude mesh, and use ncview or another tool to look through some of the fields? The convert_mpas program does have a few glitches when processing limited-area domains, but it should at least allow you to determine whether variables like theta, qv, rho, etc. all contain physically reasonable values in your initial conditions file.

 

[Yue_Ma]

Thanks again. With your reminder, I notice this surface pressure warning don't reports in global model. In IC file, it truely missing surface pressure data while I use FNL 1°×1° datasethttps://rda.ucar.edu/data/ds083.2/grib2/2013/2013.09 as interpolating data.

I am curious why this error doesn't affect global model but stops limited-area model? And how can FNL dataset works in limited-area model? What's more, this dataset missing 'PMSL' in met_em while I try using it run WRF 4.1.2.

ERROR in psfc: flag_psfc = 1, flag_soilhgt = 1, flag_slp = 0, sfcp_to_sf = F
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    1077
not enough info for a p sfc computation

It is most likely this problem causing model stop.

 

[mgduda]

The same FNL data should work for both regional and global initial conditions. Have you tried visualizing the theta, qv, rho, and other fields in the initial and lateral boundary conditions files to see whether they all look reasonable?