Real.exe crash, possibly related to input soil fields

[mmallard]

Hi,
I'm encountering some sort of IO error in real.exe that seems related to the input soil fields. See rsl.out files attached, one has the default debug=0 setup and the other has debug turned up high. The snippet below is from the rsl file with debug turned up.

d01 1983-10-01_03:00:00 flag_soil_layers at end of optional_lsm_levels is 1
d01 1983-10-01_03:00:00 flag_soil_levels at end of optional_lsm_levels is 0
forrtl: severe (27): too many records in I/O statement, unit -5, file Internal Formatted Write
Image PC Routine Line Source
real.exe 0000000003194EA8 for__io_return Unknown Unknown
real.exe 00000000031E7D37 for_write_int_fmt Unknown Unknown
real.exe 00000000016E9B3D Unknown Unknown Unknown
real.exe 00000000016DAE5C Unknown Unknown Unknown
real.exe 000000000041A02E Unknown Unknown Unknown
real.exe 00000000004187A2 Unknown Unknown Unknown
libc-2.28.so 0000147B92C63865 __libc_start_main Unknown Unknown
real.exe 00000000004186AE Unknown Unknown Unknown

This error message seems fairly non-specific. I've seen other posts about similar errors occurring because NUM_METGRID_SOIL_LEVELS doesn't match between the met_em files and the namelist.input, but that isn't the case here (both are set to 12). Note that my input data doesn't have the same soil levels used in the Noah LSM, but I've modified the Vtable and METGRID.TBL for this soil data, and the "ST" & "SM" fields (alongside the associated FLAG_ entries) and "SOIL_LAYERS" field do appear in the met_em files (see ncdump -h output attached here from one of the met_ems).
Also note that I'm trying to run WRF using ESGF CMIP6 data, where 10-m winds are not available subdaily. Therefore, I'm using the namelist setting use_surface = false so that 2-m and 10-m fields aren't needed (instead the lowest model level is used for interpolation). I've made sure that SKINTEMP is available in the met_ems as well, so I don't think that real is lacking the information to perform this interpolation.
Other notable info not captured in the attached files is that I'm using WRF version 4.5.1. I would appreciate any ideas for how to debug this further. Thanks!
Megan

 

[mmallard]

Just a quick update in case anyone wants to dig into this further.... I've taken a further look at the code that the error originates from: share/module_optional_input.F
And I think the IO error comes when it hits this line of code (line 1915 in WRF 4.5.1)

write (message,'(A,10(i3,1x))') 'st_levels_input = ', (st_levels_input(k), k=1,num_st_levels_input)
CALL wrf_debug(1,message)

It looks like, further up in this same subroutine, num_st_levels_input is first set from num_metgrid_soil_levels (which is set to 12 in my met_ems) and then st_levels_input is calculated as the mid-point of each layer

IF ( flag_um_soil == 1 ) THEN
st_levels_input(k) = (grid%soil_layers(its,num_st_levels_input + 1 - k,jts))/2
sm_levels_input(k) = (grid%soil_layers(its,num_st_levels_input + 1 - k,jts))/2
sw_levels_input(k) = (grid%soil_layers(its,num_st_levels_input + 1 - k,jts))/2
ELSE
st_levels_input(k) = (level_above + grid%soil_layers(its,num_st_levels_input + 1 - k,jts))/2
sm_levels_input(k) = (level_above + grid%soil_layers(its,num_st_levels_input + 1 - k,jts))/2
sw_levels_input(k) = (level_above + grid%soil_layers(its,num_st_levels_input + 1 - k,jts))/2
level_above = grid%soil_layers(its,num_st_levels_input + 1 - k,jts)
END IF

Note that different stanzas of code compute st_levels_input based on whether flag_soil_layers and flag_soil_levels are set (where each is true/false and denote whether the met_em stores soil info in a distinct level or within a layer average. The soil data I'm using is over a layer so flag_soil_layers is set to 1 in my case.

Since NUM_METGRID_SOIL_LEVELS, SM, ST, and SOIL_LAYERS are all present in my met_ems and the rsl file (with debug set high) show that all these fields are read with no errors, I'm still not sure why this error is popping up.

Again, any ideas of how to debug further would be appreciated,
Megan

 

[mmallard]

So even though this thread is just me talking to myself, I thought I'd post the solution in case it is of use to someone in the future for debugging because... it's a tricky one!

To better understand what was going on, I decided to add a few more lines of code within share/module_optional_input.F (near the end of subroutine optional_lsm_levels where the crash occurs) to write some additional variables to the rsl files that could possibly help me with debugging. Then I noticed this line already within the code:

write (message,'(A,10(i3,1x))') 'st_levels_input = ', (st_levels_input(k), k=1,num_st_levels_input)
CALL wrf_debug(1,message)

And a similar stanza for soil moisture, where information on the soil levels is written. Notice it's arbitrarily set to write 10 integers. Knowing I had 12 levels of soil data that span 0 to 2 m within this dataset, I bumped it up to 12 so that all my soil info would be correctly written out... and then real.exe ran and completed with no errors. So FYI some write statements can become a problem when the number of soil levels or layers exceed 10.

- Megan

 

[Ming Chen]

Megan,

Thank you for the detailed description of this issue. Would you please clarify where num_st_levels_input is set to 10?

I am looking at the codes that process soil data, and I found that the specification below might be an issue.

num_st_levels_alloc = config_flags%num_soil_layers * 3

Would you please do a quick test by changing the above line to

num_st_levels_alloc = config_flags%num_soil_layers * 6

Can it fix your problem?

Thanks!

 

[mmallard]

Megan,

Thank you for the detailed description of this issue. Would you please clarify where num_st_levels_input is set to 10?

Hi, to clarify, all I did was modify this section of code below (around line 1920 in share/module_optional_input.F for version 4.5.1)

      write (message,'(A,10(i3,1x))') 'st_levels_input = ', (st_levels_input(k), k=1,num_st_levels_input)
      CALL wrf_debug(1,message)
      write (message,'(A,10(i3,1x))') 'sm_levels_input = ', (sm_levels_input(k), k=1,num_sm_levels_input)
      CALL wrf_debug(1,message)
      write (message,'(A,10(i3,1x))') 'sw_levels_input = ', (sw_levels_input(k), k=1,num_sw_levels_input)
      CALL wrf_debug(1,message)

and changed all the format statements to '(A,12(i3,1x))' because the soil temperature/moisture data I'm using has 12 levels so the write statement would need to use 12 integers to list them all. And then real.exe was able to run instead of crashing with the error message I shared in my first post above.

I'll give your suggestion above a try as well and let you know what I find.

-Megan

 

[mmallard]

Megan,

Thank you for the detailed description of this issue. Would you please clarify where num_st_levels_input is set to 10?

I am looking at the codes that process soil data, and I found that the specification below might be an issue.

num_st_levels_alloc = config_flags%num_soil_layers * 3

Would you please do a quick test by changing the above line to

num_st_levels_alloc = config_flags%num_soil_layers * 6

Can it fix your problem?

Thanks!

I've run real with the suggested modification (setting num_st_levels_alloc = config_flags%num_soil_layers * 6 , and making the same change to num_sm_levels_alloc and num_sw_levels_alloc) and it resulted in a seg fault. Here, I've attached rsl.error files for 3 runs with real.exe with either the default code for version WRF 4.5.1 ("def"), the modification you suggested for num_st_levels_alloc and the similar sm and sw variables ("mod"), or the modification I described in my last post where I changed the write format statement ("writeMod").

The first two runs result in a crash just after the first 10 soil levels are written out (the last two levels are not listed just prior to the crash). In the 3rd test run, real.exe runs successfully (and the rsl.error file correctly lists all 12 soil levels in its "st_levels_input = ...." statement).

Note that I'm using a namelist (see first post) with debug set high. Also, these rsl.error files differ somewhat from the ones I shared in my first post due to a change in compiler (I tried changing compilers over the coarse of the debugging process hoping that might fix things since the initial error message indicated an IO issue).

- Megan