Multi-year execution: real.exe fails after processing 999 met_em files.

jhernadi

New member
Hi all.

I'm running some climatological downscaling exercises using WRF. But when I try to run an 18-month exercise, real.exe fails at the same point every time. It does not matter if I change the starting date; in the loop 1000, the program crashes and shows me the following error message:

rsl.log.0000 output:
Bash:
Domain  1: Current date being processed: 2020-04-03_18:00:00.0000, which is loop # 999 out of ****
 configflags%julyr, %julday, %gmt:        2020          94   18.0000000
d01 2020-04-03_18:00:00  Yes, this special data is acceptable to use: OUTPUT FROM METGRID V4.6.0
d01 2020-04-03_18:00:00  Input data is acceptable to use: ../preprocesing/met_em.d01.2020-04-03_18_00_00.nc
 metgrid input_wrf.F first_date_input = 2020-04-03_18_00_00
 metgrid input_wrf.F first_date_nml = 2019-12-01_00:00:00
d01 2020-04-03_18:00:00 Timing for input          8 s.
d01 2020-04-03_18:00:00          flag_soil_layers read from met_em file is  1
Using sfcprs3 to compute psfc
d01 2020-04-03_18:00:00 No average surface temperature for use with inland lakes
 Assume Noah LSM input
d01 2020-04-03_18:00:00 forcing artificial silty clay loam at   15 points, out of  75000
d01 2020-04-03_18:00:00 Timing for processing          3 s.
 LBC valid between these times 2020-04-03_15:00:00 and 2020-04-03_18:00:00
d01 2020-04-03_18:00:00 Timing for output         61 s.
d01 2020-04-03_18:00:00 Timing for loop #  999 =         72 s.

 -----------------------------------------------------------------------------

 Domain  1: Current date being processed: 2020-04-03_21:00:00.0000, which is loop #1000 out of ****
 configflags%julyr, %julday, %gmt:        2020          94   21.0000000
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:   21417
add_new_handle: no handles left
-------------------------------------------

I have tried and ruled out the following:
  • Issues with initial conditions (met_em files). This exercise has 4320 met_em files. I changed the start and end dates in namelist.input in such a way that it can scan all the files and ensure that real.exe finishes successfully.
  • Hardware limitations. Verify that the computer where you run this exercise has sufficient resources, memory (512 GB), and space (4 TB).
  • Hardcode parameters in wrf_io.F90. Changed the variables WrfDataHandleMax, MaxTimes and MaxVars, setting or validating that these parameters have values higher than 4000. Unsuccessfully.
Is there any way to resolve this impasse without resorting to a fragmented execution using restart capabilities?

Thank you all for your help.
 
Back
Top