Multi-year execution: real.exe fails after processing 999 met_em files.

J

jhernadi

New member
Hi all.

I'm running some climatological downscaling exercises using WRF. But when I try to run an 18-month exercise, real.exe fails at the same point every time. It does not matter if I change the starting date; in the loop 1000, the program crashes and shows me the following error message:

rsl.log.0000 output:
Bash: 
Domain  1: Current date being processed: 2020-04-03_18:00:00.0000, which is loop # 999 out of ****
 configflags%julyr, %julday, %gmt:        2020          94   18.0000000
d01 2020-04-03_18:00:00  Yes, this special data is acceptable to use: OUTPUT FROM METGRID V4.6.0
d01 2020-04-03_18:00:00  Input data is acceptable to use: ../preprocesing/met_em.d01.2020-04-03_18_00_00.nc
 metgrid input_wrf.F first_date_input = 2020-04-03_18_00_00
 metgrid input_wrf.F first_date_nml = 2019-12-01_00:00:00
d01 2020-04-03_18:00:00 Timing for input          8 s.
d01 2020-04-03_18:00:00          flag_soil_layers read from met_em file is  1
Using sfcprs3 to compute psfc
d01 2020-04-03_18:00:00 No average surface temperature for use with inland lakes
 Assume Noah LSM input
d01 2020-04-03_18:00:00 forcing artificial silty clay loam at   15 points, out of  75000
d01 2020-04-03_18:00:00 Timing for processing          3 s.
 LBC valid between these times 2020-04-03_15:00:00 and 2020-04-03_18:00:00
d01 2020-04-03_18:00:00 Timing for output         61 s.
d01 2020-04-03_18:00:00 Timing for loop #  999 =         72 s.

 -----------------------------------------------------------------------------

 Domain  1: Current date being processed: 2020-04-03_21:00:00.0000, which is loop #1000 out of ****
 configflags%julyr, %julday, %gmt:        2020          94   21.0000000
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:   21417
add_new_handle: no handles left
-------------------------------------------

I have tried and ruled out the following:
  • Issues with initial conditions (met_em files). This exercise has 4320 met_em files. I changed the start and end dates in namelist.input in such a way that it can scan all the files and ensure that real.exe finishes successfully.
  • Hardware limitations. Verify that the computer where you run this exercise has sufficient resources, memory (512 GB), and space (4 TB).
  • Hardcode parameters in wrf_io.F90. Changed the variables WrfDataHandleMax, MaxTimes and MaxVars, setting or validating that these parameters have values higher than 4000. Unsuccessfully.
Is there any way to resolve this impasse without resorting to a fragmented execution using restart capabilities?

Thank you all for your help.
 
Hi, Unfortunately, as you've found, there is an upper limit to the number of met* files real.exe is able to read, and that number is 1000, and simply changing the max value in the code is ineffective. So the best solution is to run real.exe in time chunks (that would read <1000 files for each run), and using the restart option to run wrf.exe. I'm sorry there's not an easier solution.
 
You must log in or register to reply here.
Back
Top