wrf.exe stops when two nested domains start at different time

[lizhenghui]

Hi,
I am running wrf.exe of WRFChem V3.6.1. The simulation is configured with two domains, with horizontal grid space of 3km(d01) and 1km(d02). If the two domains start at the same time, e.g. 03 MAY 12UTC, the simulation can be done successfully. If the d01 starts at 03 MAY 12UTC, and d02 starts at a later time, e.g. 04 MAY 00UTC, the simulation will stop at the time when d02 starts.

In rsl.out.0000 , it shows
entering mosaic_cloudchem_driver - ktau = 2881
leaving mosaic_cloudchem_driver - ktau = 2881 91
dtmax = 0.000000000000000E+000
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 87

the sentence could be found in chem/module_mosaic_cloudchem.f90 and chem/module_mosaic_therm.f90, but I don't know how to solve it.
I have attached my namelist.input and rsl* file.

Do you know how to solve the problem?
Thank you!
Zhenghui

 

[jordanschnell]

Hi Zhenghui,

I am not sure what exactly the problem is, but is there a reason you'd like to start the domains at different times? Also, v3.6.1 is pretty old code, I would highly recommend moving to a newer version (currently, v4.4.1).

Jordan

 

[lizhenghui]

Hi Zhenghui,

I am not sure what exactly the problem is, but is there a reason you'd like to start the domains at different times? Also, v3.6.1 is pretty old code, I would highly recommend moving to a newer version (currently, v4.4.1).

Jordan

Hi Jordan,
The simulation need time to spin-up in WRFChem, when the two domains start at the same (several days advance the initiation of rainfall event), the rainfall result in d02 is worse than that in d01, so I'd like to try to change the start time of d02 to improve the results of d02.
Thank you!

 

[jordanschnell]

Hi Zhenghui,

Since you are running with feedback = 0, could create a separate run for d01 only that includes the 12-hour spin up. Then run ndown.exe with the wrfout files to create a the input file for d02. You could also just run the full d01 simulation, then run the d02 simulation seperately (i.e., a one-way nested run).

You can also try increasing the debug_level (>300) to see where exactly the model is failing. I'm assuming some initialization is failing.

Jordan

 

[lizhenghui]

Hi Jordan,
As you suggested, I try to use WRFChem-v4.4.1, I run a one-domain simulation to simplify the situation. The simulation stopped immediately after the initiation.

In rsl.error.0000 file, it shows

[cn1591:396795:0:396795] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace (tid: 396795) ====
0 0x00000000000534c9 ucs_debug_print_backtrace() ???:0
1 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0
2 0x00000000047b24da array_copy_out() for_contig_array.c:0
3 0x00000000047ad879 for_array_copy_out() ???:0
4 0x00000000020b305a chem_driver_() ???:0
5 0x0000000001ee9349 solve_interface_() ???:0
6 0x00000000005f951f module_integrate_mp_integrate_() ???:0
7 0x0000000000416191 module_wrf_top_mp_wrf_run_() ???:0
8 0x000000000041614f MAIN__() ???:0
9 0x00000000004160e2 main() ???:0
10 0x0000000000023493 __libc_start_main() ???:0
11 0x0000000000415fee _start() ???:0
=================================
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
wrf.exe 00000000047C2FDA for__signal_handl Unknown Unknown
libpthread-2.28.s 000014C99AAC9B20 Unknown Unknown Unknown
wrf.exe 00000000047B24DA Unknown Unknown Unknown
wrf.exe 00000000047AD879 for_array_copy_ou Unknown Unknown
wrf.exe 00000000020B305A Unknown Unknown Unknown
wrf.exe 0000000001EE9349 Unknown Unknown Unknown
wrf.exe 00000000005F951F Unknown Unknown Unknown
wrf.exe 0000000000416191 Unknown Unknown Unknown
wrf.exe 000000000041614F Unknown Unknown Unknown
wrf.exe 00000000004160E2 Unknown Unknown Unknown
libc-2.28.so 000014C99A393493 __libc_start_main Unknown Unknown
wrf.exe 0000000000415FEE Unknown Unknown Unknown
free(): invalid next size (fast)

Do you know how to solve the problem?
I have attached the namelist.input and rsl files.
Thank you!

Zhenghui

 

[jordanschnell]

Hi Zhenghui,

It looks like you do not have any chemistry data in your input and boundary files: e.g.,

NetCDF error in wrf_io.F90, line 2883 Varname o3
NetCDF error: NetCDF: Variable not found

d01 2017-05-03_12:00:00 NetCDF error in wrf_io.F90, line 2883 Varname o3_BXS
d01 2017-05-03_12:00:00 NetCDF error: NetCDF: Variable not found

Did you run real.exe with your chem_opt selected? Otherwise, everything will start at 0 or some very small number and so chemistry may not be stable.

I see you also changed your chem_opt - may I suggest using a chem_opt that uses KPP?

Jordan

 

[lizhenghui]

Hi Jordan,
I didn't add emission files (io_form_auxinput5,6,7,8 =0), maybe that's the reason it indicated " Variable not found" for O3.
Now I add emission files (io_form_auxinput5,6,7,8 =1) and add chemistry data in input and boundary files by mozbc. But the simulation still stopped after several minutes. I have attached rsl files and namelist.input.
I run real.exe and wrf.exe with chem_opt = 32. To use a chem_opt that uses KPP, should I recompile WRFChem with "export WRF_KPP=1" rather than just change the chem_opt?

 

[jordanschnell]

Hi Zhenghui,

No, emission fields are denoted as "E_${SPECIES}"; even so, O3 is never an emitted species (only forms secondarily in the atmosphere). That said, the error is gone now that you've included chemistry fields in the IC/BCs. However, you did not set chem_in_opt = 1 so it's not clear that they were used.

Yes, to use a chem_opt that uses KPP, you will need to set WRF_KPP=1, as well as specify the YACC and FLEX library directories (see the WRF-Chem User Guide).

All of that said, the bottom of your rsl.errors shows this:

forrtl: severe (718): Cannot allocate array temporary - out of memory
and this:
double free or corruption (out)

It's possible you need to run with additional processors, but I am guessing there is something else wrong. First, ensure your IC/BCs are being used. Then try turning off/setting:

cldchem_onoff = 0
wetscav_onoff = 0
progn = 0

to see if the problem is with the cloud/aq chemistry.

Also set:
chemdiag = 0 (unless you really need those diagnostics)

If you are not running with the direct aerosol feedback, you can also set
aer_op_opt = 0

Maybe with a KPP mechanism, the problem will be fixed.

Jordan