How to write full WRF running progress into a log file?

[zizils1999]

Hi all,
I have some trouble when running WRF 4.3.3 with Mibrandt & Yau 2-moment (mp=9). It crashing but doesn't make an error report in rsl file and doesn't kill task. When using the same namelist for other mp scheme (Thompson 08, Morrison 10), WRF can complete full cycle run, with some extreme forecast value around crashing time. I tried to reduce dt but the same problem happen sooner?!? This issue happen with both GFS 0.25 deg and ECMWF forecast 8km (converted) data.
I want to check all running progress, something like debug_level (i don't know this namelist option still available or disable yet and usage of this option?), how can I do this...? Or anyone can solve this problem? Thank you all!

 

[Ming Chen]

There is no error message in the rsl file you posted. I just wonder whether you checked all the rsl files for this case? The error messages are not necessarily saved in rsl.error.0000. They may appear in any RSL files.

If convenient, please recompile WRF in debug mode, i.e., type ./clean -a in the top WRF directory, then type ./configure -D and ./compile em_real.

Once wrf.exe is created, you can rerun this case. The rsl files will tell exactly when and where the model crashed.

 

[zizils1999]

There is no error message in the rsl file you posted. I just wonder whether you checked all the rsl files for this case? The error messages are not necessarily saved in rsl.error.0000. They may appear in any RSL files.

If convenient, please recompile WRF in debug mode, i.e., type ./clean -a in the top WRF directory, then type ./configure -D and ./compile em_real.

Once wrf.exe is created, you can rerun this case. The rsl files will tell exactly when and where the model crashed.

40 rsl files has been checked without error. I checked all of them already. Beside that, I found mpirun slurm job report some error (in attached file) when running by slurm command: "mpirun -np 40 ./wrf.exe ". Hope this one can be helpful.
With debug mode (configure -D) and the same WRF option, the model crash at a slightly different time, but rsl.error doesn't report any error (like rsl file above). Do I need to turn on debug_level option on namelist.input?

 

[Ming Chen]

No, please keep debug_level = 0 because higher value of this option doesn't really provide more helpful information.

In your hailer.out.ttx, one message is

HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting

Can you find which processor is it? There souled be a file that tells at which line in which code the model crashed first.