Abnormal slowdown after WRF restart run

peng

Member
I encountered an abnormal slowdown when restarting WRF from a wrfrst file.This is a single-domain, one-way nested WRF simulation using ndown, with 100 m resolution and adaptive time stepping.
In the continuous run, after the adaptive timestep reaches dt = 1.20 s, the model runs normally:
“Timing for main (dt= 1.20): time 2026-02-18_00:00:22 on domain 1: 1.29544 elapsed seconds
Timing for main (dt= 1.20): time 2026-02-18_00:00:23 on domain 1: 1.26584 elapsed seconds
Timing for main (dt= 1.20): time 2026-02-18_00:00:24 on domain 1: 1.25132 elapsed seconds”

The output interval is 30 minutes, and the wall-clock time is also about 30 minutes per 30-minute simulation output.
However, after stopping the run and restarting from:wrfrst_d01_2026-02-20_12:00:00,the model becomes much slower, although the timestep is still dt = 1.20 s:
"Timing for main (dt= 1.20): time 2026-02-20_12:00:01 on domain 1: 6.13815 elapsed seconds
Timing for main (dt= 1.20): time 2026-02-20_12:00:02 on domain 1: 5.52845 elapsed seconds
Timing for main (dt= 1.20): time 2026-02-20_12:00:03 on domain 1: 5.57714 elapsed seconds"

So for the same dt = 1.20 s, the continuous run takes about 1.2–1.3 s per step, while the restart run takes about 5.5–6.1 s per step. After restart, it takes nearly 2 hours of wall-clock time to produce 30 minutes of simulation output.
I have checked that:
1. The number of MPI processes is the same.
2. The namelist settings are the same.
3. The timestep is the same after adaptive time stepping reaches dt = 1.20 s.
4. The previous WRF job was stopped before restarting.
5. Similar behavior has also been observed in other simulations and on other servers.
The restart file appears to be read correctly:
LBC for restart: Found the correct bounding LBC time periods for restart time = 2026-02-20_12:00:00.
Any suggestions would be appreciated.
 
I confirm that the simulation was restarted using a **wrfrst** file. Under normal circumstances, the results should not differ from running the simulation continuously. However, I am wondering whether the extended runtime will affect the simulation results.
 
Hi, Can you let me know at which stage of your ndown simulation this is happening (for e.g., is this a restart of the coarse or fine domain)? Will you please provide the namelists you use to do this, as well as your rsl.* files? Please package all rsl files into a single *.tar or zipped file and attach that. Thanks!
 
These are the namelist and rsl.error files from my direct run (WRF), as well as the restart files (wrfrst). My model is WRF version 4.4. I am currently simulating the official run after nesting down from d02 (500 m) to d03 (100 m), which is a restart of the fine-resolution domain. Since the previous log files were overwritten, these are the logs from the restart I just performed, with a certain degree of reproducibility. Do you have any idea where the problem might be?
 

Attachments

I'm sorry that I cannot provide all the directly generated RSL files, as they have been overwritten. I have checked that there is no other redundant extra information within the restart time range, and everything is running normally.
 
Thanks for providing those. It would be helpful if I can see the namelist you use for the non-restart run, to compare to the restart namelist. To get around the issue where your rsl files are overwritten, run a very short (e.g., 6-12 hours) non-restart simulation, then package the resulting rsl files into a zipped file. Then do the same for the restart simulation - afterward, zipping those rsl files, using a different file name than the one you used when zipping the non-restart rsls. Then please attach both zipped files so that I can compare the rsls for both runs.
 
Sorry for the late reply. The server resources were previously very limited, so I was not able to run this test earlier.

The current test is a 6-hour simulation using the 1-km domain. Running the 0.1-km case would take too long, and I do not currently have enough computational resources for it. I apologize for that. However, even in the 1-km restart case, a slight increase in the elapsed time per timestep can still be observed. If the same behavior occurs in the 0.1-km case, the slowdown would likely be greatly amplified.

This experiment uses a 9-km, 3-km, and 1-km nesting configuration. The 1-km domain is run after ndown, following the same procedure as the 0.1-km case; the only differences are the number of grid points and the timestep.

Attached are the namelist and rsliles for the two experiments. The wrf case is the continuous run, while the wrfrst case is the restart run using the wrfrst file. From the logs, it can be seen that the elapsed time per timestep in the wrfrst run is generally about 0.05 s longer than that in the continuous wrf run.

Do you have any idea what might be causing this issue?
 

Attachments

The rsl-namelist.zip file that I sent previously contains a short test performed on the 0.1-km domain, in which the wall-clock runtime increased dramatically after restarting from the wrfrst file.

In that test, the only difference between the non-restart run and the restart run in the namelist.input file was:

restart = .true.,
Thanks for providing those. It would be helpful if I can see the namelist you use for the non-restart run, to compare to the restart namelist. To get around the issue where your rsl files are overwritten, run a very short (e.g., 6-12 hours) non-restart simulation, then package the resulting rsl files into a zipped file. Then do the same for the restart simulation - afterward, zipping those rsl files, using a different file name than the one you used when zipping the non-restart rsls. Then please attach both zipped files so that I can compare the rsls for both runs.

All other timestep-related settings were kept unchanged.

Although the newly provided test uses a different domain configuration, the same issue has also been encountered when restarting the 0.1-km domain: the wall-clock runtime becomes significantly longer after restarting from wrfrst.

Therefore, I suspect that this may not be caused by the domain configuration itself, but may be related to the restart process, especially when restarting an ndown-generated high-resolution single-domain run.
 
Hi, and many apologies for the delay in response. My colleague and I both ran tests to see if we could repeat the issue. There aren't any significant differences in the run times between the initial run and the restart run. In the files you provided (wrf.zip and wrfrst.zip), the time steps in the rsl files are actually pretty comparable. We're unable to think of anything that could impact the runtimes between the two types of runs, unless it's related to your computing environment. We've not had any other reports of this happening in any versions of WRF, so it seems that it's likely related to your computing environment. If it continues to trouble you, I'd recommend talking to a systems administrator at your institution to see if they have any thoughts or ideas. Let us know if you figure anything out that helps.
 
Back
Top