Restart files on nested grids not exactly on output times

[RCarpenter]

Sometimes restart files on nested grids are not output exactly on output times. As an example, I have a run that starts the outer grid at 2020-06-23_09:00:00 and then launches two inner grids at 2020-06-23_12:00:00. The restart interval is 9 hours. The first namelist files written are:

wrfrst_d01_2020-06-23_18:00:00
wrfrst_d02_2020-06-23_18:00:07
wrfrst_d03_2020-06-23_18:00:07

Note that the last two are 7 seconds past the hour. Adaptive time stepping is used, and adjust_output_times and step_to_output_time are true.

Looking at rsl.out.0000, the wrfout files are not written exactly on the hour but the filenames have zero seconds:

Timing for main (dt= 16.00): time 2020-06-23_18:00:07 on domain 3: 0.21456 elapsed seconds
Timing for main (dt= 48.00): time 2020-06-23_18:00:07 on domain 2: 1.11148 elapsed seconds
Timing for main (dt=144.00): time 2020-06-23_18:00:00 on domain 1: 3.83604 elapsed seconds

Perhaps adjust_output_times and step_to_output_time are not being applied to the restart files?

I can work around this with symlinks. But the last files on the nested grids don't get written. Thanks.

 

[Ming Chen]

The two options adjust_output_times and step_to_output_time are only applied to history output files.
Can you specify start_second for D02 and D03 to make them exactly the same as that shown in wrfrst files, then try again? Please let me know whether this works. Thanks.

 

[RCarpenter]

Unfortunately it does not work. For these restart files:
wrfrst_d01_2020-06-29_15:00:00
wrfrst_d02_2020-06-29_15:00:17
wrfrst_d03_2020-06-29_15:00:17

and this namelist setting:
start_second = 0, 17, 17

I get the following:

Input data is acceptable to use: wrfinput_d02
Time in file: 2020-06-26_00:00:00
Time on domain: 2020-06-29_15:00:17
**WARNING** Time in input file not equal to time on domain **WARNING**
**WARNING** Trying next time in file wrfinput_d02 ...
2 input_wrf: wrf_get_next_time current_date: 2020-06-26_00:00:00 Status = -4
---- ERROR: Could not find matching time in input file wrfinput_d02
NOTE: 1 namelist vs input data inconsistencies found.
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 1284
NOTE: Please check and reset these options

Incidentally the d02 restart file has these values:
XTIME = 5400.287 ;
Times =
"2020-06-29_15:00:17" ;

 

[Ming Chen]

I am sorry to learn that this doesn't work. I will talk to our expert and see whether we can fix this issue.

 

[Ming Chen]

Please avoid using adaptive time step at present for the case.

 

[RCarpenter]

As an update, I am seeing hourly nested grid output that is very much off the hour. You can see that the 2020-08-15_12:00:00 file on the nested grids wasn't written until around 12:35. This is also reflected in the XTIME variable having a large fractional component. This makes me wonder about data integrity - is the data in the 12:00 file valid at 12:00 or 12:35? The performance gained by adaptive time stepping is critical for our projects.

Timing for main (dt=128.16): time 2020-08-15_12:33:20 on domain 1: 3.98878 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:33:33 on domain 3: 0.31370 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:33:47 on domain 3: 0.30673 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:00 on domain 3: 0.33181 elapsed seconds
Timing for main (dt= 40.58): time 2020-08-15_12:34:00 on domain 2: 1.20979 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:14 on domain 3: 0.31435 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:27 on domain 3: 0.31900 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:41 on domain 3: 0.32330 elapsed seconds
Timing for main (dt= 40.58): time 2020-08-15_12:34:41 on domain 2: 1.22993 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:54 on domain 3: 0.31386 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:35:08 on domain 3: 0.32217 elapsed seconds
Timing for Writing wrfout_d03_2020-08-15_12:00:00 for domain 3: 1.82772 elapsed seconds
Timing for Writing wrfsfc_d03_2020-08-15_12:00:00 for domain 3: 0.16879 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:35:21 on domain 3: 2.50633 elapsed seconds
Timing for main (dt= 40.58): time 2020-08-15_12:35:21 on domain 2: 3.43119 elapsed seconds
Timing for main (dt=121.75): time 2020-08-15_12:35:21 on domain 1: 6.03050 elapsed seconds
Timing for Writing wrfout_d02_2020-08-15_12:00:00 for domain 2: 9.37466 elapsed seconds
Timing for Writing wrfsfc_d02_2020-08-15_12:00:00 for domain 2: 0.24744 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:35:35 on domain 3: 0.30058 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:35:48 on domain 3: 0.29263 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:01 on domain 3: 0.30733 elapsed seconds
Timing for main (dt= 39.37): time 2020-08-15_12:36:01 on domain 2: 10.81757 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:14 on domain 3: 0.50503 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:27 on domain 3: 0.30630 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:40 on domain 3: 0.30287 elapsed seconds
Timing for main (dt= 39.37): time 2020-08-15_12:36:40 on domain 2: 1.66073 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:53 on domain 3: 0.29980 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:37:06 on domain 3: 0.30545 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:37:20 on domain 3: 0.32481 elapsed seconds
Timing for main (dt= 39.37): time 2020-08-15_12:37:20 on domain 2: 1.19889 elapsed seconds
Timing for main (dt=118.10): time 2020-08-15_12:37:20 on domain 1: 13.83759 elapsed seconds

 

[Ming Chen]

To answer your question, the data in the 12:00 file is valid at 12:00 if this is also the time when one step of time integration is finished. Otherwise, it is the data at the time step nearest to 12:00.