Restart files on nested grids not exactly on output times

Sometimes restart files on nested grids are not output exactly on output times. As an example, I have a run that starts the outer grid at 2020-06-23_09:00:00 and then launches two inner grids at 2020-06-23_12:00:00. The restart interval is 9 hours. The first namelist files written are:

wrfrst_d01_2020-06-23_18:00:00
wrfrst_d02_2020-06-23_18:00:07
wrfrst_d03_2020-06-23_18:00:07

Note that the last two are 7 seconds past the hour. Adaptive time stepping is used, and adjust_output_times and step_to_output_time are true.

Looking at rsl.out.0000, the wrfout files are not written exactly on the hour but the filenames have zero seconds:

Timing for main (dt= 16.00): time 2020-06-23_18:00:07 on domain 3: 0.21456 elapsed seconds
Timing for main (dt= 48.00): time 2020-06-23_18:00:07 on domain 2: 1.11148 elapsed seconds
Timing for main (dt=144.00): time 2020-06-23_18:00:00 on domain 1: 3.83604 elapsed seconds

Perhaps adjust_output_times and step_to_output_time are not being applied to the restart files?

I can work around this with symlinks. But the last files on the nested grids don't get written. Thanks.

The two options adjust_output_times and step_to_output_time are only applied to history output files.
Can you specify start_second for D02 and D03 to make them exactly the same as that shown in wrfrst files, then try again? Please let me know whether this works. Thanks.

Unfortunately it does not work. For these restart files:
wrfrst_d01_2020-06-29_15:00:00
wrfrst_d02_2020-06-29_15:00:17
wrfrst_d03_2020-06-29_15:00:17

and this namelist setting:
start_second = 0, 17, 17

I get the following:

Input data is acceptable to use: wrfinput_d02
Time in file: 2020-06-26_00:00:00
Time on domain: 2020-06-29_15:00:17
**WARNING** Time in input file not equal to time on domain **WARNING**
**WARNING** Trying next time in file wrfinput_d02 ...
2 input_wrf: wrf_get_next_time current_date: 2020-06-26_00:00:00 Status = -4
---- ERROR: Could not find matching time in input file wrfinput_d02
NOTE: 1 namelist vs input data inconsistencies found.
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: <stdin> LINE: 1284
NOTE: Please check and reset these options

Incidentally the d02 restart file has these values:
XTIME = 5400.287 ;
Times =
"2020-06-29_15:00:17" ;

I am sorry to learn that this doesn't work. I will talk to our expert and see whether we can fix this issue.

Please avoid using adaptive time step at present for the case.

As an update, I am seeing hourly nested grid output that is very much off the hour. You can see that the 2020-08-15_12:00:00 file on the nested grids wasn't written until around 12:35. This is also reflected in the XTIME variable having a large fractional component. This makes me wonder about data integrity - is the data in the 12:00 file valid at 12:00 or 12:35? The performance gained by adaptive time stepping is critical for our projects.

Timing for main (dt=128.16): time 2020-08-15_12:33:20 on domain 1: 3.98878 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:33:33 on domain 3: 0.31370 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:33:47 on domain 3: 0.30673 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:00 on domain 3: 0.33181 elapsed seconds
Timing for main (dt= 40.58): time 2020-08-15_12:34:00 on domain 2: 1.20979 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:14 on domain 3: 0.31435 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:27 on domain 3: 0.31900 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:41 on domain 3: 0.32330 elapsed seconds
Timing for main (dt= 40.58): time 2020-08-15_12:34:41 on domain 2: 1.22993 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:34:54 on domain 3: 0.31386 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:35:08 on domain 3: 0.32217 elapsed seconds
Timing for Writing wrfout_d03_2020-08-15_12:00:00 for domain 3: 1.82772 elapsed seconds
Timing for Writing wrfsfc_d03_2020-08-15_12:00:00 for domain 3: 0.16879 elapsed seconds
Timing for main (dt= 13.53): time 2020-08-15_12:35:21 on domain 3: 2.50633 elapsed seconds
Timing for main (dt= 40.58): time 2020-08-15_12:35:21 on domain 2: 3.43119 elapsed seconds
Timing for main (dt=121.75): time 2020-08-15_12:35:21 on domain 1: 6.03050 elapsed seconds
Timing for Writing wrfout_d02_2020-08-15_12:00:00 for domain 2: 9.37466 elapsed seconds
Timing for Writing wrfsfc_d02_2020-08-15_12:00:00 for domain 2: 0.24744 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:35:35 on domain 3: 0.30058 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:35:48 on domain 3: 0.29263 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:01 on domain 3: 0.30733 elapsed seconds
Timing for main (dt= 39.37): time 2020-08-15_12:36:01 on domain 2: 10.81757 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:14 on domain 3: 0.50503 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:27 on domain 3: 0.30630 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:40 on domain 3: 0.30287 elapsed seconds
Timing for main (dt= 39.37): time 2020-08-15_12:36:40 on domain 2: 1.66073 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:36:53 on domain 3: 0.29980 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:37:06 on domain 3: 0.30545 elapsed seconds
Timing for main (dt= 13.12): time 2020-08-15_12:37:20 on domain 3: 0.32481 elapsed seconds
Timing for main (dt= 39.37): time 2020-08-15_12:37:20 on domain 2: 1.19889 elapsed seconds
Timing for main (dt=118.10): time 2020-08-15_12:37:20 on domain 1: 13.83759 elapsed seconds

To answer your question, the data in the 12:00 file is valid at 12:00 if this is also the time when one step of time integration is finished. Otherwise, it is the data at the time step nearest to 12:00.