Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

restarts not producing identical output

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

Hi Dave,
I have finally gotten back to testing this. I apologize for the delay. You're correct - that I previously was not using the wrfinput_d01 file you provided, as I was able to reproduce the problem without it. This round of testing, however, I used your wrfinput file, along with the most recent namelist you provided (I think - I am attaching it so that you can check it). The only thing I modified was to set frames_per_outfile = 1 so that when I diffwrf these output files, the output file for the initial run vs. the restart should contain the same times (just makes it easier for me). I ran a test with the namelist as-is (i.e., including RRTMG radiation), and then another test without radiation, but I still get diffs with both tests (with and without radiation).

To verify, I am currently using V3.9.1.1, compiled with Intel (V17.0.1.132), out-of-the-box code - no modifications, compiled with vortex-following, and for the tropical cyclone idealized case. I initially ran the case for 4 days (2015-09-01_00 to 2015-09-05_00), and then ran a restart for only 1 day, from 2015-09-03_00 to 2015-09-04_00, and then compared output at 2015-09-04_00:00:00.

Do you notice anything I'm doing differently than you?
 

Attachments

  • namelist.input
    7.4 KB · Views: 14
Hi Kelly,

No problem, as you can tell by this delay, I have been concerned with other things.

So I drafted a long post about why yours might be different from mine.

But then to be sure I decided to use that diffwrf utility that you told me about. It turns out that some of my test cases with no radiation did in fact have differences. For example, with mp=16, mp_zero_out=2, and cu=6, but no radiation, there are differences:

[dnolan@visx test_mp16_mzo_2_norad_cu6_restart]$ /nethome/dnolan/wrf3.9.1.1/WRFV3/external/io_netcdf/diffwrf wrfout_d01_2015-09-03_00:00:00 ../test_mp16_mzo_2_norad_cu6/wrfout_d01_2015-09-03_00\:00\:00
Just plot F
Diffing wrfout_d01_2015-09-03_00:00:00 ../test_mp16_mzo_2_norad_cu6/wrfout_d01_2015-09-03_00:00:00
Next Time 2015-09-03_00:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
REFL_10CM 2592000 3 0.0000000000E+00 0.3498864970E+02 0 0.3499E+02 0.1000E+01
Next Time 2015-09-03_03:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
QNCCN 104 3 0.9919356797E+08 0.9919356797E+08 14 0.9913E-02 0.7545E-07
Next Time 2015-09-03_06:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
QNCCN 174 3 0.9915689381E+08 0.9915689381E+08 12 0.6419E-01 0.3945E-06
QNCLOUD 1 3 0.1607450049E+07 0.1607450050E+07 10 0.7454E-02 0.9736E-07
Next Time 2015-09-03_09:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
QNCCN 266 3 0.9911627451E+08 0.9911627451E+08 12 0.8265E-01 0.3101E-06
QNCLOUD 5 3 0.1716316438E+07 0.1716316439E+07 9 0.4833E-01 0.4787E-06
Next Time 2015-09-03_12:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
QNCCN 455 3 0.9907461351E+08 0.9907461351E+08 11 0.2367E+00 0.9359E-06
QNCLOUD 7 3 0.1818917030E+07 0.1818917031E+07 9 0.7378E-01 0.7209E-06
Next Time 2015-09-03_15:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
QCLOUD 1 3 0.5597900119E-05 0.5597900119E-05 15 0.8827E-17 0.1879E-10
QNCCN 725 3 0.9903596873E+08 0.9903596873E+08 13 0.2934E+00 0.1029E-05
QNCLOUD 6 3 0.1854298983E+07 0.1854298983E+07 9 0.5963E-01 0.3839E-06
QC_DIABATI 2 3 0.4138167559E-08 0.4138167559E-08 15 0.2525E-22 0.1847E-13
Next Time 2015-09-03_18:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
QCLOUD 1 3 0.6266486982E-05 0.6266486982E-05 14 0.2825E-15 0.5426E-09
QNCCN 1255 3 0.9900345890E+08 0.9900345890E+08 11 0.4295E+00 0.1056E-05
QNCLOUD 3 3 0.1855755205E+07 0.1855755206E+07 10 0.1539E-01 0.1212E-06
QC_DIABATI 58 3 0.5160490897E-08 0.5160490897E-08 12 0.1103E-17 0.7654E-09
Next Time 2015-09-03_21:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
QNCCN 2155 3 0.9895892137E+08 0.9895892137E+08 11 0.6767E+00 0.1040E-05


We can see they are all in the microphysics. However, for the even simpler mp=1, mp_zero_out=0, cu=0, no radiation, they really are identical:

[dnolan@visx test_mp1_mzo_0_norad_nocu]$ /nethome/dnolan/wrf3.9.1.1/WRFV3/external/io_netcdf/diffwrf wrfout_d01_2015-09-03_00:00:00 ../test_mp1_mzo_0_norad_nocu_restart/wrfout_d01_2015-09-03_00\:00\:00
Just plot F
Diffing wrfout_d01_2015-09-03_00:00:00 ../test_mp1_mzo_0_norad_nocu_restart/wrfout_d01_2015-09-03_00:00:00
Next Time 2015-09-03_00:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
Next Time 2015-09-03_03:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
Next Time 2015-09-03_06:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
Next Time 2015-09-03_09:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
Next Time 2015-09-03_12:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
Next Time 2015-09-03_15:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
Next Time 2015-09-03_18:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
Next Time 2015-09-03_21:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max


So my no-radiation cases had differences but not large enough to see by eye on navies after 24 hours. It appears that radiation amplifies the differences quickly so that the flow fields visibly diverge after 12 hours or so. You can see this in diffwrf too:

[dnolan@visx test_mp16_mzo_2_sw4_lw4_cu6]$ /nethome/dnolan/wrf3.9.1.1/WRFV3/external/io_netcdf/diffwrf wrfout_d01_2015-09-03_00:00:00 ../test_mp16_mzo_2_sw4_lw4_cu6_restart/wrfout_d01_2015-09-03_00\:00\:00
Just plot F
Diffing wrfout_d01_2015-09-03_00:00:00 ../test_mp16_mzo_2_sw4_lw4_cu6_restart/wrfout_d01_2015-09-03_00:00:00
Next Time 2015-09-03_00:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
REFL_10CM 2592000 3 0.3490694292E+02 0.0000000000E+00 15 0.3491E+02 0.1000E+01
Next Time 2015-09-03_03:00:00
Field Ndifs Dims RMS (1) RMS (2) DIGITS RMSE pntwise max
U 2602669 3 0.4983373169E+01 0.4983368420E+01 6 0.3183E-01 0.1678E+00
V 2593473 3 0.1724658754E+01 0.1724749825E+01 4 0.3207E-01 0.1673E+00
W 2592000 3 0.2983696652E-01 0.2990205514E-01 2 0.7679E-02 0.4069E+00
PH 2591715 3 0.6718111143E+03 0.6718180920E+03 4 0.1496E+01 0.2707E-01
T 2590601 3 0.4602067661E+02 0.4602067836E+02 7 0.2161E-01 0.1684E-01
MU 43200 2 0.5843383281E+03 0.5843373447E+03 5 0.1171E+01 0.2289E-01
P 2561592 3 0.4398090990E+03 0.4398098203E+03 5 0.1851E+01 0.8827E-01
Q2 43144 2 0.1893494736E-01 0.1893497551E-01 5 0.1271E-04 0.1975E-01
T2 42658 2 0.2997048342E+03 0.2997048241E+03 7 0.1600E-01 0.1611E-02
TH2 42521 2 0.2984602107E+03 0.2984602024E+03 7 0.1606E-01 0.1593E-02
PSFC 42896 2 0.1014764858E+06 0.1014764839E+06 7 0.1328E+01 0.2904E-03
U10 43197 2 0.5033891494E+01 0.5033881882E+01 5 0.1831E-01 0.2169E-01
V10 43199 2 0.1989033922E+01 0.1989089068E+01 4 0.1871E-01 0.3642E-01
QVAPOR 2591049 3 0.8859700030E-02 0.8859702808E-02 6 0.1419E-04 0.6978E-01
QCLOUD 62061 3 0.9910266026E-04 0.9903586233E-04 3 0.7834E-05 0.4927E+00
QRAIN 13721 3 0.6975724050E-05 0.6991432285E-05 2 0.8855E-06 0.1899E+00
QICE 205066 3 0.9461488733E-05 0.9455691198E-05 3 0.1080E-05 0.7519E+00
QSNOW 72516 3 0.1907876466E-04 0.1909885519E-04 2 0.1688E-05 0.2946E+00
QGRAUP 24437 3 0.3269896423E-05 0.3273094103E-05 3 0.4826E-06 0.2398E+00
QNCCN 2470272 3 0.9426524846E+08 0.9426550742E+08 5 0.8495E+06 0.7432E+00
QNCLOUD 62003 3 0.3441546857E+07 0.3432908647E+07 2 0.8260E+06 0.8947E+00
QNRAIN 13786 3 0.7562575983E+03 0.7949604782E+03 1 0.5317E+03 0.1000E+01
O3RAD 2417759 3 0.2117619381E-06 0.2117619378E-06 8 0.1709E-11 0.9205E-04
H_DIABATIC 626230 3 0.1192250233E-03 0.1192473186E-03 3 0.5121E-04 0.4914E+00
QV_DIABATI 732044 3 0.4636814138E-07 0.4638313171E-07 3 0.2035E-07 0.5051E+00
QC_DIABATI 831768 3 0.4434263011E-07 0.4431143736E-07 3 0.2023E-07 0.1270E+01
TSK 3986 2 0.3001415568E+03 0.3001415562E+03 8 0.6461E-04 0.9549E-05
RAINC 26977 2 0.9339385673E+01 0.9339575528E+01 4 0.1333E+00 0.3462E-01
RAINNC 2899 2 0.9568220333E+00 0.9570802033E+00 3 0.2451E-02 0.3129E-02
RAINCV 13104 2 0.1609160997E-01 0.1597093786E-01 2 0.1205E-01 0.8466E+00
RAINNCV 1254 2 0.2978218287E-02 0.2907740855E-02 1 0.1591E-03 0.5025E-01
REFL_10CM 32264 3 0.3486886807E+02 0.3486886049E+02 6 0.8514E+00 0.1147E+01
RTHRATLW 2590920 3 0.4588211029E-04 0.4587732168E-04 3 0.7246E-05 0.9633E+00
CLDFRA 163366 3 0.2044187772E+00 0.2043309812E+00 3 0.2924E-01 0.1000E+01
GLW 43146 2 0.4197072022E+03 0.4196940411E+03 4 0.1396E+01 0.7553E-01
ACSWUPT 88 2 0.1603332087E+08 0.1603332087E+08 9 0.4917E+01 0.1454E-04
ACSWUPB 69 2 0.3597488572E+07 0.3597488572E+07 10 0.1325E+00 0.2859E-05
ACSWDNB 74 2 0.4496860865E+08 0.4496860866E+08 10 0.1662E+01 0.2745E-05
ACLWUPT 35883 2 0.4835819506E+08 0.4835819994E+08 6 0.8510E+04 0.5260E-02
ACLWUPTC 26374 2 0.4975998760E+08 0.4975998681E+08 7 0.1959E+03 0.2739E-03
ACLWUPB 13694 2 0.8436437060E+08 0.8436436977E+08 8 0.8343E+02 0.4529E-04
ACLWUPBC 1474 2 0.8432284985E+08 0.8432284983E+08 9 0.1330E+02 0.6797E-05
ACLWDNB 33999 2 0.7702857166E+08 0.7702853223E+08 6 0.3691E+04 0.2235E-02
ACLWDNBC 24269 2 0.7491348798E+08 0.7491348751E+08 8 0.2127E+03 0.1139E-03
LWUPT 43119 2 0.2590928801E+03 0.2590963935E+03 4 0.4321E+01 0.3509E+00
LWUPTC 43086 2 0.2705870971E+03 0.2705869640E+03 6 0.7604E-01 0.1232E-01
LWUPB 40716 2 0.4595932397E+03 0.4595929753E+03 6 0.2796E-01 0.1444E-02
LWUPBC 39757 2 0.4593429878E+03 0.4593429823E+03 7 0.1668E-02 0.9127E-04
LWDNB 43146 2 0.4197072022E+03 0.4196940411E+03 4 0.1396E+01 0.7553E-01
LWDNBC 43128 2 0.4068400001E+03 0.4068398782E+03 6 0.8072E-01 0.5079E-02
OLR 43119 2 0.2590928801E+03 0.2590963935E+03 4 0.4321E+01 0.3509E+00
RUBLTEN 2567050 3 0.2893937383E-04 0.2836476695E-04 1 0.1456E-04 0.1461E+01
RVBLTEN 2582221 3 0.2404725191E-04 0.2377141642E-04 1 0.1377E-04 0.1237E+01
RTHBLTEN 515914 3 0.7914759521E-04 0.7915516981E-04 4 0.2992E-04 0.1709E+01
ZNT 43192 2 0.2878425665E-04 0.2878101273E-04 3 0.1142E-06 0.1634E-01
CK 43186 2 0.1204302838E-02 0.1204305854E-02 5 0.1525E-05 0.3536E-01
CD 43129 2 0.1042096016E-02 0.1042097756E-02 5 0.8832E-06 0.2065E-01
UST 43197 2 0.1799909033E+00 0.1799904783E+00 5 0.5740E-03 0.1639E-01
PBLH 43199 2 0.4497902073E+03 0.4498207894E+03 4 0.9470E+01 0.4045E+00
HFX 43200 2 0.4978486472E+01 0.4976732638E+01 3 0.1505E+00 0.1063E+00
QFX 43198 2 0.3907855491E-04 0.3907888802E-04 5 0.1545E-06 0.3731E-01
TML 3986 2 0.3001415568E+03 0.3001415562E+03 8 0.6461E-04 0.9549E-05
HML 1931 2 0.4030804280E+02 0.4030804430E+02 7 0.1786E-03 0.1391E-03
HUML 43179 2 0.1314534377E+00 0.1314460931E+00 4 0.1467E-03 0.4483E-02
HVML 43169 2 0.2491180055E+00 0.2491174991E+00 5 0.9538E-04 0.2304E-02

...and so on.

So now we have tracked the problem to a different area, microphysics.


Dave
 
Dave,
Thanks for identifying the new information. I've been working on this off/on and running various tests. I am able to reproduce what you are seeing - specifically with WDM6 and Thompson mp schemes. I have been corresponding with colleagues here about this and we are trying to track down the cause. Unfortunately so far everything we find is completely different with other cases that cause differences (e.g., the variable causing trouble with WDM6 is not the same as those that cause the problem with Thompson), and it's not clear why the differences happen to begin with. We will keep you posted with anything we are able to resolve.
 
You must log in or register to reply here.
Top