Issue Running MPAS-A from Restart File

[MattW]

Hi MPAS folks,

I've been trying to start an MPAS simulation from a restart file as part of setting up a cycled DA system with DART-MPAS. I keep running into this error:

----------------------------------------------------------------------
Beginning MPAS-atmosphere Error Log File for task 0 of 512
Opened at 2024/06/06 10:23:44
----------------------------------------------------------------------

ERROR: *******************************************************************************
ERROR: The GWDO scheme requires valid var2d, con, oa{1,2,3,4}, and ol{1,2,3,4} fields,
ERROR: but these fields appear to be zero everywhere in the model input.
ERROR: Either set config_gwdo_scheme = 'off' in the &physics namelist, or generate
ERROR: the GWDO static fields with the init_atmosphere core.
ERROR: *******************************************************************************
ERROR: Please correct issues with the model input fields and/or namelist.
CRITICAL ERROR: Core init failed for core atmosphere
Logging complete. Closing file at 2024/06/06 10:23:44

However, the restart file has nonzero values for all of these variables. I've tried copying them in from the init file in case they were incorrect, but I get the same error then. I've also tried this with MPAS 8.0.1 and 8.1.0 and gotten the same error. Setting config_gedo_scheme = 'off' leads to a segmentation fault without an interpretable error file. Could this possibly be due to an incorrect setting in my namelist.atmosphere or streams.atmosphere file? I've attached both here.

Thanks,

Matt Wilson

 

[Ming Chen]

Would you please let me know the path where you run this case in derecho? Thanks.

 

[MattW]

Sure! It's running at /glade/derecho/scratch/mawilson/mpas_OSSE/MPAS-DART/rundir/member1. Thanks!

 

[Ming Chen]

Looking at your history file, it seems that you have modified something in the MPAS?

git_version = "v8.0.0-13-g2bfdc6ba-dirty"

I did run a test using the standard version of MPASv8.0.1 and the restart works fine. However, when I ran your case, it failed with the same errors you posted.

I am not sure yet what is wrong ....

 

[MattW]

I had to make a small change to the registry file in my installation of MPASv8.0.1 so that MPAS saved the uReconstructedZonal and uReconstructedMeridional variables to the restart files, since those are required by DART. I get the same error when I use an unmodified version of MPAS 8.1.0 though.

 

[Ming Chen]

Matt,
My test case is saved at /glade/work/chenming/MPAS-V8.0.1/, where you can find namelist.atmosphere, streams.atmosphere, restart.2023-06-01_03.00.00.nc. Can you run my case using your code and let me know whether it works ?

 

[MattW]

Thanks! I just ran it with your namelist.atmosphere, streams.atmosphere, and restart file and got a different error. This time it says that 'on' is not a valid setting for the gwdo scheme. I also got the same error when using MPASv8.1.0. I've attached my log.atmosphere.out and log.atmosphere.err files.

 

[Ming Chen]

Hi Matt,

The namelist option should be:

 config_gwdo_scheme = 'bl_ysu_gwdo'

Sorry that I changed it for another test and forgot to change it back.

 

[MattW]

I was able to get MPAS to run successfully after making that change with your namelist and restart file. However, I then tried to run it with my restart file (changing the date and config_block_decomp_file_prefix in the namelist for my case) and it crashed with this error:

CRITICAL ERROR: NaN detected in 'w' field.

Does this mean there's something wrong with how I'm creating the restart file?

 

[MattW]

Also I ran this in a new directory (/glade/derecho/scratch/mawilson/mpas_OSSE/MPAS-DART/rundir/test_mpas) instead of the old member1 directory.

 

[Ming Chen]

I looked at your log file. It seems that the case crashed immediately after the restart run started, which often indicates some data issues.
Would you please run this case from scratch, i.e.,
(1) conduct a short, initial run (i.e. restart = false), and save the restart file
(2) conduct a restart run using the restart file you created at (1)
Please let me know how it works.
Please run with the official MPAS release codes.

 

[MattW]

I recreated the restart file using an unmodified version of MPASv8.1.0 (the short forecast run was completed in /glade/derecho/scratch/mawilson/mpas_OSSE/MPAS-DART/rundir/member01). However, using this new restart file led to the same error. I also tried recreating the restart file by re-running the short forecast using your namelist.atmosphere to see if an incorrect setting in my namelist was responsible for the error, and interestingly the same error shows up when trying to run the short free forecast to create the restart file using your namelist (the logs from that free forecast attempt are in /glade/derecho/scratch/mawilson/mpas_OSSE/MPAS-DART/rundir/member01/testnamelist_logs). Could there be another error in that namelist.atmosphere file that's leading to this particular issue?