Error running init_atmosphere: Invalid DateTime string (invalid time substring) initial_time

[steve-marshall-piq]

I'm having quite a bit of trouble running mpas_init_atmosphere. I am running MPAS v8.2.2, compiled within the JEDI framework (mpas-bundle, release 3.0.2). I am running this within the latest jedi development container on a Linux system (not a super-computing environment).

I have used the WPS ungrib program to convert a GDAS GRIB2 files into a file in WRF intermediate format. I have downloaded the 480km MPAS mesh and static files. I untarred these files and have links to all of these files in my current working directory (i.e. the "run" directory). NOTE: I have also tried this with the 30 km mesh, and gotten the same error, but it takes much longer to produce.

I have created the namelist.init_atmosphere and streams.init_atmosphere config files according to a tutorial found here (page 17). I've attached both of those files and the logs from my run of mpas_init_atmosphere.

I get warning messages about my namelist file, but the problems do not seem to be fatal. I've tried to resolve those issues, but some of the configuration did not work, so I have commented out the interpolation_control and decomposition sections.

The fatal problem is coming from configuration of the input stream in streams.init_atmosphere.
------
Details:
For initial testing, I am trying to make this work using the coarsest global mesh: the 480 km uniform MPAS mesh (x1.2562).

This is the command I am using to create an MPAS initialization file:
mpirun -np 1 /home/smarshall/jedi/mpas-bundle/rel3/build/bin/mpas_init_atmosphere

Here is the contents of my run directory (which I also used for running ungrib). The run directory is my current working directory when I run the command above:

-rw-r--r-- 1 smarshall smarshall 39487682 Apr 18 20:39 gdas.t00z.pgrb2.1p00.f000
lrwxrwxrwx 1 smarshall smarshall 25 Apr 21 12:47 GRIBFILE.AAA -> gdas.t00z.pgrb2.1p00.f000
lrwxrwxrwx 1 smarshall smarshall 75 Apr 21 12:47 Vtable -> /home/smarshall/jedi/mpas-bundle/rel3/WPS/ungrib/Variable_Tables/Vtable.GFS
-rw-rw-r-- 1 smarshall smarshall 709 Apr 21 12:48 namelist.wps
-rw-rw-r-- 1 smarshall smarshall 51391784 Apr 21 13:06 GFS:2024-01-01_00
-rw-rw-r-- 1 smarshall smarshall 30614 Apr 21 13:06 ungrib.log
lrwxrwxrwx 1 smarshall smarshall 100 Apr 21 14:43 x1.2562.static.nc -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.static.nc
lrwxrwxrwx 1 smarshall smarshall 101 Apr 21 18:28 x1.2562.graph.info -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.graph.info
lrwxrwxrwx 1 smarshall smarshall 109 Apr 21 18:28 x1.2562.graph.info.part.12 -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.graph.info.part.12
lrwxrwxrwx 1 smarshall smarshall 109 Apr 21 18:28 x1.2562.graph.info.part.16 -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.graph.info.part.16
lrwxrwxrwx 1 smarshall smarshall 108 Apr 21 18:28 x1.2562.graph.info.part.4 -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.graph.info.part.4
lrwxrwxrwx 1 smarshall smarshall 108 Apr 21 18:28 x1.2562.graph.info.part.2 -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.graph.info.part.2
lrwxrwxrwx 1 smarshall smarshall 108 Apr 21 18:28 x1.2562.graph.info.part.6 -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.graph.info.part.6
lrwxrwxrwx 1 smarshall smarshall 108 Apr 21 18:28 x1.2562.graph.info.part.8 -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.graph.info.part.8
lrwxrwxrwx 1 smarshall smarshall 98 Apr 21 18:28 x1.2562.grid.nc -> /home/pyxis/Test/2023-12-15_ro_development/hofx_exp_202502/mpas_cache/mesh/x1.2562/x1.2562.grid.nc
-rw-rw-r-- 1 smarshall smarshall 970 Apr 21 19:49 namelist.init_atmosphere
-rw-rw-r-- 1 smarshall smarshall 403 Apr 21 20:12 streams.init_atmosphere
-rw-rw-r-- 1 smarshall smarshall 2886 Apr 21 20:12 log.init_atmosphere.0000.out
-rw-rw-r-- 1 smarshall smarshall 441 Apr 21 20:12 log.init_atmosphere.0000.err

The crux of the error (from the .out log file) is:
Initializing MPAS_streamInfo from file streams.init_atmosphere
Reading streams configuration from file streams.init_atmosphere
Found mesh stream with filename template x1.2562.static.nc
Using default io_type for mesh stream
ERROR: Invalid DateTime string (invalid time substring) initial_time
** Attempting to bootstrap MPAS framework using stream: input
Bootstrapping framework with mesh fields from input file 'x1.2562.static.nc'
* Requested field lbc_scalars is deactivated due to packages, or is a scratch variable.
* Requested field lbc_u is deactivated due to packages, or is a scratch variable.
* Requested field lbc_w is deactivated due to packages, or is a scratch variable.
* Requested field lbc_rho is deactivated due to packages, or is a scratch variable.
* Requested field lbc_theta is deactivated due to packages, or is a scratch variable.

Parsing run-time I/O configuration from streams.init_atmosphere ...

----- found immutable stream "input" in streams.init_atmosphere -----
filename template: x1.2562.static.nc
filename interval: none
direction: input
reference time: initial_time
record interval: -
CRITICAL ERROR: xml stream parser failed: streams.init_atmosphere

The error seems to come from parsing the value of "reference_time", which I have not explicitly set, but which is getting the value "initial_time". I've tried explicitly setting filename_interval, record_interval, and reference_time for the input stream, but I cannot find a format that does not result in some other kind of error.

 

[steve-marshall-piq]

I have been able to trace where this error is coming from:
ERROR: Invalid DateTime string (invalid time substring) initial_time

It comes from mpas_set_time, in src/framework/mpas_timekeeping.F, called from mpas_expand_string, called from src/driver/mpas_subdriver.F.
The codes is going down the first branch of logic:
if ( trim(filename_interval_temp) == 'none' ) then
call mpas_expand_string(ref_time_temp, blockID, mesh_filename_temp, mesh_filename)
else
call mpas_set_time(ref_time, dateTimeString=ref_time_temp, ierr=ierr)
call mpas_set_timeInterval(filename_interval, timeString=filename_interval_temp, ierr=ierr)
call mpas_build_stream_filename(ref_time, start_time, filename_interval, mesh_filename_temp, blockID, mesh_filename, ierr)
end if

The value of ref_time_temp is "initial_time". The code in mpas_set_time assumes it is some kind of formatted time.
In my streams.init_atmosphere, I added an attribute for reference_time. I also changed to use 120 km mesh (x1.40962), since this seems to be the default in the namelist.init_atmosphere "decomposition" section. I downloaded and replaced all the mesh and static files in my run directory.

<streams>
<immutable_stream name="input"
type="input"
io_type="pnetcdf"
reference_time="2024-01-01_00:00:00"
filename_template="x1.40962.static.nc"
input_interval="initial_only" />
<immutable_stream name="output"
type="output"
filename_template="x1.40962.init.nc"
packages="initial_conds"
output_interval="initial_only" />
</streams>

This eliminated the time parsing error, but there is still a critical error at the end of log.init_atmosphere.0000.out:
CRITICAL ERROR: xml stream parser failed: streams.init_atmosphere

There is no lower-level error message describing what went wrong.

The new log and config files from the second version of my tests are attached.

 

[steve-marshall-piq]

The error from the second run comes from this code in src/driver/mpas_subdriver.F:

mgr_p = c_loc(domain_ptr % streamManager)
call xml_stream_parser(c_filename, mgr_p, c_comm, c_ierr)
if (c_ierr /= 0) then
call mpas_log_write('xml stream parser failed: '//trim(domain_ptr % streams_filename), messageType=MPAS_LOG_CRIT)
end if

mpas_init_atmosphere exits with an error status of 76.

 

[steve-marshall-piq]

I've traced this error a bit further.
The error comes from stream_mgr_add_alarm_c in src/framework/mpas_stream_manager.F

I instrumented the code to get more detail:
call mpas_log_write('mpas_add_alarm_c: setting alarmTime_local using alarmTime=' // alarmTime)
call mpas_log_write('mpas_add_alarm_c: setting alarmTime_local using alarmInterval=' // alarmInterval)
if (trim(alarmTime) == 'start' .and. trim(alarmInterval) == 'final_only') then
alarmTime_local = mpas_get_clock_time(clock, MPAS_STOP_TIME, ierr=ierr_tmp)
ierr = ior(ierr, ierr_tmp)
call mpas_get_time(alarmTime_local, dateTimeString=alarmString)
call MPAS_stream_mgr_set_property(manager, streamID, MPAS_STREAM_PROPERTY_REF_TIME, alarmString, ierr=ierr_tmp)
ierr = ior(ierr, ierr_tmp)
else if (trim(alarmTime) == 'start') then
alarmTime_local = mpas_get_clock_time(clock, MPAS_START_TIME, ierr=ierr_tmp)
ierr = ior(ierr, ierr_tmp)
else
call mpas_set_time(alarmTime_local, dateTimeString=alarmTime)
end if
if (ierr /= MPAS_STREAM_MGR_NOERR) then
call mpas_log_write('mpas_add_alarm_c: error in setting time using alarmTime='//alarmTime)
end if

These are the error messages, which indicate the code is going through the bolded line of code above.
Creating stream from c...
-- Called MPAS_stream_mgr_set_property()
-- Called MPAS_stream_mgr_set_property()
-- Called MPAS_stream_mgr_set_property()
-- Called MPAS_stream_mgr_set_property()
-- Called MPAS_stream_mgr_set_property()
-- Called MPAS_stream_mgr_get_clock()
mpas_add_alarm_c: setting alarmTime_local using alarmTime=start
mpas_add_alarm_c: setting alarmTime_local using alarmInterval=initial_only
mpas_add_alarm_c: error in setting time using alarmTime=start
mpas_add_alarm_c: error in mpas_add_clock_alarm for initial or final alarm: initial_only
-- Called MPAS_stream_mgr_add_alarm()
mpas_add_alarm_c: error in MPAS_stream_mgr_add_alarm for streamID input
stream_mgr_add_alarm_c failed with error 1, streamID = input, input, start, interval_in2 initial_only
CRITICAL ERROR: xml stream parser failed: streams.init_atmosphere

It seems like perhaps the clock is not getting initialized correctly, so we cannot extract the configured start time from the clock? However, that is just speculation. I have not figured out how to test this.

I feel like I'm digging pretty deeply into this problem, but I may be digging in the wrong place.
Has anyone else seen a problem like this with mpas_init_atmosphere before?

 

[gdicker]

You say that you've instrumented the code, have you tried stepping through the code with something like GDB? Especially since you are using only 1 MPI rank. Make sure to build with DEBUG=true first.

I'm also thinking something is going off with getting the start time. I'm curious which of the statements after mpas_timekeeping.F line 1312 is being executed and what dateTimeString contains near line 1318.

 

[mgduda]

@steve-marshall-piq Would it be possible to test outside of the JEDI framework? If you have just an MPI library as well as the PnetCDF library available, you'd be able to compile the 'init_atmosphere' core directly out of the MPAS-Dev/MPAS-Model repository; I'd also suggest using the Make build system rather than the CMake build system, as described in these slides (specifically, slides 17 - 19): https://www2.mmm.ucar.edu/projects/mpas/tutorial/Virtual2025/lectures/downloading_and_compiling.pdf .

The 120-km static file available from https://www2.mmm.ucar.edu/projects/mpas/atmosphere_meshes/x1.40962_static.tar.gz is well tested, so you could try using that in your tests.

 

[steve-marshall-piq]

Thanks @gdicker and @mgduda for the suggestions. I had already switched to using the x1.40962 static file, but found the same problems. However, I'll stick with that mesh for future tests.

I tried using the same configuration and source input GRIB data (converted using ungrib from WPS) as another colleague who did not see this problem. They were running MPAS 8.1.0, built outside of JEDI, as @mgduda described, so there are two differences to test: how the code is built and which version of MPAS is being used. For reference, mpas-bundle 3.0.2 in JEDI is using MPAS v8.2.2

I'll start with building MPAS 8.2.2 outside of JEDI, as @mgduda suggested. If I still see problems, I'll try a downgraded version.

 

[steve-marshall-piq]

Building MPAS outside of JEDI using the MPAS Makefile solved the issue. The init_atmosphere_model behaved largely as expected. Details are below.

Thank you for your help! However, I could use some more help in figuring out how to fix this within the JEDI context.

I built the same version of MPAS used by the JEDI mpas-bundle v3.0.2 in the same container I have been using for JEDI development. So I have controlled for most of the factors between the case that works and the case that does not work.

I believe the main difference is that, in JEDI, MPAS is built using CMakeLists.txt, while the case that worked built MPAS with the Makefile.

Therefore, I suspect there is either something wrong with the MPAS CMakeLists.txt, or there is some interaction between how cmake/ecbuild is used in JEDI and MPAS that is leading to a bad build. Strangely, compilation of MPAS succeeds in JEDI using ecbuild (a cmake wrapper); the problems are only revealed by strange runtime errors.

Question: Are there any known issues or testing deficiencies with the MPAS CMakeLists.txt? If so, that might point to a good place to start to fix this problem.

Ultimately, I don't think mpas-bundle v3.x.x will be fully usable until this issue gets resolved.
-------------
Details:
Within the JEDI dev container, pnetcdf is already installed, just in a weird location.

You can tell the MPAS build where to find the PNETCDF library by setting the PNETCDF environment variable, like this:
export PNETCDF=`/opt/views/view/bin/pnetcdf-config --prefix`

Then I compiled MPAS v8.2.2. like this:
make -j 8 gnu CORE=init_atmosphere AUTOCLEAN=true

Then I executed the init command in the run directory using the 120 km setup with GFS input data:
mpiexec -np 1 /full/path/to/init_atmosphere_model

This ran for a minute or so, generating this warning at the end.
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL

However, it ran successfully, generating a valid output file x1.40962.init.nc, which I could inspect with ncdump.
It created a log log.init_atmosphere.0000.out, but not a log.init_atmosphere.0000.err file.

 

[steve-marshall-piq]

@mgduda - I meant to tag you on my previous post.

 

[steve-marshall-piq]

I was finally able to run this problem to a root cause: the ESMF calendar from an external version of ESMF (v8.6.0) was not being properly initialized.

The clue was this message, issued to stderr (not the log*.out or log*.err files):
ESMF_LogWrite(): ESMF_Log not open -- cannot ESMF_LogWrite(). Log message = Wrong argument specified - , need real calendar.
ESMF_LogWrite(): ESMF_Log not open -- cannot ESMF_LogWrite(). Log message = Wrong argument specified - Internal subroutine call returned Error

The first message comes from the ESMF release/v8.6.0 branch source file src/Infrastructure/TimeMgr/src/ESMCI_Calendar.C, Calendar::convertToTime() method when the calkindflag member is not set to a valid enumerated value. This is a sign the default ESMF calendar never got set.

It looks like the built-in ESMF time library in MPAS sets the default calendar. Perhaps earlier versions of the external ESMF also did this. The mpas-bundle CMakeLists.txt expects ESMF version to be at least 8.3.0, which is probably the version first used with mpas-bundle.

However, the default initialization of the ESMF calendar does not occur automatically in ESMF 8.6.0. In this version, a call to ESMF_Initialize is required. There may be a lighter-weight way to initialize just the time keeping subsystem of ESMF.

The call to ESMF_Initialize is done within the mpas_timekeeping_init function. However, the MPAS top-level CMakeLists.txt file defines the pre-processor symbol MPAS_NO_ESMF_INIT, which precludes this call from being made.

Solution part 1:
I determine that removing the MPAS_NO_ESMF_INIT preprocessor symbol caused ESMF_Initialize to be called, and then mpas_init_atmosphere runs without error. FYI mpas_init_atmosphere is the mpas-bundle name for the init_atmosphere_model program is standalone MPAS.
That is, in the top-level MPAS CMakeLists.txt, change:
< add_definitions(-DMPAS_EXTERNAL_ESMF_LIB -DMPAS_NO_ESMF_INIT)
to
> add_definitions(-DMPAS_EXTERNAL_ESMF_LIB)

The ESMF_Initialize call uses the configuration value config_calendar_type from the nhyd section of the namelist, but this defaults to "gregorian" so it does not need to be set.

After calling ESMF_Initialize(), the MPAS programs generate another log file called PET0.ESMF_LogFile. A single process of mpas_init_atmosphere created a log file of about 6KB in size. This file warns that writing these messages will have a performance penalty and should be avoided in production.

Solution part 2:
The recommended way to avoid creating these log files (except in cases of errors) is to specify logKindFlag=ESMF_LOGKIND_Multi_On_Error in calls to ESMF_Initialize in src/framework/mpas_timekeeping.F, e.g.
call ESMF_Initialize(defaultCalendar=ESMF_CALKIND_GREGORIAN, logKindFlag=ESMF_LOGKIND_Multi_On_Error)

Making this change in mpas_timekeeping.F avoids creation of the PET0.ESMF_LogFile, while still allowing mpas_init_atmosphere to run successfully.

How to move forward

I think two changes to the MPAS source will fix this problem:

• removing -DMPAS_NO_ESMF_INIT from the compilation flags in CMakeLists.txt
• adding logKindFlag=ESMF_LOGKIND_Multi_On_Error to the ESMF_Initialize calls in src/framework/mpas_timekeeping.F that are called when the pre-processor symbol MPAS_NO_ESMF_INIT is not defined.

These changes will not affect builds using the MPAS Makefile, since those builds will still define -DMPAS_NO_ESMF_INIT.

I am happy to create a pull request with these changes, if that is acceptable to the the MPAS development team (e.g. @mgduda). However, if there is an aspect of MPAS history that I do not understand, please let me know. It looks like the top-level CMakeLists.txt files was created in May, 2024 (about a year ago), so this part of the solution is reasonably new.

 

[Ming Chen]

Hi,
The files are too large to attach here. I put them to Forum Uploads, please see whetehr you can download ( ERA5:2016-01-01_00.txt.gz and ERA5:2016-01-01_06.gz).

 

[Ming Chen]

Let me know whether my ERA5 files work for you. Thanks.

 

[steve-marshall-piq]

Hi,
The files are too large to attach here. I put them to Forum Uploads, please see whetehr you can download ( ERA5:2016-01-01_00.txt.gz and ERA5:2016-01-01_06.gz).

I was able to download the ERA5 data. Thank you for providing it.
Despite the .txt suffix on one of the files, these both look like files in WRF intermedidate format, suitable as input to the init_atmosphere_model program. Is that correct?

 

[Ming Chen]

Hi,

Thank you for your response.

I am sorry that I mistakenly upload ERA5 datafiles here, which are supposed for another user. Anyway, please ignore my post if you don't need the ERA5 data files.

Just a quick answer to your question: yes the two files are in WRF intermedidate format, suitable as input to the init_atmosphere_model.