MPAS_init_atmoshere not finding soil layers in the intermediate files

Y

yibssss

New member
Hi MPAS Forum!

I'm trying to run MPAS limited area from MERRA2 initial conditions. I have the WPS intermediate file processed from a online merra2wps GitHub repository and ready.
I have nSoilLevels = 4 and nFGSoilLevels = 5 since MERRA2 has 5 soil temp and 5-ish soil moisture layer that was interpolatted. However, init_atmosphere core crashed on "Error in interpolation of st_fg to MPAS grid: num_st = 368892". and before that message the log said Bad st_fg and Bad sm_fg from 1 to 139358 (i have 139358 cells in my mesh) so every single cell is missing its soil moisture and soil temperature :(. [The screenshot of the log is attached]

I've checked the intermediate files with WPS/utils/rd_intermediate.exe and looks like its there! I'm also doing a wrf run in the meantime so i ran metgrid with same data with a customized METGRID.TBL and the data looks good.

Anybody have any hint on how to fix it?

Thanks!
 

Attachments

  • 1789075764429.png
    1789075764429.png
    108.7 KB · Views: 5
  • 1789076094986.png
    1789076094986.png
    106.4 KB · Views: 3
Hi,
Would you please upload one single intermediate file for me to take a look? It is also helpful if you can send me your namelist.init_atmosphere and streams_init_atmopshere. In addition, which version of MPAS did you run?

If the same data works for WPS, please send me your modified METGRID.TBL. This information may help me debug what is the issue in MPAS.

By the way, please attach your log file instead of screenshots. Thanks.
 
Ming Chen said:
Hi,
Would you please upload one single intermediate file for me to take a look? It is also helpful if you can send me your namelist.init_atmosphere and streams_init_atmopshere. In addition, which version of MPAS did you run?

If the same data works for WPS, please send me your modified METGRID.TBL. This information may help me debug what is the issue in MPAS.

By the way, please attach your log file instead of screenshots. Thanks.
Click to expand...
Thanks Dr Ming Chen!!!

Yep below is the log.init_atmosphere files, MERRA2 intermediate files, a processed met_em files just in case, streams and namelist.init_atmosphere files! I also have my static.nc and grid.nc too!
Also did confirm on the WRF side ./real.exe and WRF.exe worked with this intermediate file.
It is MPAS 8.3.1 and sorry about taking the screenshots will attach the logs next time!

Uh oh how do I upload files in the forum that are too large? 0.0


Thanks!!
 

Attachments

Ming Chen said:
Hi, yibssss

If files are too large to upload, package files as a single .tar file, and upload to our Box storage. Please let me know the file name and/or link to the file. Thanks.

Ming
Click to expand...
Thanks so much again! I uploaded it looks like it doesnt have a link, the file name is
yibsss_mpas_init_atm_smst_issue_mpas_forum_sep2026
tar file! and those should be in their!

And1789148174385.png
 
@yibssss

I have downloaded your data and will look into the issue. I will keep you updated about the progess.

Because of the long waiting queue in NCAR HPC, it may take time for me to repeat your case and debug what is wrong.Thank you for your patience.
 
Hi yibssss,

Your MERRA data looks fine.

How many processors did you use to run this case? Please upload the corresponding graph.info file so that I can repeat your case. Thanks.
 
Ming Chen said:
Hi yibssss,

Your MERRA data looks fine.

How many processors did you use to run this case? Please upload the corresponding graph.info file so that I can repeat your case. Thanks.
Click to expand...
Thanks so much Dr Chen!!!
So far i only used 1 process! But once all the setup is done, I plan to use 32 processes. I attached all the partition made here with gpmetis!
 

Attachments

Hi yibssss,

I have repeated your case but ran it with 1280 processors. MPAS crashed with the error message:

CRITICAL ERROR: Error in interpolation of st_fg to MPAS grid: num_st = 240

This is from a single log.err file. Similar messages appeared in various log.err files.

I checked your mesh, and the terrain in your regional domain is shown below:
1789583035137.png

I am suspicious that your static data may be wrong, which leads to MPAS failure.

How did you create this regional mesh and produce static datafile?
 
Ming Chen said:
Hi yibssss,

I have repeated your case but ran it with 1280 processors. MPAS crashed with the error message:

CRITICAL ERROR: Error in interpolation of st_fg to MPAS grid: num_st = 240

This is from a single log.err file. Similar messages appeared in various log.err files.

I checked your mesh, and the terrain in your regional domain is shown below:
View attachment 20837

I am suspicious that your static data may be wrong, which leads to MPAS failure.

How did you create this regional mesh and produce static datafile?
Click to expand...
yikes yikes
For my static datafiles, i downloaded it from the documentation website Static Geographic Datasets — MPAS Atmosphere documentation default terestrial dataset.
For the mesh, I downloaded the 12km mesh files from Meshes & Mesh Utilities — MPAS Atmosphere documentation, then carved the conus out with the MPAS_Limited_Area python script in the same link!
To finally create the .nc, i followed the tutorial here with init_atmosphere 7. Running MPAS-Atmosphere — MPAS Atmosphere documentation

And so sorry Dr Chen! I realized I made an error and the conus.static file i sent is made with MPAS 8.3.1, but with 8.4.0 its still the same crash.
I also did some debugging on my side and i used the same mesh(grid.nc) and static(static.nc). But instead i used ERA5 condition to generate init.nc so and that worked (for a seperate run)!
Thanks so much again for the help 😭 and what's the next step i should do for debugging?
 
Hi yibssss,

Your approach sounds correct.

If the same case works with the same static datafile and ERA5 , then I would suppose that something in MERRA is wrong or at least inconsistent with MPAS.

Can you manage to change the fill value of ST and SM from 1.e+15 to zero in MERRA_2012-03-01_00, then try again? I checked the data again, and all look fine to me except the fill value of ST and SM, ---- I don't expect these values to be used. But just in case, please try and let me know whether you still get the same error.

I am running the debug mode to further investigate what is wrong. My job is hanging in the waiting queue now ....
 
Ming Chen said:
Hi yibssss,

Your approach sounds correct.

If the same case works with the same static datafile and ERA5 , then I would suppose that something in MERRA is wrong or at least inconsistent with MPAS.

Can you manage to change the fill value of ST and SM from 1.e+15 to zero in MERRA_2012-03-01_00, then try again? I checked the data again, and all look fine to me except the fill value of ST and SM, ---- I don't expect these values to be used. But just in case, please try and let me know whether you still get the same error.

I am running the debug mode to further investigate what is wrong. My job is hanging in the waiting queue now ....
Click to expand...
Hi Dr Chen!

Sorry for the slow reply! Took me a while to figure out how to edit netcdf file and the fill value. I change the fillvalue and any actual value that is 1.e+15 from 1.e+15 to 0. Then generated new intermediate file MERRA_2012-03-01_00 from it, unfortunately still the same error :( Any good news on your side?

Thanks!
 
Hi yibssss,

I recompiled MPAS in debug mode, modified the code and tried to find where the negative ST/SM appears in the domain. The model seemed to stop immediately with some weird output, for example, it shows that st_fg is negative at certain ocean points. But the intermediate file doesn't contain any negative values of ST. I am really confused by this model behavior.

I will continue to look at this issue. It will take some time because of the long waiting queue in NCAR HPC.....
 
Last edited:
Hi yibssss,

I found the issue in your case. In your MERRA intermediate file, you have soil data named as:

SM000010, SM010020, SM020040, SM040080, SM080150 (same for ST data)

In mpas_init_atm_cases.F, MPAS processes fields named as

SM010040, SM040100, SM100200, etc. The field names used in your MERRA data (except SM000010 and except ST000010) are not processed. As a result, the model crashed with the error message shown in your log file.

Please modify mpas_init_atm_cases.F. You need to:

(1) add the field names of MERRA to the code for processing

(2) change dzs_fg and zs_fg to make them consistent with your MERRA data

Then recompile and run maps_init. Please let me know if you still have issues regarding this case.
 
Ming Chen said:
Hi yibssss,

I found the issue in your case. In your MERRA intermediate file, you have soil data named as:

SM000010, SM010020, SM020040, SM040080, SM080150 (same for ST data)

In mpas_init_atm_cases.F, MPAS processes fields named as

SM010040, SM040100, SM100200, etc. The field names used in your MERRA data (except SM000010 and except ST000010) are not processed. As a result, the model crashed with the error message shown in your log file.

Please modify mpas_init_atm_cases.F. You need to:

(1) add the field names of MERRA to the code for processing

(2) change dzs_fg and zs_fg to make them consistent with your MERRA data

Then recompile and run maps_init. Please let me know if you still have issues regarding this case.
Click to expand...
Hi Dr Ming!

So I added the field names and the associated if statements to the fortran code (attached below). I don't really know any Fortran so i basically just copied what SM000010 has and pasted it, replacing the name itself, as well as the dzs_fg and zs_fg.
However, this still ended up with the same error :( Thanks so much for your help last several week, even though the issue is still here it helped me learned so much about MPAS!
 

Attachments

Your modification of mpas_init_atm_cases.F looks NOT correct.

I will send you the code later and hopefully it works for you.
 
Last edited:
Ming Chen said:
Hi yibssss,

Please replace your code by the code I attach here. then recompile MPAS and try again. Let me know if you still have some issues. Thanks.
Click to expand...
It does the trick, init_atmosphere completed the init.nc file generation and the subsequent lbc/sst_update, and the actual model is running!!! Thank you so so so much for the help from you Dr Ming Chen and NCAR!!!

Will summarize it for anyone running in the same trouble:

MERRA 2 data has Soil Moisture and Soil Temperature field names that is not processed in init_atmosphere and thus generating a init.nc would crash. The solution is adding the field to the mpas_init_atm_cases.F under the src/core_init_atmosphere. Which I (op) don't know how to do so the tagged reply above has Dr Ming Chen's Fortran code modification that can process the irregular levels for MERRA2 in IC data.
 
You must log in or register to reply here.
Back
Top