MPASSIT

[F-Colby-UML]

Has anyone used the utility MPASSIT (https://github.com/LarissaReames/MPASSIT/tree/master)? Larissa Reames was let go from NOAA in the early DOGE cuts and her email address no longer works.

I have been able to compile the code on a 4-node system but running the code results in an immediate crash without much information.

If anyone has suggestions, that would be helpful. I would love to be able to convert my MPAS output to WRF-style output.

Thanks!

 

[marencibia]

Check the error logs but in my experience you need to a bigmem partition or you'll get OOM errors.

I use 2 nodes (40 tasks each on my machine) for 15km horizontal resolution global runs. I use the following in my slurm script:
#SBATCH --qos=batch
#SBATCH --partition=bigmem
#SBATCH --nodes=2
#SBATCH -t 04:00:00
srun -n 80 MPASSIT/bin/mpassit namelist.input.fhr

One word of warning for global forecasts, the netcdf will end up with grid metadata in distances from the equator rather than angular widths needed for latlon projection:
// global attributes:

...
X = 15008.9589025252 ;
Y = 15008.9589025252 ;

If you're trying to produce grib products with UPP, you will need to hardcode the following in UPP/sorc/ncep_post.fd/INITPOST_MPAS.F and recompile:

call ext_ncd_get_dom_ti_real(DataHandle,'DX',tmp, &
1,ioutcount,istatus)
if(maptype==0)then
tmp=0.135
dxval=tmp*gdsdegr ! grid length in degrees for lat-lon proj
write(6,*) 'dxval= ',tmp
else
dxval=nint(tmp)
write(6,*) 'dxval= ',dxval
endif

call ext_ncd_get_dom_ti_real(DataHandle,'DY',tmp, &
1,ioutcount,istatus)
if(maptype==0)then
tmp=0.135
dyval=tmp*gdsdegr ! grid length in degrees for lat-lon proj
write(6,*) 'dyval= ',tmp
else
dyval=nint(tmp)
write(6,*) 'dyval= ',dyval
endif

Credit to GSL and OAR for this workaround.

 

[Ming Chen]

@ F-Colby-UML

I don't have an immediate answer to your question, Without detailed error information, It is hard to tell what is wrong.

Just for your reference, I have run a quick test using MPASSIT. Below is the namelist I used:

&config
grid_file_input_grid = "/glade/derecho/scratch/chenming/MPAS-Forum/conus.init.nc"
hist_file_input_grid = "/glade/derecho/scratch/chenming/MPAS-Forum/history.2024-03-15_00.01.00.nc"
diag_file_input_grid = "/glade/derecho/scratch/chenming/MPAS-Forum/diag.2024-03-15_00.01.00.nc"
block_decomp_file = "/glade/derecho/scratch/chenming/MPAS-Forum/conus.graph.info.part.1280"
output_file = "/glade/derecho/scratch/chenming/MPAS-Forum/mpassit_out.nc"
interp_diag = .true.
interp_hist = .true.
wrf_mod_vars = .true.
esmf_log = .false.
nx = 424
ny = 315
dx = 15000.0
dy = 15000.0
ref_lat = 39.0
ref_lon = -97.5
truelat1 = 30.0
truelat2 = 60.0
stand_lon = -97.5
target_grid_type = "lambert"
is_regional = .true.
/

History file contains 835586 meshes (global 60-3km mesh). I ran this package in NCAR HPC (derecho) using 10 nodes (1280 processors) and MPASSIT can run successfully.

I am not sure what global mesh you are using. Probably you can try with more processors just in case the crash is due to insufficient memory.

Please keep me posted how it works. Thanks.

 

[F-Colby-UML]

Hi Ming,
Thank you for your namelist and information. I did find a typo in my namelist.input file but fixing that didn't help. The program initiates on 4 processors (what I have and what I run the model on ) and then quits with pretty useless error messages. The output to the screen is below. I'll attach my namelist.input file and the histlist files. Have also included the PETO log file which I didn't find helpful but maybe there are clues there too. The grid info, history and diag files were too big for the Forum server. Am happy to send you those files directly if that would help. Maybe you can see where I'm not doing something right!

mpasuser@stratus:.../MPAS-Model/Mediterranean$ mpirun --mca orte_base_help_aggregate 0 -np 4 mpassit namelist.input > mpassit-run.txt
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode 999.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 999.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 999.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode 999.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
mpasuser@stratus:.../MPAS-Model/Mediterranean$

 

[Ming Chen]

HI,
Thank you for uploading the files. Your namelist looks fine to me. Based on the information you provided, I am suspicious that MPASSIT might fail due to insufficient memory. Can you run this program to process a smaller case, for example a case over 240-km mesh? This test can at least tell whether MPASSIT itself can work as expected.

 

[F-Colby-UML]

Hi Ming,
I ran MPASSIT on the 240km uniform grid output based on the tutorial setup. Again, the error messages are cryptic but there is one "out of resource" message - perhaps that indicates the memory issue you suggested?

mpasuser@stratus:.../MPAS-Model/240km_uniform$ mpirun -np 4 mpassit namelist.input
- NPETS IS 1
- NPETS IS 1
- NPETS IS 1
- NPETS IS 1
- FATAL ERROR:
- FATAL ERROR:
- FATAL ERROR:
READING SETUP NAMELIST.
- IOSTAT IS: 5010
- FATAL ERROR:
READING SETUP NAMELIST.
- IOSTAT IS: 5010
READING SETUP NAMELIST.
READING SETUP NAMELIST.
- IOSTAT IS: 5010
- IOSTAT IS: 5010
[stratus:966500] [[54521,0],0] ORTE_ERROR_LOG: Out of resource in file ../../../orte/util/show_help.c at line 507
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 999.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[stratus:966500] [[54521,0],0] ORTE_ERROR_LOG: Out of resource in file ../../../orte/util/show_help.c at line 507
[stratus:966500] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[stratus:966500] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

 

[Ming Chen]

I am perplexed by this error message, since the MPAS data on 240km mesh doesn't require a big memory to process....

I just did a few tests using MPASSIT to process data on various meshes, and I cannot repeat your errors, --- sorry that I don't have an immediate answer to your MPASSIT issue.

Just for your information, we provide a new tool that can create regional MPAS mesh by projecting a Cartesian mesh of perfect hexagons to the sphere using standard map projections. Please see details in MPAS-Tools/mesh_tools/hex_projection at master · MPAS-Dev/MPAS-Tools

If you want to look at MPAS results on Lambert projection, this option may provide an alternative for you. You can run regional MPAS directly over this mesh with lambert projection.

 

[F-Colby-UML]

Thanks for trying to figure out my issues! I wonder if there is some hardware-specific code buried in the MPASSIT routines. From the cryptic messages, it kind of looks like the code isn't even reading the namelist correctly. The code doesn't seem to ever get started on anything.

Let me know if you would want to see my grid, diag and history files to see if it all works on your system. It's weird, since I can run the model without any trouble, and it runs a bit faster than the WRF ( for a similar single grid) for the same real data case.

 

[Ming Chen]

Hi,
Below is the list of modules I used to build MPASSIT. This is in NCAR Derecho with intel compiler. Please take a look and see whether you can rebuild this package using same /similar modules.

#%Module

module purge


module load ncarenv/23.09 craype/2.7.23 intel/2023.2.1 ncarcompilers/1.0.0 cray-mpich/8.1.27


module load cmake/3.26.3


module load esmf/8.6.0


module load hdf5-mpi/1.12.2


module load mkl/2023.2.0


module load netcdf-mpi/4.9.2


module load parallel-netcdf/1.12.3


module load parallelio/2.6.2

 

[F-Colby-UML]

Hi Ming,

I don't have a lot of these modules. What is NCARENV? My version of netcdf is the same as yours. esmf is 8.9.0, a later version. I could never get PIO to compile when first trying MPAS, but then the MPAS code was changed to make that optional and that made it possible for me to get MPAS to run. It does run on both low res grids and relatively high res grids. I may said this too many times, but with WRF, it was easy to convert the output to GrADS which then made it easy to loop through plots of any variable, make cross sections and skewT/logP diagrams and many other very helpful images.

Nothing like that exists for MPAS. So MPASSIT was a great option. But I think it must be configured for a super computer. I don't have that. Have never really needed that for the model runs using WRF or MPAS. But I understand that MPASSIT may only run on Derecho.

I'm attaching one of the build files created during the compilation of MPASSIT. It may help you understand the environment I installed under. There is a log file that was created but it's too big for this server.

Thanks,
Frank

 

[Ming Chen]

Hi Frank,

I understand that the installation of MPASSIT could be an issue. Regarding GrADs, we didn't develop any post process packages to make MPAS output compatible with GrADs mainly because we always used NCL before. and now we are actively developing python packages for MPAS post-processing.

I am thinking whether you can run the utility convert_mpas and remap MAPS output to regular lat-lon grids? In this case you can manage to make the lat-lon gridded data compatible with GrADs. Hope this could be an solution for you.

 

[F-Colby-UML]

Thanks for the reply. I've used the convert utility and then tried to look at the output in GrADs but more needs to be done. The format didn't quite work.

I'll look forward to the python codes...!