Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

Static file creation fails on large meshes (Hera environment?)

W

wangevine

New member
Good day, I am having trouble at the step of creating the static file. I'm running on NOAA's Hera system. With the modules shown below, I can compile init_atmosphere and run the static step for meshes with grid and static files less than 2 GB. But anything bigger fails in the same way, with a segmentation fault in libpnetcdf. Details for a run of the 15km uniform mesh (x1.2621442) below. And yes, I know I don't have to create this static file, I could download it, but this is the smallest case that fails and therefore the fastest turnaround. Seems likely that the problem is with large file or large variable support in the environment, but I don't know what I'm doing wrong.

Module list (provided by NSSL):
1) intel/18.0.5.274 3) pnetcdf/1.11.2 5) hdf5parallel/1.10.6
2) impi/2018.4.274 4) szip/2.1 6) netcdf-hdf5parallel/4.7.4

Make command:
make intel-mpi CORE=init_atmosphere PRECISION=single DEBUG=true AUTOCLEAN=true PNETCDF=/apps/pnetcdf/1.11.2/intel/1.11.2 >make.out

Stack dump:
orrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpnetcdf.so.3.0 00007F9A22146DCC for__signal_handl Unknown Unknown
libpthread-2.17.s 00007F9A203A4630 Unknown Unknown Unknown
init_atmosphere_m 0000000000CFFA9C mpas_io_streams_m 2640 mpas_io_streams.F
init_atmosphere_m 0000000000B1DD45 mpas_stream_manag 3956 mpas_stream_manager.F
init_atmosphere_m 0000000000B1A746 mpas_stream_manag 3547 mpas_stream_manager.F
init_atmosphere_m 00000000004FA29A init_atm_core_mp_ 63 mpas_init_atm_core.F
init_atmosphere_m 000000000040DF12 mpas_subdriver_mp 368 mpas_subdriver.F
init_atmosphere_m 0000000000408894 MAIN__ 18 mpas.F
init_atmosphere_m 00000000004087CE Unknown Unknown Unknown
libc-2.17.so 00007F9A1FCE7555 __libc_start_main Unknown Unknown
init_atmosphere_m 00000000004086E9 Unknown Unknown Unknown

Streams file (relevant portion, NOTE that I also tried io_type=netcdf4 with the identical result):
<immutable_stream name="input"
type="input"
io_type="pnetcdf,cdf5"
filename_template="x1.2621442.grid.nc"
input_interval="initial_only" />

<immutable_stream name="output"
type="output"
io_type="pnetcdf,cdf5"
clobber_mode="overwrite"
filename_template="x1.2621442.static.nc"
packages="initial_conds"
output_interval="initial_only" />

Tail of log file:
Allocating fields ...
5096 MB allocated for fields on this task
----- done allocating fields -----

WARNING: Variable bdyMaskCell not in input file.
WARNING: Variable bdyMaskEdge not in input file.
WARNING: Variable bdyMaskVertex not in input file.
WARNING: File x1.2621442.grid.nc does not contain a seekable xtime variable. Forcing a read of the first time record.

No log...err file is created.

Can anyone who has run MPAS on Hera or Jet help? Thanks!
Wayne Angevine, CIRES / NOAA CSL
 
I should add that I am running on a whole compute node in interactive mode, so even though only one core is in use I have something like 86GB RAM available. I also tried on a bigmem node, no difference. report-mem shows that only a few percent of available memory is being used.
 
Hi Wayne,
I wonder whether you can run MPAS with multiple nodes on Hera or jet? I got the same error when I run the case will a smaller number of nodes in NCAR Cheyenne. Later I increase the number of nodes to 32 ( 1152 processors in total), and my job is able to run successfully.
Please use MPAS-v8.0, which enables parallel remapping of static fields. Hope this is helpful.
 
Thanks, Ming. I was going by the User Guide which still says that the static step must be done serially. I am running 8.0.1.

I tried with more cores and got farther. I tried up to 3840 cores which is the largest number available for nodes with 40 cores on this mesh. It fails while trying to calculate albedo (after GREENFRAC). No output is produced. Max memory use is shown as 82%.

I also tried reducing config_supersample_factor to 1 instead of 3, and running with 960 cores. (This is getting expensive). That failed sooner. I have no idea if that factor is relevant.

Here is part of my namelist:
&data_sources
config_geog_data_path = '/scratch2/BMC/calnexfc/angevine/WPS_GEOG'
config_met_prefix = 'ERA5'
config_sfc_prefix = 'SST'
config_fg_interval = 86400
config_landuse_data = 'MODIFIED_IGBP_MODIS_NOAH'
config_topo_data = 'GMTED2010'
config_vegfrac_data = 'MODIS'
config_albedo_data = 'MODIS'
config_maxsnowalbedo_data = 'MODIS'
config_supersample_factor = 3
config_use_spechumd = true

Note that I downloaded a fresh version of WPS GEOG data. I had to link topo_gmted2010_30s to landuse_30s, and rename modis_landuse_20class_30s_with_lakes to remove "with_lakes". It seems that MPAS is not up to date with the latest WPS GEOG naming.

I see the note that working with the larger meshes may require more memory, including "undersubscribing nodes." But this is the smallest possible larger mesh, if it's this difficult, how hard will it be when I want to work with the 15->3 km refined mesh that is my goal? If I need to use a non-default node/core allocation method (e.g. to get more memory), I'm going to need a recipe.
 
Hi Wayne,
Would you please upload your entire namelist.init_atmopshere and streams.init_atmosphere for me to take a look?
Also, please confirm that you download the mesh "x1.2621442.grid.nc" from the official MPAS website.
Thank you.
 
Yes, I downloaded the grid files from the official page MPAS-Atmosphere mesh downloads

namelist.init_atmosphere:
&nhyd_model
config_init_case = 7
config_start_time = '2013-07-20_00:00:00'
config_stop_time = '2013-07-21_00:00:00'
config_theta_adv_order = 3
config_coef_3rd_order = 0.25
config_interface_projection = 'linear_interpolation'
/
&dimensions
config_nvertlevels = 55
config_nsoillevels = 4
config_nfglevels = 38
config_nfgsoillevels = 4
/
&data_sources
config_geog_data_path = '/scratch2/BMC/calnexfc/angevine/WPS_GEOG'
config_met_prefix = 'ERA5'
config_sfc_prefix = 'SST'
config_fg_interval = 86400
config_landuse_data = 'MODIFIED_IGBP_MODIS_NOAH'
config_topo_data = 'GMTED2010'
config_vegfrac_data = 'MODIS'
config_albedo_data = 'MODIS'
config_maxsnowalbedo_data = 'MODIS'
config_supersample_factor = 3
config_use_spechumd = true
/
&vertical_grid
config_ztop = 30000.0
config_nsmterrain = 1
config_smooth_surfaces = true
config_dzmin = 0.3
config_nsm = 30
config_tc_vertical_grid = true
config_blend_bdy_terrain = false
/
&interpolation_control
config_extrap_airtemp = 'lapse-rate'
/
&preproc_stages
config_static_interp = true
config_native_gwd_static = true
config_vertical_grid = false
config_met_interp = false
config_input_sst = false
config_frac_seaice = false
/
&io
config_pio_num_iotasks = 0
config_pio_stride = 1
/
&decomposition
config_block_decomp_file_prefix = 'x1.2621442.graph.info.part.'
/

streams.init_atmosphere:
<streams>
<immutable_stream name="input"
type="input"
io_type="pnetcdf,cdf5"
filename_template="x1.2621442.grid.nc"
input_interval="initial_only" />

<immutable_stream name="output"
type="output"
io_type="pnetcdf,cdf5"
clobber_mode="overwrite"
filename_template="x1.2621442.static.nc"
packages="initial_conds"
output_interval="initial_only" />

<immutable_stream name="surface"
type="output"
io_type="netcdf4"
clobber_mode="overwrite"
filename_template="x1.2621442.sfc_update.nc"
filename_interval="none"
packages="sfc_update"
output_interval="86400" />

<immutable_stream name="lbc"
type="output"
io_type="netcdf4"
clobber_mode="overwrite"
filename_template="lbc.$Y-$M-$D_$h.$m.$s.nc"
filename_interval="output_interval"
packages="lbcs"
output_interval="3:00:00" />

</streams>
 
Static file creation succeeds if I use 960 tasks and 16 tasks per node. It fails with 20 tasks per node. So I conclude that static file creation for the 15km uniform grid requires about 6 GB of memory per task.

I wonder if I dare try it for the 15-3 km refined mesh?
 
Wayne,
Thank you for the update. I agree that the huge memory request is kind of annoying for MPAS run. I tried your case yesterday with 16 nodes, and the job stopped with the error "MPT: shepherd terminated: r2i6n13.ib0.cheyenne.ucar.edu - job aborting". This is definitely a machine issue possibly caused by insufficient memory. Note that 3,168 nodes have 45GB memory and the other 864 nodes have 109GB memory in Cheyenne.
Since MPASv8.0, parallel remapping of static fields is available, which is really helpful. Also, once the static data is created for a specific domain, it can be saved for future use. So I would say that the initial stage might be the hardest. Please try with other meshes and let us know if you have any issue. We are always here and would do our best to help.
 
Last edited:
You must log in or register to reply here.
Top