metgrid crashes: get_min(): No items left in the heap

Topics related to running the metgrid.exe program
Post Reply
bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Thu Apr 16, 2020 2:28 am

I can run ungrib, ghrsst-to-intermediate (writing SST:* files a bit bigger than my WRF domain), and avg_tsfc, just fine.

When I get to the metgrid stage, it gets through processing d01 (for a 5.5-day run I have 23 files), then writes 15 met_em.d02.* files, then stops with this message in both metgrid.out and metgrid.log:

ERROR: get_min(): No items left in the heap.

I'm not finding many clues using Google.

I'm using the 2019 version of PGI-CE:

# pgf90 -V

pgf90 19.10-0 64-bit target on x86-64 Linux -tp piledriver
PGI Compilers and Tools
Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.

I've attached my configure.wrf and configure.wps.

The compute nodes I'm running on have MemTotal: 65932648 kB according to /proc/meminfo. I was running 8 copies (8 consective 5-5-day periods) at the time, but the failure happens at the same stage regardless of how many other copies of metgrid are running at once.

I had accidentally set $OPAL_PREFIX to the PGI-supplied version, not the openmpi-3.1.2 version, when I compiled. But that shouldn't matter for metgrid, right? That's only a run-time thing.

Any ideas?
Attachments
configure.wrf
(20.26 KiB) Downloaded 5 times
configure.wps
(3.44 KiB) Downloaded 10 times

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Thu Apr 16, 2020 6:42 pm

Could this be related to this oddity? wrf.exe and real.exe use my version of zlib, but the WPS components do not.

Code: Select all

% ldd WRF-4.1.3/run/real.exe | grep libz
        libz.so.1 => /usr/local/src/wrf/LIBS/lib/libz.so.1 (0x00007f8e5af17000)

% ldd WRF-4.1.3/run/wrf.exe | grep libz
        libz.so.1 => /usr/local/src/wrf/LIBS/lib/libz.so.1 (0x00007f71f446f000)

% ldd WPS-4.1/ungrib.exe | grep libz
        libz.so.1 => /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libz.so.1 (0x00007ff927317000)

% ldd WPS-4.1/metgrid.exe | grep libz
        libz.so.1 => /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libz.so.1 (0x00007f4da9217000)
I told WPS to use my version, in both LD_FLAGS and COMPRESSION_LIBS:

Code: Select all

% grep -C1 -i -e wrf -e lz WPS-4.1/configure.wps

WRF_DIR = ../WRF-4.1.3

WRF_INCLUDE     =       -I$(WRF_DIR)/external/io_netcdf \
                        -I$(WRF_DIR)/external/io_grib_share \
                        -I$(WRF_DIR)/external/io_grib1 \
                        -I$(WRF_DIR)/external/io_int \
                        -I$(WRF_DIR)/inc \
                        -I$(NETCDF)/include

WRF_LIB         =       -L$(WRF_DIR)/external/io_grib1 -lio_grib1 \
                        -L$(WRF_DIR)/external/io_grib_share -lio_grib_share \
                        -L$(WRF_DIR)/external/io_int -lwrfio_int \
                        -L$(WRF_DIR)/external/io_netcdf -lwrfio_nf \
                        -L$(NETCDF)/lib -lnetcdff -lnetcdf
--
#
COMPRESSION_LIBS    = -L/usr/local/src/wrf/LIBS/lib -ljasper -lpng -lz
COMPRESSION_INC     = -I/usr/local/src/wrf/LIBS/include
FDEFS               = -DUSE_JPEG2000 -DUSE_PNG
--
FNGFLAGS            = $(FFLAGS)
LDFLAGS             = -L/usr/local/src/wrf/LIBS/lib -lz -ljasper
CFLAGS              = -O -tp=istanbul
Do I have a problem in my configure.wps?

Ming Chen
Posts: 963
Joined: Mon Apr 23, 2018 9:42 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by Ming Chen » Thu Apr 16, 2020 7:20 pm

I don't think it is a compiling issue.
I suppose you built WPS in MPI mode, please let me know if I am wrong. In this case,

(1) can you run with a single processor, e.g.,
mpirun -np 1 metered.exe, and see whether you have the same problem?

(2) You can also try to run metgrid from the time when it failed, and see whether it can successfully process the data.
WRF Help Desk

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Thu Apr 16, 2020 7:37 pm

No, I compiled WPS-4.1 using option "5", Linux x86_64, PGI compiler (serial).

Trying to keep it simple.

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Fri Apr 17, 2020 12:20 am

Because the d01 processing completed, re-starting the d02 processing at the point of failure (3/4 the way through) completes OK, and writes all 4 domains' met_em* files. If I start at the beginning and run till right before the crash, all 4 domain's met_em* files are created.

But I have a few hundred of these runs to do, and breaking them up like that would not be ideal.

I ran a test WPS run of a single 5.5-day period, on an AMD compute node with 132 GB of RAM (crash was on an Intel node with 64 GB of RAM). The AMD run completed without this error.

Ten 5.5-day WPS runs (consective 5.5-day periods) all running on the same AMD node works just fine.

A single WPS run on the Intel node crashed with the same messasge, as did a single WPS run on a different Intel node.

I was watching via 'top' as the runs on the Intel nodes progressed, and have verified it was not memory swapping - did not run out of RAM.

I used the same OS image to deploy to all compute nodes (using OpenHPC).

Conclusion: this is architecture-dependent. I have been using a compiler switch (-tp=istanbul) to maintain support for a few very old compute nodes. The Intel nodes are -tp=sandybridge.

Any suggestions for compiler switches, or how to dig into this further, would be greatly appreciated.

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Fri Apr 17, 2020 12:46 am

One more detail:

I was using my new ghrsst-to-intermediate program to use MUR SST in these runs. If I turn that off and just use the ERA5 SST field, this error does not happen, even in the Intel nodes.

https://github.com/bbrashers/WPS-ghrsst-to-intermediate
https://github.com/bbrashers/WPS-interp-intermediate

See related posts:
https://forum.mmm.ucar.edu/phpBB3/viewt ... =65&t=9050
https://forum.mmm.ucar.edu/phpBB3/viewt ... =31&t=9032

Could this problem be fixed by manipulating METGRID.TBL? I have one LANDSEA mask in the ERA5 ungribbed FILE:* files, and a different LANDSEA mask (with the same name) in the SST:* files.

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Fri Apr 17, 2020 3:01 pm

Based on the test that did NOT have this problem on AMD nodes, I ran 77 WPS runs on AMD nodes overnight, with no more than 6 runs per node.

Every single run HAD THE SAME CRASH.

Any thoughts on where I should look?

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Sat Apr 18, 2020 2:56 am

More information:

If, when I convert the GHRSST netCDF file to WPS Intermediate format, I DO NOT include the LANDSEA mask in the SST:YYYY-MM-DD_HH file, then metgrid completes WITHOUT the "get_min(): no items left in the heap" error.

From a quick viewing using ncview of the met_em* files, they look OK.

See the advice given in https://forum.mmm.ucar.edu/phpBB3/viewt ... =31&t=9032 to include the LANDSEA mask.

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Mon Apr 27, 2020 10:02 pm

Now that WRF-4.2 and WPS-4.2 are released, could we pick this thread up again?

Summary:

Writing an SST field followed by a LANDSEA mask to each SST:YYYY-MM-DD_HH Intermediate file causes metgrid to crash with the error "get_min(): No items left in the heap".

Writing just the SST field to each Intermediate file does not cause the error.

This thread https://forum.mmm.ucar.edu/phpBB3/viewt ... =31&t=9032 says it's best to include the LANDSEA mask in each Intermediate file.

I just completed an annual simulation using MUR SST with no LANDSEA mask in the Intermediate files. Performance was good, I can see no weirdness near the shorelines in the 1.33km domain.

Thanks!

Ming Chen
Posts: 963
Joined: Mon Apr 23, 2018 9:42 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by Ming Chen » Thu Apr 30, 2020 3:25 am

If your SST data is spatially continuous, e.g., there is "SST" (equivalent to "SKINTEMP") over the land, then it is OK not to include LANDSEA in your SST files. In this case, I suppose "SKINTEMP" near the coast might be used in interpolation to produce SST at points near the coast. This certainly is not that accurate, but at least the data still look reasonable.
If your SST data only have values over the ocean, then without the landsea mask will introduce problems in metgrid. This is why I suggest you should include landsea mask in your SST file.
I don't know yet why metgrid crashed if you include landsea mask in your SST file. Can you send me your intermediate file, SST file that includes landsea mask, and namelist.wps to take a look? I would like to repeat your problem first, which might give me some idea what is wrong.
WRF Help Desk

iandragaud
Posts: 11
Joined: Thu Jun 20, 2019 5:02 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by iandragaud » Sat May 09, 2020 6:49 pm

Hi Bart.

I also have some similar issues on how to create intermediate files of the MUR SST.
Another researcher and I wrote a Fortran code to read the MUR SST NetCDF files and write the intermediate files (SST:YYYY-MM-DD_HH). First, we didn't put any information in the code to write the SST mask. I thought that "missing_value" in METGRID.TBL was able to isolate the SST and, in this way, an appropriate metgrid interpolation could be done. However, after some detailed analysis, I noticed that there were some unrealistic SST gradients very close to the coastline. These SST gradients were created because, close to the coastline, the real program joined the MUR SST and the SST from the global atmospheric model (GFS in my case). I provided more information on the subject here
https://forum.mmm.ucar.edu/phpBB3/viewt ... ask#p10811.
In some periods, when the MUR SST field was almost the same as the GFS SST field, the interpolated SST field looked OK near the coast. However, when the MUR SST field was very different from the GFS SST field, the interpolated SST field presented an unrealistic gradient near the coast. In the study region that I am interested in, it occurs due to the coastal upwelling, which is better represented in the MUR SST and it is not represented it the GFS SST (https://link.springer.com/article/10.10 ... 018-0622-5).
After noticing this, I realized that the use of an SST mask was necessary and I followed the suggestions presented here: https://www2.mmm.ucar.edu/wrf/users/FAQ ... _data.html.
So, I adapted the Fortran code to process the SST mask isolated from the SST, and in that case, the code wrote the MASK:YYYY-MM-DD_HH. Then, I used the mask in the simulation by modifying the METGRID.TBL and the namelist.wps. As the SST mask field is time-invariant, I informed it using the "constants_name" in the namelist.wps, and in the METGRID.TBL I added the lines masked=land and interp_mask=MASK(2). The value I used is because the SST MASK has values equal to 2 over the land surface.
I don't touch it for a while and as you can see here https://forum.mmm.ucar.edu/phpBB3/viewt ... ask#p10811, I haven't 'yet solved the issue of properly interpolating the MUR SST to the WRF model. I probably didn't modify METGRID.TBL or the namelist.wps correctly, or even the mask were not created correctly.
I hope this information can be useful. And now, with the information you provided, I would like to try some modification to the procedure I was using to see if we solve it.

Ian

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Sat May 09, 2020 9:28 pm

Hi Ian,

Does the problem boil down to having TWO LANDSEA masks, one for the MUR SST dataset and a second one for the rest of the global data (e.g. ERA5 or GFS)?

If you give only one the higher-resolution MUR SST LANDSEA as an invariant file in the "constants_name" in namelist.wps, does that work?

Bart

iandragaud
Posts: 11
Joined: Thu Jun 20, 2019 5:02 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by iandragaud » Mon May 11, 2020 8:42 pm

Hi Bart.

Thanks for sharing your code (ghrsst-to-intermediate.f90)! I've used this code to convert a MUR SST file and it was easy to use. I will explain the tests I did.

- I successfully ran metgrid using the files FILE:YYYY-MM-DD_HH and SST:YYYY-MM-DD_HH at the same time-frequency, without the LANDSEA mask.

- I successfully ran metgrid using the files FILE:YYYY-MM-DD_HH with a 3-hour temporal resolution and a constant SST:YYYY-MM-DD_HH. In this case, I didn't include the LANDSEA mask. For this case, I considered
fg_name = 'FILE',
constants_name= = 'SST:YYYY-MM-DD_HH',

- I ran metgrid without success using the files FILE:YYYY-MM-DD_HH and SST:YYYY-MM-DD_HH at the same time-frequency, with the LANDSEA mask. When the domain d03 was being processed, the model stoped with the same error you got: ERROR: get_min(): No items left in the heap.

After getting these results, my idea was to give the higher-resolution MUR SST LANDSEA as an invariant file in the "constants_name" in namelist.wps. According to the information I found at Q5 (https://www2.mmm.ucar.edu/wrf/users/FAQ ... _data.html), I thought it was possible to do this. Then, I did the following tests:

- I ran metgrid without success using the files FILE:YYYY-MM-DD_HH with a 3-hour temporal resolution and a constant SST:YYYY-MM-DD_HH including the LANDSEA mask in the same file. For this case I considered
fg_name = 'FILE',
constants_name= = 'SST:YYYY-MM-DD_HH',

- I ran metgrid without success using the files FILE:YYYY-MM-DD_HH with a 3-hour temporal resolution and a constant SST:YYYY-MM-DD_HH and LANDSEA:YYYY-MM-DD_HH. In this case, the LANDSEA mask wasn't in the same file as the SST. For this, I considered
fg_name = 'FILE',
constants_name= = 'SST:YYYY-MM-DD_HH', 'LANDSEA:YYYY-MM-DD_HH',

In the last two tests, I received the following error message: "ERROR: Cannot combine time-independent data with time-dependent data for field LANDSEA.mask". I still don't know how to avoid this error.

Ian

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Mon May 11, 2020 9:09 pm

Hi Ian,

When using a constants_name file, it should be constant - contain ONLY the land-sea mask. My program can generate a file like that for you, for example:

Code: Select all

# ghrsst-to-intermediate -l /path/to/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc
That will create a file named LANDSEA:YYYY-MM-DD_HH, without any SST fields in it. Try using that file for the constants_name file.

My latest idea is to avoid a possible name-collision, and rename (in the ghrsst-to-intermediate.f90 code) LANDSEA to SST_MASK, then refer to that in METGRID.TBL, something like this:

Code: Select all

========================================
name=SKINTEMP
mpas_name=skintemp
        interp_option=sixteen_pt+four_pt+wt_average_4pt+wt_average_16pt+search
        masked=both
        interp_land_mask  = SST_MASK(1)
        interp_water_mask = SST_MASK(0)
        fill_missing=0.
========================================
name=SST
        interp_option=sixteen_pt+four_pt
        fill_missing=0.
        missing_value=-1.E30
        interp_mask=SST_MASK(1)
        flag_in_output=FLAG_SST
========================================
Perhaps METGRID is getting confused by finding two different fields named LANDSEA, one from UNGRIB and one from ghrsst-to-intermediate.

I have confirmed that using the --nolandsea flag runs. I have not found any really bad-looking SST gradients near the coast, but I have not looked very closely.

Bart

Code: Select all

ghrsst-to-intermediate --help
 Usage:  ghrsst-to-intermediate [Options] file.nc
 Options:
     -g geo_em.d01.nc        Output sub-grid that covers this domain.
     -b Slat,Nlat,Wlon,Elon  Specify sub-grid manually (no spaces).
     -s | --sst              Output SST:YYYY-MM-DD_HH files.
     -i | --seaice           Output SEAICE:YYYY-MM-DD_HH files.
     -l | --landsea          Output LANDSEA:YYYY-MM-DD_HH files.
     -n | --nolandsea        Don't append LANDSEA mask to SST ICE files.
     -d | --debug            Print debug messages to the screen.
     -V | --version          Print version number and exit.
     -h | --help             Show this help message.
 Required:
     file.nc                GHRSST file downloaded from the JPL PODAAC.

 Converts GDS version 2 GHRSST datasets from JPL PODAAC (in netCDF format)
 to WPS intermediate format. See
 https://podaac.jpl.nasa.gov/datasetlist?ids=ProcessingLevel&values=*4*&search=GHRSST&view=list

 Example:
 ghrsst-to-intermediate -sst -l -g ../ERA5/geo_em.d01.nc
 2014/20140101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc

iandragaud
Posts: 11
Joined: Thu Jun 20, 2019 5:02 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by iandragaud » Tue May 12, 2020 6:06 pm

Hi Bart,

Thanks for the information!
In the second test I did, I considered the SST to be constant. I did this to try to control all the possibilities and understand the errors. In addition, it's a common practice to consider some fields constant in short simulations, such as the SST
(slide 40, https://www2.mmm.ucar.edu/wrf/users/tut ... eneral.pdf).
In the last test, I considered what you suggested. I converted the MUR SST file and created a file named LANDSEA:YYYY-MM-DD_HH, without any SST fields in it. To do it, I used the -l option. However, I received the following error message: "ERROR: Cannot combine time-independent data with time-dependent data for field LANDSEA.mask".

The idea of avoiding a possible name-collision sounds good!

Ian

bartbrashers
Posts: 59
Joined: Wed Aug 08, 2018 2:21 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by bartbrashers » Tue Jun 09, 2020 6:41 pm

I can confirm that this METGRID crash - get_min(): No items left in the heap - does not occur if the land-sea mask in SST WPS Intermediate files is not named the same as in the main output from UNGRIB. That is, if you are writing your own Intermediate files, don't name the land-sea mask LANDSEA. I named it SST_MASK, and made the changes to METGRID.TBL as in my previous post, and avoided this crash.

It makes sense that LANDSEA is "visible" globally in METGRID. I had assumed when reading in an SST Intermediate file it would be a local variable.

iandragaud
Posts: 11
Joined: Thu Jun 20, 2019 5:02 pm

Re: metgrid crashes: get_min(): No items left in the heap

Post by iandragaud » Wed Jun 24, 2020 2:20 pm

Thank you very much for providing this useful information!

Post Reply

Return to “metgrid”