insufficient virtual memory error while running geogrid.exe

[melika1389]

Hello there,

I have created one of the inputs of the geogrid and wanted to create the geogrid file for that specific input to make sure if the projections and everything is fine or not. Still, after running geogrid.exe, I am getting the error that there is not enough virtual memory, which I think is not because of not having enough CPU or memory because, in my last run, I gave it 32 cores with almost 32 Gb per core. It gave me the same error, so I think it must be something wrong with the procedure I have done until now. Q1. Usually, how many cores with how much memory per each core is needed for running the geogrid.exe? Q2. Can I run the geogrid.exe file multiple times to check the procedure and each time add more parameters to it?

I am adding all my steps to make sure I haven't missed anything:

The input data is a three arc-seconds Hydrosheds DEM which using ArcMap, after downloading the data and giving the wps84 as the geographical information system, I resampled the grids to a 30 arc-seconds grid. Then I export it as a tiff file, and using the following command; I saved it as an ENVI file.

gdal_translate -of ENVI -co INTERLEAVE=BSQ D:\ArcMaps\GL_dem_30s.tif D:\ArcMaps\GL_dem_30s.envi

Then I renamed the file to 0001-2526.0001-1248.

Q3. How can I give this dem file with the original resolution to the wps instead of resampling it to 30 arc-seconds? Can I add the original dx and dy whenever I need to add them?

Next I created the following index file for the data:

type = continuous
signed = yes
projection = regular_ll
dx = 0.00833333
dy = 0.00833333
known_x = 1.0
known_y = 1.0
known_lat = 45.579165
known_lon = -82.688335
wordsize=2
tile_x = 2526
tile_y=1248
tile_z=1
missing_value = 0
units="meter"
interp_option = default:average_gcell(4.0)+four_pt+average_4pt
rel_path = default:/DEM/hydrosheds_30s/
description="DEM"

And this is my namelist.wps:

&share
wrf_core = 'ARW',
max_dom = 1,
start_date = '2000-01-01_00:00:00'
end_date = '2000-12-31_21:00:00'
interval_seconds = 31611600
io_form_geogrid = 2,
/

&geogrid
parent_id = 1, 1,
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORTANT NOTE !!!!!!!!!!!!!!!!!!!!!!!!!!!!
! The default datasets used to produce the HGT_M, GREENFRAC,
! and LU_INDEX/LANDUSEF fields have changed in WPS v3.8. The HGT_M field
! is now interpolated from 30-arc-second USGS GMTED2010, the GREENFRAC
! field is interpolated from MODIS FPAR, and the LU_INDEX/LANDUSEF fields
! are interpolated from 21-class MODIS.
!
! To match the output given by the default namelist.wps in WPS v3.7.1,
! the following setting for geog_data_res may be used:
!
! geog_data_res = 'gtopo_10m+usgs_10m+nesdis_greenfrac+10m','gtopo_2m+usgs_2m+nesdis_greenfrac+2m',
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORTANT NOTE !!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
geog_data_res = 'default','default',
dx = 0.00833333,
dy = 0.00833333,
map_proj = 'lat-lon',
ref_lat = 45.579165,
ref_lon = -82.688335,
stand_lon = 82.688335,
geog_data_path = '/home/mel1389/GRIP/'
/

&ungrib
out_format = 'WPS',
prefix = 'FILE',
/

&metgrid
fg_name = 'FILE'
io_form_metgrid = 2,
/

And my GEOGRID.TBL:

# See options.txt for a (somewhat up to date) list of the
# options that may be specified here.

===============================
name = HGT_M
priority = 1
dest_type = continuous
smooth_option = smth-desmth_special; smooth_passes=1
fill_missing=0.
interp_option = default:average_gcell(4.0)+four_pt+average_4pt
rel_path = default:/DEM/hydrosheds_30s/
===============================

In the above namelist.wps, I gave the "stand_lon" as "-ref_lon" based on the WPS documentation as the case study is in N.H. Q4. Am I right?
Q5. Is the size of my input file, OK? My tile_x is 2526, which is more than 1200 than the usual wps files, and my tile_y is 1248, which makes the whole shape a rectangle, not a square, or do I have to create smaller files? Q6. Does the "gdal_translate" command, automatically fill the outside of the clipped tiff file with NAs? My tiff file has been clipped for my region.
Q7. What does "tile_bdr" stand for? I didn't define the parameters, so probably the default (zero) has been picked. Although in WPS examples, it was given, three. Do I need to define it?
Q8. About the "wordsize" parameter, my tile_x*tile_y is equal to 3152448, while the final file size is 6157 KB, which means each grid has almost 2 bytes. Is giving wordsize=2, OK?

I am looking forward to hearing from you.
Please do not hesitate to email me.

Regards,
Melika

 

[Ming Chen]

Melika,
Please see the following answers to your questions:
(1) Usually, how many cores with how much memory per each core is needed for running the geogrid.exe?
This depends on your grid size.
(2) Can I run the geogrid.exe file multiple times to check the procedure and each time add more parameters to it?
Yes you can. Please modify GEOGRID.TBL to keep only those parameters you want to process.
(3) How can I give this dem file with the original resolution to the wps instead of resampling it to 30 arc-seconds? Can I add the original dx and dy whenever I need to add them?
I am not sure of this question without fully understanding what gdal_translate can do.
(4) I gave the "stand_lon" as "-ref_lon" based on the WPS documentation as the case study is in N.H. Q4. Am I right?
This is not correct. ref_lon should be equal to stand_lon
(5) All the input tile must have the same x- and y-dimensions. In your case, the tile shape is not correct. Please change.
(7) What does "tile_bdr" stand for?
It is an integer specifying the halo width, in grid points, for each tile of data. Default value is 0
(8) About the "wordsize" parameter, my tile_x*tile_y is equal to 3152448, while the final file size is 6157 KB, which means each grid has almost 2 bytes. Is giving wordsize=2, OK?
Yes it is correct.

 

[melika1389]

Hello again,

I was working on the solution, so I had a couple of modifications.
1- My whole domain's grid size is 2526 * 1249, which after converting it to WPS's binary files, currently the tile_x and tile_y are equal to 500.
2- Thank you
3- Thanks
4- Thank you.
5- I did it.
7- Thanks
8- Thank you

 

[Ming Chen]

Please keep me updated about your progress. Thanks.

 

[melika1389]

Hello Ming,

I could hopefully run the geogrid.exe for my DEM and land cover data.
I have faced a couple of new questions that I was wondering if you can answer?
As I am creating WRF-Hydro's inputs for NoahMP LSM, I don't know how I should modify the GEOGRID.TBL for four different soil layers.
I am using a gridded Global Soil Dataset for use in Earth System Models (GSDE) data, which has eight different layers. Using the weighting average, I have created four different soil layers with the Noah-MP soil depths. So the four layers depth are 10, 30, 60, and 100 cm. I have converted the initial soil percentages to the Noah-MP's soil table.
Q1. What are the terms for soil_layer1, soil_layer2, soil_layer3, and soil_layer4? Where should I define the soil depths?
Q2. There are a couple of mandatory inputs in the GEOGRID.TBL that I am not sure if I need to include them on my GEOGRID.TBL.They are:
CON, VAR, OA1, OA2, OA3, OA4, OL1, OL2, OL3, OL4, VAR_SSO.
Q3. If they are mandatory for my case too, I don't know what the difference between OA1, OA2, OA3, and OA4 is. Also, the difference between OL1, OL2, OL3, and OL4. And the overall definition and equations of CON, VAR, OA1, OA2, OA3, OA4, OL1, OL2, OL3, OL4, and VAR_SSO are not clear. Therefore, I was wondering if you can introduce documentation or website to help me figure them out?
Q4. Can I calculate the above parameters myself? If I wanted to use the available binary data on the ucar website, is it OK that my created files' tile is different from their tile?
Q5. The soil, land cover, and DEM binary files that I have created are entirely clipped to the boundary of the region, which is not a rectangle. Does it make a problem for me? How can I clip the ucar file to the shapefile of my region? Do I need to do some sort of data conversions? In order to clip the ucar data to a smaller rectangle, not the exact boundary, I was thinking of a process like:
known_x and know_y represent the index that their corresponding latitude and longitude are known_lat and known_lon, and by knowing dx and dy and the number of tiles, I should be able to clip the data. Am I correct? If I leave the ucar data as a rectangle and my data like the region's boundary, does it make a problem?

Thanks,
Melika