Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

file sizes after metgrid.exe (version 4.2.2)

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

R

rameshv

New member
Upto Version 4, the file sizes were uniform at increasing time stamps, why it is different in the latest version 4.2.2?

-rw-r--r-- 1 rameshv cccr-res 81773075 Feb 3 15:27 met_em.d01.2019-08-05_00:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81752886 Feb 3 15:27 met_em.d01.2019-08-05_01:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81735482 Feb 3 15:27 met_em.d01.2019-08-05_02:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81735237 Feb 3 15:27 met_em.d01.2019-08-05_03:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81753763 Feb 3 15:27 met_em.d01.2019-08-05_04:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81774083 Feb 3 15:28 met_em.d01.2019-08-05_05:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81796465 Feb 3 15:28 met_em.d01.2019-08-05_06:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81844889 Feb 3 15:28 met_em.d01.2019-08-05_07:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81896102 Feb 3 15:28 met_em.d01.2019-08-05_08:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81945896 Feb 3 15:28 met_em.d01.2019-08-05_09:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 82074140 Feb 3 15:28 met_em.d01.2019-08-05_10:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 82040531 Feb 3 15:28 met_em.d01.2019-08-05_11:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 82014752 Feb 3 15:28 met_em.d01.2019-08-05_12:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81990699 Feb 3 15:28 met_em.d01.2019-08-05_13:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81985872 Feb 3 15:28 met_em.d01.2019-08-05_14:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81980597 Feb 3 15:29 met_em.d01.2019-08-05_15:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81956731 Feb 3 15:29 met_em.d01.2019-08-05_16:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81937213 Feb 3 15:29 met_em.d01.2019-08-05_17:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81937513 Feb 3 15:29 met_em.d01.2019-08-05_18:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81926427 Feb 3 15:29 met_em.d01.2019-08-05_19:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81921154 Feb 3 15:29 met_em.d01.2019-08-05_20:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81905691 Feb 3 15:29 met_em.d01.2019-08-05_21:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81958402 Feb 3 15:29 met_em.d01.2019-08-05_22:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81929207 Feb 3 15:29 met_em.d01.2019-08-05_23:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81912707 Feb 3 15:30 met_em.d01.2019-08-06_00:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81881766 Feb 3 15:30 met_em.d01.2019-08-06_01:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81871694 Feb 3 15:30 met_em.d01.2019-08-06_02:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81867997 Feb 3 15:30 met_em.d01.2019-08-06_03:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81879980 Feb 3 15:30 met_em.d01.2019-08-06_04:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81899502 Feb 3 15:30 met_em.d01.2019-08-06_05:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 81921951 Feb 3 15:30 met_em.d01.2019-08-06_06:00:00.nc
 
Did you use different forcing data? Also, if these files work fine for REAL, then the slight difference in the size shouldn't be a concern.
 
No, the same (ERA5) inputs sent to metgrid.exe (versions prior to 4.0) and version 4.2.2

File sizes of met_em.d* files after metgrid.exe (4.2.2) is smaller and also different for each time stamp.
 
The problem is: real.exe (4.2.2) does not accept the smaller files. It gives an error saying my model domain may be a global one, etc.,
 
For the same input files generated from ungrib.exe, the following is the file sizes from V4.0 and V4.2.2,
What am I missing. This generates a lot of problem with real.exe of V4.2.2

File sizes (from version 4.0)

-rw-r--r-- 1 rameshv cccr-res 383364816 Feb 7 18:20 met_em.d01.2019-08-05_00:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 383364816 Feb 7 18:20 met_em.d01.2019-08-05_01:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 383364816 Feb 7 18:20 met_em.d01.2019-08-05_02:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 383364816 Feb 7 18:20 met_em.d01.2019-08-05_03:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 383364816 Feb 7 18:20 met_em.d01.2019-08-05_04:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 383364816 Feb 7 18:20 met_em.d01.2019-08-05_05:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 383364816 Feb 7 18:20 met_em.d01.2019-08-05_06:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 383364816 Feb 7 18:20 met_em.d01.2019-08-05_07:00:00.nc

File sizes (from version 4.2.2)

-rw-r--r-- 1 rameshv cccr-res 96064270 Feb 7 18:17 met_em.d01.2019-08-05_00:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 96039103 Feb 7 18:18 met_em.d01.2019-08-05_01:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 96028530 Feb 7 18:18 met_em.d01.2019-08-05_02:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 96032750 Feb 7 18:18 met_em.d01.2019-08-05_03:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 96060299 Feb 7 18:18 met_em.d01.2019-08-05_04:00:00.nc
-rw-r--r-- 1 rameshv cccr-res 53418915 Feb 7 18:18 met_em.d01.2019-08-05_05:00:00.nc
 
Can you look at the contents in your met_em files produced by WRFV4.0 and WRFV4.2.2? Please use ncdump -h and save the output for me to take a look.

Also, If you run with GFS, did you see the same large differences between the two versions?
 
The default for netcdf has been to have compression activated, starting with release v4.0 (SHA a19f3d6424). Here is that modification.

Code: 
commit a19f3d64244d4435cd1aa9b2a8459ba296a9b3c9
Author: Dave Gill <gill@ucar.edu>
Date:   Mon Mar 26 18:23:00 2018 -0600

    NETCDF4 compression is now default (#415)

    TYPE: enhancement

    KEYWORDS: netcdf4, compression

    SOURCE: internal

    DESCRIPTION OF CHANGES:
    1. Modify the configure script to make the (already existing) NETCDF4 compression option the default.
    2. Clean up the logic in the configure script that had outdated references.
    3. Clean up the logic in the top-level Makefile when testing NETCDF4.
    4. The traditional NETCDF "classic" format is available with setting an environment variable:
    ```
    setenv NETCDF_classic 1
    ```
    or
    ```
    export NETCDF_classic=1
    ```

    LIST OF MODIFIED FILES:
    M   Makefile
    M   configure

    TESTS CONDUCTED:
     - [x] Regression test OK.
     - [x] Logic correctly identifies if users ONLY have non-compressing NETCDF.
    ```
    ************************** W A R N I N G ************************************
    NETCDF4 IO features are requested, but this installation of NetCDF
      /usr/local/netcdf-3.6.3-gfortran
    DOES NOT support these IO features.

    Please make sure NETCDF version is 4.1.3 or later and was built with
    --enable-netcdf4

    OR set NETCDF_classic variable
       bash/ksh : export NETCDF_classic=1
          echo  csh : setenv NETCDF_classic 1
    ```
     - [x] The new code vs the original code gives bit-for-bit answers in a number of ways using diffwrf for wrfinput (d01 and d02), wrfbdy, wrfout (d01 and d02, several time slices):
    1. new code classic vs original code classic
    2. new code compressed vs original code compressed
    3. new code compressed vs new code classic (I didn't know this worked!)
     - [x] Files are compressed
    ```
    File                             Classic  Compressed
    Name                              (bytes)   (bytes)
    =====================================================
    wrfbdy_d01                       16608908   8733475
    wrfinput_d01                     13495844   6414076
    wrfinput_d02                      2817476   2103253
    wrfout_d01_2000-01-24_12:00:00   12953172   6338101
    wrfout_d01_2000-01-24_12:03:00   12953172   7019622
    wrfout_d01_2000-01-24_12:06:00   12953172   7058111
    wrfout_d01_2000-01-24_12:09:00   12953172   7075389
    wrfout_d02_2000-01-24_12:00:00    2706324   2019162
    wrfout_d02_2000-01-24_12:01:00    2706324   2080210
    wrfout_d02_2000-01-24_12:02:00    2706324   2078364
    wrfout_d02_2000-01-24_12:03:00    2706324   2079096
    wrfout_d02_2000-01-24_12:04:00    2706324   2079120
    wrfout_d02_2000-01-24_12:05:00    2706324   2077147
    wrfout_d02_2000-01-24_12:06:00    2706324   2078344
    wrfout_d02_2000-01-24_12:07:00    2706324   2079854
    wrfout_d02_2000-01-24_12:08:00    2706324   2079070
    wrfout_d02_2000-01-24_12:09:00    2706324   2079789
    ```
     -[x] Time and size tests were conducted on cheyenne with domain=1501x1201x35, with 1008 cores (28 nodes at 36 cores/node).
   
    No Compression
    ```
    Timing for Writing wrfout_d01_2005-06-04_00:00:00 for domain        1:   12.96644 elapsed seconds
    Timing for Writing wrfout_d01_2005-06-04_00:00:15 for domain        1:   12.66845 elapsed seconds
    Timing for Writing wrfout_d01_2005-06-04_00:00:30 for domain        1:   12.68951 elapsed seconds
   
    4317696 -rw-r--r-- 1 gill p66770001 4421272056 Mar 23 09:02 wrfout_d01_2005-06-04_00:00:00
    4317696 -rw-r--r-- 1 gill p66770001 4421272056 Mar 23 09:02 wrfout_d01_2005-06-04_00:00:15
    4317696 -rw-r--r-- 1 gill p66770001 4421272056 Mar 23 09:02 wrfout_d01_2005-06-04_00:00:30
    ```
    Compression level = 2
    ```
    Timing for Writing wrfout_d01_2005-06-04_00:00:00 for domain        1:   73.21764 elapsed seconds
    Timing for Writing wrfout_d01_2005-06-04_00:00:15 for domain        1:   76.12779 elapsed seconds
    Timing for Writing wrfout_d01_2005-06-04_00:00:30 for domain        1:   76.23354 elapsed seconds
   
    1430784 -rw-r--r-- 1 gill p66770001 1464888707 Mar 23 09:11 wrfout_d01_2005-06-04_00:00:00
    1635840 -rw-r--r-- 1 gill p66770001 1674906321 Mar 23 09:12 wrfout_d01_2005-06-04_00:00:15
    1638912 -rw-r--r-- 1 gill p66770001 1678168921 Mar 23 09:13 wrfout_d01_2005-06-04_00:00:30
    ```
   
    Compression level = 1
    ```
    Timing for Writing wrfout_d01_2005-06-04_00:00:00 for domain        1:   70.44591 elapsed seconds
    Timing for Writing wrfout_d01_2005-06-04_00:00:15 for domain        1:   74.08282 elapsed seconds
    Timing for Writing wrfout_d01_2005-06-04_00:00:30 for domain        1:   74.24098 elapsed seconds
   
    1442816 -rw-r--r-- 1 gill p66770001 1477427814 Mar 23 09:23 wrfout_d01_2005-06-04_00:00:00
    1648384 -rw-r--r-- 1 gill p66770001 1687774626 Mar 23 09:25 wrfout_d01_2005-06-04_00:00:15
    1651712 -rw-r--r-- 1 gill p66770001 1691152421 Mar 23 09:26 wrfout_d01_2005-06-04_00:00:30
    ```
   
    Smaller domain: 425x300x35, 144 cores (4 nodes x 36 cores/node):
    ```
    Timing for Writing wrfout_d01_2001-10-24_00:00:00 for domain        1:    1.02305 elapsed seconds
    Timing for Writing wrfout_d01_2001-10-24_00:01:12 for domain        1:    1.02047 elapsed seconds
    Timing for Writing wrfout_d01_2001-10-24_00:02:24 for domain        1:    0.97939 elapsed seconds
    Timing for Writing wrfout_d01_2001-10-24_00:03:36 for domain        1:    0.99052 elapsed seconds

    303872 -rw-r--r-- 1 gill p66770001 311013852 Mar 23 09:49 wrfout_d01_2001-10-24_00:00:00
    303872 -rw-r--r-- 1 gill p66770001 311013852 Mar 23 09:49 wrfout_d01_2001-10-24_00:01:12
    303872 -rw-r--r-- 1 gill p66770001 311013852 Mar 23 09:49 wrfout_d01_2001-10-24_00:02:24
    303872 -rw-r--r-- 1 gill p66770001 311013852 Mar 23 09:49 wrfout_d01_2001-10-24_00:03:36
    ```
   
    Compression = 2

    ```
    Timing for Writing wrfout_d01_2001-10-24_00:00:00 for domain        1:    5.27841 elapsed seconds
    Timing for Writing wrfout_d01_2001-10-24_00:01:12 for domain        1:    5.43014 elapsed seconds
    Timing for Writing wrfout_d01_2001-10-24_00:02:24 for domain        1:    5.49076 elapsed seconds
    Timing for Writing wrfout_d01_2001-10-24_00:03:36 for domain        1:    5.43229 elapsed seconds

    108544 -rw-r--r-- 1 gill p66770001 111023826 Mar 23 09:58 wrfout_d01_2001-10-24_00:00:00
    121344 -rw-r--r-- 1 gill p66770001 124159720 Mar 23 09:58 wrfout_d01_2001-10-24_00:01:12
    122112 -rw-r--r-- 1 gill p66770001 124843642 Mar 23 09:58 wrfout_d01_2001-10-24_00:02:24
    122112 -rw-r--r-- 1 gill p66770001 124983509 Mar 23 09:58 wrfout_d01_2001-10-24_00:03:36
    ```

You can get the fixed file size, as is mentioned in the commit message, with the env variable NETCDF_classic. The amount of time spent compressing the data can be substantial for larger files.
 
You must log in or register to reply here.
Top