Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

icon-eu download data

Hi, Julianrvogel,

All upper air variables included in the intermediate files should be on the same levels. Otherwise metgrid cannot run successfully. I believe this is the reason why your case failed. In your yesterday's post (December 4), I found two variables both named "GHT" but their values are different as shown below:
Code: 
================================================
FIELD = GHT
UNITS = m DESCRIPTION = Height
DATE = 2023-08-11_21:00:00 FCST = 0.000000
SOURCE = DWD
LEVEL = 66.000000
I,J DIMS = 1214, 745
IPROJ = 0 PROJECTION = LAT LON
REF_X, REF_Y = 1.000000, 1.000000
REF_LAT, REF_LON = 43.180000, 356.059998
DLAT, DLON = 0.020000, 0.020000
EARTH_RADIUS = 6371.229492
DATA(1,1)=189.067993

================================================
FIELD = GHT
UNITS = m DESCRIPTION = Height
DATE = 2023-08-11_21:00:00 FCST = 0.000000
SOURCE = DWD
LEVEL = 65.000000
I,J DIMS = 1214, 745
IPROJ = 0 PROJECTION = LAT LON
REF_X, REF_Y = 1.000000, 1.000000
REF_LAT, REF_LON = 43.180000, 356.059998
DLAT, DLON = 0.020000, 0.020000
EARTH_RADIUS = 6371.229492
DATA(1,1)=209.088318
================================================

I guess both are GHT but the first one (66-level) are GHT on full model levels, while the second one (65-level) should be GHT on half levels. If this is indeed the case, then you can simply remove the first full-level GHT and use the one on half-levels.

Please let me know whether this works for you.
 
Dear Ming,

Thank you for clarifying that all upper air variables must be on the same levels. Then the only way is to destagger the level heights beforehand.

The level heights (GHT) of ICON forcing are given only on full model levels (as is W), while most other variables (P, T, U, V, Q) are given on half model levels. Level 66 is actually the ground surface, 65 is the first level above ground and level 1 is top of atmosphere.

Note: I am using ICON-D2, which has 65 half and 66 full levels. The original download script was for ICON-EU, which has 74 half and 75 full levels.

The reason why the ICON forcing has worked so far, is that the download script omits the last level 66 for ICON-D2 (or level 75 for ICON-EU). When removing the last level, metgrid works, but it actually assigns the staggered level heights to the non-staggered variables.

What I did now is to destagger the 66 full levels to the 65 half levels, using eccodes in Python directly on the grib files, before ingesting them to ungrib. Now metgrid works and the level heights should be correct. I believe this is a step everyone needs to do when using ICON forcing.
 
julianrvogel said:
Dear Ming,

Thank you for clarifying that all upper air variables must be on the same levels. Then the only way is to destagger the level heights beforehand.

The level heights (GHT) of ICON forcing are given only on full model levels (as is W), while most other variables (P, T, U, V, Q) are given on half model levels. Level 66 is actually the ground surface, 65 is the first level above ground and level 1 is top of atmosphere.

Note: I am using ICON-D2, which has 65 half and 66 full levels. The original download script was for ICON-EU, which has 74 half and 75 full levels.

The reason why the ICON forcing has worked so far, is that the download script omits the last level 66 for ICON-D2 (or level 75 for ICON-EU). When removing the last level, metgrid works, but it actually assigns the staggered level heights to the non-staggered variables.

What I did now is to destagger the 66 full levels to the 65 half levels, using eccodes in Python directly on the grib files, before ingesting them to ungrib. Now metgrid works and the level heights should be correct. I believe this is a step everyone needs to do when using ICON forcing.
Click to expand...
do you have a script that does all the steps including the eccodes step?
 
Ming Chen said:
Please see the attached script for icon download.
Click to expand...
julianrvogel said:
Hello everyone,

I am currently testing to run WRF V4.6.1 using ICON data as input. I have tested only the European model ICON-EU so far. After a few struggles, it works for me. However, I found a few things that seem strange to me using this download script.

First of all, there is a newer version of the download script from Michael Graf on Github that acknowledges that the European model forecasts 1-hourly data only for up to 30 hours (in the newer script the forecast hours fff reach until 24 instead of 72). But the script still downloads only the model levels from 1 to 60, while there are actually 74 model levels. It may be the case that an older ICON-EU model had fewer levels or maybe there was another reason for this? As far as I can see, if we download only levels 1 to 60, the atmospheric data between 20 m (level 74) and ~ 1.9 km (level 60) is missing. The data is probably still sufficient so that WPS/WRF wouldn't complain and just interpolate everything in between, but this seems not ideal.

Finally, the soil level 0 temperature is not used by the VTable, so I removed it from the download. However, there are 2 more soil levels in the Vtable that we actually need to download.

I am attaching my corrected version of the download script. Hopefully, it may be of use to others and maybe someone else here can confirm my modifications.

Thanks a lot to the contributors who implemented the ICON support! Very useful :)
Click to expand...

Good morning @Ming Chen @julianrvogel


I used your shell scripts as a guide to build a Python-based ICON-EU downloader for WRF, and I’m attaching the script here in case it’s useful for others or for further refinement.

Here’s what the Python code does:



1. Handles ICON’s real forecast schedule (1-hourly vs 3-hourly)​


One of the confusing parts of ICON-EU is the time structure: the model doesn’t give you 1-hourly data for the full forecast. The script lets the user choose:


  • “1-hr native” mode
    • Downloads hourly data from 0 to 78 h,
    • Then automatically switches to 3-hourly (81, 84, …) up to the user-specified max hour.
      This matches how the files actually appear on the DWD server and avoids hammering the server for non-existent steps (e.g., 79, 80).
  • “3-hr only” mode
    • Downloads every 3 hours from 0 to max_hour (0, 3, 6, …, 120, …)
      Useful if you just want a coarse time series or smaller output volume.

You just answer a simple prompt:

Time step mode (1 = 1-hr native, 3 = every 3 hours) [1]:


Under the hood, a small helper builds the exact list of forecast hours, so the rest of the script stays clean.



2. Downloads all 74 model levels, not just 60​


As you pointed out, limiting ICON-EU to levels 1–60 leaves a vertical gap between ~20 m (level 74) and ~1.9 km (level 60). WPS can interpolate around it, but it’s not ideal scientifically.


In this Python version:

model_levels = range(1, 75) # 1..74 inclusive


So we now get the full vertical column that ICON-EU provides. If DWD changes the number of levels in the future, you only need to tweak that one line.



3. Aligns soil levels with the ICON Vtable​


I followed the same reasoning from your post:


  • Soil level 0 temperature is not used by the Vtable, so it’s excluded.
  • Additional soil levels that are referenced in the Vtable are included.

This is all controlled via two lists:

soil_t_levels = [5, 2, 6, 18, 54, 162, 486]
soil_w_levels = [0, 1, 3, 9, 27, 81, 243, 729]


If anyone’s using a slightly different Vtable or soil layering scheme, they can adapt it by editing just these lists—no need to touch the rest of the logic.



4. Parallel downloads with graceful skipping of existing files​


To make repeated runs more practical (and kinder to the DWD server), the script uses a ThreadPoolExecutor:


  • All URLs for a given forecast hour (model levels, soil, single-level fields, time-invariant fields) are collected into one list.
  • These are downloaded in parallel using multiple threads.
  • Each file is:
    • Downloaded as .bz2
    • Decompressed to .grib2
    • The .bz2 is deleted
  • If a .grib2 file already exists, the script can skip re-downloading it.

So if you re-run for the same run date and init time, it only downloads what’s missing and then re-assembles the per-hour GRIB files.



5. One GRIB2 file per forecast hour, ready for WPS​


For each forecast hour FFF, the script:


  1. Downloads and decompresses all the needed GRIB chunks into a temporary directory.
  2. Concatenates them into a single file:
    icon_YYYYMMDDHH_FFF.grib2

  3. Cleans up the per-variable GRIB chunks for that hour to save disk space.

The final directory looks like:

icon/
icon_2025120812_000.grib2
icon_2025120812_001.grib2
...
icon_2025120812_120.grib2


From there, the WPS/WRF workflow is standard:


  • Link these files into WPS,
  • Use the ICON Vtable,
  • Run ungrib.exe, then metgrid.exe, real.exe, and finally wrf.exe.


Closing thoughts​


So this Python script is basically a drop-in modernization of the original ICON shell logic:


  • It respects the actual ICON-EU time structure,
  • It restores full vertical coverage (all 74 levels),
  • It corrects the soil level selection for the Vtable,
  • And it speeds everything up with parallel downloads and clean file management.

Here it is attached—feedback and improvements are very welcome!
 

Attachments

  • Download_Icon.py.txt
    17.3 KB · Views: 9
Hi @William.Hatheway

Here is my code for destaggering the grib files. However, it is a bit specific to what I am doing. I do not use forecasts, but only run on historical analysis time steps. So my file structure does not contain the FF field and would need adjustment for preparing forecast data. Furthermore, I also use my own python download tool that combines the grib data into less files. But I still have multiple files per time step, where one of the files contains only HHL (height) and wz (vertical wind), from which we only need HHL for WRF. I am not sure if the code works well when everything is in one file, as HHL has 1 more level than all other vars. I did not find the time to test the code for other setups.

The code modifies only the specified fields defined in "TARGETS", which is just "HHL" in this case. A new file is created for every processed step appending "_destag" to the file name.

I tested this on one case so far and it seems to work well.
 

Attachments

  • batch_destag_dwd_icon.py.txt
    5.3 KB · Views: 3
You must log in or register to reply here.
Top