Running wrf at 12x12 km how to avoid cfl errors?

[satjemkes]

Hi,
Apparently the cfl errors are popping up with me as well and it is not trivial to detect a trend how the system responds to changes in the namelist.

Input is ECMWF data at T799 resolution (from IFS), running through the wps system to generate 27x27 km data. Running this with WRF 4.3.3 seems to be fine. timestep is below the 6*dx threshold, process runs smooth.

Same exercise but now using the sameT799 data to generate input for 12x12 km grid.

This time it seems very difficult to find appropriate settings to avoid the cfl errors.

namelist.input is attached
output from real.exe is enclosed

running wrf on a 16 node cpu with the ulimit -s unlimited
16 nodes should be fine the rule listed indicates number of nodes between 3 and 51.

reducing timestep 60 => 25 does not really help
the smooth_cg_topo true or false does not really help

changing the number of vertical grid levels, or lowering top does not really help
If i reduce the number of grid cells in ew-ns direction should result in a positive (ie. successful) run

So i am looking for some guidance. physical package is standard conus.
The meteorological situation is of a low pressure north or ireland and the domain includes the alps (=elevated area)

I have tried namelist from others in this forum, but no success. So i am kind of stuck.

thanks
stephen

 

[Ming Chen]

Your namelist.input indicates that you run with the option

real_data_init_type = 3

Note that the default option is 1, which is also what we use most frequently. Is there any special reason you choose this option?

Thanks.

 

[satjemkes]

no not really, will this help to avoid the issue with the cfl?
i could change this and try later this week.
thanks
stephen

 

[satjemkes]

A long story and more for future reference: The issue i with the ECMWF from the IFS at T799 resolution and full model levels. That is
1. ungrib works with global data no matter what.
2. ungrib does not work with ecmwf data at T1279 spectral truncation no matter what (global or regional: regional data does not work see 1.)

3 using ungrib and ECMWF ifs data with T799 spectral resolution to generate input data for WRF at a 12 x 12 km spatial sampling works, however WRF does not like this no matter what combination of the parameters.
4. using ungrib and ECMWF ifs data at T799 spectral resolution to generate WRF data at 27x27 works, and WRF does run, but not always. In particular certain combinations of skeb and sppt stochastic perturbations will cause wrf to fail
5. So far i have had success with the GDAS data from NCAR at 0.25x0.25 resolution to generate input for WRF at 12x12 km spatial sampling. I have made successful runs with a basic setup and with very selected combination of skeb and sppt perturbations, but it looks okay.

Note i am running with dzbot at 10, dzstretch of 1.2 and a p_top at 1000 with 55 levels (2&3 are also based on different combinations of these)

I suspect that the fine vertical stratification of ecmwf might cause the problems with wrf. (i have tried epssm = 0.9 but this does not help)
The gdas data has 34 levels, i have not tried to retrieve ecmwf ifs data at T799 and with only 34 levels to double check, or at the same levels of the ensemble system. This would also be a nice exersice.

 

[Ming Chen]

Hi Stephen
Thank you for the update. If ungrib cannot process ecmwf data at T1279 spectral truncation, it is possibly because the record length of this data is too long that WPS cannot handle. We are aware of this issue but haven't fixed it yet.
For the 12km resolution case, this case failed due to CFL violation. Your namelist looks fine to me except that epssm = 0.2. Please try with a larger value of epssm (e.g. 0.6 or even 0.9) and a smaller value of time step (e.g., 45 ). Let me know whether it works.
Thanks.

 

[satjemkes]

Dear Ming,
Thank you for the reply.

Unfortunately my experience is not positive. The setting of the record length was tried. But did not resulted in a successful execution. Reason is that yes changing the record length will enable the system to correctly read the file, but then you hit a decision tree in the grib1 reader which indicates that if the nx parameter is equal to a certain value, AND you are not providing data from some specific source, then the ungrib routine will stop as it can ONLY handle data from these specific sources.

in other words T1279 directly from ECMWF MARS system can not be processed by ungrib. The atmospheric part is fine as this is grib2, the surface part is the blocking issue as this is grib1.

The CFL violation is more tricky over the last couple of days i have tried several things including your suggestions and many more. Bottom line is that it does not work. I suspect, but have not demonstrated this, that it has nothing to do with the settings in WRF, but with the high vertical resolution of the ECMWF data. The data comes at the model resolution which means 137 levels in the vertical. As i am interpolating from T799 (i believe about 25x25 km) to 12x12 km, it is not clear to me what happens over areas with pronounced orography. ECMWF data is on hybrid sigma coordinates just like wrf is using but what is under the hood to generate the high resolution input data is unnknown to me.

If time permits i will try to get ECMWF data from the MARS system at a fewer vertical levels (91 or 60) still as this will not change the vertical stratification near the surface i am not very hopeful as my experience with e.g. the stochastic perturbations sppt, indicates that its the PBL.

SPPT perturbations acts mainly on the physics near the surface, and with ECMWF at T799 (which can be processed by ungrib) at 27x27 km resolution using either the conus or tropical setup frequently lead to a crash of the system. I have seen from the bgerr v2 output (in netcdf) that for the successful runs the spread in the ensemble generated by SPPT and SKEB is mainly near the surface, hence my interpretation that SPPT modifies the physics near the surface. And thus if WRF run fails due to the inclusion of SPPT (with SKEB i have not encountered problems), it is likely that the perturbations in the PBL are going wild. And we know that ECMWF has many model levels near the surface.

The suggestion is therefore that the issues with the CFL and the SPPT are linked and are caused by some sort of transformation of the 137 level ECMWF to the 55 leves adopted for the wrf runs.

I have not tried a wrf run with many more levels (91 or so) to see if that solves the problem. Would be interesting though.

Bottom line trying to generate high spatial resolution data from WRF using the T799 or T1279 spectral truncation provided by the ECMWF MARS system is causing a lot of trouble and so far I was not able to get this going. T799 at 27x27 resolution seems to be fine for un perturbed runs, problems can be expected when stochastic perturbations are applied.

We now have GDAS data, and i converted that to wrf input at 12x12 km resolution, and will next try to prepare an ensemble by applying the skeb-sppt stochastic perturbations to see if this is more robust. GDAS data comes at 34 levels and at 0.25x0.25 deg res. so some sort of pre-processing took already place.

best
stephen