WRFV4.0.2 LES-run

Hi all,

at the moment I am trying to start a WRF-LES run for a real scenario, with a spatial resolution of 200m using WRF 4.0.2 with the set up as you can see in the attached namelist.

Short overview: 4 Domains (9km, 3km, 1km and 200m) for 15 days in May/June 2018 in Europe, with the inner domain being run with LES.

For setting up the 4th Domain I have tried two different ways, 1.) start a full run starting at D01, using one-way nesting, and 2.) using ndown using the output from D03 as boundary. Both approaches lead to two different errors.

1.) Full run:

For the full run I am starting the attached namelist.input.full_run, after ~20sec run time the run stops with the following output:


Trying to allocate more memory (I am already using ulimit -s to increase the stack size) for the job and decreasing the time step both yield the same result.

2.) NDOWN:

The second way I tried is to first run all Meso-scale Domains (D01 – D03) and afterwards run the 4th Domain creating wrfbdy and wrfinput via ndown looking at the tutorial (http://www2.mmm.ucar.edu/wrf/OnLineTutorial/CASES/NestRuns/ndown.htm)

Creating all meso-scale Domains works fine and give expected results, also running ndown works fine and creates the needed files. Staring wrf for the LES Domain now also crashes after about 5sec of runtime. Everytime at the same point:


This holds for the namelist.input.LES as attached to this post. I was trying to use another option for sf_surcafe_physics and setting it to 1 instead of 2 makes the model run till the end. But I have almost no variation of temperature over day (i.e., no diurnal cycle whatsoever, which is unrealistic in this case) in this run, anyways I would like to use the Option used before, so 2.

Has anyone a hint what causes this error and how I could solve it? I would prefer a way using ndown, but getting the full run to work would also be very nice.

Thanks a lot,
Sarah

Hi Sarah,

I am getting a SIGSEGV error similar to your full job for an LES run on a 250 m domain. I have been trying to diagnose the issue for a month now and am still not sure why it is occurring.

I think it has more to do with the underlying data than anything else because my same configuration that fails in one 30-hr case runs for another with no issue.

Andre

Hi Sarah (and Andre),
Grid distances >=200m (up to about 1km) are kind of a "grey-zone", where using an LES option may work, but will not have correctly partitioned resolved/sub-grid energy fractions. They also just may not work at all, depending on several different components. It's actually recommended that instead of trying to run this in LES mode, that you, instead use the Shin-Hong PBL (option 11), which is based on YSU, designed for these particular grid distances. So this could be what is going on; however, a segmentation fault can also be caused by several other factors, such as the number of processors you're using, etc. I would first recommend taking a look at this physics presentation that is given at our WRF Tutorial each year, with a lot of information regarding LES runs:
http://www2.mmm.ucar.edu/wrf/users/tutorial/201901/dudhia_physics.pdf

Perhaps you will want to make modifications to your runs and see if any of the problems are resolved. If not, we can dig a little deeper.

Hi Kevin,

I understand what you are saying about the grey zone but I have seen many groups do LES simulations in this range with success. Is the real issue the lack of vertical resolution in the LES domain?

Andre

Update: I changed bl_physics to 11 for domains 1-4 and the simulation still failed at a similar spot in the integration with a SIGSEGV.

Hi Andre,
Okay, for this new failed run, can you please attach your new namelist along with all of your out/error files (e.g., rsl.error.*). If there are a lot of them, please package them all into one *.tar file. Can you also let me know which version of the model you are running? You mention that this same set-up worked fine for another colleague. Was the only difference between your runs the input data? i.e., were the domains identical, exact same dates, namelists identical, same computing environment, same number of processors, etc.? If so, have you taken a look at your input data to see that everything looks reasonable?

Thanks,
Kelly

Hi Kelly,

I am running v3.8.1 on a Cray cluster. Each node has 128 GB of RAM. I ran with the same namelist using all five domains (down to 50m) on a case with no issues. This same case that I am having trouble with also finished with LSM=1 and 4 with no problems.

For whatever reason I cannot upload my tar file at all. I have been trying for quite some time. I don't know if it is on my end or not. the file size is quite small.

Andre,
Hmm, I'm not sure why you're unable to attach files here. Is it just hanging, or are you getting an error when trying to do so? If you get an error related to the file extension name, add .txt to the end and see if that helps. You can also take a look at the home page for this forum, and in the top section there is information on uploading to a different source:
http://forum.mmm.ucar.edu/phpBB3/

This is typically only necessary for larger files, but perhaps it will help you to get around your upload problem.

Kelly

Hi Kelly,

It must be an issue with the firewall I am behind. I get http errors when uploading on the thread and even on the nextcloud site it gives me problems.

I resigned to running the job piecemeal with debug mode on and hope it will finish the whole run but I am still troubled by the seg faults as I feel they are related to the choice of LSM somehow.

Andre

Andre,
I'm sorry you're having trouble with the firewall. That is frustrating. If you are still having problems with this case, and you have another way to get the files to me (e.g., Google Drive, Dropbox, a web page, etc.), let me know!

kwerner said:

Hi Sarah (and Andre),
Grid distances >=200m (up to about 1km) are kind of a "grey-zone", where using an LES option may work, but will not have correctly partitioned resolved/sub-grid energy fractions. They also just may not work at all, depending on several different components. It's actually recommended that instead of trying to run this in LES mode, that you, instead use the Shin-Hong PBL (option 11), which is based on YSU, designed for these particular grid distances. So this could be what is going on; however, a segmentation fault can also be caused by several other factors, such as the number of processors you're using, etc. I would first recommend taking a look at this physics presentation that is given at our WRF Tutorial each year, with a lot of information regarding LES runs:
http://www2.mmm.ucar.edu/wrf/users/tutorial/201901/dudhia_physics.pdf

Perhaps you will want to make modifications to your runs and see if any of the problems are resolved. If not, we can dig a little deeper.

Click to expand...

Hello,
thanks a lot for your help. A full run with the suggested physics option seems to work for me.

Sarah

Sarah,
Great! Thank you for letting me know.

Kelly