[Question] WRF Workflow Management – Best Practice Recommendations

[lnpilz]

Hi everyone,

I’m currently starting my PhD project which will focus a lot on simulating with WRF. More precisely: one goal is to generate CO2 fields with WRF-Chem at a 1x1km resolution in 5 different regions for one year each.

As I don’t want to manage time chunking and simulation restarting by hand, I’m currently searching for best practice recommendations for workflow management tools. My requirements are:

• must be Open Source
• must work with local/custom installations of WRF
• must be actively developed
• configuration/state of simulation must be version controllable (pref. using git)
• must easily support ensemble runs (parallel ensemble runs should work with 1 WRF installation and not 15 separate ones)
• must be able to submit jobs to SLURM and ideally monitor their progress
• should also track input files

Unfortunately, I couldn’t find any best practice guides or community resources and so I already did some digging and found some tools, which probably won’t work for me:
GSD External Workflow Manager (WFM) – not actively developed, no SLURM-integration
WRF-Portal – not actively developed, not well version-controllable, no SLURM-integration
mkexp – no SLURM-integration
UEMS – requires semi-manual modification of WRF source code, no SLURM-integration

I also found some tools which might work for me:
WRF4G - developed by University of Cantabria
ecFlow - developed by ECMWF, might not easily support ensembles, unfortunately not on Github
Rose/Cylc - developed by UK MetOffice, might not easily support ensembles
ESM-Tools - developed by Alfred Wegener Institute, does not support WRF (yet), don’t know if one can use local WRF installations

I am currently leaning towards using WRF4G since it natively supports WRF, but wanted to ask if any of you had any experience in using one of these tools in conjunction with WRF, knew other tools or had any general advice regarding WRF workflow management.

If I missed any tools or resources or misjudged any of the listed ones, please let me know

Thanks in advance,

Lukas

 

[kwerner]

Hi Lukas,
I'm not sure if this meets all of your criteria, but our group worked closely with a cloud computing company - Scala Computing, which has a WRF application for continuous runs. It allows you to be mostly hands-off, while keeping things monitored. They have scripts that run everything, but you are able to make modifications to those scripts to suit your needs. If you're interested in it, take a look at this basic web page we put together. You will need to contact Scala to get everything set up on your end.

If this doesn't meet your needs, hopefully someone else out there will have a good suggestion!

 

[lnpilz]

Hi Kelly,
thanks for the hint, this project looks very interesting indeed. Unfortunately it's not quite what I had in mind. I have a WRF installation and a cluster (accessible via SLURM) at my disposal and I was just wondering how to manage my simulations on this setup.

Do you at NCAR use WRF-Portal for more complex setups and ensembles or any other tools maybe or are you now migrating to Scala?

Thanks in advance!

 

[kwerner]

Lukas,
We currently don't primarily use any specific tools. My group doesn't even use Scala. We only contributed to a project where we created Cloud-WRF for users interested in various cloud computing applications, and Scala was part of that. We currently mostly use our NCAR supercomputer, and I'm sure people have scripts they've written for purposes similar to yours, but I'm not aware of who may have these. I'm hoping someone else will chime in to this post and give you some helpful resources!