Using 'getvar' to extract all WRF variables

[wr1989]

I'm running 1km simulations (soon to move down to 250m) and I'm having memory issues using 'getvar' to extract the variables.

I can get T2, rh2, td2, uvmet10, uvmet10_wspd_wdir and slp for all times but the remaining variables (excluding lat/lon/ter), require a single time index. I've attempted to create a loop to go through each time index and combine the individual variables into one dataset but it crashes my kernel in Jupyter due to memory issues. For example, I have 181 time steps so for 'p', I loop through all 181 wrfout files, use 'getvar' to extract 'p' and then combine them into one new dataset with all times for variable 'p'. I've also tried using SLURM on my university's HPC but that crashes as well.

Is there any way to set this up to process after WRF finishes running using a similar .csh or .sh command to run on Linux so the variables are pre-processed and ready for visualization rather than having to try and process them in a Jupyter notebook?

I appreciate any help!

 

[Ming Chen]

The memory issue is quite common when we use Jupyter Notebook.

In NCAR HPC, when we launch Jupyter Notebook, we have an option to require larger memory. I suppose you should have similar option in your university HPC system. Can you talk to your computer manager, asking about option for large memory?

 

[wr1989]

I have that option as well but even maxing out memory (216Gb), it still prompts the kernel to crash.

 

[Ming Chen]

216GB is large enough, I guess. The maximum memory I used before is 40GB and it is sufficient for my large cases.

I am not sure whether something else is involved, ----- Would you please upload your script for me to take a look? Also, what is the grid number of your wrfout files (e.g., east-west and north-south grid numbers and vertical levels.)

 

[wr1989]

My .ipynb is too big to upload but I can type the gist of my scripts:

P = getvar(wrf_1km_files, 'p', timeidx = ALL_TIMES, method = 'cat')

My input requests 50 vertical levels and my grid is 579x719, if I remember correctly - it's a 1km grid so the files are very large.