Altered SST Cells Cause Simulation to Break Several Years into Run?

[cjoh513]

Hello there,
This post is tangentially related to a previous post I made (found here). In that post I detailed how the ERA5 data I was using showed the Guadeloupe, Dominica, Martinique, etc. islands within the SSTs (being 0 Kelvin) but did not appear in the landmask.

After being stumped on how to fix this, I got around it through python altering the met_em SSTs for those few cells at each timestep. I pulled the average value of the surrounding cells and set the islands' SSTs from 0 Kelvin to the average. Below are two images showing this as an example. (Left is the pre-altered met_em SST, right is the altered one)


I altered the SSTs the way I did because the landmask/landsea thinks it's water and that spot so close to Puerto Rico (the research area) of 0 Kelvin kept showing up, impacting other variables such as T2 and Skintemp.


That's all for context. I altered the met_ems, ran real, and got the simulation running with no snafus. The real trouble I'm having now is figuring out why 4 years into the simulation (2013-2018 is the full run. It breaks in 2017) it breaks. It breaks on 08/25/2017

I can't find a CFL error, I can't find any error in the rsl.* files. I do however think it's tied to my manipulation of the SSTs because when I tried redoing the WPS process, I did not alter the met_ems in any way and the simulation ran fine. When I alter the SSTs it breaks at the same point every time.

I don't understand why it's breaking now after running the previous years just fine. I've also run plenty of other simulations with altered SSTs and had no problems so this feels very strange to me. Are there other variables that the SSTs might be tied to that I didn't consider? I should also say that this simulation is in wrf3.8. I've attached my namelist and some of the rsl_out files but couldn't really upload many as even compressed they were too large, but if it's necessary I can find a workaround. Same for the met_ems themselves if interested in comparing those.

Thanks for any help or information that can be provided.

 

[kwerner]

Hi,
I have a few questions.

1) First, set debug_level to 0. This parameter was removed from the default namelist in more recent versions of the model because it rarely produces any useful information, and ends up just adding a lot of junk to the error/out files, making them very large and difficult to read.
2) Are you able to reproduce the error if you start the model from the most recent restart file?
3) Does this issue still exist in a newer version of the model. Can you try with the latest version? If the problem is reproducible with a restart file, use that, instead of running the entire simulation again. Note that to run a newer version of WRF with older (pre-V4) WPS input, you'll need to modify your namelist for backward compatibility.

If the issue still happens with the latest model, since the problem doesn't occur with the standard met_em files, you may have to do some debugging to determine what is causing the issue.

 

[cjoh513]

Thanks for the response!
1. I've tinkered around with various debug levels prior to my post and could not find an error with the debug = 0, 10, 100, 1000 but I do understand what you mean.
2. Yes, I have 3 restart files I've tested it from. 9/17/2017, 8/08/2017, and 8/25/2017 (I had it generate a new restart file to test the day of). They all break at about the same time (maybe a few minutes difference).
3. I'll definitely try running it on an updated version. I didn't realize I could make it backwards compatible so thanks for that piece of info!

I'm also wondering if this is a cfl error in disguise and if the SSTs are merely the domino that set off a cascading effect.
Looking at the U/V wind data, I realized there were signs of large-scale cyclonic winds. Then I realized that I was looking at Hurricane Harvey right at the edge of the domain. There are some very subtle differences between low-pressure locations of the altered and non-altered models.

Possibly, without the SST alteration, the model resolves everything just inside the domain correctly. With the altered SSTs, the model cascades just differently enough to throw a wrench into the gears. I knew someone who ran into a similar issue with a hurricane being at the edge of their domain so that went into my thought process too.

 

[cjoh513]

Update!
So after some tinkering, I found that the simulation still breaks with an updated WRF but I also found a workaround. Although, it may not be the best practice.
However, it seems to have been an issue with Hurricane Harvey causing some kind of cfl error. Below are the U and V winds showing Harvey being bisected by the edge of the domain.



So the problem seems to have been a domino caused by altering the SSTs originally that knocks over others until Harvey is in such a position to cause an error.
The solution that worked was to throw another domino in. I altered 4 cells worth of SSTs southwest offshore of Nicaragua and caused another slight cascading effect. This time Harvey resolved a little further away from the domain and created no error, I immediately switched the Nicaraguan SSTs back and the model has run perfectly fine since then.

The moral I suppose is don't bisect hurricanes with your domain, and always check what major weather phenomena are going to be within the simulation time period.

 

[kwerner]

Hi,
Thank you so much for posting this update. We do advise to keep domain boundaries away from complex terrain because of the possibility of CFL errors, and since there are strong updrafts in hurricane's, they can cause a similar issue. So your method for resolving the problem should be a good solution. I'm glad you were able to track this down and to share with us how you resolved it.