High resolution (43m)

[jmcabrera]

Hi everyone!

I'm trying to run a model with 4 domains (5400 m, 1080 m, 216 m, and 43.2 m).

I was able to simulate 1 hour, but it took 1 hour using 49 processors. Now, I'm looking for ways to reduce the processing time.

My first approach was to reduce the parent_time_step_ratio in the first_namelist.input file.

time_step = 18,
parent_time_step_ratio = 1, 2, 3, 5

The second approach is using adaptive time stepping (second_namelist.input). This works for 3 domains, but when I include the 4th domain, I get a segmentation fault error.

use_adaptive_time_step = .true.,
step_to_output_time = .true.,
target_cfl = 1.2, 1.2, 1.2, 1.2
target_hcfl = .84, .84, .84, .84
max_step_increase_pct = 5, 51, 51, 51
starting_time_step = -1, -1, -1, -1
max_time_step = 43, 43, 43, 10
min_time_step = 16, 16, 1, 1
adaptation_domain = 1,

Any recommendations on how to solve this issue or further reduce processing time would be greatly appreciated.

Thanks!

 

[meteoadriatic]

In your first attempt, a time step ratio of 2 means that the nest use only half of what parent domain timestep use. Which, if parent is 18s, first nest would be 9s which is probably too large as it has 5 times smaller dx. Maybe it will survive, but as you go down you again have too small time step scaling (3x) which leads to pretty large time steps for d3 (18/2/3 = 3 seconds for dx = 0.216 km, and "safe" value considering 6xdx gives 1.3 seconds). For your d4, with dx = 43 m, you would need approx 0.25 seconds and you have 18/2/3/5 = 0.6 seconds). That surely leads to crash. Instead you probably want to increase time step ratio over 5 (let's say, 1,6,6,6), to get to much smaller time steps for nests not reduce it.

In your second attempt with adaptive time step your target_cfl and target_hcfl are too large. Try 0.8 and 0.6 respectively and see if that is enough to keep model stable. min_time_step of 16 for d2 (dx = 1.08 km) is also huge, so seting min_time_step to -1, -1, -1, -1 (defaults) should also be necessary or set min values to be = 2xdx or even 1xdx.

There are more nounces that you should pay attention and experiment by trial and error on such small scales. Too much to write in one post, expect to have many many attempts before that work out.

EDIT: Now I see that you want to decrease simulation wall-time. Probably there is not much you can do. Maybe simpler physics and if you can reducing domain area. Apart from that if you want to keep that grid size, probably nothing you can do.

 

[jmcabrera]

Hi Meteoadriatic,

Thanks for you reply.

About the the second attempt, the option of -1 doesn't work, I tested with 10, 2, 1, 1. For 43m the min time should be around 0.09 but I can't use decimal numbers and finally the processing stop.

How you decided the target_cfl and target_hcfl values? Always I see the same values for those fields.

 

[meteoadriatic]

Hello, there is min_time_step_den parameter which is a denominator for the min_time_step.

So, the combination of:
min_time_step = 10, 2, 4, 8
min_time_step_den = 1, 1, 10, 100

yields: 10/1, 2/1, 4/10, 8/100, or equal to, 10, 2, 0.4, 0.08 seconds. The same goes for other adaptive time step entries.

Mentioned target_cfl and target_hcfl are my wild guesses, that steam from problematic simulations. Those will just keep CFL conditions on lower side, which means that adaptive time step will be lower than when using default values. That, in case you have instability. If you don't, then feel free to increase them until model is still stable, but set minimum values to those suggested above, or lower (theoretically you can go as low as you want with minimums).

 

[kwerner]

@jmcabrera,
When you ran this originally, why did you have to use a time_step of 18? If you set that larger (e.g., 30), did you get CFL errors? If not, then using adaptive time step probably won't help much. If you weren't getting those errors, try setting to 30 and see what happens.

I would not recommend changing the parent_time_step_ratio. It's usually best to keep that set to the same as the parent_grid_ratio. You should be able to use a few more processors. You could try something like 81 (9x9) to see if that speeds it up.

 

[ehsantaghizadeh]

@jmcabrera,
When you ran this originally, why did you have to use a time_step of 18? If you set that larger (e.g., 30), did you get CFL errors? If not, then using adaptive time step probably won't help much. If you weren't getting those errors, try setting to 30 and see what happens.

I would not recommend changing the parent_time_step_ratio. It's usually best to keep that set to the same as the parent_grid_ratio. You should be able to use a few more processors. You could try something like 81 (9x9) to see if that speeds it up.

I was curious about the reasoning behind setting the basis to 9x9.
Could you please explain why you chose the number 9 and the benefits or specific considerations for selecting this configuration?

 

[kwerner]

I was curious about the reasoning behind setting the basis to 9x9.
Could you please explain why you chose the number 9 and the benefits or specific considerations for selecting this configuration?

I was simply suggesting a total number of processors that aligns with the specifications in Choosing an Appropriate Number of Processors, and then to make the processing grid squared, I chose a perfect square of 9x9. So it was somewhat arbitrary. It's not necessary to have a perfect square, but you also don't want to have a grid that is something like 11x2. Closer to a square can sometimes be more successful.

 

[jmcabrera]

I was simply suggesting a total number of processors that aligns with the specifications in Choosing an Appropriate Number of Processors, and then to make the processing grid squared, I chose a perfect square of 9x9. So it was somewhat arbitrary. It's not necessary to have a perfect square, but you also don't want to have a grid that is something like 11x2. Closer to a square can sometimes be more successful.

I tried using 16 processors (the recommended option), and it took less time than when using 48 processors.