Strange behavior with MPAS-Atmosphere 8.0.1

[mantovanijr]

Hi everyone.

Basically, I ran 48-h simulations with 60-3 km mesh using convection-permitting physics suit, with 576 MPI tasks x 12 nodes, and took 280 min to finish.

The strange behavior is this: I create the same configuration, but initialize a day earlier, so now it is a 72-h simulation. The total time takes more than 12 hours to finish. Why is this happening? Is it something to do with the cluster? Note that all the configuration is the same as well as the output frequency.

Thank you.

 

[mgduda]

It might be interesting to see if every time step is slower for the 72-h simulation, or whether the simulation is slower only during certain periods of the integration. If you run something like

grep "Timing for integration" log.atmosphere.0000.out

for the original 48-h run, and again for the 72-h run, are there any notable differences in the time series of the time step times?

 

[mgduda]

It might also be worth comparing the total time for individual code regions in the timer summaries at the end of the log files. Are all of the timer values proportionately larger in the 72-h simulation compared with the 48-h simulation? Or are some code regions taking disproportionately longer?

 

[mantovanijr]

It might also be worth comparing the total time for individual code regions in the timer summaries at the end of the log files. Are all of the timer values proportionately larger in the 72-h simulation compared with the 48-h simulation? Or are some code regions taking disproportionately longer?

Thank you. I re-ran the case and it took ~430 min to finish, which is the expected time based on a 48-h run. It seems to me the cluster presented some instability and now it's ok.

 

[mgduda]

Great to hear the runtimes are now in line with expectations, and thanks very much for following up!