MPAS restart run with different numbers of CPUs

[JiaWang]

I have a restart file from a MPAS run using 128 CPUs.
When I use this restart file to initiate a restart run while using 256 CPUs, the job just hangs there without any error. The log file is attached.
But if I still use 128 CPUs for the restart run, it runs successfully.
Why is this?

 

[abishekg]

Hi Jia,

Could you describe more about the experiment you are trying to run? And are you able to run this specific case on 256 CPUs from a cold initialization (without restarting) and does the problem only occur when restarting?

Also, if you can attach the log file from the successful run, that might be useful info.

 

[JiaWang]

For a cold start, I can use any number of CPUs, as long as I have a compatible "x4.535554.graph.info.part.xxx"

The problem occurs when I generate the restart file using 128 CPUs, then use the restart file to initialize a new run but using 256 CPUs.

 

[JiaWang]

My mistake.
For a cold start run, the 256 CPUs configuration also does not work, but the 128 CPUs and 512 CPUs can work.
I do not know why the simulation just hangs there when using 256 CPUs, no matter cold restart or restart.

 

[abishekg]

Thanks for checking! That is useful info regarding the 256 CPUs case. Could you say more about the type of experiment you are running and namelist options used (you can attach the namelist file)? I'll see if we can reproduce this issue on our end.

 

[JiaWang]

Attached is the namelist I use.
Both 128 CPUs and 512 CPUs can work, but not 256 CPUs.
I use the variable mesh 60-15 km, "x4.535554".

 

[abishekg]

Thank you! I will have a look at this today and get back to you.