MPT Error running MPAS atmosphere_model

[vortexdude]

Hi,

I've been trying to run atmosphere_model off a restart file and I'm getting an MPT Error that I do not know how to address. I'm running MPAS version 6.1 with 4 nodes on Cheyenne (I can provide further information such as the namelist if that is helpful). The command I'm running is: mpiexec_mpt dplace -s 1 ./atmosphere_model || exit 2

I don't know if the error message below is related to something wrong with compiling the model? Or if I set something bad in the namelist? Any insight would be extremely appreciated since I do not know how to proceed. Much thanks in advance!

Error Message:

MPT ERROR: Rank 0(g:0) is aborting with error code 1046803809.
Process ID: 64529, Host: r6i6n8, Program: /glade/work/jdberman/MPAS_v6.1/atmosphere/atmosphere_model
MPT Version: HPE MPT 2.19 02/23/19 05:30:09

MPT: --------stack traceback-------
MPT: Attaching to program: /proc/64529/exe, process 64529
MPT: Try: zypper install -C "debuginfo(build-id)=4e96cf37d52b9c2f3648e691878b682da5abfa42"
MPT: (no debugging symbols found)...done.
MPT: Try: zypper install -C "debuginfo(build-id)=5eb2f40ad3b0125943aba8f08dd08609351a2967"
MPT: (no debugging symbols found)...done.
MPT: Try: zypper install -C "debuginfo(build-id)=b115bb26e97505a5bd3b56d70d20459aa1206ac9"
MPT: (no debugging symbols found)...done.
MPT: Try: zypper install -C "debuginfo(build-id)=4f3d05f200db29c6835a48e466e0378a8541fd36"
MPT: (no debugging symbols found)...done.
MPT: [Thread debugging using libthread_db enabled]
MPT: Using host libthread_db library "/glade/u/apps/ch/os/lib64/libthread_db.so.1".
MPT: Try: zypper install -C "debuginfo(build-id)=93c4deac1088eb84fbd01cf2a2c54399f516e9a7"
MPT: (no debugging symbols found)...done.
MPT: Try: zypper install -C "debuginfo(build-id)=5f9ec139af58fa59c33f72d1b3e56f083f1613ae"
MPT: (no debugging symbols found)...done.
MPT: Try: zypper install -C "debuginfo(build-id)=bc347d1c2dd56b51057fbac71e84906135d02da5"
MPT: (no debugging symbols found)...done.
MPT: Try: zypper install -C "debuginfo(build-id)=4c08f43bb18e99a7df4bad5c4a52bac67ddf9b8d"
MPT: (no debugging symbols found)...done.
MPT: Try: zypper install -C "debuginfo(build-id)=3ae04b58bd81ea7745dba789d89937e719309568"
MPT: (no debugging symbols found)...done.
MPT: 0x00002b44f920c6da in waitpid () from /glade/u/apps/ch/os/lib64/libpthread.so.0
MPT: Missing separate debuginfos, use: zypper install glibc-debuginfo-2.22-49.16.x86_64
MPT: (gdb) #0 0x00002b44f920c6da in waitpid ()
MPT: from /glade/u/apps/ch/os/lib64/libpthread.so.0
MPT: #1 0x00002b44f994bdb6 in mpi_sgi_system (
MPT: #2 MPI_SGI_stacktraceback (
MPT: header=header@entry=0x7fff99ab6d10 "MPT ERROR: Rank 0(g:0) is aborting with error code 1046803809.\n\tProcess ID: 64529, Host: r6i6n8, Program: /glade/work/torn/MPAS_v6.1/atmosphere/atmosphere_model\n\tMPT Version: HPE MPT 2.19 02/23/19 05"...) at sig.c:340
MPT: #3 0x00002b44f9894419 in print_traceback (ecode=ecode@entry=1046803809)
MPT: at abort.c:246
MPT: #4 0x00002b44f98946ea in PMPI_Abort (comm=<optimized out>,
MPT: errorcode=1046803809) at abort.c:68
MPT: #5 0x00002b44f98949cc in pmpi_abort__ ()
MPT: from /glade/u/apps/ch/opt/mpt/2.19/lib/libmpi.so
MPT: #6 0x0000000000ab14cb in mpas_log_mp_mpas_log_write_ ()
MPT: #7 0x0000000000532941 in mpas_atmphys_utilities_mp_physics_error_fatal_ ()
MPT: #8 0x0000000000531a41 in mpas_atmphys_control_mp_physics_tables_init_ ()
MPT: #9 0x000000000053f915 in atm_core_mp_atm_mpas_init_block_ ()
MPT: #10 0x000000000054082f in atm_core_mp_atm_core_init_ ()
MPT: #11 0x000000000040a3e2 in mpas_subdriver_mp_mpas_init_ ()
MPT: #12 0x0000000000407969 in MAIN__ ()
MPT: #13 0x000000000040791e in main ()
MPT: (gdb) A debugging session is active.
MPT:
MPT: Inferior 1 [process 64529] will be detached.
MPT:
MPT: Quit anyway? (y or n) [answered Y; input not from terminal]
MPT: Detaching from program: /proc/64529/exe, process 64529

MPT: -----stack traceback ends-----
MPT ERROR: MPI_COMM_WORLD rank 0 has terminated without calling MPI_Finalize()
aborting job

 

[mcurry]

Its hard to tell from the stack traceback. The namelist.atmosphere and the streams.atmosphere files would be beneficial if you have them.

It would also be helpful if you could attach the full log file of log.atmosphere.0000.out if it is available. Likewise if the model was able to create a log.atmosphere.xxxx.err file (which it might have made many), including one that includes the error would also be helpful. Sometimes, however, the model isn't able to produce a log.atmosphere.xxxx.err file on a crash.

Lastly, do you know what compilers and libraries were used to compile MPAS? If you do, including those would also be helpful!

 

[vortexdude]

Thanks for the reply!

Attached is the namelist.atmosphere, namelist.init_atmosphere and the streams.atmosphere and streams.init_atmosphere files (I had to include an ".txt" extension to upload them).

So the MPAS model was compiled by someone else (who ran the model successfully previously) and gave me the restart files, from which I've been trying to run the simulations. The model was compiled in March 2019, shortly after Cheyenne was updated to MPT 2.19, which I don't know if that could be a potential issue. I couldn't locate the log.atmosphere.0000.out or log.atmosphere.xxxx.err files, but if I can later I'll upload them.

I believe MPAS was compiled using ifort and single precision and using pio. But I'm not sure where exactly I might find that. Let me know if the attached files and this information is enough to help understand it, or if there is other information that would help solve the problem.

Thank you so much for offering to help!

 

[mcurry]

o the MPAS model was compiled by someone else (who ran the model successfully previously) and gave me the restart files, from which I've been trying to run the simulations. The model was compiled in March 2019, shortly after Cheyenne was updated to MPT 2.19, which I don't know if that could be a potential issue.

Ah yes, that is most likely the cause of the issue. Here and here are the notices that CISL put out about the update. Note:

Most users’ programs and executables will need to be rebuilt following the update, as many system libraries will change.

So, unfortunately, any MPAS executable built before the MPT update will have to be re-compiled.

Was the person who gave you these executables able to successfully run them recently (after the MPT update)?

 

[vortexdude]

Yes, the person did re-compile the model after the update. I'm not sure if he tested running the executables afterwards (I just emailed him to ask).

Is this likely the reason why I'm getting the error message? Or could it be another reason? I assume the namelist and stream files didn't provide any information that could explain it? Or is there any other test I could do to help solve this MPT error message?

Thanks for your help and time!

 

[vortexdude]

An update:

I asked the person to recompile MPAS, and then I tried running the model again from the restart file, which resulted in the same error.

I finally located the log.atmosphere.out and log.atmosphere.err files, and -- to no surprise -- there is an error message in there:

----------------------------------------------------------------------
Beginning MPAS-atmosphere Error Log File for task 0 of 144
Opened at 2019/10/17 14:15:25
----------------------------------------------------------------------

ERROR:
ERROR: ------------------------------ FATAL CALLED ------------------------------
ERROR: --- tables to run the Thompson cloud microphysics scheme do not exist: run build_tables first.
CRITICAL ERROR: MPAS core_physics abort
Logging complete. Closing file at 2019/10/17 14:15:25


It appears that I did not have the following Thompson microphysics scheme tables built:

MP_THOMPSON_freezeH2O_DATA.DBL@ MP_THOMPSON_QIautQS_DATA.DBL@ MP_THOMPSON_QRacrQG_DATA.DBL@ MP_THOMPSON_QRacrQS_DATA.DBL@

So I ran ./build_tables, which then created these files. And then I ran the model - and it worked!!

Thank you mcurry for your help in trying to debug this! I now realize to check (and find) the log.atmosphere* files first and understand them before looking at any MPT errors.

Thanks!

 

[mcurry]

Yes! The log files are very important. Glad you were able to get things working!