Atmosphere model run failed using MPI+OMP

Hello everyone,

I'm running the atmosphere model using MPI+OMP. The mesh resolution is 60 km. The run succeeded using only MPI, while failed using MPI+OMP. Since I submitted my job using nohup, followed are contents of the nohup log file (see also the attachment run_atmosphere.log).

nohup: ignoring input
Copyright (C) 2009-2019 Intel Corporation. All rights reserved.
Intel(R) Advisor 2019 (build 602216)
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred


I used 20 MPI tasks with each having 2 threads to run the atmosphere model. Followed is my job submission script (see also the attachment run_atmosphere):

ulimit -s unlimited
export OMP_NUM_THREADS=2
mpirun -n 20 -machinefile hostfile ./atmosphere_model


The atmosphere model was installed supporting OpenMP using command "make ifort CORE=atmosphere USE_PIO2=true DEBUG=true OPENMP=true". I used Intel 2020 basic compiler (ifort, icc and icpc) and mvapich MPI compiler (mpif90, mpicc, mpicxx) to compile the model. The mvapich and other MPAS libruary source codes were downloaded from the site (https://www2.mmm.ucar.edu/people/duda/files/mpas/sources/).


The input data I used is CFSR:2010-10-23_00, which is the Sample real-data input file provided by the MPAS website http://mpas-dev.github.io/. Please find log.atmosphere.0000.out, streams.atmosphere and namelist.atmosphere files in the attached. I also attached the namelist.init_atmosphere files and streams.init_atmosphere files for both static data interpolation and initial condition interpolation.


Thank you in advance!

The problem was solved by adding "export KMP_STACKSIZE=20480000000" to the job submission script.

Thanks very much for following-up with the solution you've found!