Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

init_atmosphere_model with error

D

dlu

New member
I am trying to setup MPAS model in a workstation with Ubuntu OS. I setup all libs according to example script file from Index of /people/duda/files/mpas/sources including: mpich-3.3.1, zlib-1.2.11, hdf5-1.10.5, pnetcdf-1.11.2, netcdf-c-4.6.3, netcdf-fortran-4.5.2, and pio. Both pgi and nvhpc works fine and made all libs successful. We used the latest version of MPAS by making a clone of the MPAS-Model repository. We can successfully create init_atmosphere_model and atmosphere_model. But when running init_atmosphere_model, the error info will be like: 0: nvf_to_cfi_type: unexpected nvf kind 9197808 (0: CFI_is_contiguous: unsupported array rank 72). The rank number keep changing every time. Do you have any suggestion?

Thanks


Duanjun
 
Which version of the NVHPC compilers are you using? It might be worth trying newer versions of the MPICH and PnetCDF libraries. In the current release of MPAS, only an MPI and PnetCDF library are needed, so it should be easy enough to install MPICH 4.3.1 and PnetCDF 1.14.0 if you've already been able to install older versions of these libraries. When compiling MPAS, just ensure that the $PIO environment variable is not set so that MPAS won't try to link with the PIO library (which you won't have, since you will have installed just MPICH and PnetCDF).
 
Hi Duanjun,

Would you please clarify:

(1) which version of MPAS did you run?

(2) How did you compile MPAS and what compiler did you choose?

(3) Is your case a global MPAS run? If so, what did did you ungrib to provide initial condition for MPAS? Which mesh did you choose?

(4) Please upload your namelist.init_atmopshere and streams.init_atmosphere for me to take a look.

Thanks.
 
Hi @dlu,

I recall a colleague at NCAR having a similar problem when trying to containerize MPAS. I'm checking with them what they may have done to solve it. It was suggested they try switching the I/O library (they were using SMIOL, so they should try PIO).

What versions of PGI and NVHPC are you using? On our Derecho machine, I've had good results with software stacks our sys. admins installed based on NVHPC v24.3, v24.11, and v25.1 (not much testing with v25.5). I haven't touched PGI in years.

Also make sure to check core dumps or use debugging tools, both with a version of the code compiled with DEBUG=true
 
The NVHPC I used is "nvfortran 25.5-0 64-bit target on x86-64 Linux". I don't know if this the the latest one. I check later. The MPAS should be the latest one too (As I mentioned that I used clone), which shall be 8.3.1. At this time I just want to test the model using example case you guys provided: x1.40962. This what I used to compile initi_atmosphere_model: "make -f Makefile nvhpc CORE=init_atmosphere PRECISION=single AUTOCLEAN=true". I have two files in current fold: x1.40962.graph.info and x1.40962.grid.nc. The namelist.init_atmosphere and streams.init_atmosphere are attached. If I run the command as: ./init_atmosphere_model, it will produce: "0: nvf_to_cfi_type: unexpected nvf kind 9197808"
 

Attachments

  • namelist.init_atmosphere.txt
    1.6 KB · Views: 0
  • streams.init_atmosphere.txt
    1.1 KB · Views: 0
I tried what mgduda's suggestion with nvhpc two versions: 24.3 and 25.5. I only installed install MPICH 4.3.1 and PnetCDF 1.14.0, which were successfully through. Then I compiled MPAS v8.3.1. The Makefile of MPAS cannot went through by stopping with:
The following compilation command failed with errors:
mpicc pnetcdf.c -I/home/dlu/mpass_home/mpas-libs/include -O3 -DSINGLE_PRECISION -O3 -L/home/dlu/mpass_home/mpas-libs/lib -lpnetcdf -o pnetcdf.out
Then I have to change "CC_PARALLEL = mpicc" to "CC_PARALLEL = nvc", which init_atmosphere was created successfully. As I run init_atmosphere again, it will stop as: 0: CFI_is_contiguous: unsupported array rank 200.
I also tried by using pgi and I can successfully made MPICH4.3.1 and Pnetcdf as well as MPAS-A model (v8.3.1). As I run ./init_atmosphere, it stopped as:

Abort(406412805) on node 0: Fatal error in internal_Comm_dup: Invalid communicator, error stack:
internal_Comm_dup(97): MPI_Comm_dup(comm=0x4c000406, newcomm=0x3ae582f0) failed
internal_Comm_dup(43): Invalid communicator

Do you guys have any idea?
 
You must log in or register to reply here.
Top