Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

MPAS domain partition issue

M

MattW

New member
Hi MPAS support folks,

I recently ran into an issue with trying to run MPAS on a 3km limited area domain over the Ohio Valley with ERA5 ICs / BCs. I was able to create the static file and interpolated ICs / LBCs fine, and I used metis to create a graph.info.part file for the way I'll be running MPAS on derecho (in this case, using 1 node with 128 processors). However, when I run MPAS, I get a segmentation fault:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

I'm wondering if this is due to something being wrong with my graph.info.part file? I've attached my output log, the error file, and my namelist.atmosphere, streams.atmosphere, and graph.info.part files in the folder below. Also, the directory I'm running this in on derecho is /glade/derecho/scratch/mawilson/mpas_tests/ohio_mesh if that helps.

Thanks!

-Matt Wilson
 

Attachments

  • MPAS_OH.zip
    112.3 KB · Views: 5
From your log.atmosphere.0000.out file, it looks like the model crashed while taking the first time step. It's possible that there's an issue with the ICs or LBCs, or a problem with namelist settings. Taking a quick look at your namelist.atmosphere file, you've got a 30 s time step. In some cases, you may be able to get by with that time step on a 3-km mesh, but when cold-starting the model with ICs that were interpolated from another source, it's probably safest to choose a smaller time step -- typically around 5 - 6 times your minimum model grid distance in km. Can you try reducing config_dt to 15.0 to see whether that allows the model to get through some time steps?
 
Thanks! I just tried running the model with a 15 second timestep and a 5 second timestep and both of those runs failed with the same error. Is there anything else in the namelist.atmosphere that looks off?
 
I also tried running it with 256 cores and 512 cores (with graph.info files for each) in case the issue was that there weren't enough cores for the domain, but that resulted in the same error as well.
 
Did you run the case in derecho? if so, please tell me where the case is located. Also, what data did you use to create initial and boundary conditions for this regional MPAS run? Thanks.
 
Ming Chen said:
Did you run the case in derecho? if so, please tell me where the case is located. Also, what data did you use to create initial and boundary conditions for this regional MPAS run? Thanks.
Click to expand...
I'm running this case on derecho at /glade/derecho/scratch/mawilson/mpas_tests/ohio_mesh. The initial and boundary conditions were created from ERA5 data. Thanks!
 
Hi Matt,
I have repeated your case, Below is what I found:
(1) When I use GFS, the case can run successfully.
(2) When I use ERA5 over a different period (in my test case it is initialized at 2017-04-26_00:00:00), and the case can run successfully
(3) However, when I use ERA5 over your time period (2022-09-15), the case failed.
Based on above tests, I can confirm that the model works fine, but the input data somehow is not correct.

Would you please clarify how you process the ERA5 data?
 
I downloaded all of the necessary fields from the NCAR RDA archive of ERA5 and ran them through ungrib. I then ran the static data file for ERA5 through ungrib and combined it with the previously-ungribbed data. Maybe something is wrong with the data from this particular day in ERA5?
 
How did you download and process the ERA5 data? I just tried the test case that worked for you (2017-04-26_00:00:00) and got the same segmentation fault error.
 
Sorry for the additional questions, but was your successful 2017-04-26 run conducted using the same domain and model version I'm using for my 2022-09-15 experiment? I just tried running the 2022-09-15 case using GFS IC/BCs and got the same error, so I think the problem is either in how I'm processing the data to create the IC/BCs or in how I've configured the domain.
 
My case is over a different domain. This case was created years ago in cheyenne and I didn't use ERA5 since then.

After all the codes and data are migrated to derecho, I am not sure whether something has changed that caused the trouble you have.
 
Hi Matt,

I just run a global MPAS case using ERA5 2023 data as input. It works just fine and I suppose it should also work for regional MPAS.

I am suspicious that your input data are not correct. Please confirm that you download upper level data of U, V, T, Q and Z ?
 
I think I may have figured out what was going wrong with my limited area run using ERA5. The ERA5 data I was using was missing a surface field (I think it was called SOILHGT) which prevented it from constructing the blended terrain at the boundary of the limited-area mesh. I was trying to get around that earlier by setting config_blend_bdy_terrain to false, which allowed me to create ICs and BCs. However, when I tried to use those ICs and BCs to run MPAS, I got the segfault error I initially wrote about. Repeating that process using GFS data resulted in the same error; however, the GFS data has all of the surface data it needs to set config_blend_bdy_terrain to true, and creating the initial conditions with that setting allows me to successfully run MPAS. I'm guessing that might also by why you're not running into problems with your ERA5-forced global run--either your ERA5 data has all of the fields it needs, or the lack of the fields needed to run the blended terrain doesn't matter, since you don't need the blended terrain for the global run.
 
MattW said:
I think I may have figured out what was going wrong with my limited area run using ERA5. The ERA5 data I was using was missing a surface field (I think it was called SOILHGT) which prevented it from constructing the blended terrain at the boundary of the limited-area mesh. I was trying to get around that earlier by setting config_blend_bdy_terrain to false, which allowed me to create ICs and BCs. However, when I tried to use those ICs and BCs to run MPAS, I got the segfault error I initially wrote about. Repeating that process using GFS data resulted in the same error; however, the GFS data has all of the surface data it needs to set config_blend_bdy_terrain to true, and creating the initial conditions with that setting allows me to successfully run MPAS. I'm guessing that might also by why you're not running into problems with your ERA5-forced global run--either your ERA5 data has all of the fields it needs, or the lack of the fields needed to run the blended terrain doesn't matter, since you don't need the blended terrain for the global run.
Click to expand...
Hi @MattW : Do you find any solution on this? I tried with ERA5 the problems is missing of a variable SOILHGT.
However, I still want to use ERA5 as the input.
 
@ frfajary

I saw your post of the same question somewhere else. Please don't post same questions in different forums.
Just in case you still need information for the ERA5 SOILHGT issue, please see the answer here.
 
You must log in or register to reply here.
Top