Error while running wrf.exe forrtl: severe (174): SIGSEGV, segmentation fault occurred

[anuradha]

Dear Users,
I am running WRF ARW V4.0 on a HPC using GFS input data.
While running WRF.exe I am getting this error
forrtl: severe (174): SIGSEGV, segmentation fault occurred
I have already kept "ulimit -s unlimited" in my bashrc and the script before running real.exe and wrf.exe
Still I encounter same problem.
I am attaching my namelist and rsl files.
Please suggest.

 

[kwerner]

Hi,
There are a couple of things that may be causing the problem.

1) To my knowledge, the highest resolution of available GFS data is 0.25 degree, meaning that the resolution is a little more than 27 km. This would mean that from the large-scale input first-guess data to your domain 01, that is ~9:1 ratio, which is potentially too large. If you are interested in looking at a domain with a resolution of 3 km, it may be necessary to put another coarser domain around your current d01 to make sure the transition is smooth. See this page for best practices on domain set-up: https://www2.mmm.ucar.edu/wrf/users/namelist_best_prac_wps.html

2) Your domain is very large (1000x1000) with a very high resolution, meaning that each output is likely going to be large and possibly exceed the size limit of 4GB per output. You currently have the frames_per_outfile parameter in your namelist set to 1000, meaning that for the full 24 hour time period in your namelist, all of the history write times (history_interval = 60, so 24 total hours) will be put into a single wrfout file. Try setting that to something much smaller. For example, if you set frames_per_outfile = 1, then each wrfout* time would go into its own file and you should have 24 wrfout* files at the end.

After correcting these things, there is a chance it still will not run correctly, but at least we will know that your set-up is better. At that point, we can start looking at other things.

 

[anuradha]

Dear Sir,
Thanks for your reply.
For checking I have run the model with 18 km resolution (by giving GFS input data 25 km), but still the same error is coming
I have increased the debug level to 300 in namelist file to know the error
The model coming out when it calls rrtmg_lw
I am attaching the corresponding namelist, rsl error files
Please suggest

 

[kwerner]

Can you try these things:
1) Don't turn debug_level on. Set that back to 0. We recently removed that option from the default namelist because it rarely gives any useful information and simply adds a lot of junk to the rsl* files, making them difficult to read through.
2) set frames_per_outfile = 1,
3) Can you do a search through all of your rsl* files to make sure there aren't any CFL errors:

grep cfl rsl.*

If you get any print-outs for that, then the problem is likely an instability issue and you'll need to set your time_step lower.
4) Check your available space on the disk you are using. If you are close to, or out of space, then the model will crash because there's no room to write more files.

If none of those help, will you please send me a couple of times of the met_em.d01* files so that I can try to repeat this problem? If the files are too large to attach here, please see the home page of this forum for instructions on uploading large files. Thanks!

 

[anuradha]

I am unable to attach my met files... Even from the cloud

 

[anuradha]

Sir,
I have tried all the options u addressed , but same error is coming.
I am attaching the met files.
Pls suggest

 

[anuradha]

Dear Sir,
This is the error shows in my rsl error file
************************************************************************************************
taskid: 10 hostname: iitmc09n22
module_io_quilt_old.F 2931 F
Quilting with 1 groups of 0 I/O tasks.
Ntasks in X 4, ntasks in Y 4
WRF V4.0 MODEL
*************************************
Parent domain
ids,ide,jds,jde 1 167 1 167
ims,ime,jms,jme 77 132 77 132
ips,ipe,jps,jpe 84 125 84 125
*************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
alloc_space_field: domain 1, 86238608 bytes allocated
med_initialdata_input: calling input_input
Input data is acceptable to use:
Max map factor in domain 1 = 1.09. Scale the dt in the model accordingly.
INPUT LandUse = "MODIFIED_IGBP_MODIS_NOAH"
LANDUSE TYPE = "MODIFIED_IGBP_MODIS_NOAH" FOUND 33 CATEGORIES 2 SEASONS WATER CATEGORY = 17 SNOW CATEGORY = 15
INITIALIZE THREE Noah LSM RELATED TABLES
d01 2020-05-26_00:00:00 Input data is acceptable to use:
Tile Strategy is not specified. Assuming 1D-Y
WRF TILE 1 IS 84 IE 125 JS 84 JE 125
WRF NUMBER OF TILES = 1
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
wrf.exe 0000000003D039E9 Unknown Unknown Unknown
wrf.exe 0000000003D022BE Unknown Unknown Unknown
wrf.exe 0000000003CA8742 Unknown Unknown Unknown
wrf.exe 0000000003C3FD53 Unknown Unknown Unknown
wrf.exe 0000000003C4693B Unknown Unknown Unknown
libpthread.so.0 00002B76889A4790 Unknown Unknown Unknown
wrf.exe 00000000022C7DEB rrtmg_lw_taumolta 5289 module_ra_rrtmg_lw.f90
wrf.exe 00000000022C3848 rrtmg_lw_taumol_m 4935 module_ra_rrtmg_lw.f90
wrf.exe 00000000022FC9F3 rrtmg_lw_rad_mp_r 10980 module_ra_rrtmg_lw.f90
wrf.exe 0000000002317953 module_ra_rrtmg_l 12480 module_ra_rrtmg_lw.f90
wrf.exe 0000000001A0E6AD module_radiation_ 1629 module_radiation_driver.f90
wrf.exe 0000000001BAD128 module_first_rk_s 262 module_first_rk_step_part1.f90
wrf.exe 00000000013A514F solve_em_ 892 solve_em.f90
wrf.exe 0000000001208030 solve_interface_ 123 solve_interface.f90
wrf.exe 00000000004D06DB module_integrate_ 338 module_integrate.f90
wrf.exe 000000000040D067 module_wrf_top_mp 324 module_wrf_top.f90
wrf.exe 000000000040C965 MAIN__ 44 wrf.f90

 

[kwerner]

Hi,
I ran a test simulation using your namelist setup and your met_em* files. I also get a segmentation fault. However, I then downloaded my own GFS data and ran WPS with your set-up (I think), and I'm able to run my simulation to completion. I did have to reduce the time_step a little, but that doesn't seem to be what is causing the problems with your simulation.
1) Where did you obtain your GFS data?
2) Can you send your namelist.wps file so that I can confirm that I'm setting it up correctly?

 

[anuradha]

Dear Sir,
Please find the attachment and do the needful


I am downloding GFS data from the below ftp site
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/

 

[kwerner]

Hi,
Even when using your namelist.wps file, and your namelist.input file, I am able to run without any problems with the GFS data I am downloading. I'm not sure if there is a problem with the data you're using. This is where I'm getting my data:
https://rda.ucar.edu/datasets/ds084.1/
Try getting it from there and see if it helps. Just start with a few times, and run a short run to see if you can get past the problem. You didn't happen to modify your WRF code at all, did you?

 

[Yonghang Xie]

Dear Sir,
Thanks for your reply.
For checking I have run the model with 18 km resolution (by giving GFS input data 25 km), but still the same error is coming
I have increased the debug level to 300 in namelist file to know the error
The model coming out when it calls rrtmg_lw
I am attaching the corresponding namelist, rsl error files
Please suggest

Hi anuradha

I also had the same problem, the wrf-run stopped at 'calls rrtmg_lw'.
Have you known the reson? Thanks for your reply.

 

[kwerner]

@Yonghang,
That is a very common place for the model to stop, when it stops right at the beginning. It's likely a different problem than the user above was experiencing. Can you repost your question as a new topic, and make sure to include the version of the model you're using, your namelist.input file, and your error files (e.g., rsl.error.*) in the post.

 

[Ming Chen]

If the model stops at the right beginning, it often indicates the input data is wrong.
I would suggest you download data from the website here:
https://rda.ucar.edu/datasets/ds084.1/

Then restart from WPS (you only need to rerun ungrib and metgrid, if your model domain is not changed).

Please try and let us know if you still have problems.

 

[yogeshkumkar]

*SOLVED*
I had the similar problem with RegCM+CLM where I used KF cumulus parameterization. The model run well, I got restart files. I needed to continue the simulation after 3 months of inactive model code. I tried all of the options discussed here. Finally I reconfigured the mode, make clean all, and then recreated all the executables. Then it worked. Save the old executables somewhere before you 'make clean all'.
Looks like it is noting to do with the KF options and model code. Its about compilers and modules.
Thanks.
Yogesh