"segmentation fault" in wrf.exe using shcu_physics=5

[yutinghe]

Hello all,
I'm using WRF-Solar shcu_physics=5 (Deng) to simulate the shortwave radiation conditions over the whole China, but got a "segmentation fault" error, it seems like this error is related to the Deng shallow cumulus scheme. I'm not sure if I might have missed some parameter settings. I've attached my namelist.input and rsl.error files below. My WRF version is 4.7.1, and I'm using 512 cpu to run my case.
By the way, I found this post describes a situation very similar to mine, but even after setting epssm = 0.9, I'm still encountering the same error.
Hope you can give me some help, many thanks!
Yuting

[e2113r5n4:51615:0:51615] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7ffec35c3918)
==== backtrace (tid: 51615) ====
0 0x0000000000053553 ucs_debug_print_backtrace() /root/jpsun-test/ucx-1.9.0/src/ucs/debug/debug.c:656
1 0x0000000002a9afa4 module_shcu_deng_mp_deng_shcu_() ???:0
2 0x0000000002a93327 module_shcu_deng_mp_deng_shcu_driver_() ???:0
3 0x000000000223c563 module_shallowcu_driver_mp_shallowcu_driver_() ???:0
4 0x000000000172072f module_first_rk_step_part1_mp_first_rk_step_part1_() ???:0
5 0x000000000122d036 solve_em_() ???:0
6 0x00000000010398c8 solve_interface_() ???:0
7 0x00000000004299ad module_integrate_mp_integrate_() ???:0
8 0x0000000000410cc1 module_wrf_top_mp_wrf_run_() ???:0
9 0x0000000000410c7f MAIN__() ???:0
10 0x0000000000410c1e main() ???:0
11 0x00000000000223d5 __libc_start_main() ???:0
12 0x0000000000410b29 _start() ???:0
=================================
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
wrf.exe 00000000031DC154 for__signal_handl Unknown Unknown
libpthread-2.17.s 00002B80CA65B5D0 Unknown Unknown Unknown
wrf.exe 0000000002A9AFA4 Unknown Unknown Unknown
wrf.exe 0000000002A93327 Unknown Unknown Unknown
wrf.exe 000000000223C563 Unknown Unknown Unknown
wrf.exe 000000000172072F Unknown Unknown Unknown
wrf.exe 000000000122D036 Unknown Unknown Unknown
wrf.exe 00000000010398C8 Unknown Unknown Unknown
wrf.exe 00000000004299AD Unknown Unknown Unknown
wrf.exe 0000000000410CC1 Unknown Unknown Unknown
wrf.exe 0000000000410C7F Unknown Unknown Unknown
wrf.exe 0000000000410C1E Unknown Unknown Unknown
libc-2.17.so 00002B80CA88A3D5 __libc_start_main Unknown Unknown
wrf.exe 0000000000410B29 Unknown Unknown Unknown

 

[akilag]

Hello, we have the same issue with WRF version 4.6.1. We are using the WRF reference configuration provided here: WRF-Solar® Reference Configuration | Research Applications Laboratory. Using this configuration as is results in a segmentation fault.

As per this post, I have removed icloud_bl = 0 for testing. Then I also had to remove shcu_physics setting (previously set to 5 for the Deng shallow convection scheme), then the simulation proceeds normally.

But the post linked here recommends not to set icloud_bl to 0.

Any suggestions from the moderators would be greatly appreciated!

 

[Ming Chen]

@ yutinghe

The Deng cumulus scheme actually handles both shallow and deep convections, although its deep convection part is not very active. For this reason, we don't recommend using this scheme because it may interfere with cumulus scheme (if turned on), or insufficiently describe deep convection (if cumulus scheme is off). In addition, MYNN PBL has been updated significantly, and these updates are tested with a focus on boundary processes. I don't think a broad test has been done to ensure it works with various combinations of other schemes including the Deng cumulus scheme.
Based on above reasons, I would suggest that you avoid using the Deng cumulus scheme at present.
If for some reason you must run with this scheme, then I would suggest that you compile WRF in debug mode, then rerun your case. The log file will tell exactly when and where the model crashes first, which will give you some hints what is wrong.

 

[akilag]

Hello, thanks very much for the reply. I am sharing what I have found so far in case it is useful for others.

Following your advice on another post, I compiled WRF version 4.6.1 in debug mode and found that the source of the segmentation fault was that the allocatable variable, QC_BL2D was not allocated.

Upon further investigation, in module_bl_mynn_wrapper.F, a few arrays - qc_bl2d, qi_bl2d, and cldfra_bl2d are allocated only when icloud_bl is greater than 0 even though they are passed to the mynn_bl_driver which does not use those arrays when icloud_bl is 0. I modified the wrapper to allocate the arrays, initialize, and deallocate the arrays for icloud_bl >=0. This resolved the segmentation fault issue and the run is now proceeding normally.

I have noted your recommendation about the use of Deng cumulus scheme.

 

[yutinghe]

Hello, thanks very much for the reply. I am sharing what I have found so far in case it is useful for others.

Following your advice on another post, I compiled WRF version 4.6.1 in debug mode and found that the source of the segmentation fault was that the allocatable variable, QC_BL2D was not allocated.

Upon further investigation, in module_bl_mynn_wrapper.F, a few arrays - qc_bl2d, qi_bl2d, and cldfra_bl2d are allocated only when icloud_bl is greater than 0 even though they are passed to the mynn_bl_driver which does not use those arrays when icloud_bl is 0. I modified the wrapper to allocate the arrays, initialize, and deallocate the arrays for icloud_bl >=0. This resolved the segmentation fault issue and the run is now proceeding normally.

I have noted your recommendation about the use of Deng cumulus scheme.

Hello, followed your post, I modified module_bl_mynn_wrapper.F in four locations:

!Allocate any arrays being used
if (icloud_bl >= 0) then
   allocate(qc_bl2d(ims:ime,kms:kme))
   allocate(qi_bl2d(ims:ime,kms:kme))
   allocate(cldfra_bl2d(ims:ime,kms:kme))
   qc_bl2d=0.0
   qi_bl2d=0.0
   cldfra_bl2d=0.0
endif

!need sgs cloud info input for diagnostic-decay
if (icloud_bl >= 0) then
   do k=kts,ktf
   do i=its,itf
      qc_bl2d(i,k)     = qc_bl(i,k,j)
      qi_bl2d(i,k)     = qi_bl(i,k,j)
      cldfra_bl2d(i,k) = cldfra_bl(i,k,j)
   enddo
   enddo
endif

!- Collect 3D ouput:
if (icloud_bl >= 0) then
   do k=kts,ktf
   do i=its,itf
      qc_bl(i,k,j)     = qc_bl2d(i,k)/(1.0 - sqv(i,k))
      qi_bl(i,k,j)     = qi_bl2d(i,k)/(1.0 - sqv(i,k))
      cldfra_bl(i,k,j) = cldfra_bl2d(i,k)
   enddo
   enddo
endif

if (icloud_bl >= 0) then
   deallocate(qc_bl2d)
   deallocate(qi_bl2d)
   deallocate(cldfra_bl2d)
endif

but the segmentation fault is still occured, can you give me more detailed modifications? and would you mind posting your namelist.input file? Many thanks!

 

[akilag]

Hello,

A few suggestions:

1) I only updated the code to allocate and deallocate the arrays (for icloud_bl >= 0) so just the first and the last code blocks you have. It appears we do not use these variables in module_bl_mynn.F as the arrays are initialized to 0 within that code (but we do need to allocate and deallocate the arrays as we are passing those in the subroutine call).
2) After the code changes, I had to do ./clean -a to clean the *.f90 files and recompile WRF (./configure and ./compile em_real). This time, I built the normal version, not the debug version (so run ./configure and not ./configure -D).
3) Based on Ming Chen's reply above, we will turn off Deng cumulus scheme (in WRF 4.6.1). We are still deciding whether we need to set icloud_bl to 0 or 1.
4) I also built WRF v4.7.1 yesterday and our current namelist.input works without any issues. Perhaps you could give this a try too.
5) Our namelist.input is directly from the WRF solar reference configuration here. We just replicated it for three domains using this as a reference.

 

[yutinghe]

Hello,

A few suggestions:

1) I only updated the code to allocate and deallocate the arrays (for icloud_bl >= 0) so just the first and the last code blocks you have. It appears we do not use these variables in module_bl_mynn.F as the arrays are initialized to 0 within that code (but we do need to allocate and deallocate the arrays as we are passing those in the subroutine call).
2) After the code changes, I had to do ./clean -a to clean the *.f90 files and recompile WRF (./configure and ./compile em_real). This time, I built the normal version, not the debug version (so run ./configure and not ./configure -D).
3) Based on Ming Chen's reply above, we will turn off Deng cumulus scheme (in WRF 4.6.1). We are still deciding whether we need to set icloud_bl to 0 or 1.
4) I also built WRF v4.7.1 yesterday and our current namelist.input works without any issues. Perhaps you could give this a try too.
5) Our namelist.input is directly from the WRF solar reference configuration here. We just replicated it for three domains using this as a reference.

Thank you for your reply! So you modified the code to allocate and deallocate the arrays for icloud_bl >= 0, and run successfully with Deng cumulus scheme? Because I'm simulating the solar resources using WRF-Solar, and I saw that Deng cumulus scheme is recommended, so I want to use this scheme if it can work fine.

 

[akilag]

Yes, exactly. After recompiling the code after the update, I was able to keep both shcu_physics = 5 and icloud_bl = 0 in the WRF namelist with version 4.6.1.

We are leaning towards 4.7.1 however which didn't produce any seg fault issues with the same namelist (and because it is the latest version with bug fixes).

If you are still getting a segmentation fault with 4.6.1, you can try configuring with the debug flag (./configure -D and then ./compile em_real). This help me to find where the segmentation fault was coming from.

 

[yutinghe]

Thank you so much! I'll try as you recommended!