I get segment faults when doing trajectory with WRF,

[Peter]

Hi everyone,

I am working on the trajectory based on WRF v 4.3.1. As I know, the trajectory code has been renewed since WRF3.9, the renewed information is:

Trajectory diagnostics:

The original "trajectory" code in routine module_em.F only computed the trajectory spatial coordinates per trajectory, per time step.
The new code, in share/module_trajectory.F, linearly interpolates in x,y, and z to get the value of up 100 variables per trajectory per time step.

And the User Guide can be found in

https://www2.acom.ucar.edu/sites/default/files/documents/Trajectory.desc_.pdf

Following the guidance, I modified the namelist.input and wrfinput_traj_d01.
For namelist.input, the modification is:

&domains
num_traj = 1000,

&physics
traj_opt = 1
dm_has_traj = .true., .true., .true.,

For wrfinput_traj_d01,

&traj_default
traj_def%start_time = '2018-07-04_06:00:00',
traj_def%stop_time = '2018-07-04_09:00:00',
traj_def%dyn_name(1:6) = 'p', 'T', 'z', 'u', 'v', 'w',
traj_def%hyd_name(1) = 'QVAPOR',
/

&traj_spec
traj_type%start_time = '2018-07-04_06:00:00', '2018-07-04_06:00:00',
'2018-07-04_06:00:00', '2018-07-04_06:00:00',
'2018-07-04_06:00:00', '2018-07-04_06:00:00',
'2018-07-04_06:00:00', '2018-07-04_06:00:00',
'2018-07-04_06:00:00', '2018-07-04_06:00:00',
'2018-07-04_06:00:00',
traj_type%stop_time = '2018-07-04_09:00:00', '2018-07-04_09:00:00',
'2018-07-04_09:00:00', '2018-07-04_09:00:00',
'2018-07-04_09:00:00', '2018-07-04_09:00:00',
'2018-07-04_09:00:00', '2018-07-04_09:00:00',
'2018-07-04_09:00:00', '2018-07-04_09:00:00',
'2018-07-04_09:00:00',
traj_type%lev = 60., 60., 60., 60., 60., 60., 60., 60., 60., 60., 60.,
traj_type%lon = 125.0, 126.0, 127.0, 128.0,
129.0, 130.0, 131.0, 132.0,
133.0, 134.0, 135.0,
traj_type%lat = 29.0, 29.0, 30.0, 30.0,
31.0, 31.0, 32.0, 32.0,
33.0, 33.0, 34.0,
/

When I use 1 core to execute wrf.exe, i.e. serial, the program works well. However, when I use more cores and compute in parallel, a segment fault is reported with no detail information.
I have tried to changing my timestep smaller and use "ulimit -s unlimited" command before wrf.exe, it still doesn't work. I also tried to
export MP_STACK_SIZE=64000000, nothing works.

Does anyone have successful experience to run trajectory using WRF? Any suggestions will be grateful. Thanks!

best,
Deming

 

[kwerner]

Hi Deming,
1) Are you running this with WRF-chem, or basic WRF?
2) If you turn off the trajectory option, are you able to run with multiple processors without any issues?
3) Can you attach your namelist.input file, and package up your error/out files (e.g., rsl.*) and attach those, as well?
Thanks!

 

[Peter]

Hi Deming,
1) Are you running this with WRF-chem, or basic WRF?
2) If you turn off the trajectory option, are you able to run with multiple processors without any issues?
3) Can you attach your namelist.input file, and package up your error/out files (e.g., rsl.*) and attach those, as well?
Thanks!

Hi Kwerner,
Thanks for replying.
I am running basic WRF, and it runs well with multiple processors when I turn off the trajectory option.
I will attach all the output files below for detailed diagnoses. Thank you very much for the help!
Deming

 

[kwerner]

Hi,
Thanks for sending those. As a test, can you let me know if it works if you set the number of trajectories to something much lower (like 25)?