Hello,
I am encountering two distinct and reproducible crashes with WRF-Chem V4.4.2, on a single domain (Central Africa/Sahel region, 55 km, 100x47x40, running from 2022-11-28 to 2023-03-01). After extensive diagnosis (detailed below), both appear to originate from the code rather than from my input data, and I would appreciate your help confirming this and identifying a possible fix.
Base configuration:
- chem_opt = 9 (CBMZ_MOSAIC_4BIN_AQ)
- biomass_burn_opt = 2 (MOZCART plume-rise, biomassb_mozc package)
- Fire emissions: FINNv2.5.1 modvirs, processed via fire_emis, mapping corrected to exactly match the ebu_in: species list of the biomassb_mozc package (Registry.chem, line 4093)
- Anthropogenic emissions: emiss_opt = 4 (ecbmz_mosaic)
- Chemical IC/BC: chem_in_opt=1, gas_ic_opt=1, gas_bc_opt=1, aer_ic_opt=1, aer_bc_opt=1, prepared via mozbc
- wetscav_onoff=1, mp_physics tested with both 2 (Lin et al.) and 10 (Morrison), same result
=== BUG 1: systematic crash in the plume-rise routine (biomassb_mozc) at the model top ===
With biomass_burn_opt=2 active, the run crashes systematically at the very first integration step, inside module_plumerise1.F, with the following message:
DEBUG Z FROM WRF: z(44,39,25)= NaN
DEBUG Z FROM WRF: z_at_w(44,39,25)= NaN
DEBUG Z FROM WRF: p_phy/t_phy/rho_phy = NaN
DEBUG ZMID: z(39),z_at_w(kts) = NaN 1.22364473
DEBUG PLUME ERROR: znz >= zt(nkp)
DEBUG PLUME ERROR: znz,zt(nkp) = NaN 19850.0000
DEBUG PLUME ERROR: k1,k2,kmt,nkp = 1 39 1 200
*** floating-point exceptions: IEEE_INVALID_FLAG IEEE_DIVIDE_BY_ZERO IEEE_OVERFLOW_FLAG
STOP envir stop 12
This NaN appears systematically at the topmost vertical level (k=39, e_vert=40), at the exact same grid cell (i=44,j=39), regardless of the following changes (each tested individually, with no effect on the crash location or values):
- time_step reduced from 330s to 100s
- epssm increased from 0.3 to 0.5
- damp_opt=3 disabled (damp_opt=0)
- mp_physics switched from 10 (Morrison) to 2 (Lin et al.)
I have verified in detail:
- The content of wrfinput_d01 at this exact column (P, PB, PH, PHB, T, QVAPOR, U, V): all values are physically consistent, no NaN, no discontinuity.
- The FIRESIZE_AG*/MEAN_FCT_AG* fields in wrffirechemi at this cell: mutually consistent (no size=0/fraction>0 mismatch).
- Real and reasonable fire activity is indeed present at this location (ebu_in_co=4.65 mole km-2 hr-1).
I found two forum threads that appear directly related:
- Plumerise bug (excessive sulfate concentrations near the model top above active fire locations, with biomass_burn_opt=2, present in 4.1.3 and 4.2.1)
- The discussion confirming that FINN emissions processed through fire_emis are only officially supported with MOZART chemistry options (scale_fire_emiss restricted to MOZART_KPP/MOZCART_KPP/etc. in module_plumerise1.F), which is not my case (CBMZ_MOSAIC_4BIN_AQ).
Question: is the combination biomass_burn_opt=2 + chem_opt=9 (CBMZ-MOSAIC, non-KPP) officially supported? Is there a known fix for this systematic NaN at the model top?
=== BUG 2: independent LSODES crash, unrelated to fire emissions ===
With fire emissions fully disabled (biomass_burn_opt=0), the crash location shifts: it now occurs inside gasodesolver_lsodes, at a different grid cell (i=97, j=2, k=1 — near the surface, southeast corner of the domain), with repeated solver failures (up to *** gasodesolver_lsodes failure no. 33) followed by NaN across all emitted species (no, no2, co, so2, nh3, eth, hcho, ch3oh, c2h5oh, ald2, hcooh, par, aone, tol, xyl, cres, isop, dms...), while species without direct emissions remain at a fixed non-NaN initialization value.
At this exact cell I have verified:
- wrfchemi_d01 (E_CO, E_NO, E_NO2, E_SO2, E_NH3, E_TOL, E_KET, E_ALD — both domain-wide min/max and the local value): all values are clean and physically reasonable.
- wrfbdy_d01 (so2_BYS, no2_BYS, no_BYS, o3_BYS, co_BYS at the same point): consistent with the wrfinput_d01 values just inside the domain, no discontinuity.
- wrfinput_d01 (so2, no2, no, o3, co at the same column): clean.
This cell falls within the lateral boundary relaxation zone (spec_zone=1, relax_zone=4, jds=1, so j=2 is in the buffer zone). I wonder whether the LSODES solver is particularly sensitive to a concentration gradient in this IC/BC transition zone, but I have not found any anomalous value that would explain the failure on its own.
=== Request ===
Could you please help me determine:
1. Whether Bug 1 (plume-rise/biomassb_mozc) is a known issue for CBMZ-MOSAIC, and whether there is a known fix or workaround other than disabling biomass_burn_opt or migrating to a MOZART mechanism (which would require recompiling with KPP)?
2. Any leads for diagnosing Bug 2 (LSODES) more precisely, which appears to be independent of Bug 1?
I can provide the full rsl.error/rsl.out log files, my namelist.input, and my fire_emis.inp file if that would help.
Thank you very much for your help,
Best regards,
David
I am encountering two distinct and reproducible crashes with WRF-Chem V4.4.2, on a single domain (Central Africa/Sahel region, 55 km, 100x47x40, running from 2022-11-28 to 2023-03-01). After extensive diagnosis (detailed below), both appear to originate from the code rather than from my input data, and I would appreciate your help confirming this and identifying a possible fix.
Base configuration:
- chem_opt = 9 (CBMZ_MOSAIC_4BIN_AQ)
- biomass_burn_opt = 2 (MOZCART plume-rise, biomassb_mozc package)
- Fire emissions: FINNv2.5.1 modvirs, processed via fire_emis, mapping corrected to exactly match the ebu_in: species list of the biomassb_mozc package (Registry.chem, line 4093)
- Anthropogenic emissions: emiss_opt = 4 (ecbmz_mosaic)
- Chemical IC/BC: chem_in_opt=1, gas_ic_opt=1, gas_bc_opt=1, aer_ic_opt=1, aer_bc_opt=1, prepared via mozbc
- wetscav_onoff=1, mp_physics tested with both 2 (Lin et al.) and 10 (Morrison), same result
=== BUG 1: systematic crash in the plume-rise routine (biomassb_mozc) at the model top ===
With biomass_burn_opt=2 active, the run crashes systematically at the very first integration step, inside module_plumerise1.F, with the following message:
DEBUG Z FROM WRF: z(44,39,25)= NaN
DEBUG Z FROM WRF: z_at_w(44,39,25)= NaN
DEBUG Z FROM WRF: p_phy/t_phy/rho_phy = NaN
DEBUG ZMID: z(39),z_at_w(kts) = NaN 1.22364473
DEBUG PLUME ERROR: znz >= zt(nkp)
DEBUG PLUME ERROR: znz,zt(nkp) = NaN 19850.0000
DEBUG PLUME ERROR: k1,k2,kmt,nkp = 1 39 1 200
*** floating-point exceptions: IEEE_INVALID_FLAG IEEE_DIVIDE_BY_ZERO IEEE_OVERFLOW_FLAG
STOP envir stop 12
This NaN appears systematically at the topmost vertical level (k=39, e_vert=40), at the exact same grid cell (i=44,j=39), regardless of the following changes (each tested individually, with no effect on the crash location or values):
- time_step reduced from 330s to 100s
- epssm increased from 0.3 to 0.5
- damp_opt=3 disabled (damp_opt=0)
- mp_physics switched from 10 (Morrison) to 2 (Lin et al.)
I have verified in detail:
- The content of wrfinput_d01 at this exact column (P, PB, PH, PHB, T, QVAPOR, U, V): all values are physically consistent, no NaN, no discontinuity.
- The FIRESIZE_AG*/MEAN_FCT_AG* fields in wrffirechemi at this cell: mutually consistent (no size=0/fraction>0 mismatch).
- Real and reasonable fire activity is indeed present at this location (ebu_in_co=4.65 mole km-2 hr-1).
I found two forum threads that appear directly related:
- Plumerise bug (excessive sulfate concentrations near the model top above active fire locations, with biomass_burn_opt=2, present in 4.1.3 and 4.2.1)
- The discussion confirming that FINN emissions processed through fire_emis are only officially supported with MOZART chemistry options (scale_fire_emiss restricted to MOZART_KPP/MOZCART_KPP/etc. in module_plumerise1.F), which is not my case (CBMZ_MOSAIC_4BIN_AQ).
Question: is the combination biomass_burn_opt=2 + chem_opt=9 (CBMZ-MOSAIC, non-KPP) officially supported? Is there a known fix for this systematic NaN at the model top?
=== BUG 2: independent LSODES crash, unrelated to fire emissions ===
With fire emissions fully disabled (biomass_burn_opt=0), the crash location shifts: it now occurs inside gasodesolver_lsodes, at a different grid cell (i=97, j=2, k=1 — near the surface, southeast corner of the domain), with repeated solver failures (up to *** gasodesolver_lsodes failure no. 33) followed by NaN across all emitted species (no, no2, co, so2, nh3, eth, hcho, ch3oh, c2h5oh, ald2, hcooh, par, aone, tol, xyl, cres, isop, dms...), while species without direct emissions remain at a fixed non-NaN initialization value.
At this exact cell I have verified:
- wrfchemi_d01 (E_CO, E_NO, E_NO2, E_SO2, E_NH3, E_TOL, E_KET, E_ALD — both domain-wide min/max and the local value): all values are clean and physically reasonable.
- wrfbdy_d01 (so2_BYS, no2_BYS, no_BYS, o3_BYS, co_BYS at the same point): consistent with the wrfinput_d01 values just inside the domain, no discontinuity.
- wrfinput_d01 (so2, no2, no, o3, co at the same column): clean.
This cell falls within the lateral boundary relaxation zone (spec_zone=1, relax_zone=4, jds=1, so j=2 is in the buffer zone). I wonder whether the LSODES solver is particularly sensitive to a concentration gradient in this IC/BC transition zone, but I have not found any anomalous value that would explain the failure on its own.
=== Request ===
Could you please help me determine:
1. Whether Bug 1 (plume-rise/biomassb_mozc) is a known issue for CBMZ-MOSAIC, and whether there is a known fix or workaround other than disabling biomass_burn_opt or migrating to a MOZART mechanism (which would require recompiling with KPP)?
2. Any leads for diagnosing Bug 2 (LSODES) more precisely, which appears to be independent of Bug 1?
I can provide the full rsl.error/rsl.out log files, my namelist.input, and my fire_emis.inp file if that would help.
Thank you very much for your help,
Best regards,
David