Observation nudging (obs_nudge_opt) has zero measurable effect on d02 despite NSTA>0, correct alarm setup, and correct argument passing

[hirose]

Body:
WRF version: 4.7.1 (WRF-Chem build)

Setup:

Two-domain 2-way nest (feedback=1):
- d01: dx=15 km, grid_fdda=1 (analysis/grid nudging toward an MSM+FNL hybrid analysis; this is working correctly and shows a measurable effect vs. a no-nudging baseline)
- d02: dx=3.75 km, grid_fdda=0

I am trying to add surface-station (AMEDAS, Japan) observation nudging (obs_nudge_opt) on both domains, using an identical OBS_DOMAIN101/OBS_DOMAIN201 pair (per the guide, identical content across domains is fine since WRF ignores obs outside the domain). Current relevant &fdda settings:

obs_nudge_opt = 1, 1,
max_obs = 150000,
fdda_start = 0, 0,
fdda_end = 132120, 132120,
obs_nudge_wind = 1, 1,
obs_coef_wind = 6.0e-2, 6.0e-2, ! intentionally 100x normal, for diagnosis
obs_nudge_temp = 0, 0,
obs_coef_temp = 0.0, 0.0,
obs_nudge_mois = 0, 0,
obs_coef_mois = 0.0, 0.0,
obs_rinxy = 30.0, 30.0,
obs_rinsig = 0.1,
obs_twindo = 0.5, 0.5,
obs_npfi = 10,
obs_ionf = 2, 2,
obs_idynin = 0,
obs_dtramp = 40.0,
obs_ipf_errob = .true.,
obs_ipf_nudob = .true.,
obs_ipf_in4dob = .true.,

and in &time_control:

auxinput11_interval = 1, 1,

(Originally I had auxinput11_interval_s = 60, 60, and obs_rinxy/obs_twindo/obs_ionf given as a single value instead of one per domain — both of which I found documented as pitfalls in Brian Reen's "A Brief Guide to Observation Nudging in WRF" and have since fixed, per below.)

Symptom:

Comparing this run's d02 output against an otherwise-identical run with obs_nudge_opt=0,0 (grid nudging on d01 only, no obs nudging at all) — extracting 10-m wind at ~150 AMEDAS station locations and computing bias/RMSE/MAE against observations — the two cases are bit-for-bit identical (matching to 16 significant digits in the aggregate statistics, and exact at every station/hour when diffing the raw extracted time series). This holds even with obs_coef_wind set to 100x a normal value (6.0e-2 s⁻¹, e-folding time ~17 s) and run for multiple days. If the nudging were having any effect at all, even a small one from imperfect propagation, I would expect at least some tiny nonzero divergence given the chaotic nature of the model — not exact agreement to double precision.

What I've verified so far (all correct):

1. OBS_DOMAIN101/OBS_DOMAIN201 are present and opened correctly ("opening first fdda obs file, fonc=01 inest= 1" / "inest= 2" both appear).
2. rsl.error shows CALL ERROB for both INEST=1 and INEST=2, with NSTA consistently > 0 (~130 for d01, ~120 for d02).
3. The "OBSERVATION NUDGING IS ACTIVATED FOR MESH 2" / "END SETUP DESCRIPTION ... MESH 2" block prints correctly, with TWINDO (hrs) = 0.500 etc., confirming the per-domain namelist values are being read as intended for d02.
4. I traced the full call chain (wrf_fddaobs_in -> in4dob -> ... -> fddaobs_driver -> errob/nudob in module_fddaobs_rtfdda.F) and manually verified, by extracting and positionally diffing the SUBROUTINE nudob(...) dummy-argument list against the actual CALL nudob(...) argument list in module_fddaobs_driver.F (84 arguments), that every argument lines up correctly — in particular iswind<-nudge_wind, giv<-coef_wind, tfaci<-fdob%tfaci are all correctly wired, and tfaci is confirmed to stay at 1.0 (since obs_idynin=0).
5. The n=n+njcsnd loop increment (over NSTAT observations) is present and correct — it is not stuck reprocessing a single station.
6. rindx prints as 2.0 for d01 (30 km / 15 km dx, correct), and the horizontal weighting setup looks consistent for d02 as well.

Question:

Given all of the above checks out, what else could cause the observation-nudging tendency to have zero measurable effect on the model state, even at 100x the normal coefficient over multiple days? Is there a known interaction between grid_fdda on the parent domain + feedback=1 + obs_nudge_opt on the child domain that could suppress this? Or is there another namelist/registry pitfall (in the vein of the auxinput11_interval_s and per-domain-array ones I already found and fixed) that I might be missing?

Happy to share full rsl.error excerpts or the complete namelist on request.

 

[kwerner]

Hi, I'd first like to apologize for the delay in response. I want to let you know I'm running some tests and trying to track down the issue. I hope to get back to you with a solution, or at least some suggestions. Have you made any progress since posting this?

 

[hirose]

Hi, thanks for following up, and no worries about the delay. Yes — we tracked down the root cause on our end.

The issue was a FORMAT mismatch in our own observation-file generation, not a WRF bug. In wrf_fddaobs_in.F's in4dob routine, the data-record layout branches on is_sound:

- is_sound=.TRUE. (upper-air/sounding): FORMAT 104, 6 variable/QC pairs
- is_sound=.FALSE. (surface, single-level): FORMAT 105, 9 variable/QC pairs (SLP, reference pressure, height, temperature, U, V, RH, surface pressure, precipitation)

Our converter correctly wrote is_sound=.FALSE. headers (our obs are surface stations) but only emitted 6 variable/QC pairs of data per record — FORMAT 104 content under a FORMAT 105 header. WRF reads the header, sees is_sound=F, and expects a 217-character FORMAT 105 data line; our line was only 145 characters (FORMAT 104), so the parser silently overran into the next record's header. There's no crash and no error message: the run completes normally, NSTA is correctly nonzero, and IN4DOB/ERROB get called on schedule — but every observation after the first one or two is silently corrupted, so the innovations end up effectively meaningless and nudging has no measurable effect on the model state. This matches the symptom in my original post exactly (NSTA>0, correct alarm/argument setup, zero measurable effect).

We confirmed this by adding temporary print statements around VAROBS/ERRF inside ERROB. After fixing the converter to emit the full 9-pair FORMAT 105 record (unused slots set to the standard missing value, -888888), the innovations became physically reasonable, and a before/after comparison of nudged vs. non-nudged U10/V10 showed the expected signature of working nudging: essentially zero difference at t=0, growing smoothly over the first several hours, then settling to a steady nonzero offset.

For anyone hitting the same "zero effect despite NSTA>0" symptom with surface observations: check the byte length of your data lines against what in4dob expects for your is_sound setting (105 for surface, 104 for sounding) before looking anywhere else — this failure mode produces no diagnostic output at all.

Happy to share more detail on the converter or debugging approach if useful.

 

[kwerner]

Thank you so much for the update! Hopefully that will help someone else in the future.