Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

Obs nudging runtime issues

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

L

luna_123456

New member
I am attempting to nudge a simulation using WRFv3.8.1 towards a sounding but am finding that there are no changes between the nudged simulation and the simulation without observation nudging. The sounding has 250 levels. The observation time is 1 hour into the simulation and it appears to read in correctly. Below is the nudging section from the namelist.input and output from the rsl files which I believe indicates that the observations are read in properly. Is there anything obviously wrong here or any thoughts on what can be adjusted to activate nudging?

Namelist.input:
&fdda
obs_nudge_opt = 1,0,0,0,
max_obs = 150000,
fdda_start = 0., 0., 0., 0.,
fdda_end = 99999., 99999., 99999., 99999.,
obs_nudge_wind = 1,1,1,1,
obs_coef_wind = 6.E-4,6.E-4,6.E-4,6.E-4,
obs_nudge_temp = 1,1,1,1,
obs_coef_temp = 6.E-4,6.E-4,6.E-4,6.E-4,
obs_nudge_mois = 1,1,1,1,
obs_coef_mois = 6.E-4,6.E-4,6.E-4,6.E-4,
obs_rinxy = 125.,25.,5.,1.,
obs_rinsig = 0.1,
obs_twindo = 0.6666667,0.6666667,0.6666667,0.6666667,
obs_npfi = 10,
obs_ionf = 1, 2, 2, 2,
obs_idynin = 0,
obs_dtramp = 40.,
obs_prt_freq = 1, 10, 10, 10,
obs_prt_max = 10,
obs_ipf_errob = .true.,
obs_ipf_nudob = .true.,
obs_ipf_in4dob = .true.,
obs_ipf_init = .true.,
obs_max_sndng_gap = 30,
/


Reading in obs:

OBS NUDGING: Reading new obs for time window TBACK = -0.667 TFORWD = 0.667 for grid = 1
opening first fdda obs file, fonc=01 inest= 1
ifon= 1
Saving index of first ob after end of current time window for nest = 1:
ob index = 1, time of ob = 1.0000 hrs, end of time window = 0.6667 hrs
Saving index of first ob after end of current time window for nest = 1:
ob index = 2, time of ob = 1.0000 hrs, end of time window = 0.6667 hrs
Saving index of first ob after end of current time window for nest = 1:
ob index = 3, time of ob = 1.0000 hrs, end of time window = 0.6667 hrs
Saving index of first ob after end of current time window for nest = 1:
ob index = 4, time of ob = 1.0000 hrs, end of time window = 0.6667 hrs
Saving index of first ob after end of current time window for nest = 1:
ob index = 5, time of ob = 1.0000 hrs, end of time window = 0.6667 hrs
.
.
.
Saving index of first ob after end of current time window for nest = 1:
ob index = 250, time of ob = 1.0000 hrs, end of time window = 0.6667 hrs
.
.
.
.
.
.
****** CALL IN4DOB AT KTAU = 0 AND XTIME = 0.00: NSTA = 0 ******
*****NUDGING INDIVIDUAL OBS ON MESH # 1 WITH RINXY = 0.125E+03 KM, RINSIG = 0.100E+00 AND TWINDO (HALF-PERIOD) = 0.400E+02 MIN
NUDGING IND. OBS WINDS WITH GIV = 0.600E-01
NUDGING IND. OBS TEMPERATURE WITH GIT = 0.600E-01

NUDGING IND. OBS MOISTURE WITH GIQ = 0.600E-01
ABOVE THE SURFACE LAYER, OBS NUDGING IS PERFORMED ON PRESSURE LEVELS,
WHERE RINXY VARIES LINEARLY FROM 0.125E+03 KM AT THE SURFACE TO 0.250E+03 KM AT 500.00 MB AND ABOVE
IN THE SURFACE LAYER, WXY IS A FUNCTION OF DPSMX = 75.00 MB WITH DCON = 0.133E+00 - SEE SUBROUTINE NUDOB
FOR EFFICIENCY, THE OBS NUDGING FREQUENCY FOR MESH # 1 IS 1 CGM TIMESTEPS
.
.
.
.
.
.
REPORTING OBS MASS-PT LOCS FOR NEST 1 AT XTIME= 0.0 MINUTES
FREQ= 1, MAX= 10 LOCS, NEWLY READ OBS ONLY, -999 => OBS OFF PROC

OBS# I J K OBS LAT OBS LON XLAT(I,J) XLONG(I,J) TIME(hrs) OBS STATION ID
1 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
2 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
3 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
4 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
5 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
6 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
7 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
8 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
9 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
10 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1

OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 0 0.00 3 3 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 0 0.00 4 4 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 0 0.00 1 1 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 0 0.00 2 2 rindx=20.0


Starting obs nudging 40 minutes prior to sounding collection:

++++++CALL ERROB AT KTAU = 34 AND INEST = 1: NSTA = 250 ++++++
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 34 20.40 3 3 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 34 20.40 4 4 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 34 20.40 1 1 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 34 20.40 2 2 rindx=20.0


Obs nudging at sounding time:

++++++CALL ERROB AT KTAU = 100 AND INEST = 1: NSTA = 250 ++++++
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 100 60.00 3 3 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 100 60.00 4 4 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 100 60.00 1 1 rindx=20.0
OBS NUDGING FOR IN,J,KTAU,XTIME,IVAR,IPL: 1 10 100 60.00 2 2 rindx=20.0
 
Hi,
Can you please attach your full namelist.input file, as well as the rsl.out.0000 file?

Thanks,
Kelly
 
Hi,
Do you mind running this again and turning debug_level to 0? This is an option that we've recently removed in newer versions of the code because it rarely gives any useful information and ends up just adding a ton of junk to the rsl* files, making them difficult to read. My computer keeps stalling when trying to advance in this file. Once you do that, can you attach the new namelist and rsl file? I apologize for the inconvenience.
 
Attached is the rsl.out file. The only change that was made to the namelist.input was debug_level = 0 instead of debug_level = 1000.
I stopped the simulation after ~5.5 hours since this is well after the time that I was trying to implement observation nudging.
 

Attachments

  • rsl.out.txt
    418 KB · Views: 79
Is it possible that the data are being ingested incorrectly? I'm reviewing this and the observations that are being printed out seem like they might indicate that there are 250 observations at the surface at that time. What does K represent in this case? If it's the vertical coordinate, shouldn't K increase for each observation?

OBS# I J K OBS LAT OBS LON XLAT(I,J) XLONG(I,J) TIME(hrs) OBS STATION ID
1 20.106 19.948 1.000 35.200 -112.050 35.200 -112.050 1.00 TEST1
 
Hi,
Thanks for sending that. I just ran a test with some observational data I have and it's also showing K set to 1.000 for every one listed, as I'm seeing in your out file too. I am seeing differences in my case, though. What are you using to look at the differences? If you're not already, can you run a case without obs nudging, and then one with it, and then compare the output at the same times with the diffwrf tool - you can find this in WRF/external/io_netcdf/diffwrf. Just copy diffwrf to your running directory and then use this syntax to compare:

./diffwrf wrfout_d01.date_and_time_of_file_without_nudging wrfout_d01.date_and_time_of_file_with_nudging >& diffwrf.txt

You can take a look at that diffwrf.txt file to see if any differences were found.
 
My diffwrf.txt comes back empty:

Code: 
 Just plot  F
Diffing wrfout_d01_2016-03-03_01:00:00 nudge_applied/wrfout_d01_2016-03-03_01:00:00
 Next Time 2016-03-03_01:00:00
     Field   Ndifs    Dims       RMS (1)            RMS (2)     DIGITS    RMSE     pntwise max

Did you use OBSGRID or the RT_fdda_reformat_obsnud.pl to generate your OBS_DOMAIN101 file? Would you mind sharing your working OBS_DOMAIN file with me? I found that if I use OBSGRID to generate the file, K now has different values that increase with each observation but there is no difference in the model output.
 
I used the OBSGRID program to create my OBS_DOMAIN101 file. I'll attach that file here, but my simulation is different than yours. I'll also attach the other files necessary to run this (this forum requires the .txt extension, so just remove those once you download the files). If you're unable to find anything significant, then perhaps you can also attach your files so that I can try to repeat your case on my end. If so, please send me your OBS_DOMAIN101, wrfinput_d01, wrfbdy_d01, as well as the files I can use to try to recreate your OBS_DOMAIN101 file (namelist.oa, your met_em* files for a short run, and your input observation file).

Thanks,
Kelly
 

Attachments

  • namelist.input
    5 KB · Views: 99
  • OBS_DOMAIN101.txt
    258 KB · Views: 129
  • wrfbdy_d01.txt
    18.4 MB · Views: 65
  • wrfinput_d01.txt
    14.8 MB · Views: 65
Thanks for sharing all of that! I found that I can nudge the simulation with a surface observation based on what you shared (I just ran a test using the first point from the sonde data) and get differences in the nudged versus non-nudged simulations. So the model and the data input are working for the surface obs but there is something wrong when I try to include more vertical levels for the observations. As far as I can tell there is nothing wrong with the sounding observations when I compare them with the example on page 30 provided in A Brief Guide to Observation Nudging in WRF by Brian Reen (link here: http://www2.mmm.ucar.edu/wrf/users/docs/ObsNudgingGuide.pdf ).


This file works for the surface obs nudging:

Code: 
 20160303010000
    35.1982 -114.0480
  TEST1                                                 
  FM-12                                  1356.     F     F      1
   85000.000       0.000    1356.000       0.000     282.200       0.000       3.322       0.000     -11.583       0.000      43.909       0.000 -888888.000 -888888.000 -888888.000 -888888.000 -888888.000 -888888.000

This file does not work for the sounding obs nudging:

Code: 
 20160303010000
    35.1982 -114.0480
  TEST1                                                 
  FM-35                                  1356.     T     F      7
   85000.000       0.000    1356.000       0.000     282.200       0.000       3.322       0.000     -11.583       0.000      43.909       0.000
   82500.000       0.000    1665.915       0.000     279.642       0.000      -0.418       0.000     -11.501       0.000      49.020       0.000
   80000.000       0.000    1916.999       0.000     277.732       0.000      -3.996       0.000     -13.939       0.000      46.816       0.000
   77500.000       0.000    2174.999       0.000     275.023       0.000      -2.784       0.000     -11.803       0.000      58.928       0.000
   75000.000       0.000    2439.998       0.000     275.017       0.000      -8.106       0.000      -8.298        0.000      46.702       0.000
   72500.000       0.000    2712.113       0.000     272.850       0.000      -9.226       0.000      -7.896        0.000      52.808       0.000
   70000.000       0.000    2991.996       0.000     271.350       0.000      -4.547       0.000      -9.464        0.000      52.157       0.000


Have you had any success with running a simulation nudged with a sounding and do you have an OBS_DOMAIN101 for a sounding?
 
Hi,
I actually have not tried using a sounding before, and unfortunately don't have an example file to share; however, can you try a test:

Try to split up the levels into their own full data record. For instance, instead of
Code: 
 20160303010000
    35.1982 -114.0480
  TEST1                                                 
  FM-12                                  1356.     F     F      1
   85000.000       0.000    1356.000       0.000     282.200       0.000       3.322       0.000     -11.583       0.000      43.909       0.000 -888888.000 -888888.000 -888888.000 -888888.000 -888888.000 -888888.000
   82500.000       0.000    1665.915       0.000     279.642       0.000      -0.418       0.000     -11.501       0.000      49.020       0.000

Try
Code: 
 20160303010000
    35.1982 -114.0480
  TEST1                                                 
  FM-12                                  1356.     F     F      1
   85000.000       0.000    1356.000       0.000     282.200       0.000       3.322       0.000     -11.583       0.000      43.909       0.000 -888888.000 -888888.000 -888888.000 -888888.000 -888888.000 -888888.000
 20160303010000
    35.1982 -114.0480
  TEST1                                                 
  FM-35                                  1356.     T     F      7
   82500.000       0.000    1665.915       0.000     279.642       0.000      -0.418       0.000     -11.501       0.000      49.020       0.000

Just test it out with 2 levels first to see if this will even work before trying to recreate the entire file this way. Let me know if that makes a difference.

Thanks,
Kelly
 
Thanks Kelly,

I have given that a try and found that there isn't any difference between the wrfout from the different OBS_DOMAIN files. With the sounding split like that though, the surface nudging is still active even if the upper air obs aren't contributing to nudging.

I'm starting to print out information from the module_fddaobs_rtfdda to see if I can pick up any clues there.
 
Hmm. Okay, do you mind attaching your OBS_DOMAIN1O1 file, and your original observation (sounding) file so that I can poke around and see if I can find any differences? If you send your namelist.wps file, that would also be useful for me to set up the same domain.

Thanks,
Kelly
 
Attached is my OBS_DOMAIN101 and the original sounding in little_r format which can be used in obsgrid or RT_fdda_reformat_obsnud.pl.

From the additional print statements from the WRF code, I'm finding that the model isn't having any difficulty reading in the observation and the obs are passing the quality tests throughout the code (e.g. T isn't greater than 20 C off from the model at that time or level).
 

Attachments

  • OBS_DOMAIN101.txt
    1.3 KB · Views: 109
  • sounding.txt
    49.9 KB · Views: 82
Hi,
I just ran some tests with V3.8.1 using your OBS_DOMAIN101 file. I used my own namelist set-up, but made sure to include the point that is in your observation file (I copied your entire &fdda section into my namelist. I also used your same settings for the two auxinput11 parameters). I conducted the following 3 tests:

1) I ran with obs_nudge_opt =1, and then again, only changing obs_nudge_opt = 0. I ran for 6 hours, starting 1 hour prior to the observation time. At the end of the 2 different runs, the wrfout* files were different.

2) I set obs_nudge_opt = 1, using your exact 0BS_DOMAIN101 file, running for 6 hours. I ran the exact same test again, but only kept the 1st level in the OBS_DOMAIN101 file. Comparing the wrfout* from the 2 tests after 6 hours, the files were different.

3) I then noticed that in your OBS_DOMAIN101 file, you have the date as
20160303010000
but the namelist.input file that you previously gave me used a start date/time of
2017-03-03_00:00:00

So then I conducted tests 1) and 2) above again, but changing the year in the OBS_DOMAIN101 file to be different from my namelist.input file (just to see if this would work). The tests did complete, with WRF being successful; however, at the end of my 6 hour periods, the wrfout* files were identical with/without nudging on, regardless of whether I used 1 or 8 lines in the observation file.

Could it be that you had the year wrong in many of these tests, and that it didn't match the namelist?

Kelly
 
Thanks for checking that, it's reassuring to know that the OBS file is correctly setup for a sounding. I did check the dates and they are correct across the obs and namelist. I must have corrected that error sometime earlier. WRF is definitely reading in the file for the right time since XTIME = 1.0 in the namelist and NSTA = 8 for the time window of nudging.

I think it's coming down to how the model is compiled. I just tried compiling with smpar and found that the upper air nudging will work (I get a difference between the two wrfout* files for nudge vs no nudge) but the model crashes at random times around the full strength nudging (usually afterwards). Throughout this forum I have been running with smpar+dmpar.

Could you share what kind of machine you're running on and how you have WRFv3.8.1 compiled?
 
Hi,
Interesting. We typically recommend either using an smpar or dmpar compile. There has not been a lot of thorough testing with the dm+sm build, and we have actually seen some odd results or problems with it (which may explain what is happening for you, as well). I ran this on my local machine, which is a Mac and a GNU build with dmpar, and I also tested on our supercomputing system, which is a Linux machine, using an Intel/dmpar build of WRF.
 
This is very similar to the trouble I am having right now. It might be related to the handling of multiple tiles by the CPUs. In my case, the tiles were handled sequentially, so the calculation of rko is neglected in the subroutine errob in phys/module_fddaobs_rtfdda.F . The rko and errf(9,:) are set to 0. The errf(9,:) is used as the vertical indices in nudob subroutine for actual nudging tendencies calculation. More details are listed in my post below.
http://forum.mmm.ucar.edu/phpBB3/viewtopic.php?f=45&t=5391
I compiled my wrf.exe with dmpar only, which means no OPENMP I believe and the sequential handling of tiles? I tried dm+sm option for wrf.exe but I have segmentation errors from time to time for no obvious reasons. Might be related to the CPUs and the compiler compatibility as well?
I have some "fix" listed in my post that works for multiple soundings. But that might mess up other nudging cases so not a very good solution.

Zheng
 
You must log in or register to reply here.
Top