Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

Possible Bug with MOZART-MOSAIC scheme while writing output

R

ramousta

New member
Hi everybody!
I am having some issues running WRF (from V4.3 to V4.4.2) with MOZART-MOSAIC chemistry (chem_opt = 202). The simulation starts correctly and works fine until the model attempts to write wrfout, the simulation crashes without any message. The lines where the model stops:
d01 2019-12-03_01:00:00 DEBUG wrf_timetoa(): returning with str = [2019-12-03_01:00:00]
d01 2019-12-03_01:00:00 output_wrf: begin, current_date=2019-12-03_01:00:00
d01 2019-12-03_01:00:00 module_io.F (md_calls.m4) : in wrf_put_dom_ti_char_arr
.................
d01 2019-12-03_01:00:00 module_io.F (md_calls.m4) : in wrf_put_dom_ti_real_sca
d01 2019-12-03_01:00:00 output_wrf: end, fid = 4
d01 2019-12-03_01:00:00 back from outsub in open_w_dataset
d01 2019-12-03_01:00:00 calling wrf_open_for_write_commit in open_w_dataset
d01 2019-12-03_01:00:00 module_io.F: in wrf_open_for_write_commit
d01 2019-12-03_01:00:00 Information: NOFILL being set for writing to wrfout_d01_2019-12-03_01:00:00

If I use iofields_filename option, I got this message:

d01 2019-12-03_01:00:00 calling outsub in open_w_dataset
d01 2019-12-03_01:00:00 output_wrf: begin, fid = 4
W A R N I N G : Unable to modify mask for alt. Variable not found. File: iofields.chem at line 1
W A R N I N G : Unable to modify mask for gsw. Variable not found. File: iofields.chem at line 2
W A R N I N G : Unable to modify mask for extaer1. Variable not found. File: iofields.chem at line 3
W A R N I N G : Unable to modify mask for extaer2. Variable not found. File: iofields.chem at line 4
....................................................................................
W A R N I N G : Unable to modify mask for brnch_rto. Variable not found. File: iofields.chem at line 19
W A R N I N G : Unable to modify mask for ccn1. Variable not found. File: iofields.chem at line 20
Ignoring problems reading iofields.chem
Continuing. To make this a fatal error, set ignore_iofields_warn to false in namelist

d01 2019-12-03_01:00:00 module_io.F: in wrf_inquire_filename
............................................................................
d01 2019-12-03_01:00:00 module_io.F (md_calls.m4) : in wrf_put_dom_ti_integer_arr
d01 2019-12-03_01:00:00 module_io.F (md_calls.m4) : in wrf_put_dom_ti_integer_arr
d01 2019-12-03_01:00:00 module_io.F (md_calls.m4) : in wrf_put_dom_ti_integer_arr

The simulation work very well for the version WRFV4.2.2.
I really believe that this is a bug that appears from V4.3. Perhaps it is related to photolysis changes made by the developers !!!!
There are other persons that reported this problem before:
forum.mmm.ucar.edu

WRF4.3.2 crashes while writing output

Hi all, I am having some issues running WRF V4.3.2 with MOZART-MOSAIC chemistry (chem_opt = 201). The simulation starts correctly and works fine until the model attempts to write wrfout. In particular, when reaching the history_interval the simulation crashes without any message. Playing a bit...
forum.mmm.ucar.edu forum.mmm.ucar.edu

Problem while running WRF-Chem with MOZART-MOSAIC scheme (chem_opt = 202 and 201)

groups.google.com groups.google.com

Thank you very much for offering your assistance and for reporting this problem to WRFchem developers.
Rachid
 

Attachments

  • namelist.input
    21.3 KB · Views: 53
Last edited:
Hi Rachid,

Thank you for contacting the forum. I'd like to work with you in an attempt to solve this problem that others might also be experiencing. I've been looking through possible changes made to 4.3.

First, it's possible this problem is related to the KPP optimization change: Switched off optimization for KPP compilation, to stop buffer overflow. by douglowe · Pull Request #1471 · wrf-model/WRF
Please try to edit the file: chem/KPP/configure_KPP by changing CC_FLAGS="-O0" to CC_FLAGS="-O".

Then perform a "./clean -a", recompile and test. If that works, yay! If not...

Second, please turn off these options so that we can ensure the problem might be related to I/O:

emiss_opt = 0
phot_opt = 0
wetscav_onoff = 0
cldchem_onoff = 0
conv_tr_wetscav = 0
conv_tr_aqchem = 0
seas_opt = 0
dust_opt = 0
dmsemis_opt = 0
biomass_burn_opt = 0
aer_ra_feedback = 0

a) If the model runs without issue with these options, off, please try to add them back iteratively to see which one might be causing the issue.
-Large changes were made to the dust code, so I might start with that first.

b) If the model does not run with these options off, please perform a "./clean -a", then "./configure", then change the FC_DEBUG line in configure.wrf to be "-g $(FCNOOPT) -traceback", recompile, and try running again. Then please reply with what line the model fails on (in the rsl.out.0000)

Finally, when you say "Perhaps it is related to photolysis changes made by the developers", which changes are you talking about? In any case, it seems this problem might be occurring with both phot_opt = 3 and 4.

Jordan
 
Last edited:
Thank you very much Jordan for your help. I will follow these instructions and i will get back to you soon with further update.
 
Hi Jordan!

What I mean by "Perhaps it is related to photolysis changes ...", is that since version 4.3 there are many changes related to dust and photolysis rates that have been made and the problem we have may be related to theses changes. Perhaps I am wrong, I just put a question about that?!!

The results of the tests are:

Test 1: configure_kpp by changing CC_FLAGS="-O0" to CC_FLAGS="-O" --> It does not resolve the problem.

Test 2: turn off emiss_opt, phot_opt, wetscav_onoff, cldchem_onoff, conv_tr_wetscav, conv_tr_aqchem, seas_opt, dust_opt, dmsemis_opt, biomass_burn_opt and aer_ra_feedback.

The program got stuck at the beginning without any error message. I had to kill the program forcibly.The message at the end of log files
.....................................................
d01 2019-12-03_00:00:00 call HALO_RK_CHEM
d01 2019-12-03_00:00:00 calling inc/HALO_EM_CHEM_E_5_inline.inc
.....................................
d01 2019-12-03_00:00:00 calling kpp_mechanism_driver
d01 2019-12-03_00:00:00 kpp_mechanism_driver: calling mozart_mosaic_4bin_aq_interface
Forced exit from Rosenbrock due to the following error:
Step size too small
T= 2.880000000000002E-022 and H= 2.880000000000002E-022
.....................................................
Forced exit from Rosenbrock due to the following error:
Step size too small
T= 3.010585434151566E-028 and H= 3.010585434151566E-028

Test 4: install wrfchem in the debug mode (with option FCDEBUG = -g $(FCNOOPT) -traceback). In this case the simulation runs 6 times longer than in normal mode, but I did not see significant changes in printed messages in rsl files.

I attached the rsl+namelist output files when installing WRFchem normal and in debug mode.

Thank you, Jordan, so much for your support
 

Attachments

  • rsl.zip
    440.8 KB · Views: 1
  • rsl_turnedOff_Param.zip
    145.6 KB · Views: 0
  • rsl_Debug.zip
    443.4 KB · Views: 1
  • rsl_Debug_turnedOff_Param.zip
    147.9 KB · Views: 1
Last edited:
Hi Rachid,

Thank you for performing these tests. I've briefly taken a look and will continue to dig - in the meantime, can you check to make sure there aren't any unique messages in any of the other rsl.* files? Unfortunately the *.0000 are not very helpful...

Test 2 is worrying.. with it failing at an indetermined spot .. can you try testing those again? At least attempt to turn off dust. It would be good to try to find which one is causing the hang.

You could also try a smaller time step (just to see if it makes it through the first wrfout write)

Also, can you please attach the iofields file you tested previously?

Jordan
 
Last edited:
Thank you very much Jordan!
mozart_ph_diag = 1 --> Solves the problem
Tested for history_interval = 2 and 60.
in this case worfout files are generated with good data.
What is the utility of mozart_ph_diag? I Could not find it in the wrfchem guide
 
Hi Rachid,

Hmmmm.. very interesting. That option outputs pH diagnostics, see: Add pH diagnostics for mozart_mosaic_4bin_aq chemistry by jordanschnell · Pull Request #1342 · wrf-model/WRF

It seems the variables are not packaged correctly. I will figure this out and remedy this for the upcoming release.

Can you try one last thing for me? Try to use chem_opt = 201 without the mozart_ph_diag = 1 and see if it fails as before. Then test adding that namelist option back. It should only be defined for 202 but it might be causing issues with 201 as well. Thanks for working through this.

Jordan
 
These are this changes I'd like you to try:

in chem/aerosol_driver.F

change this:
! output of aerosol pH from MOSAIC 4-bin
REAL, DIMENSION( ims:ime , kms:kme , jms:jme ) , &
INTENT(OUT ) :: &
ph_aer01, ph_aer02, ph_aer03, ph_aer04


to this:
! output of aerosol pH from MOSAIC 4-bin
REAL, DIMENSION( ims:ime , kms:kme , jms:jme ) , &
OPTIONAL, &
INTENT(OUT ) :: &
ph_aer01, ph_aer02, ph_aer03, ph_aer04


in chem/cloudchem_driver.F

change this:
REAL, DIMENSION( ims:ime, kms:kme, jms:jme ), &
INTENT(OUT) :: ph_cw ! ph_cw - pH of cloud water

To this:
REAL, DIMENSION( ims:ime, kms:kme, jms:jme ), &
OPTIONAL, &
INTENT(OUT) :: ph_cw ! ph_cw - pH of cloud water


In chem/module_mosaic_cloudchem.F

Change this:

real, intent(out), &
dimension( ims:ime, kms:kme, jms:jme ) :: &
ph_cw

to this:

real, intent(out), &
OPTIONAL, &
dimension( ims:ime, kms:kme, jms:jme ) :: &
ph_cw

in chem/module_mosaic_driver.F

change this:

real, intent(out), &
dimension( ims:ime, kms:kme, jms:jme ) :: &
ph_aer01, ph_aer02, ph_aer03, ph_aer04

to this:

real, intent(out), &
OPTIONAL, &
dimension( ims:ime, kms:kme, jms:jme ) :: &
ph_aer01, ph_aer02, ph_aer03, ph_aer04


in chem/module_mosaic_therm.F

change this:

real, intent(out), dimension(kms:kme) :: &
ph_aer1, ph_aer2, ph_aer3, ph_aer4 ! pH of the aerosols

to this:

real, &
OPTIONAL, &
intent(out), dimension(kms:kme) :: &
ph_aer1, ph_aer2, ph_aer3, ph_aer4 ! pH of the aerosols


Once you make these changes, perform a "./clean -a", recompile, and test again without the new namelist option.

To explain, it seems the code for the ph_diagnostics is expecting those arrays to always exist. However, unless mozart_ph_diag = 1, those arrays are never activated and so when the model attempts to write them out, they aren't there. Adding the "OPTIONAL" arguments should fix this.
 
Test 1: chem_opt = 201 without the mozart_ph_diag = 1 ---> The same peroblem as before wrfout not created
Got error: ERROR: cloud chemistry option requires chem_opt = 8 through 13 or 31 to 36 or 41 to 43 or 109 to function.
I remoeve these from the namelists
!wetscav_onoff = 1, 1, 1, ! wet scavenging in stratocumulus clouds, 1 = on
!cldchem_onoff = 1, 1, 1, ! aqueous chemistry in stratocumulus clouds, 1 = on

Test 2: chem_opt = 201 with the mozart_ph_diag = 1 ---> works and worfout files are generated with good data.

These tests are made with history_interval = 2
I am using the last version WRFV4.4.2.
OK Jordan, I will do that and and I will get back to you with further update.
 
Last edited:
Hi Jordan!
I made the changes you asked for to the files
chem/aerosol_driver.F chem/cloudchem_driver.F chem/module_mosaic_cloudchem.F chem/module_mosaic_driver.F chem/module_mosaic_therm.F and clean and recompile WRFchem again. I got the same results as before for chem_opt = 201 and 202: To generate wrfout files we need to set mozart_ph_diag = 1

A quick search in the WRF folder for files having "mozart_ph_diag=" give these results

./inc/scalar_indices.inc: IF (model_config_rec%mozart_ph_diag==1)THEN
./inc/allocs.inc: IF ( setinitval .EQ. 3 ) grid%mozart_ph_diag=0
./inc/namelist_script.inc: if test ! -z "$NL_MOZART_PH_DIAG"; then echo "mozart_ph_diag=${NL_MOZART_PH_DIAG}," >> $FILE;fi
./Registry/registry.chem: package do_mozart_ph_diag mozart_ph_diag==1 - state : h_cw,ph_aer01,ph_aer02,ph_aer03,ph_aer04
./frame/module_alloc_space_0.f90: IF ( setinitval .EQ. 3 ) grid%mozart_ph_diag=0
./frame/module_configure.f90: IF (model_config_rec%mozart_ph_diag==1)THEN
./inc/nl_config.inc: mozart_ph_diag = model_config_rec%mozart_ph_diag
./inc/nl_config.inc: model_config_rec%mozart_ph_diag = mozart_ph_diag
./inc/namelist_defaults.inc: mozart_ph_diag = 0
./frame/module_state_description.f90: INTEGER, PARAMETER :: do_mozart_ph_diag = 1
./frame/xx0.f90: model_config_rec%mozart_ph_diag = mozart_ph_diag
./frame/yy0.f90: mozart_ph_diag = model_config_rec%mozart_ph_diag
./frame/module_configure.f90: mozart_ph_diag = 0
./frame/module_state_description.F: INTEGER, PARAMETER :: do_mozart_ph_diag = 1

May be this can help!!
Her I can see that mozart_ph_diag = 0 by default in the inc/namelist_defaults.inc file. May be we need simply to add a condition to set it to 1 in case MOZART-MOSAIC is used

​

 
Last edited:
Hi Rachid,

While that solution would fix the problem, I don't want to output those arrays by default since they are 3D arrays and thus require additional storage. Can you send me your wrfinput_d01 and wrfbdy_d01 file (or upload to google drive?) - my email is
 
Last edited:
Hi Rachid,

I believe I have a solution to the problem. I would very much appreciate it if you could test this as well. Please add these changes in addition to the changes I've already suggested (i.e., the 'optional' arguments).

In /chem/module_mosaic_cloudchem.F, add the lines in bold

iradical_onoff = 0
if ( config_flags%mozart_ph_diag .eq. 1 ) then
! Initialize pH of CW ph_cw to a FillValue value
do jt = jts, jte
do kt = kts, kte
do it = its, ite
ph_cw(it,kt,jt) = -9999.
end do
end do
end do
end if
do 3920 jt = jts, jte
....

gas_aqfrac(it,kt,jt,: ) = gas_aqfrac_box( : )
if ( config_flags%mozart_ph_diag .eq. 1 ) then
ph_cw(it,kt,jt) = ph_aq_box
end if
3800 continue

In chem/module_mosaic_driver.F

call aerchemistry( it, jt, kclm_calcbgn, kclm_calcend, &
dtchem, idiagaa_dum, vbs_nbin, &
ph_aer01(it,kms:kme,jt), ph_aer02(it,kms:kme,jt), &
ph_aer03(it,kms:kme,jt), ph_aer04(it,kms:kme,jt), &
kms,kme,config_flags%mozart_ph_diag )

In chem/module_mosaic_therm.F


subroutine aerchemistry( iclm, jclm, kclm_calcbgn, kclm_calcend, &
dtchem_sngl, idiagaa,vbs_nbin, &
ph_aer1, ph_aer2, ph_aer3, ph_aer4,kms,kme,do_ph )
...
...
real dtchem_sngl
integer kms, kme
integer, intent(in) :: do_ph

...

call specialoutaa( iclm, jclm, k, m, 'after_movesect' )

if ( do_ph .eq. 1 ) then
ph_aer1(k) = mc(1,1)
ph_aer2(k) = mc(1,2)
ph_aer3(k) = mc(1,3)
ph_aer4(k) = mc(1,4)
end if

Once you make the changes, recompile. Please let me know if you have questions regarding where the changes need to occur.

Jordan
 
Ok Jordan, now the simulation runs correctly without using mozart_ph_diag = 1 for both options (chem_opt = 201 and 202).
Thank you very much.
 
Last edited:
jordanschnell said:
@wrf_alessandro I apologize for not completing the help with you here before the forum switchover: WRF4.3.2 crashes while writing output

Note that other users seem to be having a similar problem. Were you able to fix this? If not, please stay tuned to this thread as we are attempting to figure this out.
Click to expand...
Hi Jordan,
I was didn't solve this issue and due to other problems with recent WRF versions I decided to stay on v4.2.2. Anyway, glad to know you were able to fix this bug.
Alessandro
 
HI Rachid,

I'm pleased to hear the solution works. To confirm, once you made the requested changes to the code, it works with mozart_ph_diag = 0?

For this new problem, please open up a new ticket. I don't think this is related to your previous problem (especially since you were able to run for d02).

Jordan
 
You must log in or register to reply here.
Top