Error in running WRF-Chem (for version 3.9.1.) while using chem_opt=202 though WRF-Chem is running successfully for chem_opt=112 and 201

[Ankan]

Hello everyone ,
I am giving a test run with two domains (d01:20 km and d02:4 km) with one-way nesting by setting 'feedback' option to '0' for just 2 days. The WRF-Chem ran successfully when I used the MOZCART option (chem_opt=112) and MOSART-MOSAIC (chem_opt=201). But, some specific errors only occur for MOZART-MOSAIC, including the aqueous phase chemistry option (chem_opt=202). I am discussing my errors (in brief) below.
I checked the 'wrfrun.log' where the following lines are printed:
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 107575 RUNNING AT durga
= EXIT CODE: 11
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
Intel(R) MPI Library troubleshooting guide:
Documentation Library
===================================================================================
In addition, some warning messages are coming in rsl files.
For example, the following warning messages are printed several times in the 'rsl.error.0000' file.
WARNING: Large total lw optical depth of ******** at point i,j,nb= 1 1 1
Diagnostics 1: k, tauaerlw1, tauaerlw16
1 0.00 0.00
2 0.00 0.00
3****************
4****************
5****************
6****************
7****************
8 0.00 0.00
9 0.00 0.00
10 0.00 0.00
11 0.00 0.00
12****************
13****************
14****************
15****************
16****************
17****************
18****************
19****************
2029210.94********
21 121.24 788.67
22 13.10 88.81
23 0.00 0.00
24 0.00 0.00
25 0.00 0.00
26 0.00 0.00
27 0.00 0.00
28 0.00 0.00
29 0.00 0.00
30 0.00 0.00
31 0.00 0.00
32 0.00 0.00
33 0.00 0.00
34 0.00 0.00
-------------------------
-------------------------

And the following warnings are printed 'rsl.out.0000' file:
entering mosaic_cloudchem_driver - ktau = 2
leaving mosaic_cloudchem_driver - ktau = 2 0
ASTEM internal steps exceeded 200
ASTEM internal steps exceeded 200
ASTEM internal steps exceeded 200
ASTEM internal steps exceeded 200
ASTEM internal steps exceeded 200
ASTEM internal steps exceeded 200

Also, a 'fort.67' file generated containing the same line " ASTEM internal steps exceeded 200".
I checked a previous post (Possible Bug with MOZART-MOSAIC scheme while writing output and Error running WRF-Chem), there it was suggested to set
&chem
mozart_ph_diag=1
However, in that case, the WRF-Chem version was from V4.3 to V4.4.2, while my version is 3.9.1. I have tried to include this option in the "&chem" section, but even real.exe didn't run (after linking the required emission input files in the WRFChem/run directory and before running mozbc). I checked Registry/registry.chem, but I didn't find any options there. So, I really doubt that this would work for the earlier versions.
I also gave a test run by setting 'cldchem_onoff' and 'conv_tr_aqchem' to '0' for both domains, and the simulation lasted a little bit longer (1 hour and 20 minutes) than before (20 minutes). But the same errors came again. I again gave a test run by setting 'chemdt' to '0', 'bioemdt' and 'photdt' to '2' and '0.4' (2*chemdt), 'wetscav_onoff', 'cldchem_onoff', 'conv_tr_aqchem' and 'conv_tr_wetscav' to '0' for both domains. I will give an update on the run if it runs successfully or not. I suspect that there could be a possible bug in 'module_mosaic_cloudchem.F' or related driver files ('.F' files) for my WRF-Chem version. Therefore, I am attaching my 'namelist.input', 'wrfrun.log', 'module_mosaic_cloudchem.F', 'module_mosaic_driver.F' and 'module_mosaic_therm.F' files for my version. The rsl files are too large (before generating the rsl files, the /home/ directory was at 39% and after generating the rsl files, it reached 70%). So, I am attaching some portions of them where it was showing some warnings.
Can anyone please take a look at these and let me know the solution for my case? I have been really struggling with this 'chem_opt=202' option for the last few months, and I haven't found any solutions yet after trying so many test runs. So, I will be very grateful if anyone kindly help me to solve this issue.If you need anything else, please let me know. Thank you for your time.
With regards,
Ankan

 

[jordanschnell]

Hi Ankan,

the ph_diag option was not introduced until v4.3 so it is not an issue with v3.9 and you will get an error trying to use it. It is still not clear form your errors what is causing your problem and the limited rsl information you have provided does not indicate anything. My suggestion is to turn off as many options as possible to see if the basic gas and aerosol chemistry are working.

i.e.,

&phys
progn = 0,
phot_opt = 0,
bio_emiss_opt = 0,
cldchem_onoff = 0,
chem_conv_tr = 0,
conv_tr_wetscav = 0,
conv_tr_aqchem = 0,
seas_opt = 0,
dust_opt =0,
dmsems_opt = 0,
biomass_burn_opt = 0,
have_bcs_chem = .false.,
have_bcs_chem_upper = .false.
aer_ra_feefback = 0,

Then you can try to turn back on options one at a time until the error presents itself.


Jordan

 

[Ankan]

Hello @jordanschnell,
Thank you for your response.Okay I will give the trial runs as you suggested and let you know the results. Actually, rsl files are so large that I couldn't upload here. If you want to see the rsl files, please let me know, then I will try to upload as a link of Google drive maybe.
Regards,
Ankan

 

[Ankan]

Hello @jordanschnell,
I gave a test run by setting 'chemdt' to '0', 'bioemdt' and 'photdt' to '2' and '0.4' (2*chemdt), 'wetscav_onoff', 'cldchem_onoff', 'conv_tr_aqchem' and 'conv_tr_wetscav' to '0' for both domains (as I mentioned earlier). But the simulation stopped again, and the same errors occurred as before. However, the simulation lasted a little bit longer than before (1 hour, 40 minutes, and 12 seconds). So, that means maybe there is nothing wrong with cloud chemistry and wet scavenging drivers. There is something else wrong. Right? What do you think? I will do more test runs to find out the real reason behind the problem and let you know the results of the trial experiments. For now, I am discussing the errors below since rsl files are so large (around 1.5 GB each):
In the rsl.error.0000 file, the following lines are printed after 20 minutes of simulation time:
Timing for main: time 2015-12-22_00:20:00 on domain 2: 300.25681 elapsed seconds
Timing for main: time 2015-12-22_00:20:00 on domain 1: 1511.25403 elapsed seconds
-------------------------
WARNING: Large total lw optical depth of ******** at point i,j,nb= 1 1 1
Diagnostics 1: k, tauaerlw1, tauaerlw16
1 1.61 10.52
2 12.96 84.57
3 54.11 353.02
4 231.57 1510.85
5 1234.29 8052.79
620493.77********
7****************
8****************
9****************
10****************
11****************
12****************
13****************
14****************
15****************
16****************
17 487.56 3181.13
18 0.00 0.00
19 0.00 0.00
20 0.00 0.00
21 0.00 0.00
22 0.00 0.00
23 0.00 0.00
24 0.00 0.00
25 0.00 0.00
26 0.00 0.00
27 0.00 0.00
28 0.00 0.00
29 0.00 0.00
30 0.00 0.00
31 1.69 11.05
32****************
33 6481.4043166.81
34 6.84 44.81
-------------------------
-------------------------
-------------------------
-------------------------
Again, almost before stopping the simulation, the following warnings are printed:
WARNING: Large total sw optical depth of ******** at point i,j,nb= 80 48 11
Diagnostics 1: k, tauaer300, tauaer400, tauaer600, tauaer999, tauaer
1****************************************
2****************************************
3****************************************
4****************************************
5****************************************
6****************************************
7****************************************
8****************************************
9****************************************
10****************************************
11****************************************
12****************************************
13****************************************
14****************************************
15****************************************
16****************************************
17****************************************
18****************************************
19****************************************
20****************************************
21****************************************
22****************************************
23****************************************
24****************************************
25****************************************
26****************************************
27****************************************
28****************************************
29****************************************
30****************************************
31****************************************
32****************************************
33****************************************
34****************************************
That means the errors are related to large lw and sw radiative fluxes at some points. Right? What does it mean, actually? Does it mean that there is an issue with the optical driver? Or, is this related to large aerosol concentrations? Can you please help me understand this? That will be really helpful for me to find out the reason. Thank you for your time.
Regards,
Ankan

 

[Ankan]

Hello @jordanschnell ,
In the last few days, I ran the following sensitivity simulations over a single domain with a resolution of 20 km for just one day to reduce the computational costs.

First, after going through one previous thread (Plumerise bug), I did a sensitivity simulation by switching off the biomass burning options (i.e., biomass_burn_opt = 0, scale_fire_emiss = .false., plumerisefire_frq = 0).
And the simulation stopped after 40 minutes, with the same error occurring as before. That concludes that there must not be anything wrong with biomass-burning emissions in my case.
Then, I kept all options on and switched off only aerosol chemistry (i.e., 'aerchem_onoff = 0').
And, finally, it gave me a successful run! No error, no warnings, no fort.67 file generation and the size of the rsl files are also in KB only.
I checked the output file using ncview to see the output variables, and they all look realistic to me.
So, what should I do now? Can turning off the aerosol chemistry cause significant changes in the output variables? For your information, I want to study aerosol-cloud interaction over the Indian region. Is it okay to switch off this option for my case? Or is there anything I have to do now?
I am attaching my 'namelist.input' file. Can you please check this and let me know if the setup is correct? Also, please guide me if there is something I have to do for my study. It will be very helpful for me. If you need anything else regarding this, please let me know. I am really looking forward to hearing from you. Thank you for your time and consideration.
With regards,
Ankan

 

[jordanschnell]

Hi Ankan,

This indicates the problem is with MOSAIC and not the cloud chemistry, resolved (wet), or convective scavenging. To be clear, the model will not run if you set aerchem_onoff = 1, and

progn = 0,
wetscav_onoff = 0
conv_tr_wetscav = 0
conv_tr_aqchem = 0

?

You will not be able to investigate aerosol interactions if the aerosol chemistry is turned off.

Jordan

 

[Ankan]

Hello @jordanschnell,
Thank you so much for your response. I also gave a test run with the same setup but for two domains, keeping 'aerchem_onoff = 0' and it is still running without giving any warnings. That means, yes, the problem should be with MOSAIC, as you said. However, earlier WRF-Chem ran successfully while using chem_opt=201, which is also MOSAIC but, without any aqueous chemistry. Does that mean something else is wrong in the driver files that deal with these processes? What should I do now? I want to study aerosol-cloud-climate interaction, and if I will not be able to investigate this without using aerosol chemistry, then it is a very big problem for me. Can you please guide me in this regard? I suspect that there may be any bug present in the driver files in the WRFChem/chem/ directory. Do I need to modify any mosaic driver files that have to do with the aerosol chemistry and then recompile the model? Can you please help me to solve this issue? If you need any driver files for my version, I will share them with you. Please let me know. I am eagerly waiting for your reply. Thank you for your time.
With regards,
Ankan

 

[jordanschnell]

Ankan,

this is an old version of the model that many people have sucessfully used so it is not a problem with the code. Will the model run if you set:

aerchem_onoff = 1,
progn = 0,
wetscav_onoff = 0
conv_tr_wetscav = 0
conv_tr_aqchem = 0

???

I also asked you test turning off these options, one at a time, leaving the others on:

phot_opt = 0,
bio_emiss_opt = 0,
cldchem_onoff = 0,
chem_conv_tr = 0,
conv_tr_wetscav = 0,
conv_tr_aqchem = 0,
seas_opt = 0,
dust_opt =0,
dmsems_opt = 0,
biomass_burn_opt = 0,
have_bcs_chem = .false.,
have_bcs_chem_upper = .false.
aer_ra_feefback = 0,

I'm trying to help, but I need you to understand that you need to. narrow down the process that is causing the issue. By completely turning off aerchem, yes it is likely MOSAIC, but now I am not clear if you tested turning off the other options first. If you turn off aerchem, then many of the toher options will also be turned off, so it's not clear what the issue is. You need to turn them off one a time.

Jordan

 

[Ankan]

Hello @jordanschnell,
Thank you so much for your guidance. No, I haven't done that. I will do this experiment and I will do the test runs as you suggested by turning off these options one at a time while keeping others on and give you updates of all the simulations. I hope it will help to narrow down the issue. Thank you for helping me.
With regards,
Ankan

 

[Ankan]

Hello @jordanschnell,
I ran a series of trial WRF-Chem simulations using a single domain (d01:20 km) for only one day after considering your suggestions. The results of these experiments are as follows:

Experiments	Results
EX 1. progn = 0, aerchem_onoff = 1, wetscav_onoff = 0 conv_tr_wetscav = 0 conv_tr_aqchem = 0	Failed! The simulation stopped within a few seconds. No 'fort.67' file was generated. No output file was created.
EX 2. progn =1, The rest of the options are kept the same as EX 1.	Failed! The simulation stopped after 40 minutes of simulation period. Some warnings are printed after 40 minutes in 'rsl.error.0000' file, such as "WARNING: Large total lw optical depth of ******** at point i,j,nb= ". In the 'rsl.out.0000' file, starting at 2 minutes, several times "ASTEM internal steps exceeded 200" lines appear in between the simulations. A 'fort.67' file was generated containing the line "ASTEM internal steps exceeded 200" multiple times. In the 'wrfrun.log' file "BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES" was printed at the end.

Since the model runs only after setting 'progn=1' for 'chem_opt=202', the following experiments are done by setting 'progn=1' and turning off the suggested options one at a time, leaving the others on.

EX 3. have_bcs_upper = .false.,	Failed! Same results as EX 2, except warnings of Large total sw optical depth diagnostics (not lw) are printed in 'rsl.error.0000' file.
EX 4. have_bcs_chem = .false.,	Failed! Same results as EX 2, except the simulation lasted a little bit longer (1 hour simulation time), and warnings of both large total sw and lw optical depth diagnostics are printed in 'rsl.error.0000' file.
EX 5-11. dust_opt, seas_opt , dmsemis_opt, biomass_burn_opt, bio_emiss_opt, phot_opt, chem_diag = 0 (one at a time, keeping others on)	Failed! Same results as EX 2.
EX 12. cldchem_onoff = 0 and conv_tr_aqchem = 0	Failed! Simulation lasted a little bit lonnger (3 hour 10 minutes of simulation). No "BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES" line was printed at the end of the 'wrfrun.log' file. The size of each rsl files are so large (around 580-750 MB each). The warnings of both large total sw and lw optical depth diagnostics are printed in 'rsl.error.0000' file. In the 'rsl.out.0000' file, starting at 2 minutes, several times "ASTEM internal steps exceeded 200" lines appear in between the simulations, and a fort.67 file was generated containing the same line multiple times as before. Output file was generated. No value for some species, particularly aerosol (e.g. bc_a0*,so4_a0*, sulf, na_a0*, cl_a0*, nh4_a0*, water_a04* etc.), but value for PM10, PM2.5_Dry, num_a0*, no2, n03, ch30h etc.
EX 13. aer_ra_feedback = 0	Partly success! Ran successfully without printing any warnings of large lw or sw optical depth in the rsl.error.0000 file. However, a 'fort.67' file generated as before. But there is no value for emission variables or aerosol species in the output file! only output value for dust!
EX 14. aerchem_onoff = 0	Success! No error, no warnings, no fort.67 file generation and the size of the rsl files are also in KB only. After checking the output file using ncview, the output variables look realistic to me.

So, only 'aerchem_onoff = 0' gives me a successful run. That makes me suspect that something is going wrong while using only aerosol chemistry driver files. Right? So, what do you think? And what should I do now if I want to study aerosol-cloud interaction? Can you please guide me? That will be very helpful for me. I am eagerly waiting for your reply. Thank you for your time and consideration.
For your convenience, some rsl.errror.000 files are attached herewith. Please find the same.
With regards,
Ankan

 

[jordanschnell]

Hi Ankan,

This is extremely helpful - thank you for attempting these tests. I have one last request - please turn off the anthropogenic emissions (i.e., set emiss_opt = 0). There should not be a problem with the MOSAIC code in that version of the model, so something else is causing it to fail. I thought maybe it was one of the online emissions but maybe it is the anthropogenic. I am also still wondering if the problem is related to your system infrastructure/# cores/memory available.

Jordan

 

[Ankan]

Hello @jordanschnell,
Thank you for your response. I performed the experiment you suggested by setting emiss_opt = 0, and the same errors occurred as in EX 4 (simulation stopped after 1.5 hours of simulation time). So, does that mean something is wrong with the MOSAIC code only? I am also wondering that there may be nothing wrong with MOSAIC code, as I completed a successful WRF-Chem run when I used MOZART-MOSAIC without aqueous chemistry, i.e., chem_opt=201. That is why I wanted to verify with another widely used MOSAIC option available, CBMZ-MOSAIC, just to be sure that there is no problem with MOSAIC code for this old version. But I don't know how to set 'emis_map' variable in the input file for running the anthro_emis utility using CBMZ-MOSAIC. Therefore, I asked in another thread here in this forum (Query on the input file for CBMZ-MOSAIC chemistry option for running anthro_emis to generate anthropogenic emission input files (wrfchemi* files)), but I am still waiting for a reply. Can you help me with that too? That will be very helpful for solving this issue.
And, I don't think that this error would be because of memory issues because all these tests ran with only a single domain, and during the simulation, it only took around 25 GB out of the total 378 GB of memory. I am not sure though, since I am very new to this model. And, for your information, I am running WRF-Chem using 28 out of 32 processors on a Linux server. And I noticed after doing several test simulations for the last two to three months that the simulations took a lot of time when I used nested domains, but for a single domain, the simulation time is quite normal. So, what do you think? Could it be because of infrastructure issues?

And, it has been really nice talking to you, as your suggestions helped me a lot to understand this model. So, I am really looking forward to talking to you. Thank you for your time.
With regards,
Ankan

 

[Ankan]

Hello @jordanschnell,
Since among all of the test runs (sensitivity experiments) mentioned in the previous messages (including the experiment by setting emiss_opt = 0), WRF-Chem ran successfully only by setting 'aerchem_onoff = 0', I highly suspect that there is something wrong with the mosaic aerosol chemistry driver files. I also learned from my senior at another institute that WRF-Chem version 3.9.1 ran successfully with 'chem_opt=202' in her case only after some modifications were made to some files in the chem/ directory. But she forgot what had changed that time. Therefore, I am clueless and hence need your help. After going through many documents, I found some lines in a document entitled "WRF-Chem 3.6.1:MOZART gas-phase chemistry with MOSAIC aerosols" (attached here) that may give any hints about this. The lines are as follows:

"2) Glyoxal uptake into aqueous aerosols to form SOA (Knote et al., 2014a)
Currently, glyoxal SOA is turned on by default for mozart_mosaic_4bin (chem_opt = 201) or
mozart_mosaic_4bin_aq (chem_opt = 202).
- with mozart_mosaic_4bin, a simple surface uptake is used (“simple” in Knote et al. 2014a).
- with mozart_mosaic_4bin_aq the complex version is used ("hybrid" in Knote et al. 2014a).
A new module was added called module_mosaic_gly.F which contains the glyoxal SOA
formation and partitioning code.
- The glyoxal routine is called during ASTEM in module_mosaic_therm.F
- Available parameterizations are defined in module_data_mosaic_therm.F.
- The choice of glyoxal parametrization called for various chem_opts can be found in the
mosaic_aerchem_driver routine in module_mosaic_driver.F If one would like to modify this
choice set glysoa_param to a different option in module_mosaic_driver.F, and add additional
tracers to the registry.chem."

Here, it is clearly mentioned that the glyoxal routine is called during ASTEM in module_mosaic_therm.F. And if you remember, I was getting some warnings such as "ASTEM internal steps exceeded 200" .
I also found some lines on another webpage (Frequently Asked Questions), where the following lines are mentioned as these bugs are fixed for WRF-Chem version 4 or later (screenshot is also attached):

• glysoa not needed as no longer uses to_toa variable which has the summation error (module_mosaic_driver.F).

So, maybe something is wrong in the module_mosaic_therm.F. file, which is related to glysoa. Though I am unsure, I am sharing this information with you in the hopes that it will assist in resolving this issue. I am attaching all the documents I mentioned and the driver files that are mentioned. I also attached two files that contain some lines on ASTEM in 'module_mosaic_therm.F' and 'module_data_mosaic_therm.F' files.
Therefore, I kindly request that, if you find some time, could you please look at this and help me with it? That would be greatly appreciated. I am eagerly waiting for your reply. Thank you for your time and consideration.
With regards,
Ankan

 

[k.kedia]

Hi Ankan,

I have received similar errors as described by you in the above posts and I reached the same conclusion that the problems are related to the ASTEM module in MOSAIC and one the major differences in chem_opt 201 and 202 is the inclusion of gloxal chemistry.
Have you arrived at any solution to address the ASTEM exceeded 200 error message and abrupt termination of model simulation with chem_opt = 202?

Thanks
Krishna

 

[Ankan]

Hello @k.kedia,
No, I haven't been able to solve this problem yet since I didn't get any reply after that (as you can see above). I strongly suspect that there is some bug present in the case of chem_opt=202 that may be related to glyoxal chemistry. Are you performing simulations with WRF-Chem version 3.9.1.1, just like me? Do you have similar kinds of system configurations as I do? And can you please post here if you find any solution to resolve this issue? That will be really helpful for all of us who have been struggling with this issue. Thank you for your time.
With regards,
Ankan

 

[imy]

Hi Ankan,

I encountered the same problem as you did while running wrf using versions 4.2.2 and 4.5. Have you been able to find a solution to this issue? Additionally, I observed that the time at which my simulation terminated was related to the photdt setting. Specifically, when I set it to 30, the simulation stopped at 00:32:30. When I set it to 60, the simulation stopped at 01:02:30.

Thanks

 

[Ankan]

Hi @imy,
Did you try after adding
&chem
mozart_ph_diag=1
in the namelist for your version of WRF-Chem? Please run the model after adding this and let me know if it works or not for your version of WRF-chem.
With regards,
Ankan

 

[peng]

Hello, have you solved the problem of "Large total sw optical depth of"? Although I used a different chem_opt, the same error occurred. My chem_opt=9 still enabled MOSAIC code, and the same problem occurred. I am very distressed about this problem, and I would appreciate it if you could enlighten me

 

[Ankan]

HI @peng,
I couldn't solve the problem while using chem_opt=202. So, I switched to other chemistry options. I think you can ask the question separately in this forum, as per your case. Maybe someone with expertise can help you.
With regards,
Ankan