ERROR: The mandatory field TT was not found in any input data.

[katierobinson]

Hi, I am trying to run metgrid for a future simulation using CCSM4 as input data, I downloaded the data from here: https://rda.ucar.edu/datasets/ds316.1/?hash=access#!description

I skipped running ungrib since it did not seem necessary for this file type but if this was a mistake please let me know. If not, this is the error I am getting when I try to run metgrid:

Processing domain 1 of 1
Processing 2040-01-01_00
CCSM4_CMIP5_MOAR_BC_RCP45
WARNING: Couldn't open file CCSM4_CMIP5_MOAR_BC_RCP45:2040-01-01_00 for input.
ERROR: The mandatory field TT was not found in any input data.

Thanks for your help!

 

[Ming Chen]

The CCSM4 data is binary format data, I think. Pease let me know if I am wrong.
You need to read the data, write it to intermediate format, then run metgrid.exe to process this data. You are right that it is no necessary to run ungrib in this case.

Please see detailed information about the intermediate format:
https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_v4/v4.0/users_guide_chap3.html#_Writing_Meteorological_Data

We also provide a sample code to convert CAM data to intermediate format data. Please go to the website
https://www2.mmm.ucar.edu/wrf/users/utilities/util.htm

At the bottom of this page, you will find (CAM2WRF.tgz). You may modify this case to make it for converting CCSM4 data.

 

[katierobinson]

Ming,

Thanks for your response. I am fairly new to WRF but I think that according to the site where I downloaded the CCSM data - it should already be in intermediate format;

https://rda.ucar.edu/datasets/ds316.1/index.html#!description says:
"The dataset contains all the variables needed for the initial and boundary conditions for simulations with the Weather Research and Forecasting model (WRF) or the Model for Prediction Across Scales (MPAS), provided in the Intermediate File Format specific to WRF and MPAS"

In this case where the input data is already in intermediate format, is there a way to know why the TT field is still missing??

 

[Ming Chen]

Based on the information in the website, I agree that those files are in intermediate format. Let's check the content in those files.

Please run the executable file (WPS/util/rd_intermediate.exe) and save the log file for me to take a look.

rd_intermediate.exe filename &> log (in bash)

(filename should be the file you download from the website)

 

[katierobinson]

Ming,

I ran rd_intermediate like you explained. The only line printed in the log file is the following:
SUCCESSFUL COMPLETION OF PROGRAM RD_INTERMEDIATE

Is there another place where details about the content of the files would be stored besides log?
Thanks.

 

[Ming Chen]

The utility is robust to read any data in intermediate format. However, it doesn't check whether all required fields for running WRF are available. The 'SUCCESS' message only tells that the data is in intermediate format. Please save the output, which includes all variables available in the file. This output file can well tell whether some required fields are to available.

 

[katierobinson]

Ming,

By output do you mean the log file? The log only has one line in it which is: SUCCESSFUL COMPLETION OF PROGRAM RD_INTERMEDIATE

Sorry I'm not exactly sure if that's what you meant by output.

 

[Ming Chen]

I attach a sample log file. Pleas take a look.

 

[katierobinson]

Okay, here is the log file that rd_intermediate produces.

And this is the command I used to run the executable:
./rd_intermediate.exe /home/krobin/scratch/DATA/CESM4.5/CCSM4_CMIP5_MOAR_BC_RCP45\:2040-01-01_00 &> rd_intermediate.log

 

[Ming Chen]

Hi, please see the attached log file, which is the output of the command:

./rd_intermediate.exe /gpfs/fs1/collections/rda/data/ds316.1/RCP4.5/2040/CCSM4_CMIP5_MOAR_BC_RCP45:2040-01-01_00 > log.txt

This log file indicates that the data is fine and it includes variables required for running WRF

I am really perplexed by the output you have. Please try to re-download the data.

 

[katierobinson]

Hi Ming,

I am perplexed as well!! I tried redownloading the RCP4.5 data and also tried rd_intermediate with a different year and with RCP8.5 data, all gave the same one line output as before... I'm stumped!

 

[Ming Chen]

Let's start from just a single file. Let's take the file CCSM4_CMIP5_MOAR_BC_RCP45:2040-01-01_00 as an example:

I download this file from CISL RDA server, and issue the following command:
ls -l CCSM4_CMIP5_MOAR_BC_RCP45:2040-01-01_00

The file size shown is:
-rw-r--r-- 1 rdadata decs 33433816 Nov 21 2014 /gpfs/fs1/collections/rda/data/ds316.1/RCP4.5/2040/CCSM4_CMIP5_MOAR_BC_RCP45:2040-01-01_00

Can you look at the file you have, and check the file size?

 

[katierobinson]

Hi, here is the file size that prints when I use the command ls -l

I tried with a file from RCP4.5 and 8.5 but got the same results:

-rw-rw-r-- 1 krobin krobin 37253 Feb 25 14:42 CCSM4_CMIP5_MOAR_BC_RCP45:2040-01-01_00

-rw-rw-r-- 1 krobin krobin 37253 Feb 23 16:47 CCSM4_CMIP5_MOAR_BC_RCP85:2040-01-01_00

 

[katierobinson]

Hi again,

I think I see where the problem is happening, I downloaded the same file locally to my laptop instead of to the remote cluster which I am using to run WRF, on my laptop the file size is much different than the version that is downloaded on the remote cluster:

-rw-r--r--@ 1 katie staff 33433816 26 Feb 13:44 CCSM4_CMIP5_MOAR_BC_RCP45-2040-01-01_00.dms locally, compared to:

-rw-rw-r-- 1 krobin krobin 37253 Feb 25 14:42 CCSM4_CMIP5_MOAR_BC_RCP45:2040-01-01_00 in the remote cluster

So, perhaps the problem is coming from my shell script which I use to download the files from the internet, I've attached the script here since I am am unsure where the problem is coming from in this script! Thanks for your help

 

[katierobinson]

Hi again,

The problem has been resolved!!! The problem was occurring because wget calls to download data from RDA require an auth cookie, which means you need to authenticate with your RDA username and password before calling wget.

I downloaded a csh script from the RDA website that allowed me to input my password and then the CMIP files downloaded fully to the remote cluster I am using. After this metgrid ran successfully with no missing fields.

Just wanted to follow up in case anyone else has the same problem!

Thanks.

 

[pengyuan]

I attach a sample log file. Pleas take a look.

Hi Ming,
I got the same error info.
I attached the log file after running rd_intermidate.exe.
Please give me some suggestions how can i solve this problem.
I use RCP60 and RCP85 data on January 1 2022.

 

[pengyuan]

I solved the problem. There is a typo in the namelist.wps.

 

[Ming Chen]

@pengyuan @katierobinson
Thanks for the update. I am glad the problems are solved.