Scheduled Downtime
On Friday 21 April 2023 @ 5pm MT, this website will be down for maintenance and expected to return online the morning of 24 April 2023 at the latest

"Bad return value from PIO" in 3 km simulation

This post was from a previous version of the WRF&MPAS-A Support Forum. New replies have been disabled and if you have follow up questions related to this post, then please start a new thread from the forum home page.

EMMANUEL

Member
I am having an issue in running ./atmosphere . Can you kindly suggest me on this

The error is indicates in log.out file
--- specifics on cloud microphysics option microp_scheme = mp_thompson
--- dt_microp = 18.0000000000000
--- n_microp = 1
--- has_reqc = 0
--- has_reqi = 0
--- has_reqs = 0
skipping over lutype = OLD
skipping over lutype = USGS
landuse type = MODIFIED_IGBP_MODIS_NOAH found 33 categories 2 seasons

--- initialize NOAH LSM tables
skipping over lutype = USGS
landuse type = MODIFIED_IGBP_MODIS_NOAH found 20 categories
end read VEGPARM.TBL
input soil texture classification = STAS
soil texture classification = STAS found 19 categories
end read SOILPARM.TBL
end read GENPARM.TBL
--- end initialize NOAH LSM tables

min/max of meshScalingDel2 = 1.00000000000000 1.00000000000000
min/max of meshScalingDel4 = 1.00000000000000 1.00000000000000
ERROR: MPAS IO Error: Bad return value from PIO

*************************************In log error

Beginning MPAS-atmosphere Error Log File for task 0 of 8208
Opened at 2021/08/25 16:28:06
----------------------------------------------------------------------

ERROR: MPAS IO Error: Bad return value from PIO


**************************************
streams file

<streams>
<immutable_stream name="input"
type="input"
filename_template="x1.65536002.init.nc"
input_interval="initial_only" />

<immutable_stream name="restart"
type="input;output"
io_type="pnetcdf,cdf5"
filename_template="restart.$Y-$M-$D_$h.$m.$s.nc"
input_interval="initial_only"
clobber_mode="overwrite"
output_interval="none" />

<stream name="output"
type="output"
io_type="pnetcdf,cdf5"
filename_template="history.$Y-$M-$D_$h.$m.$s.nc"
clobber_mode="overwrite"
output_interval="1:00:00" >

<file name="stream_list.atmosphere.output"/>
</stream>

<stream name="diagnostics"
type="output"
io_type="pnetcdf,cdf5"
filename_template="diag.$Y-$M-$D_$h.$m.$s.nc"
clobber_mode="overwrite"
output_interval="1:00:00" >

<file name="stream_list.atmosphere.diagnostics"/>
</stream>

<stream name="surface"
type="input"
filename_template="x1.65536002.sfc_update.nc"
filename_interval="none"
input_interval="86400" >

<file name="stream_list.atmosphere.surface"/>
</stream>

<immutable_stream name="iau"
type="input"
filename_template="x1.65536002.AmB.$Y-$M-$D_$h.$m.$s.nc"
filename_interval="none"
packages="iau"
input_interval="initial_only" />

<immutable_stream name="lbc_in"
type="input"
filename_template="lbc.$Y-$M-$D_$h.$m.$s.nc"
filename_interval="input_interval"
packages="limited_area"
input_interval="none" />

</streams>
 
The point at which the "Bad return value from PIO" message is being written is approximately the point at which the initial output files (history and diagnostics) should be written by the model. There's nothing in your streams.atmosphere file that looks incorrect to me. Could you post the &io namelist settings from your namelist.atmosphere file? Are there any partial history or diagnostics NetCDF files that have been written to your run directory?

(Note: I also modified the title of this thread to be more descriptive of the specific problem that's being discussed.)
 
Thank you for the update of the title

These are the settings I have set in namelist.atmosphere
/
&io
config_pio_num_iotasks = 0
config_pio_stride = 1
/

I have removed before the new run all the history and diagnostic files of previous run
 
/
&io
config_pio_num_iotasks = 0
config_pio_stride = 1
/
&decomposition
config_block_decomp_file_prefix = 'x1.65536002.graph.info.part.'
/
&restart
config_do_restart = false
/
&printout
config_print_global_minmax_vel = true
config_print_detailed_minmax_vel = false
/
&IAU
config_IAU_option = 'off'
config_IAU_window_length_s = 21600.
/
&physics
config_sst_update = true
config_sstdiurn_update = false
config_deepsoiltemp_update = false
config_radtlw_interval = '00:30:00'
config_radtsw_interval = '00:30:00'
config_bucket_update = '24:00:00' ******I have edited this line too
config_physics_suite = 'convection_permitting'
/
&soundings
config_sounding_interval = 'none'
/
 
Kindly suggest me, I would like to know if it is correct
in the case of ncdump -v xtime init. file or static files for a 5days run shows only initial time steps that is

data:
xtime =
"2014-01-01_00:00:00"; is it fine or do I need to make changes in the output_interval from "initial_only" to other
 
Sorry, I should have clarified: I was asking about partial output files because I was interested to see whether, e.g., the model had successfully written the first history file and was failing while writing the first diag file, or whether the first history file was zero bytes or contained definitions of the variables in the file (but without the actual data for those variables).

I don't have any good ideas at the moment as to why MPAS was able to write a (probably large) initial conditions file with the init_atmosphere_model program, but is now apparently failing to write the initial model output file. As a test -- just to see whether the model output is the only source of problems -- could you try a simulation that runs for just one timestep (i.e., set config_run_duration = '18' in your namelist to run for 18 seconds, assuming you're using config_dt = 18), but with output_interval = "none" for all of your output streams in the streams.atmosphere file?
 
Thank you for your response
Yes it has produced a history file for first-time step and variables also. It's size is not zero.

I would like to inform that

I have provided input data for 3days with 6hour intervals for surface and pressure level
whereas in case of sst_update.nc I have used 1day(86400) interval in update (do i need to produce for 6 hour interval too?)

Is it due to this the model could not go to next time step?

In the namelist
config_dt = 18.0
config_start_time = '2014-01-01_00:00:00'
config_run_duration = '3_00:00:00'




mgduda said:
Sorry, I should have clarified: I was asking about partial output files because I was interested to see whether, e.g., the model had successfully written the first history file and was failing while writing the first diag file, or whether the first history file was zero bytes or contained definitions of the variables in the file (but without the actual data for those variables).
 

Attachments

  • l.txt
    18.8 KB · Views: 24
I have run with 18 seconds run duration it ran successfully without an error with setting all the output intervals =none
 
I have tried running for duration of 2 days again in this case hist and diag file are written for time steps first

But the error comes to be
Beginning MPAS-atmosphere Error Log File for task 0 of 8208
Opened at 2021/08/27 16:57:12
----------------------------------------------------------------------

ERROR: MPAS IO Error: Bad return value from PIO
ERROR: ********************************************************************************
ERROR: Error writing one or more output streams
CRITICAL ERROR: ********************************************************************************
Logging complete. Closing file at 2021/08/27 17:26:22
********However I have removed all the previous nc file that was wriiten
 
If I understand correctly, the model was now able to write the first history and diag files, then to take time steps, but failed when writing later history or diag files? Is it possible that you're running out of filesystem space or hitting a user quota on the filesystem?

Which versions of libraries have you installed (HDF5, PnetCDF, NetCDF-C, NetCDF-Fortran, PIO)? Given the apparently intermittent output problems, perhaps there's some issue with one or more of the I/O libraries?

When running the model at larger scales we have found improvements in I/O performance by using fewer I/O tasks, often using just one I/O task per node. Could you also try namelist settings like the following:
Code:
&io
config_pio_num_iotasks = 513
config_pio_stride = 16
/
The key point is that the product of config_pio_num_iotasks and config_pio_stride should equal the total number of MPI tasks in your simulation (in this case, 513 * 16 = 8208). More generally, you can try setting config_pio_num_iotasks to the number of nodes and config_pio_stride to the number of processors per node that your simulation uses.
 
hdf5/1.10.6.0/
netcdf/4.6.3.0/
parallel-netcdf/1.11.1.1/

I tried with the Pio task too as you have suggested it indicates the same error.
 
The versions of libraries that you're using seem fine to me.

Is the model able to get past the initial output to the point of taking timesteps if you turn off the "output" stream (by setting output_interval="none"), but keep the "diagnostics" stream active? If the model runs with just the "diagnostics" stream, you could try enabling the "output" stream, but writing just a few fields to the stream by editing the stream_list.atmosphere.output file so that it contains, e.g.,
Code:
scalars
initial_time
xtime
w
pressure
surface_pressure
rho
theta
uReconstructZonal
uReconstructMeridional

Also, have you verified that the model is able to run and write output for coarser meshes? For example, does your intended model configuration run if you use the 10-km quasi-uniform mesh rather than the 3-km quasi-uniform mesh? Here, you'd only need to take a few timesteps to verify that the 10-km simulation can start up, write initial output, and integrate.
 
Thank you Dr. Mgduda for your suggestion and support. I could finally work out.
with regards

mgduda said:
The versions of libraries that you're using seem fine to me.

Is the model able to get past the initial output to the point of taking timesteps if you turn off the "output" stream (by setting output_interval="none"), but keep the "diagnostics" stream active? If the model runs with just the "diagnostics" stream, you could try enabling the "output" stream, but writing just a few fields to the stream by editing the stream_list.atmosphere.output file so that it contains, e.g.,
Code:
scalars
initial_time
xtime
w
pressure
surface_pressure
rho
theta
uReconstructZonal
uReconstructMeridional

Also, have you verified that the model is able to run and write output for coarser meshes? For example, does your intended model configuration run if you use the 10-km quasi-uniform mesh rather than the 3-km quasi-uniform mesh? Here, you'd only need to take a few timesteps to verify that the 10-km simulation can start up, write initial output, and integrate.
 
Top