So recently I posted a topic regarding generating intermediate files using the GEFS reforecast set via two python packages (PyGrib / PyWinter), and had to solve an interpolation error in the set, however, I have now stumbled onto a new issue with this set. For some unknown reason metgrid is taking a very long time to process the soil parameters (ST/SM) using these generated intermediate files (Even on HPC environments it can take minutes to process a single field using double digit processor counts). As a comparative test, I ran through the same domain using CFSv2 processed via two ungrib runs and these fields are processed almost immediately (WRF/WPS V4.2).
The only thing I'm noticing in the metgrid output is that unlike the CFS processing each field is immediately processed fully (IE: TT @ 1000, TT @ 975, TT @ 950, so on) instead of what I normally see in the processing where each level is processed (IE: TT @ 1000, UU @ 1000, VV @ 1000, so on). Running rd_intermediate does not appear to show anything out of the ordinary for the intermediate files.
The only strange thing I have noticed in the files deals with some kind of value translation aspect (printing the array in python shows incorrect values, and a lot of values close to the extremes (IE: 0.9999997, 0.000001), but not quite touching the fractional limits (1.0, 0.0) as I have seen from other sets. I'm also noticing some strangeness with masked array behavior from some of the files when loading the GRIB data in PyGrib, but I'm not sure what is happening here to be exact.
I have attached a sample of the intermediate file (Note the file extension) generated by the GEFS reforecast set along with the metgrid logs for both GEFS View attachment 1 and CFSv2View attachment 2 test I did, and the rd_intermediate outputView attachment 3 from the sample GEFS file. Any assistance to pinning what's going on here, or some suggestions on things to try to fix the issue would be greatly appreciated. I can provide the python script I used to generate these intermediate files if you feel that would help to identify potential issues.
Thanks!
The only thing I'm noticing in the metgrid output is that unlike the CFS processing each field is immediately processed fully (IE: TT @ 1000, TT @ 975, TT @ 950, so on) instead of what I normally see in the processing where each level is processed (IE: TT @ 1000, UU @ 1000, VV @ 1000, so on). Running rd_intermediate does not appear to show anything out of the ordinary for the intermediate files.
The only strange thing I have noticed in the files deals with some kind of value translation aspect (printing the array in python shows incorrect values, and a lot of values close to the extremes (IE: 0.9999997, 0.000001), but not quite touching the fractional limits (1.0, 0.0) as I have seen from other sets. I'm also noticing some strangeness with masked array behavior from some of the files when loading the GRIB data in PyGrib, but I'm not sure what is happening here to be exact.
I have attached a sample of the intermediate file (Note the file extension) generated by the GEFS reforecast set along with the metgrid logs for both GEFS View attachment 1 and CFSv2View attachment 2 test I did, and the rd_intermediate outputView attachment 3 from the sample GEFS file. Any assistance to pinning what's going on here, or some suggestions on things to try to fix the issue would be greatly appreciated. I can provide the python script I used to generate these intermediate files if you feel that would help to identify potential issues.
Thanks!