Why GHI from NWP cloud cover instead of DSWRF?

[jarataraj]

Hello!

Why do the NWP forecast functions for the GFS (such as solarforecastarbiter.reference_forecasts.models.gfs_quarter_deg_to_hourly_mean) calculate GHI from the NWP model cloud cover, rather than using the GFS output DSWRF (Downward Short-Wave Radiation Flux [W/m^2] ). Are DSWRF and GHI not the same thing?

Also, the docstring for _resample_using_cloud_cover says that the GFS cloud cover "is an interval average with ending label." I have searched thoroughly but cannot find this information anywhere else. Does anyone have any idea where to find more detailed documentation on the GFS outputs? I'd really like to find similar information for other outputs, such as whether DSWRF is an average of the previous or following hour.

Thank you for your help!

[williamhobbs]

On your first question, see https://forecastarbiter.epri.com/benchmarks/ and https://pvlib-python.readthedocs.io/en/stable/user_guide/forecasts.html?#cloud-cover-and-radiation. DSWRF and GHI are the same thing, but most of the NWPs, with the noted exception of the HRRR, have (or have had?) issues with solar position, interval labeling, or interval averaging when it comes to irradiance.

On your second question, you've probably seen this, but it looks like a good answer: https://earthscience.stackexchange.com/questions/24235/understanding-gfs-parameter-averaging-times. Also see https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/ncep_norm.html. And in this package there are more details in comments (see example linked/inserted below), but I don't know where the supporting info came from.

solarforecastarbiter-core/solarforecastarbiter/reference_forecasts/forecast.py

Lines 168 to 182 in d4d85a3

	"""Convert mixed interval averages into pure interval averages.
	For example, the GFS 3 hour output contains the following data:
	* forecast hour 3: average cloud cover from 0 - 3 hours
	* forecast hour 6: average cloud cover from 0 - 6 hours
	* forecast hour 9: average cloud cover from 6 - 9 hours
	* forecast hour 12: average cloud cover from 6 - 12 hours
	and so on. This function returns:
	* forecast hour 3: average cloud cover from 0 - 3 hours
	* forecast hour 6: average cloud cover from 3 - 6 hours
	* forecast hour 9: average cloud cover from 6 - 9 hours
	* forecast hour 12: average cloud cover from 9 - 12 hours

[jarataraj]

Thank you for your answer! The links really clear things up. I ended up using DSWRF from the GFS for my GHI forecast but will change things when I'm able given the information in your answer. I have seen that stack exchange question - I am the one who asked it!

[williamhobbs]

GFS averaging interval just came up on the NCEP NOMADS mailing list. I don't think there's any new information (it's the same person that responded to your stackexchange question), but this just shows that it is common question. Here's the exchange (with email addresses and personal info redacted, since I'm not sure what NCEP's rules are about distributing listserv emails):

From: NCEP.List.Nomads-Users * on behalf of Felix * <*@*.de>
Sent: Tuesday, December 20, 2022 8:36 AM
To: ncep.list.nomads-users*; *@*.com <*@*.com>
Subject: Re: [NCEP.List.Nomads-Users] Recent issues with data downloads via OpenDAP

Hi Dan,

Re 1.): It sounds like you might be looking at values stored as the NCEP-typical running 1 to 6 hour average, which is reset every 6 hours. I know this well from GFS, it is probably used in NAM or HiResW as well for some parameters. This is a regular pattern, although it keeps confusing people and is not so well documented. I wrote a reply on a question on this just last week in the Earth Sciences stack exchange, maybe that helps: https://earthscience.stackexchange.com/questions/24235/understanding-gfs-parameter-averaging-times/

Re 2.): Not sure. We load NCEP/NOAA data either from OpenDAP with the dods java lib (with retries etc) or as grib with grib_filter. I can't recall that entire parameters were missing in downloads, but the rest was complete. Maybe ncks has issues with some of the data in certain runs? Maybe someone else here has more experience with NCO and NAM or HiResW.

Regards,

Felix

Am 20.12.2022 um 08:02 schrieb Dan *:
Hello,

Over the past 2 weeks, I have been downloading the following 5 variables 2 or 3 times daily from various models via OpenDAP using ncks in the NCO toolkit:

tmp2m
dpt2m
rh2m
apcpsfc
cfrzrsfc

I have been encountering the following problems:

1.) Precip accumulation (apcpsfc) in some of the hourly models (HIRESW, NAM NEST) is happening in irregular groups of timesteps. It will start out accumulating every 3 timesteps, then go to 1 timestep, then 5 timesteps, then back to 3 timesteps, and so on. There is no pattern, making it impossible to write a decumulation function to obtain single-timestep precipitation amounts.

2.) I have experienced missing data several times. Two specific examples:

-- In the 20221219 18z HRRR hourly, the entire tmp2m variable was missing (values for all coordinates at all timesteps were zero).

-- In the 20221216 18z NAM NEST, precipitation data was missing (zero) in large geographical areas (for example in the states of Ohio and Indiana).

In all of the above instances, I re-downloaded the data multiple times and received the same result each time. There were no server errors returned. This issue did not affect any of the public model data display sites such as COD and Pivotal. Is this a problem with NOMADs, OpenDAP, ncks and/or NCO? Are there more reliable ways to download model data?

Thank you in advance for any assistance!

Dan