Is it possible to see which fxx/steps are available for a model run?

[kingkong-cmd]

For example, this gets the default fxx = 0 for ecmwf/ifs.

H = Herbie("2024-08-5 12:00", model="ecmwf")

Is it possible to infer or access a list of the available forecast steps somehow? Somehing like fxx_list = [0,1,2,3, ... 144, 147, 150]

[williamhobbs]

I don't think this currently exists, but would certainly be useful. I have mostly resorted to looking at model documentation from various sources to know which fxx values are available for a given model and initialization time. It's not always easy to find, though. And at least once think I have found errors in official documentation (see #421).

For ECMWF IFS I have these notes:

    # From https://www.ecmwf.int/en/forecasts/datasets/open-data
    # For times 00z &12z: 0 to 144 by 3, 150 to 240 by 6.
    # For times 06z & 18z: 0 to 90 by 3.
    # From:
    # https://confluence.ecmwf.int/display/DAC/ECMWF+open+data%3A+real-time+forecasts+from+IFS+and+AIFS
    # Product "oper" runs 00z, 12z, 0h to 144h by 3h, 144h to 240h by 6h
    # Product "scda" runs 06z, 18z, 0h to 90h by 3h
    # **BUT**, see https://github.com/blaylockbk/Herbie/discussions/421
    # Starting 2024-11-12 06:00, 'scda' runs to 144h by 3h

Used in some code here: https://github.com/williamhobbs/energy-forecasting-tools/blob/661d0bfc4c5d77605a8c61de412b3ef916c1653a/forecast_solar.py#L82-L119

This is somewhat related, but not exactly the same: https://herbie.readthedocs.io/en/stable/user_guide/tutorial/latest.html.

[kingkong-cmd]

All right, thanks.

I do this in my honky tonky homemade NOAA gfs downloader by parsing timestamps from the available filenames, but I guess the filename convention is far from uniform across models. Also for gfs, even if it is supposed to provide forecasts for x hourly steps, they make them available incrementally so you have to continuously check how many steps it has published.

However, given that Herbie already knows the filename conventions for each model, I'd assume it wouldn't be far fetched for it to be able to parse available files and create a fxx_list. However, I have no idea how it works under the hood.

[williamhobbs]

I think the filename conventions you are thinking of are in the files in https://github.com/blaylockbk/Herbie/tree/main/src/herbie/models. For example, https://github.com/blaylockbk/Herbie/blob/main/src/herbie/models/gfs.py for GFS, e.g.,

Herbie/src/herbie/models/gfs.py

Line 58 in 43792b0

post_root = f"gfs.{self.date:%Y%m%d/%H}/atmos/gfs.t{self.date:%H}z.{self.product}.f{self.fxx:03d}"

And then one example for ECMWF,

Herbie/src/herbie/models/ecmwf.py

Lines 60 to 63 in 43792b0

	post_root = (
	f"{self.date:%Y%m%d/%Hz}/ifs/{self.resolution}/{self.product}"
	f"/{self.date:%Y%m%d%H%M%S}-{self.fxx}h-{self.product}-{product_suffix}.grib2"
	)

[blaylockbk]

@williamhobbs is right, this doesn't exist, but could.

I mainly haven't added this feature due to the maintenance burden. For example, the early versions of HRRR only have hourly forecasts 0-18, and then later versions have longer lead times, but only for certain hours. Lead times for other models can be complicated and change in the future.

I would entertain a pull request to add this feature to any model you are interested in. Changes would go in the model template file. You would add some logic that checked the user's fxx value against a list of known valid options. I would prefer only raising a warning if the user provided an invalid lead time, and tell the user what the expected fxx values are (and preferrably point to documentation describing the model's output interval).

[blaylockbk]

Herbie already knows the filename conventions for each model, I'd assume it wouldn't be far fetched for it to be able to parse available files and create a fxx_list. However, I have no idea how it works under the hood.

Herbie knowns where to look for files, and given user input, checks if a files exist. I've thought about including a feature where if a user doesn't supply a value like fxx or product, Herbie would list all avaialble files (from a single source). You're right, this isn't far fetched.

[kingkong-cmd]

As you say, I don't think this would be a good fit for hard coding based on the naming conventions. Better to implement something more dynamic that checks whats available. Should/could probably be a user triggered function .fetch_all_fxx() instead of something very deeply integrated.

I did a quick attempt on a pr, but I got lost in the data structures for models and sources and urls which I found hard to understand.

I also discovered that it's likely not as simple as a generic catch-all function for all models since they are likely in different file system structures, so would maybe require a bit customizing for each model.

Since I were not able to implement into Herbie itself, i made some quick and dirty snippets for myself. Examples below.

ecmwf

A bit finicky to read from aws s3

today = datetime.today()
custom_time = today.replace(hour=12, minute=0, second=0, microsecond=0)
date_str = custom_time.strftime('%Y-%m-%-d %H:%M') #yyyy-mm-d 12:00

herbie = Herbie(date_str, model="ecmwf")
url = herbie.grib
parsed = urlparse(url)
base_url = f"{parsed.scheme}://{parsed.netloc}" # 'https://ecmwf-forecasts.s3.eu-central-1.amazonaws.com'
path_parts = parsed.path.lstrip('/').split('/') 
prefix = '/'.join(path_parts[:-1]) + '/' # '20240805/12z/ifs/0p25/oper/'

# Params for listing objects under prefix
params = {
'list-type': '2',
'prefix': prefix
}
response = requests.get(base_url, params=params)
soup = BeautifulSoup(response.content, 'xml')
items = soup.find_all('Key')

time_steps = []
for item in items:
if item.text.endswith('.grib2'):
	match = re.search(r'(\d+)h', item.text)
	time_steps.append(int(match.group(1)))

time_steps = sorted(time_steps)

ECCC

Simpler, and could of course have skipped BeautifulSoup

recent = pd.Timestamp("now").normalize() # This creates a timestamp for current day at 00:00 (beginning of day)

herbie = Herbie(recent, model="gdps", fxx=0, product="15km/grib2/lat_lon", variable='UGRD', level='TGL_10')

grib_dir_url = herbie.SOURCES['msc'].rsplit('/', 2)[0] + '/'

response = requests.get(grib_dir_url)

soup = BeautifulSoup(response.content, 'lxml')
items = soup.find_all('a')

time_steps = []
for item in items:
    try:
        time_steps.append(int(item.text.replace('/','')))
    except:
        continu