404 error with River Flood Hazards from GloFAS Discharge Data

[francescodalbello]

Hi everyone @peanutfun , I'm trying to replicate the tutorial https://climada-petals.readthedocs.io/en/v6.1.0/glofas_rf.html# and the setup_all section, let's say, works. I had to manually add the 100 MB gumbell_fit file but I dont' get errors.
I installed both climada and climada-petals via mamba version 6.1.

The problem arises when I launch the second block:

from climada_petals.hazard.rf_glofas import (
    RiverFloodInundation,
    hazard_series_from_dataset,
)

forecast_date = "2023-08-01"
rf = RiverFloodInundation()
rf.download_forecast(
    countries="Switzerland",
    forecast_date=forecast_date,
    lead_time_days=5,
    preprocess=lambda x: x.max(dim="step"),
)
ds_flood = rf.compute()
hazard = hazard_series_from_dataset(ds_flood, "flood_depth", "number")`.

It returns an HTTP error saying that the dataset does not exist. I correctly placed the .cdsapirc in $HOME. Has anything changed in the api structure?

I quote the error here:

/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/river_flood_computation.py:182: UserWarning: The specified chunks separate the stored chunks along dimension "latitude" starting at index 1377. This could degrade performance. Instead, consider rechunking after loading.
self.flood_maps = xr.open_dataarray(
/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/river_flood_computation.py:182: UserWarning: The specified chunks separate the stored chunks along dimension "longitude" starting at index 3480. This could degrade performance. Instead, consider rechunking after loading.
self.flood_maps = xr.open_dataarray(
/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/river_flood_computation.py:186: FutureWarning: In a future version, xarray will not decode timedelta values based on the presence of a timedelta-like units attribute by default. Instead it will rely on the presence of a timedelta64 dtype attribute, which is now xarray's default way of encoding timedelta64 values. To continue decoding timedeltas based on the presence of a timedelta-like units attribute, users will need to explicitly opt-in by passing True or CFTimedeltaCoder(decode_via_units=True) to decode_timedelta. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
self.gumbel_fits = xr.open_dataset(data_dir / "gumbel-fit.nc", chunks="auto")
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/cds_glofas_downloader.py", line 154, in glofas_request_single
client.retrieve(product, request, outfile)
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/legacy_client.py", line 167, in retrieve
submitted = self.client.submit_and_wait_on_results(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/client.py", line 414, in submit_and_wait_on_results
return self._retrieve_api.submit(collection_id, request).get_results()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 735, in submit
return self.get_process(collection_id).submit(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 724, in get_process
return Process.from_request("get", url, **self._request_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 176, in from_request
cads_raise_for_status(response)
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 99, in cads_raise_for_status
raise requests.HTTPError(message, response=response)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://cds.climate.copernicus.eu/api/retrieve/v1/processes/cems-glofas-forecast
process not found
dataset cems-glofas-forecast not found
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/francesco/Repositories/project_crat/preprocessing_pipeline/glofas_wsl.py", line 6, in
rf.download_forecast(
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/river_flood_computation.py", line 354, in download_forecast
forecast = download_glofas_discharge(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/transform_ops.py", line 318, in download_glofas_discharge
files = glofas_request(
^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/cds_glofas_downloader.py", line 289, in glofas_request
return glofas_request_multiple(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/cds_glofas_downloader.py", line 174, in glofas_request_multiple
return pool.starmap(
^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/climada_petals/hazard/rf_glofas/cds_glofas_downloader.py", line 154, in glofas_request_single
client.retrieve(product, request, outfile)
^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/legacy_client.py", line 167, in retrieve
submitted = self.client.submit_and_wait_on_results(
^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/client.py", line 414, in submit_and_wait_on_results
return self._retrieve_api.submit(collection_id, request).get_results()
^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 735, in submit
return self.get_process(collection_id).submit(request)
^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 724, in get_process
return Process.from_request("get", url, **self._request_kwargs)
^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 176, in from_request
cads_raise_for_status(response)
^^^^^^^^^^^^^^^^^
File "/home/francesco/micromamba/envs/climada_wsl/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 99, in cads_raise_for_status
raise requests.HTTPError(message, response=response)
^^^^^^^^^^^^^^^
requests.exceptions.HTTPError: [Errno None] None

[peanutfun]

@francescodalbello Thank you for your interest in the module and your report! The GloFAS module in the current release is outdated. You receive the error because the download URL changed (the data moved from the Climate Data Store to the Emergency Data Store). Similarly, the URL for the Gumbel fit data changed.

We are currently updating the module to GloFAS v4 in #167. I recommend you install Climada from source and check out the rf-glofas-v4 branch for Petals. Run setup_all again and you should be good to go!

[francescodalbello]

Hello, and thank you very much for your prompt reply. I did as suggested, and now it downloads the 100 MB fit file on its own, but it still cannot access glofas. Am I doing something wrong?

from climada.util import log_level
from climada_petals.hazard.rf_glofas import setup_all

with log_level("DEBUG", "climada_petals"):
    setup_all()

from climada_petals.hazard.rf_glofas import RiverFloodInundation

countries = ["Italy"]

rf = RiverFloodInundation()
rf.download_forecast(
    countries=countries,
    forecast_date="2025-11-12",
    preprocess=lambda x: x.max(dim="step"),
    system_version="operational",  # Version mislabeled
)

ds_flood = rf.compute()
print(ds_flood)

2025-11-10 14:43:08,079 - climada_petals.hazard.rf_glofas.setup - DEBUG - Downloading FLOPROS database
2025-11-10 14:43:17,538 - climada_petals.hazard.rf_glofas.setup - DEBUG - Downloading Gumbel fit parameters
/home/francescodalbello/workspace/climada_petals/climada_petals/hazard/rf_glofas/river_flood_computation.py:209: FutureWarning: In a future version, xarray will not decode the variable 'step' into a timedelta64 dtype based on the presence of a timedelta-like 'units' attribute by default. Instead it will rely on the presence of a timedelta64 'dtype' attribute, which is now xarray's default way of encoding timedelta64 values.
To continue decoding into a timedelta64 dtype, either set decode_timedelta=True when opening this dataset, or add the attribute dtype='timedelta64[ns]' to this variable on disk.
To opt-in to future behavior, set decode_timedelta=False.
self.gumbel_fits = xr.open_dataset(
Traceback (most recent call last):
File "/home/francescodalbello/Repositories/project_crat/preprocessing_pipeline/tmp.py", line 12, in
rf.download_forecast(
File "/home/francescodalbello/workspace/climada_petals/climada_petals/hazard/rf_glofas/river_flood_computation.py", line 458, in download_forecast
forecast = download_glofas_discharge(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francescodalbello/workspace/climada_petals/climada_petals/hazard/rf_glofas/transform_ops.py", line 326, in download_glofas_discharge
files = glofas_request(
^^^^^^^^^^^^^^^
File "/home/francescodalbello/workspace/climada_petals/climada_petals/hazard/rf_glofas/cds_glofas_downloader.py", line 290, in glofas_request
glofas_request_single(
File "/home/francescodalbello/workspace/climada_petals/climada_petals/hazard/rf_glofas/cds_glofas_downloader.py", line 158, in glofas_request_single
client.retrieve(product, request, outfile)
File "/home/francescodalbello/micromamba/envs/climada_env/lib/python3.11/site-packages/ecmwf/datastores/legacy_client.py", line 167, in retrieve
submitted = self.client.submit_and_wait_on_results(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francescodalbello/micromamba/envs/climada_env/lib/python3.11/site-packages/ecmwf/datastores/client.py", line 414, in submit_and_wait_on_results
return self._retrieve_api.submit(collection_id, request).get_results()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francescodalbello/micromamba/envs/climada_env/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 735, in submit
return self.get_process(collection_id).submit(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francescodalbello/micromamba/envs/climada_env/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 724, in get_process
return Process.from_request("get", url, **self._request_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francescodalbello/micromamba/envs/climada_env/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 176, in from_request
cads_raise_for_status(response)
File "/home/francescodalbello/micromamba/envs/climada_env/lib/python3.11/site-packages/ecmwf/datastores/processing.py", line 99, in cads_raise_for_status
raise requests.HTTPError(message, response=response)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://cds.climate.copernicus.eu/api/retrieve/v1/processes/cems-glofas-forecast
process not found
dataset cems-glofas-forecast not found

[peanutfun]

@francescodalbello Please make sure the first line in your .cdsapirc file contains this URL:

url: https://ewds.climate.copernicus.eu/api

[francescodalbello]

That was it, thank you! Now the problem is the constant timeouts...even with less than 2 years of history. Is this a known issue? Is there anything I can do to help work around this issue?

My code:

from climada_petals.hazard.rf_glofas import (
    RiverFloodInundation,
    hazard_series_from_dataset,
)

rf = RiverFloodInundation()
rf.download_reanalysis(
    countries="Italy",
    year=2024
)
ds_flood = rf.compute()
hazard = hazard_series_from_dataset(ds_flood, "flood_depth")
#hazard = hazard_series_from_dataset(ds_flood, "flood_depth", "number")

and I got an api 'complete' return.

and

Even if the request is processed successfully, Python cannot retrieve it and remains in timeout (I waited for 3 attempts).

(climada_env) francescodalbello@4RCJNN3:~/Repositories/deda_crat/preprocessing_pipeline/riverflood$ python official_riverflood_glofas.py 
/home/francescodalbello/workspace/climada_petals/climada_petals/hazard/rf_glofas/river_flood_computation.py:209: FutureWarning: In a future version, xarray will not decode the variable 'step' into a timedelta64 dtype based on the presence of a timedelta-like 'units' attribute by default. Instead it will rely on the presence of a timedelta64 'dtype' attribute, which is now xarray's default way of encoding timedelta64 values.
To continue decoding into a timedelta64 dtype, either set `decode_timedelta=True` when opening this dataset, or add the attribute `dtype='timedelta64[ns]'` to this variable on disk.
To opt-in to future behavior, set `decode_timedelta=False`.
  self.gumbel_fits = xr.open_dataset(
Recovering from connection error [('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))], attempt 1 of 500
Retrying in 480 seconds

[peanutfun]

@francescodalbello I am not sure about the timeouts. I have received similar errors, which is why I already raised the timeout limit to 480 secs. Were you able to download the data successfully in the meantime?

[peanutfun]

Closing due to inactivity