ESMValTool Tutorial: recipe_python extract_location preprocessor: GeocoderInsufficientPrivileges Error

[rbeucher]

Hi All,

I am preparing for an Introduction to ESMValTool workshop and I came across an error while trying to run the recipe_python.

The extract_location preprocessor returns an error and the recipe fails:

"""
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/geocoders/base.py", line 368, in _call_geocoder
    result = self.adapter.get_json(url, timeout=timeout, headers=req_headers)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/adapters.py", line 447, in get_json
    resp = self._request(url, timeout=timeout, headers=headers)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/adapters.py", line 475, in _request
    raise AdapterHTTPError(
geopy.adapters.AdapterHTTPError: Non-successful status code 403

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 816, in _run_task
    output_files = task.run()
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 264, in run
    self.output_files = self._run(input_files)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 670, in _run
    product.apply(step, self.debug)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 470, in apply
    self.cubes = preprocess(self.cubes, step,
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 379, in preprocess
    result.append(_run_preproc_function(function, item, settings,
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 328, in _run_preproc_function
    return function(items, **kwargs)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/preprocessor/_regrid.py", line 377, in extract_location
    geolocation = geolocator.geocode(location)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/geocoders/nominatim.py", line 297, in geocode
    return self._call_geocoder(url, callback, timeout=timeout)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/geocoders/base.py", line 388, in _call_geocoder
    res = self._adapter_error_handler(error)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/geopy/geocoders/base.py", line 411, in _adapter_error_handler
    raise exc_cls(str(error)) from error
geopy.exc.GeocoderInsufficientPrivileges: Non-successful status code 403
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_main.py", line 524, in run
    fire.Fire(ESMValTool())
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_main.py", line 411, in run
    self._run(recipe, session)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_main.py", line 453, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_main.py", line 125, in process_recipe
    recipe.run()
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_recipe/recipe.py", line 1294, in run
    self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks'])
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 738, in run
    self._run_parallel(address, max_parallel_tasks)
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 782, in _run_parallel
    _copy_results(task, running[task])
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/site-packages/esmvalcore/_task.py", line 805, in _copy_results
    task.output_files, task.products = future.get()
  File "/g/data/xp65/public/apps/med_conda/envs/access-med-0.1/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
geopy.exc.GeocoderInsufficientPrivileges: Non-successful status code 403

main_log_debug.txt

I am attaching the main_log_debug.txt

Any help is appreciated as the tutorial day is fast approaching.

@bouweandela @remi-kazeroni ?

[rbeucher]

I am using geopy v2.3.0 which when running the following code:

from geopy.geocoders import Nominatim
geolocator = Nominatim()

returns a ConfigurationError:

ConfigurationError: Using Nominatim with default or sample `user_agent` "geopy/2.3.0" is strongly discouraged, as it violates Nominatim's ToS https://operations.osmfoundation.org/policies/nominatim/ and may possibly cause 403 and 429 HTTP errors. Please specify a custom `user_agent` with `Nominatim(user_agent="my-application")` or by overriding the default `user_agent`: `geopy.geocoders.options.default_user_agent = "my-application"`.

The following works

import geopy
from geopy.geocoders import Nominatim
geopy.geocoders.options.default_user_agent = "esmvaltool"
address = "90, Park Avenue, New York City, New York, 10016"
geolocator = Nominatim()
location = geolocator.geocode(address, timeout=10, exactly_one=False)

While the extract_location preprocessor uses something like this:

import geopy
from geopy.geocoders import Nominatim
import ssl
address = "90, Park Avenue, New York City, New York, 10016"
ssl_context = ssl.create_default_context()
geolocator = Nominatim(user_agent="esmvalcore", ssl_context=ssl_context)
location = geolocator.geocode(address, timeout=10, exactly_one=False)

And that fails. I tried on a couple of systems and I can reproduce the issue

[rbeucher]

geopy/geopy#555

[rbeucher]

Actually, the error seems to occur randomly. When it crashes, I can get it to work by changing the user_agent to something else than esmvalcore. Are we hitting some sort of limit on the number of requests?
Anyway, that is annoying.

[remi-kazeroni]

@rbeucher - I have tried to reproduce the issue you encountered but couldn't get this failure. I have installed the Tool with mamba env create -n esmvaltool esmvaltool python=3.10 to reproduce your environment. Running examples/recipe_python.yml and the last code snippet from this #2171 (comment) work fine for me. I wonder if this issue is similar to the one seen at the MetOffice (see #2012) due to custom installations on clusters with specific restrictions...

As we had issues with this preprocessor in our nightly tests where examples/recipe_python.yml was run frequently, we made a copy of that recipe: testing/recipe_python_for_CI.yml which is identical to the former one but with a hard-coded location (see ESMValGroup/ESMValTool#3157). Perhaps running testing/recipe_python_for_CI.yml could be a temporary workaround for you.

[schlunma]

Hi @rbeucher, do you still have problems with this? Me and a colleague had similar problems today on Levante (#2188), but now it works fine again. Could be related to an issue on the server.

[rbeucher]

Hi @schlunma

I have not tried since last week. Will let you know

[bouweandela]

Maybe we should reconsider using the extract_location preprocessor function in the example recipe. It is nice that participants of trainings can change the location, but the service is a bit unreliable and that can be confusing for people new to ESMValTool. @ESMValGroup/userengagementteam Would you have ideas for which preprocessor functions are 'nice' and interesting for the examples/recipe_python.yml recipe?

[alistairsellar]

Maybe we should reconsider using the extract_location preprocessor function in the example recipe. It is nice that participants of trainings can change the location, but the service is a bit unreliable and that can be confusing for people new to ESMValTool. @ESMValGroup/userengagementteam Would you have ideas for which preprocessor functions are 'nice' and interesting for the examples/recipe_python.yml recipe?

@nhsavage thought this might interest you, since you had difficulties with extract_location for CORDEX emample recipe...

[nhsavage]

@nhsavage thought this might interest you, since you had difficulties with extract_location for CORDEX emample recipe...

thanks - my problem is unrelated to this (very specific to using rotated pole or projected coordinate systems)