Server Error

[bejaeger]

Hi openml team,

I'm experiencing server errors since yesterday.
E.g.

import openml
openml.tasks.get_task(168910, download_splits=True)

Returns an

OpenMLServerError: Unexpected server error when calling https://openml.org/api_splits/get/168910/Task_168910_splits.arff. Please contact the developers!
Status code: 412
failed to perform action generate_folds. Evaluation Engine result send to EMAIL_API_LOG account.

Is there anything we can do about it?

Thanks for working on this project!

Full traceback:

    task = openml.tasks.get_task(int(tid), download_splits=True)
packages/benchmarking/.venv/lib/python3.10/site-packages/openml/tasks/functions.py:441: in get_task
    raise e
packages/benchmarking/.venv/lib/python3.10/site-packages/openml/tasks/functions.py:438: in get_task
    task.download_split()
packages/benchmarking/.venv/lib/python3.10/site-packages/openml/tasks/task.py:191: in download_split
    self._download_split(cached_split_file)
packages/benchmarking/.venv/lib/python3.10/site-packages/openml/tasks/task.py:175: in _download_split
    openml._api_calls._download_text_file(
packages/benchmarking/.venv/lib/python3.10/site-packages/openml/_api_calls.py:275: in _download_text_file
    response = __read_url(source, request_method="get", md5_checksum=md5_checksum)
packages/benchmarking/.venv/lib/python3.10/site-packages/openml/_api_calls.py:342: in __read_url
    return _send_request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

request_method = 'get'
url = 'https://openml.org/api_splits/get/168910/Task_168910_splits.arff'
data = {}, files = None, md5_checksum = None

    def _send_request(  # noqa: C901, PLR0912
        request_method: str,
        url: str,
        data: DATA_TYPE,
        files: FILE_ELEMENTS_TYPE | None = None,
        md5_checksum: str | None = None,
    ) -> requests.Response:
        n_retries = max(1, config.connection_n_retries)
    
        response: requests.Response | None = None
        delay_method = _human_delay if config.retry_policy == "human" else _robot_delay
    
        # Error to raise in case of retrying too often. Will be set to the last observed exception.
        retry_raise_e: Exception | None = None
    
        with requests.Session() as session:
            # Start at one to have a non-zero multiplier for the sleep
            for retry_counter in range(1, n_retries + 1):
                try:
                    if request_method == "get":
                        response = session.get(url, params=data, headers=_HEADERS)
                    elif request_method == "delete":
                        response = session.delete(url, params=data, headers=_HEADERS)
                    elif request_method == "post":
                        response = session.post(url, data=data, files=files, headers=_HEADERS)
                    else:
                        raise NotImplementedError()
    
                    __check_response(response=response, url=url, file_elements=files)
    
                    if request_method == "get" and not __is_checksum_equal(
                        response.text.encode("utf-8"), md5_checksum
                    ):
                        # -- Check if encoding is not UTF-8 perhaps
                        if __is_checksum_equal(response.content, md5_checksum):
                            raise OpenMLHashException(
                                f"Checksum of downloaded file is unequal to the expected checksum"
                                f"{md5_checksum} because the text encoding is not UTF-8 when "
                                f"downloading {url}. There might be a sever-sided issue with the file, "
                                "see: https://github.com/openml/openml-python/issues/1180.",
                            )
    
                        raise OpenMLHashException(
                            f"Checksum of downloaded file is unequal to the expected checksum "
                            f"{md5_checksum} when downloading {url}.",
                        )
    
                    return response
                except OpenMLServerException as e:
                    # Propagate all server errors to the calling functions, except
                    # for 107 which represents a database connection error.
                    # These are typically caused by high server load,
                    # which means trying again might resolve the issue.
                    if e.code != DATABASE_CONNECTION_ERRCODE:
                        raise e
                    retry_raise_e = e
                except xml.parsers.expat.ExpatError as e:
                    if request_method != "get" or retry_counter >= n_retries:
                        if response is not None:
                            extra = f"Status code: {response.status_code}\n{response.text}"
                        else:
                            extra = "No response retrieved."
    
>                       raise OpenMLServerError(
                            f"Unexpected server error when calling {url}. Please contact the "
                            f"developers!\n{extra}"
                        ) from e
E                       openml.exceptions.OpenMLServerError: Unexpected server error when calling https://openml.org/api_splits/get/168910/Task_168910_splits.arff. Please contact the developers!
E                       Status code: 412
E                       failed to perform action generate_folds. Evaluation Engine result send to EMAIL_API_LOG account.

[kindler-king]

Hi there,

Thanks for reporting this issue! The HTTP 412 error with the message "failed to perform action generate_folds" indicates that the backend service responsible for generating task splits has encountered a failure.

This is a known issue (see also #1108 in the main repo). The Evaluation Engine needs to successfully generate the cross-validation splits before they can be downloaded, and when this process fails on the server, there's nothing the client can do.

Potential workaround for now:
You can still work with the task by skipping the splits download and creating your own cross-validation:

import openml
from sklearn.model_selection import StratifiedKFold

# Get task without splits
task = openml.tasks.get_task(168910, download_splits=False)
dataset = openml.datasets.get_dataset(task.dataset_id)
X, y, _, _ = dataset.get_data(target=task.target_name)

# Use your own CV strategy
cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=1)

I don't know if there is a cleaner solution as of now. Hope it helps provide some context.

[PGijsbers]

Thanks for the suggestion @kindler-king, and sorry we are slow to reply @bejaeger. In this case, it was actually related to the server maintenance we were doing at that time. During the week we replaced one of our backend servers and temporarily had to route traffic to a server with limited functionalities. If you try to fetch the task splits now, it should work as expected (I tried). If you still encounter issues with this task, feel free to re-open! Thanks for you patience!