feature: artifact-helper client

Allow uploading artifacts using:
```
from openeo.extra.artifacts import build_artifact_helper

artifact_helper = build_artifact_helper(connection)
storage_uri = artifact_helper.upload_file(path, object_name)
presigned_uri = artifact_helper.get_presigned_url(storage_uri)
```

Author: Peter Van Bouwel

docs/api-artifacts.rst

@@ -0,0 +1,200 @@
+.. _api-openeo-extra-artifacts:
+
+====================================
+API: openeo.extra.artifacts
+====================================
+
+.. warning::
+    This is a new experimental API, subject to change.
+
+
+.. important::
+    The artifacts functionality relies on extra Python packages. They can be installed using:
+
+    .. code-block:: shell
+
+     pip install "openeo[artifacts]" --upgrade
+
+
+When running openEO jobs it is not uncommon to require artifacts that should be accessible during job execution. This
+requires the artifacts to be accessible from within the openEO processing environment. :py:mod:`openeo.extra.artifacts` tries
+to perform the heavy lifting for this use case by allowing staging artifacts to a secure but temporary location using 3
+simple steps:
+
+1. Connect to your openEO backend
+2. Create an artifact helper from your openEO connection
+3. Upload your file using the artifact helper and optionally get a presigned URI
+
+So in code this looks like:
+
+.. code-block:: python
+
+    import openeo
+    from openeo.extra.artifacts import build_artifact_helper
+
+    connection = openeo.connect("my-openeo.prod.example").authenticate_oidc()
+
+    artifact_helper = build_artifact_helper(connection)
+    storage_uri = artifact_helper.upload_file(path, object_name)
+    presigned_uri = artifact_helper.get_presigned_url(storage_uri)
+
+Note:
+
+* The presigned_uri should be used for accessing the objects. It has authentication details embedded so if your data is
+  sensitive you must make sure to keep this URL secret. You can lower expires_in_seconds in
+  :py:meth:`openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC.get_presigned_url`
+  to limit the time window in which the URI can be used.
+
+* The openEO backend must expose additional metadata in its capabilities doc to make this possible. Implementers of a
+  backend can check the extra documentation :ref:`advertising-capabilities`.
+
+
+User facing API
+===============
+
+
+.. autofunction:: openeo.extra.artifacts.build_artifact_helper
+
+
+.. autoclass:: openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC
+    :members: upload_file, get_presigned_url
+    :no-index:
+
+
+How does it work ?
+==================
+
+1) :py:meth:`openeo.extra.artifacts.build_artifact_helper` is a factory method that
+   will create an artifact helper where the type is defined by the config type. The openEO connection object is used to
+   see if the openEO backend advertises a preferred config.
+2) :py:meth:`openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC.upload_file` and
+   :py:meth:`openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC.get_presigned_url` do the heavy lifting to
+   store your artifact in provider managed storage and to return references that can be used. In case the backend uses
+   an Object storage that has an S3 API it will:
+
+   1. Get temporary S3 credentials based on config advertised by the backend and the session from your connection
+   2. Upload the file into object storage and return an S3 URI which the backend can resolve
+   3. Optional the :py:meth:`openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC.get_presigned_url` makes a
+      URI signed with the temporary credentials such that it works standalone (Some tools and execution steps do not
+      support handling of internal references. presigned URLs should work in any tool).
+
+
+Documentation for backend providers
+===================================
+
+This section and its subsection is for engineers who operate an openEO backend. If you are a user of an openEO platform
+this is unlikely to be of value to you.
+
+.. _advertising-capabilities:
+
+Advertising capabilities from the backend
+-----------------------------------------
+
+It is expected that the backend advertises in its capabilities a section on artifacts. The following is an example
+for the S3STSConfig (of the :py:mod:`openeo.extra.artifacts._s3sts` package).
+
+.. code-block:: json
+
+    {
+       // ...
+       "artifacts": {
+         "providers": [
+           {
+             // This id is a logical name
+             "id": "s3",
+             // The config type of the ArtifactHelper
+             "type": "S3STSConfig"
+             // The config block its keys can differ for other config types
+             "config": {
+               // The bucket where the artifacts will be stored
+               "bucket": "openeo-artifacts",
+               // The role that will be assumed via STS
+               "role": "arn:aws:iam::000000000000:role/S3Access",
+               // Where S3 API calls are sent
+               "s3_endpoint": "https://my.s3.test",
+              // Where STS API calls are sent
+               "sts_endpoint": "https://my.sts.test"
+             },
+           }
+         ]
+       },
+       // ...
+    }
+
+
+Extending support for other types of artifacts
+----------------------------------------------
+
+.. warning::
+    This is a section for developers of the `openeo-python-client` Python package. If you want to walk this road it is
+    best to create an issue on github and detail what support you are planning to add to get input on feasibility and
+    whether it will be mergeable early on.
+
+Ideally the user-interface is simple and stable. Unfortunately implementations themselves come with more complexity.
+This section explains what is needed to provide support for additional types of artifacts. Below the steps we show
+the API that is involved.
+
+1. Create another internal package for the implementation. The following steps should be done inside that package.
+   This package resides under :py:mod:`openeo.extra.artifacts`
+2. Create a config implementation which extends :py:class:`openeo.extra.artifacts._config.ArtifactsStorageConfigABC`
+   and should be a frozen dataclass. This class implements the logic to determine the configuration used by the
+   implementation `_load_connection_provided_config(self, provider_config: ProviderConfig) -> None` is used for that.
+
+   When this method is called explicit config is already put in place and if not provided default config is put in
+   place.
+   Because frozen dataclasses are used for config `object.__setattr__(self, ...)` must be used to manipulate the
+   values.
+
+   So per attribute the same pattern is used. For example an attribute `foo` which has a default `bar` that can be kept
+   constant would be:
+
+    .. code-block:: python
+
+        if self.foo is None:
+            try:
+                object.__setattr__(self, "foo", provider_config["foo"])
+            except NoDefaultConfig:
+                object.__setattr__(self, "foo", "bar")
+
+   Here we use :py:exc:`openeo.extra.artifacts.exceptions.NoDefaultConfig`
+
+3. Create an implementation of :py:class:`openeo.extra.artifacts._uri.StorageURI` to model the internal URIs to the
+   stored artifact
+4. Create an ArtifactHelper implementation which extends :py:class:`openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC`
+5. Add a key value pair to the :py:obj:`openeo.extra.artifacts.artifact_helper.config_to_helper` dictionary. The key is
+   the class created in 2 and the value is the class created in step 3
+
+.. autoclass:: openeo.extra.artifacts._config.ArtifactsStorageConfigABC
+    :members:
+    :private-members: _load_connection_provided_config
+
+.. autoclass:: openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC
+    :members:
+    :private-members: _get_default_storage_config, _from_openeo_connection
+
+.. autoclass:: openeo.extra.artifacts._uri.StorageURI
+    :members:
+
+
+Artifacts exceptions
+--------------------
+
+When using artifacts your interactions can result in the following exceptions.
+
+.. autoexception:: openeo.extra.artifacts.exceptions.ArtifactsException
+    :members:
+
+.. autoexception:: openeo.extra.artifacts.exceptions.NoAdvertisedProviders
+    :members:
+
+.. autoexception:: openeo.extra.artifacts.exceptions.UnsupportedArtifactsType
+    :members:
+
+.. autoexception:: openeo.extra.artifacts.exceptions.NoDefaultConfig
+    :members:
+
+.. autoexception:: openeo.extra.artifacts.exceptions.InvalidProviderConfig
+    :members:
+
+.. autoexception:: openeo.extra.artifacts.exceptions.ProviderSpecificException
+    :members:

docs/index.rst

@@ -61,6 +61,7 @@ Table of contents
    cookbook/index
    api
    api-processes
+   api-artifacts
    process_mapping
    development
    best_practices

openeo/extra/artifacts/__init__.py

@@ -0,0 +1 @@
+from openeo.extra.artifacts.artifact_helper import build_artifact_helper

openeo/extra/artifacts/_artifact_helper_abc.py

@@ -0,0 +1,95 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from openeo.extra.artifacts._backend import ProviderConfig
+from openeo.extra.artifacts._config import ArtifactsStorageConfigABC
+from openeo.extra.artifacts._uri import StorageURI
+from openeo.rest.connection import Connection
+
+
+class ArtifactHelperABC(ABC):
+    """
+    This class defines the *interface* that an artifact helper should implement and support. This is used by OpenEO users
+    willing to manage artifacts.
+
+    Instances that implement it get created by the `openeo.extra.artifacts.build_artifact_helper` factory
+    """
+
+    @classmethod
+    def from_openeo_connection(
+        cls,
+        connection: Connection,
+        provider_config: ProviderConfig,
+        *,
+        config: Optional[ArtifactsStorageConfigABC] = None,
+    ) -> ArtifactHelperABC:
+        """
+        Create a new Artifact helper from the OpenEO connection. This is the starting point to upload artifacts.
+        Each implementation has its own builder
+        """
+        if config is None:
+            config = cls._get_default_storage_config()
+        config.load_connection_provided_config(provider_config)
+        return cls._from_openeo_connection(connection, config)
+
+    @abstractmethod
+    def upload_file(self, path: str | Path, object_name: str = "") -> StorageURI:
+        """
+        A method to store an artifact remotely and get a StorageURI which points to the stored data.
+
+        :param path: Location of the file to be uploaded absolute path or relative to current
+                     working directory.
+        :param object_name: Optional name you want to give to the object. If not specified the filename will be
+                            used.
+
+        :return: If you want to use the StorageURI in a processgraph convert it using Python's built-in `str()`
+                 function which is understood by the OpenEO processor.
+        """
+
+    @abstractmethod
+    def get_presigned_url(self, storage_uri: StorageURI, expires_in_seconds: int = 7 * 3600 * 24) -> str:
+        """
+        A method to get a signed https URL for a given StorageURI which can be accessed via normal http libraries.
+
+        These URIs should be kept secret as they provide access to the data.
+
+        :param storage_uri: URI to the artifact that is stored by a previous `upload_file` call
+        :param expires_in_seconds: Optional how long expressed in seconds before the returned signed URL becomes invalid
+
+        :return: The signed https URI.
+
+        """
+
+    def __init__(self, config: ArtifactsStorageConfigABC):
+        if not config.is_openeo_connection_metadata_loaded():
+            raise RuntimeError("config should have openeo connection metadata loaded prior to initialization.")
+        self._config = config
+
+    @classmethod
+    @abstractmethod
+    def _get_default_storage_config(cls) -> ArtifactsStorageConfigABC:
+        """
+        A method that provides a default storage config for the Artifact Helper. It will return a class that
+        extends `ArtifactsStorageConfigABC` and just provides default values which are defined in code no fancy
+        resolvement from the backend yet. The config does not need to be usable by itself yet.
+
+        If a config value can be advertised by the backend it should be initialized to a sentinel value and the actual
+        value should be put in place if not advertised by the backend which happens in an implementation of
+        :func:`~openeo.extra.artifacts._artifact_helper_abc.ArtifactsStorageConfigABC._load_connection_provided_config`
+        """
+
+    @classmethod
+    @abstractmethod
+    def _from_openeo_connection(cls, connection: Connection, config: ArtifactsStorageConfigABC) -> ArtifactHelperABC:
+        """
+        The implementation that creates an artifact helper. This method takes a config which has already been
+        initialized from the metadata of the OpenEO connection.
+
+        This method is internal as it is always called via `ArtifactHelperABC.from_openeo_connection`
+
+        :param connection: A valid instance of a connection object to an OpenEOBackend
+        :param config: object that specifies configuration for Artifact storage.
+        """