Initial version of Zenodo connector

.gitignore

@@ -1,3 +1,14 @@
+# configurations
+.env.override
+.prod.env
+.test.env
+
+# Certificates
+certs/
+
+# IDE
+.idea/
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[codz]
@@ -204,4 +215,4 @@ cython_debug/
 # Marimo
 marimo/_static/
 marimo/_lsp/
-__marimo__/
+__marimo__/

Dockerfile

@@ -0,0 +1,27 @@
+FROM python:3.13.7-trixie
+
+# Install CA infrastructure
+RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
+
+# Add AIoD cert and build custom bundle
+RUN mkdir -p /certs
+COPY certs/aiod-insight-centre.crt /certs/aiod-insight-centre.crt
+RUN cat /etc/ssl/certs/ca-certificates.crt /certs/aiod-insight-centre.crt > /certs/custom-ca-bundle.crt
+
+# Make Python requests use the custom CA bundle
+ENV REQUESTS_CA_BUNDLE=/certs/custom-ca-bundle.crt
+
+RUN useradd -m appuser
+USER appuser
+WORKDIR /home/appuser
+RUN python -m venv .venv
+RUN .venv/bin/python -m pip install uv
+
+COPY pyproject.toml /app/pyproject.toml
+RUN .venv/bin/uv pip install -r /app/pyproject.toml
+
+COPY . /app
+RUN .venv/bin/uv pip install file:///app
+
+ENTRYPOINT [".venv/bin/python", "/app/src/connector.py"]
+CMD ["--mode", "all"]

README.md

@@ -1,2 +1,52 @@
-# zenodo-connector
-The connector indexing metadata from Zenodo and registering it with AI-on-Demand
+# AI-on-Demand Zenodo connector
+Collects dataset metadata from [Zenodo](https://zenodo.org) and uploads it to AI-on-Demand.
+
+This package is not intended to be used directly by others, but may serve as an example of how to build a connector for the AI-on-Demand platform.
+For more information on how to test this connector locally as a springboard for developing your own connector, reference the [Development](#Development) section below.
+
+### TODO
+This package is work in progress.
+
+- [ ] Automatically publish to DockerHub on release
+- [ ] Add tests
+
+## Installation
+You can use the image directly from Docker Hub (TODO) or build it locally.
+
+From Docker Hub: `docker pull aiondemand/zenodo-connector`.
+
+To build a local image:
+
+ - Clone the repository: `git clone https://github.com/aiondemand/zenodo-connector && cd zenodo-connector`
+ - Build the image: `docker build -t aiondemand/zenodo-connector -f Dockerfile .`
+
+### Configuring Client Credentials
+You will need to configure what server the connector should connect to, as well as the credentials for the client that allow you to upload data.
+The connector requires a `config.toml` file with a valid [aiondemand configuration](https://aiondemand.github.io/aiondemand/api/configuration/),
+the default configuration can be found in the [`/script/config.prod.toml`](/script/config.prod.toml) file.
+You will also need to have the 'Client Secret' for the client, which can be obtained from the keycloak administrator.
+The client secret must be provided to the Docker container as an environment variable or in a dotenv file *similar to* [`script/.local.env`](script/.local.env) but named `script/.prod.env`.
+
+Please contact the Keycloak service maintainer to obtain said credentials you need if you are in charge of deploying this Zenodo connector.
+
+## Running the Connector
+You will need to mount the aiondemand configuration to `/home/appuser/.aiod/config.toml` and provide environment variables directly with `-e` or through mounting the dotfile in `/home/appuser/.aiod/zenodo/.env`. The [`script/run.sh`](script/run.sh) script provides a convenience that automatically does this. 
+It takes one positional argument that has to be `local`, `test`, or `prod` to use the respective files in the `script` folder for configuration.
+Any following arguments are interpreted as arguments to the main script.
+For the latest commandline arguments, use `docker run aiondemand/zenodo-connector --help`.
+Some example invocations that use the `script/run.sh` script:
+
+ - `script/run.sh local --mode id --value 12345 --app-log-level debug` syncs one specific Zenodo record, and produces debug logs for the connector only.
+ - `script/run.sh test --mode since --value 2025-09-30T00:00 --root-log-level debug` syncs all records created or modified after 2025-09-30T00:00 (in reverse chronological order).
+ - `script/run.sh prod --mode all --root-log-level info` indexes all records on Zenodo, producing info logs for the connector and all its dependencies (this is the default).
+
+## Development
+You can test the connector when running the [metadata catalogue](https://github.com/aiondemand/aiod-rest-api) locally.
+The default configurations for this setup can be found in the [`.local.env`](script/.local.env) and [`config.local.toml`](script/config.local.toml) files.
+
+When connecting to the AI-on-Demand test or production server, you will need to a dedicated client registered in the keycloak instance which is connected to the REST API you want to upload data to. 
+See [this form]() to apply for a client. The client will need to have a `platform_X` role attached, where `X` is the name of the platform from which you register assets. 
+When a client is created, you will need its 'Client ID' and 'Client Secret' and update the relevant configuration and environment files accordingly.
+
+## Disclaimer
+This project is not affiliated with Zenodo in any way.

pyproject.toml

@@ -0,0 +1,20 @@
+[project]
+name="aiod-zenodo-connector"
+dependencies = [
+    "aiondemand",
+    "python-dotenv",
+]
+dynamic=["version"]
+
+[build-system]
+requires = ["hatchling >= 1.26"]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+path = "src/__version__.py"
+
+[tool.hatch.metadata]
+allow-direct-references = true
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/"]

script/.local.env

@@ -0,0 +1,4 @@
+CLIENT_SECRET=S2zo0zW6QMy8ffcqCozYbHkj0JajcWtQ
+PLATFORM_NAME=example
+STOP_ON_UNEXPECTED_ERROR=true
+PER_DATASET_DELAY=

script/config.local.toml

@@ -0,0 +1,3 @@
+api_server = 'http://localhost/'
+auth_server = 'http://localhost/aiod-auth/'
+client_id = 'sdk-service'

script/config.prod.toml

@@ -0,0 +1,3 @@
+api_server = 'https://api.aiod.eu/'
+auth_server = 'https://auth.aiod.eu/aiod-auth/'
+client_id = 'platform_openml'

script/config.test.toml

@@ -0,0 +1,3 @@
+api_server = 'https://aiod.insight-centre.org/'
+auth_server = 'https://ai4europe.test.fedcloud.eu/aiod-auth/'
+client_id   = 'platform_zenodo'

script/run.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+WHICH="${1:-prod}"
+shift
+
+ABSOLUTE_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${ABSOLUTE_SCRIPT_DIR}/.${WHICH}.env"
+CONFIG_FILE="${ABSOLUTE_SCRIPT_DIR}/config.${WHICH}.toml"
+
+if [[ ! -f "$ENV_FILE" ]]; then
+  echo "Error: expected env file not found: $ENV_FILE" >&2
+  exit 1
+fi
+
+if [[ ! -f "$CONFIG_FILE" ]]; then
+  echo "Error: expected config file not found: $CONFIG_FILE" >&2
+  exit 1
+fi
+
+docker run \
+  -v "${ENV_FILE}":/home/appuser/.aiod/zenodo/.env \
+  -v "${CONFIG_FILE}":/home/appuser/.aiod/config.toml \
+  --network=host \
+  -it aiondemand/zenodo-connector "$@"

src/__version__.py

@@ -0,0 +1 @@
+version="0.0.0"