feat: download of artifact without version (uses lates) and group (all artifacts and their latest version)

Author: Integer-Ctrl

Dockerfile

@@ -1,6 +1,6 @@
 FROM python:3.10-slim
 
-WORKDIR /app
+WORKDIR /data
 
 COPY . .
 

README.md

@@ -25,13 +25,16 @@ Commands:
   deploy
   downoad
 ```
+
+## Docker Image Usage
+
+A docker image is available at [dbpedia/databus-python-client](https://hub.docker.com/r/dbpedia/databus-python-client). See [download section](#usage-of-docker-image) for details.
+
 ### Deploy command
 ```
 databusclient deploy --help
 ```
 ```
-
-
 Usage: databusclient deploy [OPTIONS] DISTRIBUTIONS...
 
 Arguments:
@@ -40,14 +43,14 @@ Arguments:
                     content variants of a distribution, fileExt and Compression can be set, if not they are inferred from the path  [required]
 
 Options:
-  --versionid TEXT    target databus version/dataset identifier of the form <h
+  --versionid TEXT    Target databus version/dataset identifier of the form <h
                       ttps://databus.dbpedia.org/$ACCOUNT/$GROUP/$ARTIFACT/$VE
                       RSION>  [required]
-  --title TEXT        dataset title  [required]
-  --abstract TEXT     dataset abstract max 200 chars  [required]
-  --description TEXT  dataset description  [required]
-  --license TEXT      license (see dalicc.net)  [required]
-  --apikey TEXT       apikey  [required]
+  --title TEXT        Dataset title  [required]
+  --abstract TEXT     Dataset abstract max 200 chars  [required]
+  --description TEXT  Dataset description  [required]
+  --license TEXT      License (see dalicc.net)  [required]
+  --apikey TEXT       API key  [required]
   --help              Show this message and exit.
 ```
 Examples of using deploy command
@@ -65,6 +68,93 @@ A few more notes for CLI usage:
   * For complete inferred: Just use the URL with `https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml`
   * If other parameters are used, you need to leave them empty like `https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml||yml|7a751b6dd5eb8d73d97793c3c564c71ab7b565fa4ba619e4a8fd05a6f80ff653:367116`
 
+### Download command
+```
+python3 -m databusclient downoad --help
+```
+
+```
+Usage: python3 -m databusclient download [OPTIONS] DATABUSURIS...
+
+Arguments:
+  DATABUSURIS...  databus uris to download from https://databus.dbpedia.org,
+                  or a query statement that returns databus uris from https://databus.dbpedia.org/sparql
+                  to be downloaded [required]
+
+  Download datasets from databus, optionally using vault access if vault
+  options are provided.
+
+Options:
+  --localdir TEXT  Local databus folder (if not given, databus folder
+                   structure is created in current working directory)
+  --databus TEXT   Databus URL (if not given, inferred from databusuri, e.g.
+                   https://databus.dbpedia.org/sparql)
+  --token TEXT     Path to Vault refresh token file
+  --authurl TEXT   Keycloak token endpoint URL  [default:
+                   https://auth.dbpedia.org/realms/dbpedia/protocol/openid-
+                   connect/token]
+  --clientid TEXT  Client ID for token exchange  [default: vault-token-
+                   exchange]
+  --help           Show this message and exit.        Show this message and exit.
+```
+
+Examples of using download command
+
+**File**: download of a single file
+```
+python3 -m databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01/mappingbased-literals_lang=az.ttl.bz2
+```
+
+**Version**: download of all files of a specific version
+```
+python3 -m databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01
+```
+
+**Artifact**: download of all files with latest version of an artifact
+```
+python3 -m databusclient download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals
+``` 
+
+**Group**: download of all files with lates version of all artifacts of a group
+```
+python3 -m databusclient download https://databus.dbpedia.org/dbpedia/mappings
+```
+
+If no `--localdir` is provided, the current working directory is used as base directory. The downloaded files will be stored in the working directory in a folder structure according to the databus structure, i.e. `./$ACCOUNT/$GROUP/$ARTIFACT/$VERSION/`.
+
+**Collcetion**: download of all files within a collection
+```
+python3 -m databusclient download https://databus.dbpedia.org/dbpedia/collections/dbpedia-snapshot-2022-12
+```
+
+**Query**: download of all files returned by a query (sparql endpoint must be provided with `--databus`)
+```
+python3 -m databusclient download 'PREFIX dcat: <http://www.w3.org/ns/dcat#> SELECT ?x WHERE { ?sub dcat:downloadURL ?x . } LIMIT 10' --databus https://databus.dbpedia.org/sparql
+```
+
+#### Authentication with vault
+
+For downloading files from the vault, you need to provide a vault token. See [getting-the-access-refresh-token](https://github.com/dbpedia/databus-vault-access?tab=readme-ov-file#step-1-getting-the-access-refresh-token) for details. You can come back here once you have a `vault-token.dat` file. To use it, just provide the path to the file with `--token /path/to/vault-token.dat`.
+
+Example:
+```
+python3 -m databusclient download https://databus.dbpedia.org/dbpedia-enterprise/live-fusion-snapshots/fusion/2025-08-23 --token vault-token.dat
+```
+
+If vault authentication is required for downloading a file, the client will use the token. If no vault authentication is required, the token will not be used.
+
+#### Usage of docker image
+
+A docker image is available at [dbpedia/databus-python-client](https://hub.docker.com/r/dbpedia/databus-python-client). You can use it like this:
+
+```
+docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia/mappings/mappingbased-literals/2022.12.01
+```
+If using vault authentication, make sure the token file is available in the container, e.g. by placing it in the current working directory.
+```
+docker run --rm -v $(pwd):/data dbpedia/databus-python-client download https://databus.dbpedia.org/dbpedia-enterprise/live-fusion-snapshots/fusion/2025-08-23/fusion_props=all_subjectns=commons-wikimedia-org_vocab=all.ttl.gz --token vault-token.dat
+```
+
 ## Module Usage
 
 ### Step 1: Create lists of distributions for the dataset

databusclient/cli.py

@@ -63,15 +63,15 @@ def app():
 
 @app.command()
 @click.option(
-    "--version-id",
+    "--versionid",
     required=True,
     help="Target databus version/dataset identifier of the form "
          "<https://databus.dbpedia.org/$ACCOUNT/$GROUP/$ARTIFACT/$VERSION>",
 )
 @click.option("--title", required=True, help="Dataset title")
 @click.option("--abstract", required=True, help="Dataset abstract max 200 chars")
 @click.option("--description", required=True, help="Dataset description")
-@click.option("--license-uri", required=True, help="License (see dalicc.net)")
+@click.option("--license", required=True, help="License (see dalicc.net)")
 @click.option("--apikey", required=True, help="API key")
 @click.argument(
     "distributions",

databusclient/client.py

@@ -533,10 +533,12 @@ def __handle_databus_file_query__(endpoint_url, query) -> List[str]:
         yield value
 
 
-def __handle_databus_file_json__(json_str: str) -> List[str]:
+def __handle_databus_artifact_version__(json_str: str) -> List[str]:
     """
     Parse the JSON-LD of a databus artifact version to extract download URLs.
     Don't get downloadURLs directly from the JSON-LD, but follow the "file" links to count access to databus accurately.
+
+    Returns a list of download URLs.
     """
 
     databusIdUrl = []
@@ -549,6 +551,48 @@ def __handle_databus_file_json__(json_str: str) -> List[str]:
     return databusIdUrl
 
 
+def __get_databus_latest_version_of_artifact__(json_str: str) -> str:
+    """
+    Parse the JSON-LD of a databus artifact to extract URLs of the latest version.
+
+    Returns download URL of latest version of the artifact.
+    """
+    json_dict = json.loads(json_str)
+    versions = json_dict.get("databus:hasVersion")
+    
+    # Single version case {}
+    if isinstance(versions, dict):
+        versions = [versions]
+    # Multiple versions case [{}, {}]
+
+    version_urls = [v["@id"] for v in versions if "@id" in v]
+    if not version_urls:
+        raise ValueError("No versions found in artifact JSON-LD")
+
+    version_urls.sort(reverse=True)  # Sort versions in descending order
+    return version_urls[0]  # Return the latest version URL
+
+
+def __get_databus_artifacts_of_group__(json_str: str) -> List[str]:
+    """
+    Parse the JSON-LD of a databus group to extract URLs of all artifacts.
+
+    Returns a list of artifact URLs.
+    """
+    json_dict = json.loads(json_str)
+    artifacts = json_dict.get("databus:hasArtifact", [])
+    
+    result = []
+    for item in artifacts:
+        uri = item.get("@id")
+        if not uri:
+            continue
+        _, _, _, _, version, _ = __get_databus_id_parts__(uri)
+        if version is None:
+            result.append(uri)
+    return result
+
+
 def wsha256(raw: str):
     return sha256(raw.encode('utf-8')).hexdigest()
 
@@ -558,7 +602,7 @@ def __handle_databus_collection__(uri: str) -> str:
     return requests.get(uri, headers=headers).text
 
 
-def __handle_databus_artifact_version__(uri: str) -> str:
+def __get_json_ld_from_databus__(uri: str) -> str:
     headers = {"Accept": "application/ld+json"}
     return requests.get(uri, headers=headers).text
 
@@ -607,6 +651,7 @@ def download(
     client_id: Client ID for token exchange
     """
 
+    # TODO: make pretty
     for databusURI in databusURIs:
         host, account, group, artifact, version, file = __get_databus_id_parts__(databusURI)
 
@@ -627,15 +672,31 @@ def download(
                 __download_list__([databusURI], localDir, vault_token_file=token, auth_url=auth_url, client_id=client_id)
             # databus artifact version
             elif version is not None:
-                json_str = __handle_databus_artifact_version__(databusURI)
-                res = __handle_databus_file_json__(json_str)
+                json_str = __get_json_ld_from_databus__(databusURI)
+                res = __handle_databus_artifact_version__(json_str)
                 __download_list__(res, localDir, vault_token_file=token, auth_url=auth_url, client_id=client_id)
             # databus artifact
             elif artifact is not None:
-                print("artifactId not supported yet")  # TODO
+                json_str = __get_json_ld_from_databus__(databusURI)
+                latest = __get_databus_latest_version_of_artifact__(json_str)
+                print(f"No version given, using latest version: {latest}")
+                json_str = __get_json_ld_from_databus__(latest)
+                res = __handle_databus_artifact_version__(json_str)
+                __download_list__(res, localDir, vault_token_file=token, auth_url=auth_url, client_id=client_id)
+                
             # databus group
             elif group is not None:
-                print("groupId not supported yet")  # TODO
+                json_str = __get_json_ld_from_databus__(databusURI)
+                artifacts = __get_databus_artifacts_of_group__(json_str)
+                for artifact_uri in artifacts:
+                    print(f"Processing artifact {artifact_uri}")
+                    json_str = __get_json_ld_from_databus__(artifact_uri)
+                    latest = __get_databus_latest_version_of_artifact__(json_str)
+                    print(f"No version given, using latest version: {latest}")
+                    json_str = __get_json_ld_from_databus__(latest)
+                    res = __handle_databus_artifact_version__(json_str)
+                    __download_list__(res, localDir, vault_token_file=token, auth_url=auth_url, client_id=client_id)
+
             # databus account
             elif account is not None:
                 print("accountId not supported yet")  # TODO