feat: add --file-availability option for disk available filtering (#163)

Author: nasseralbess

.gitignore

@@ -106,3 +106,6 @@ venv.bak/
 
 # _cache folder with archived websites
 cache
+
+# Claude Code
+.claude

.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "1.0.1"
+  ".": "1.0.2"
 }

CHANGELOG.md

@@ -2,6 +2,43 @@
 
 # Changelog
 
+## [1.0.2](https://github.com/cernopendata/cernopendata-client/compare/1.0.1...1.0.2) (2025-12-17)
+
+
+### Bug fixes
+
+* **download:** use record IDs for all local data paths ([#167](https://github.com/cernopendata/cernopendata-client/issues/167)) ([9b97f7c](https://github.com/cernopendata/cernopendata-client/commit/9b97f7caebf2c049f2734ec1af399c6aeb027341)), closes [#166](https://github.com/cernopendata/cernopendata-client/issues/166)
+* **verifier:** zero-pad Adler32 checksums to 8 hex characters ([#169](https://github.com/cernopendata/cernopendata-client/issues/169)) ([b6daa50](https://github.com/cernopendata/cernopendata-client/commit/b6daa508c3d190585f84853687ff0cfb8acbe792))
+
+
+### Code refactoring
+
+* **searcher:** remove Python 2 compatibility code ([#170](https://github.com/cernopendata/cernopendata-client/issues/170)) ([cc3bb8b](https://github.com/cernopendata/cernopendata-client/commit/cc3bb8b80b6cf383168e0edf93ae17fafe65ab5c))
+* **validator:** remove Python 2 compatibility code ([#170](https://github.com/cernopendata/cernopendata-client/issues/170)) ([b03de4f](https://github.com/cernopendata/cernopendata-client/commit/b03de4fb7519864a1e0f1f1380467007c5a73b17))
+
+
+### Test suite
+
+* **conftest:** add fixture for automatic directory cleanup ([#168](https://github.com/cernopendata/cernopendata-client/issues/168)) ([b847121](https://github.com/cernopendata/cernopendata-client/commit/b84712178cd143d86d66642b09e45d1ffdd9f6b0))
+* **conftest:** add shared CLI runner fixture ([#168](https://github.com/cernopendata/cernopendata-client/issues/168)) ([b387fde](https://github.com/cernopendata/cernopendata-client/commit/b387fde1569c7e81c825fe3b8601a4cbefe51fbd))
+* **downloader:** add unit tests for file filtering functions ([#170](https://github.com/cernopendata/cernopendata-client/issues/170)) ([a93d2be](https://github.com/cernopendata/cernopendata-client/commit/a93d2be9528016868a06cf0622c13f83493a834d))
+* **get-metadata:** add test for filter without output-value ([#170](https://github.com/cernopendata/cernopendata-client/issues/170)) ([7c65a79](https://github.com/cernopendata/cernopendata-client/commit/7c65a79de09b675847067db7c6f036cf0f62588c))
+* **global:** add [@pytest](https://github.com/pytest).mark.local marker for local-only tests ([#170](https://github.com/cernopendata/cernopendata-client/issues/170)) ([59fb1da](https://github.com/cernopendata/cernopendata-client/commit/59fb1da2a4ffd6160ad7cd8ce291d88c4774e807))
+* **list-directory:** add test for empty directory ([#170](https://github.com/cernopendata/cernopendata-client/issues/170)) ([22e75e5](https://github.com/cernopendata/cernopendata-client/commit/22e75e56cab423905d701a4de05295fcfd319d3d))
+* **metadater:** add tests for filter edge cases ([#170](https://github.com/cernopendata/cernopendata-client/issues/170)) ([b1855b9](https://github.com/cernopendata/cernopendata-client/commit/b1855b9e12b1c8365cdcf1c3f6e73227dedbfba6))
+* **validator:** correct typos in test assertions ([#170](https://github.com/cernopendata/cernopendata-client/issues/170)) ([07b7b00](https://github.com/cernopendata/cernopendata-client/commit/07b7b00376921288fbb3aed66fac9ce04866858c))
+
+
+### Continuous integration
+
+* **commitlint:** fix local running of commit linter on macOS ([#168](https://github.com/cernopendata/cernopendata-client/issues/168)) ([6f411b1](https://github.com/cernopendata/cernopendata-client/commit/6f411b1b4ed859b18a54365d43a0f748dbfa7c9c))
+
+
+### Documentation
+
+* **claude:** add initial Claude Code configuration ([#164](https://github.com/cernopendata/cernopendata-client/issues/164)) ([7d14b38](https://github.com/cernopendata/cernopendata-client/commit/7d14b3845b859af2f2282f565f1c040898adc7f6))
+* **claude:** expand file change instructions for all file types ([#168](https://github.com/cernopendata/cernopendata-client/issues/168)) ([5652939](https://github.com/cernopendata/cernopendata-client/commit/5652939b50c1b0dd60471961e80476dcad64ab0a))
+
 ## [1.0.1](https://github.com/cernopendata/cernopendata-client/compare/1.0.0...1.0.1) (2025-11-10)
 
 

CLAUDE.md

@@ -0,0 +1,147 @@
+# CLAUDE.md
+
+This file provides guidance for Claude Code when working with this repository.
+
+## Project Overview
+
+`cernopendata-client` is a command-line tool to download files from the CERN
+Open Data portal. It enables querying datasets and downloading/verifying
+individual data files.
+
+## Development Setup
+
+```bash
+# Using mise (recommended) - installs Python versions 3.8-3.14
+mise install
+
+# Create virtual environment and install in editable mode
+python3 -m venv env
+source env/bin/activate
+pip install -e '.[tests]'
+```
+
+## Testing and Code Quality
+
+The project uses `run-tests.sh` for all quality checks. Run all checks with:
+
+```bash
+./run-tests.sh
+```
+
+### Individual Check Commands
+
+| Command                               | Description                            |
+| ------------------------------------- | -------------------------------------- |
+| `./run-tests.sh --python-tests`       | Run pytest test suite (all tests)      |
+| `./run-tests.sh --python-tests-local` | Run pytest test suite (local only)     |
+| `./run-tests.sh --format-black`       | Check Python formatting (black)        |
+| `./run-tests.sh --lint-flake8`        | Lint Python code (flake8)              |
+| `./run-tests.sh --lint-pydocstyle`    | Check Python docstrings                |
+| `./run-tests.sh --format-prettier`    | Check Markdown/YAML formatting         |
+| `./run-tests.sh --format-shfmt`       | Check shell script formatting          |
+| `./run-tests.sh --lint-shellcheck`    | Lint shell scripts                     |
+| `./run-tests.sh --lint-markdownlint`  | Lint Markdown files                    |
+| `./run-tests.sh --lint-yamllint`      | Lint YAML files                        |
+| `./run-tests.sh --lint-jsonlint`      | Lint JSON files                        |
+| `./run-tests.sh --lint-commitlint`    | Check commit message format            |
+| `./run-tests.sh --lint-manifest`      | Check Python manifest (check-manifest) |
+| `./run-tests.sh --docs-sphinx`        | Build and test Sphinx documentation    |
+| `./run-tests.sh --docker-build`       | Build Docker image                     |
+| `./run-tests.sh --docker-tests`       | Run tests in Docker container          |
+| `./run-tests.sh --lint-hadolint`      | Lint Dockerfile                        |
+
+### Using tox for Multi-Python Testing
+
+```bash
+tox           # Run tests across all Python versions (3.8-3.14)
+tox -e py312  # Run tests for specific Python version
+```
+
+### Local vs Remote Tests
+
+Tests that run locally without network access are marked with
+`@pytest.mark.local`. Use pytest markers or run-tests.sh to run subsets:
+
+```bash
+# Run only local tests (no network access)
+./run-tests.sh --python-tests-local
+pytest -m local
+
+# Run only remote tests (connect to CERN Open Data portal)
+pytest -m "not local"
+
+# Run all tests
+./run-tests.sh --python-tests
+pytest
+```
+
+## Project Structure
+
+- `cernopendata_client/` - Main Python package
+  - `cli.py` - Command-line interface (Click-based)
+  - `downloader.py` - File download functionality
+  - `searcher.py` - Dataset search functionality
+  - `verifier.py` - File verification
+  - `validator.py` - Input validation
+  - `metadater.py` - Metadata handling
+  - `config.py` - Configuration
+  - `printer.py` - Output formatting
+  - `utils.py` - Utility functions
+  - `walker.py` - Directory traversal
+- `tests/` - Test suite
+- `docs/` - Sphinx documentation
+
+## Code Style
+
+- Python formatting: black
+- Python linting: flake8
+- Docstrings: pydocstyle (ignores D413, D301)
+- Commit messages: conventional commits (commitlint)
+- Shell scripts: shellcheck + shfmt
+- Markdown/YAML/JSON: prettier + markdownlint + yamllint + jsonlint
+
+## When Making Changes
+
+When modifying files in this repository:
+
+1. **Update copyright years**: If a file has a copyright header, add the current
+   year to the list if not already present (e.g.
+   `Copyright (C) 2019, 2020, 2021, 2023, 2025 CERN.`)
+
+2. **Python files** (`.py`): Run `./run-tests.sh --format-black` to check
+   formatting, and `./run-tests.sh --lint-flake8` and
+   `./run-tests.sh --lint-pydocstyle` to check linting
+
+3. **Shell scripts** (`.sh`): Run `./run-tests.sh --format-shfmt` and
+   `./run-tests.sh --lint-shellcheck` to check formatting and linting
+
+4. **Markdown files** (`.md`): Run `./run-tests.sh --format-prettier` and
+   `./run-tests.sh --lint-markdownlint` to check formatting and linting
+
+5. **YAML files** (`.yml`, `.yaml`): Run `./run-tests.sh --format-prettier` and
+   `./run-tests.sh --lint-yamllint` to check formatting and linting
+
+6. **JSON files** (`.json`): Run `./run-tests.sh --format-prettier` and
+   `./run-tests.sh --lint-jsonlint` to check formatting and linting
+
+7. **Dockerfile**: Run `./run-tests.sh --lint-hadolint` to check linting, and
+   `./run-tests.sh --docker-build` to verify the Docker image builds correctly
+
+8. **Documentation** (`docs/`): Run `./run-tests.sh --docs-sphinx` to verify
+   Sphinx documentation builds correctly
+
+9. **Python package structure** (`setup.py`, `MANIFEST.in`): Run
+   `./run-tests.sh --lint-manifest` to verify the package manifest is correct
+
+10. **Commit messages**: Run `./run-tests.sh --lint-commitlint` to verify commit
+    messages follow conventional commits format
+
+11. **Tests** (`tests/`): Run `./run-tests.sh --python-tests-local` for local
+    tests without network access, or `./run-tests.sh --python-tests` for all
+    tests including those that connect to the CERN Open Data portal
+
+## Key Notes
+
+- License: GPLv3
+- Python support: 3.8 - 3.14
+- Main branch: `master`

Dockerfile

@@ -1,6 +1,6 @@
 # This file is part of cernopendata-client.
 #
-# Copyright (C) 2020, 2022, 2023, 2024 CERN.
+# Copyright (C) 2020, 2022, 2023, 2024, 2025 CERN.
 #
 # cernopendata-client is free software; you can redistribute it and/or modify
 # it under the terms of the GPLv3 license; see LICENSE file for more details.
@@ -77,5 +77,5 @@ LABEL org.opencontainers.image.title="cernopendata-client"
 LABEL org.opencontainers.image.url="https://github.com/cernopendata/cernopendata-client"
 LABEL org.opencontainers.image.vendor="cernopendata"
 # x-release-please-start-version
-LABEL org.opencontainers.image.version="1.0.1"
+LABEL org.opencontainers.image.version="1.0.2"
 # x-release-please-end

cernopendata_client/__init__.py

@@ -2,7 +2,7 @@
 #
 # This file is part of cernopendata-client.
 #
-# Copyright (C) 2019 CERN.
+# Copyright (C) 2019, 2020, 2025 CERN.
 #
 # cernopendata-client is free software; you can redistribute it and/or modify
 # it under the terms of the GPLv3 license; see LICENSE file for more details.

cernopendata_client/cli.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 # This file is part of cernopendata-client.
 #
-# Copyright (C) 2019, 2020, 2021, 2023 CERN.
+# Copyright (C) 2019, 2020, 2021, 2023, 2025 CERN.
 #
 # cernopendata-client is free software; you can redistribute it and/or modify
 # it under the terms of the GPLv3 license; see LICENSE file for more details.
@@ -70,9 +70,9 @@ def version():
 
 
 @cernopendata_client.command()
-@click.option("--recid", type=click.INT, help="Record ID")
-@click.option("--doi", help="Digital Object Identifier")
-@click.option("--title", help="Title of the record")
+@click.option("--recid", type=click.INT, help="Record ID (exact match)")
+@click.option("--doi", help="Digital Object Identifier (exact match)")
+@click.option("--title", help="Record title (exact match, no wildcards)")
 @click.option(
     "--output-value",
     is_flag=False,
@@ -147,9 +147,9 @@ def get_metadata(server, recid, doi, title, output_value, filters):
 
 
 @cernopendata_client.command()
-@click.option("--recid", type=click.INT, help="Record ID")
-@click.option("--doi", help="Digital Object Identifier")
-@click.option("--title", help="Record title")
+@click.option("--recid", type=click.INT, help="Record ID (exact match)")
+@click.option("--doi", help="Digital Object Identifier (exact match)")
+@click.option("--title", help="Record title (exact match, no wildcards)")
 @click.option(
     "--protocol",
     default="http",
@@ -230,9 +230,9 @@ def _validate_and_load(server, recid, doi, title, retry_limit, retry_sleep):
 
 
 @cernopendata_client.command()
-@click.option("--recid", type=click.INT, help="Record ID")
-@click.option("--doi", help="Digital Object Identifier")
-@click.option("--title", help="Record title")
+@click.option("--recid", type=click.INT, help="Record ID (exact match)")
+@click.option("--doi", help="Digital Object Identifier (exact match)")
+@click.option("--title", help="Record title (exact match, no wildcards)")
 @click.option(
     "--protocol",
     default="http",
@@ -347,6 +347,7 @@ def download_files(
     record_json = _validate_and_load(
         server, recid, doi, title, retry_limit, retry_sleep
     )
+    record_recid = record_json["metadata"]["recid"]
     file_locations_info = get_files_list(server, record_json, protocol, expand)
     if expand:
         if not file_availability and any(f[3] != "online" for f in file_locations_info):
@@ -402,7 +403,7 @@ def download_files(
         sys.exit(0)
 
     total_files = len(download_file_locations)
-    path = str(recid)
+    path = record_recid
     if not os.path.isdir(path):
         try:
             os.mkdir(path)
@@ -439,11 +440,11 @@ def download_files(
         if verify:
             file_info_remote = get_file_info_remote(
                 server,
-                recid,
+                record_recid,
                 protocol=protocol,
                 filtered_files=[file_location],
             )
-            file_info_local = get_file_info_local(recid)
+            file_info_local = get_file_info_local(record_recid)
             verify_file_info(file_info_local, file_info_remote)
     display_message(
         msg_type="info",
@@ -452,19 +453,21 @@ def download_files(
 
 
 @cernopendata_client.command()
-@click.option("--recid", type=click.INT, help="Record ID")
+@click.option("--recid", type=click.INT, help="Record ID (exact match)")
+@click.option("--doi", help="Digital Object Identifier (exact match)")
+@click.option("--title", help="Record title (exact match, no wildcards)")
 @click.option(
     "--server",
     default=SERVER_HTTP_URI,
     type=click.STRING,
     help="Which CERN Open Data server to query? [default={}]".format(SERVER_HTTP_URI),
 )
-def verify_files(server, recid):
+def verify_files(server, recid, doi, title):
     """Verify downloaded data file integrity.
 
-    Select a CERN Open Data bibliographic record by a record ID and
-    verify integrity of downloaded data files belonging to this
-    record.
+    Select a CERN Open Data bibliographic record by a record ID, a
+    DOI, or a title and verify integrity of downloaded data files
+    belonging to this record.
 
     Examples: \n
     \t $ cernopendata-client verify-files --recid 5500
@@ -474,24 +477,28 @@ def verify_files(server, recid):
     if recid is not None:
         validate_recid(recid)
 
+    # Get record metadata and resolve recid from DOI/title if needed
+    record_json = get_record_as_json(server, recid, doi, title)
+    record_recid = record_json["metadata"]["recid"]
+
     # Get remote file information
-    file_info_remote = get_file_info_remote(server, recid)
+    file_info_remote = get_file_info_remote(server, record_recid)
 
     # Get local file information
-    file_info_local = get_file_info_local(recid)
+    file_info_local = get_file_info_local(record_recid)
     if not file_info_local:
         display_message(
             msg_type="error",
             msg="No local files found for record {}. Perhaps run `download-files` first? Exiting.".format(
-                recid
+                record_recid
             ),
         )
         sys.exit(1)
 
     # Verify number of files
     display_message(
         msg_type="info",
-        msg="Verifying number of files for record {}... ".format(recid),
+        msg="Verifying number of files for record {}... ".format(record_recid),
     )
     display_message(
         msg_type="note",

cernopendata_client/metadater.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 # This file is part of cernopendata-client.
 #
-# Copyright (C) 2020 CERN.
+# Copyright (C) 2023, 2025 CERN.
 #
 # cernopendata-client is free software; you can redistribute it and/or modify
 # it under the terms of the GPLv3 license; see LICENSE file for more details.