kira-biodata-manager (kira-bm)

Reproducible bio-data manager with a project-local store and a shared global cache. kira-bm it's like npm/cargo/pip for bioinformatics.

Prerequisites

• Rust 1.95+ (Edition 2024)
• Network access to NCBI (Datasets, E-utilities, GEO), RCSB PDB, UniProt, ENA, Ensembl, AlphaFold DB, InterPro/Pfam, GO, KEGG, Reactome

Installation

Install from crates.io (Rust 1.95+ / Windows / Linux / MacOS):

cargo install kira-biodata-manager

Or build from source:

cargo build --release

Config (kira-bm.json)

It's like package.json/Cargo.toml

{
  "schema_version": 1,
  "proteins": [
    "1LYZ",
    { "id": "4V6X", "format": "cif" }
  ],
  "genomes": [
    {
      "accession": "GCF_000005845.2",
      "include": ["genome", "gff3", "protein", "seq-report"]
    }
  ],
  "srr": [
    "SRR014966",
    { "id": "DRR051805", "format": "fasta", "paired": true }
  ],
  "uniprot": [
    "P69905",
    { "id": "P04637-2" }
  ],
  "doi": [
    "10.1038/s41586-020-2649-2"
  ],
  "expression": [
    "GSE102902"
  ],
  "expression10x": [
    "GSE128033"
  ],
  "alphafold": ["P69905"],
  "ensembl": ["homo_sapiens@113"],
  "pfam": ["PF00001"],
  "knowledge": ["go", "go@2024-01-17", "kegg", "reactome"]
}

Notes:

• schema_version defaults to 1 if omitted. Unknown keys are rejected (a typo never silently drops datasets).
• Protein format defaults to cif. Supported: cif, pdb, bcif. Large entries (e.g. 4V6X) exist only as cif/bcif; asking for pdb fails with a clear error. Extended wwPDB ids (pdb_00001lyz) are accepted.
• Genome include defaults to ["genome","gff3","protein","seq-report"] and is part of the cache identity: a cached copy with a different include is not reused.
• SRR accepts SRR, ERR and DRR runs. Format defaults to fastq; output is always gzip-compressed (reads.fastq.gz or reads_1.fastq.gz/reads_2.fastq.gz, plus reads_unpaired.fastq.gz when present). Paired layout is detected automatically; paired: true only asserts it.
• UniProt accepts accessions like P69905 and isoforms like P04637-2.
• DOI accepts 10.…, doi:10.… or https://doi.org/10.… (normalised to lowercase).
• expression downloads series-level supplementary files; add --samples to include per-sample files. expression10x downloads every complete 10x triplet (*matrix.mtx.gz + *barcodes.tsv.gz + *features/genes.tsv.gz), which on GEO are per-sample files with a GSM…_ prefix.
• knowledge: go (current), go@YYYY-MM-DD (pinned release from release.geneontology.org), kegg, reactome. KEGG's REST API is for academic use only; the licence note is recorded in the dataset metadata.

Usage

Fetch from config (kira-bm.json) in the current directory:

kira-bm fetch
or just
kira-bm

In case you have no kira-bm.json file in project and run kira-bm -- you'll see an interactive terminal user interface.

Fetch a specific dataset (add dataset to project's dataset directory):

kira-bm fetch protein:1LYZ
kira-bm fetch protein:1LYZ --format bcif
kira-bm fetch genome:GCF_000005845.2
kira-bm fetch srr:SRR014966
kira-bm fetch uniprot:P69905
kira-bm fetch alphafold:P69905
kira-bm fetch ensembl:homo_sapiens@113
kira-bm fetch pfam:PF00001
kira-bm fetch expression:GSE102902
kira-bm fetch expression10x:GSE128033
kira-bm fetch go
kira-bm fetch go@2024-01-17
kira-bm fetch kegg
kira-bm fetch reactome

Routing:

• Protein structures (protein:<PDB_ID>) are fetched from RCSB PDB (bcif from models.rcsb.org).
• Genomes are fetched from NCBI Datasets; the embedded md5sum.txt is verified after extraction.
• SRR/ERR/DRR runs are fetched from the ENA FASTQ mirror first (MD5-verified .fastq.gz, no external tools); runs that ENA does not mirror fall back to the NCBI SRA Toolkit (prefetch + fasterq-dump --split-3).
• UniProt accessions (uniprot:<ACCESSION>) are fetched from UniProt.
• AlphaFold models come from the AlphaFold DB prediction API (latest version, every fragment, mmCIF + PDB + PAE).
• Ensembl genomes come from the Ensembl FTP (primary_assembly FASTA when available, otherwise toplevel, plus GTF).
• DOI-based discovery (doi:<DOI>) resolves metadata via Crossref and hydrates public dataset IDs.
• GEO expression datasets (expression:<GSE>, expression10x:<GSE>) are fetched from NCBI GEO.
• Knowledge bases (go, kegg, reactome) are fetched from their official sources.

A batch fetch continues past failures: every item reports action: "error" with the message, the summary counts them, and the exit code is 3 when at least one dataset failed.

List datasets (JSON in non-interactive mode):

kira-bm list --non-interactive

Show dataset info:

kira-bm info protein:1LYZ --non-interactive

Verify the project against kira-bm.lock, or clear the project store:

kira-bm verify
kira-bm clear

Export for workflow engines:

kira-bm export --target manifest  --output kira-bm.manifest.json
kira-bm export --target snakemake --output Snakefile.kira
kira-bm export --target nextflow  --output samplesheet.csv

All exported paths are relative to the project directory.

Reproducibility

• kira-bm.lock records, per dataset and format, a SHA-256 digest, the upstream release when known (PDB release date, UniProt entry version, AlphaFold model version, GO/KEGG/Reactome release, Ensembl release, assembly version) and the request options (genome include, GEO samples).
• Paths in the lockfile, in metadata/ records and in exports are relative to the project directory, so a project can be moved or checked out elsewhere and still verify.
• Directory digests skip kira-bm's own metadata.json (timestamps, tool versions), so re-downloading identical data yields the same digest.
• Unchanged datasets are not re-hashed on subsequent runs.

Environment variables

Variable	Meaning
KIRA_BM_CACHE_DIR	Global cache location (default $XDG_CACHE_HOME/kira-biodata-manager or ~/.cache/kira-biodata-manager). Useful on HPC with a shared cache.
KIRA_BM_LINK_MODE	hardlink (default) or copy. With hard links the project store and the cache share file contents; treat dataset files as read-only.
KIRA_BM_DOWNLOAD_JOBS	Parallel downloads for multi-file datasets (GEO), default 4.
NCBI_API_KEY	NCBI API key (10 req/s instead of 3), sent to Datasets and E-utilities.
NCBI_EMAIL	Contact e-mail passed to E-utilities as recommended by NCBI.
KIRA_BM_CONFIG_DIR	Location of settings.toml (download mirrors).

Concurrent kira-bm processes (e.g. Snakemake jobs sharing a cache) are safe: each dataset is locked while it is being materialised and the project store has its own lock.

Every download applies the mirrors from kira-bm settings add-mirror <from-prefix> <to-prefix>, retries transient failures with backoff (Retry-After honoured), resumes interrupted transfers with HTTP ranges and checks Content-Length/Content-Range, plus MD5 where the provider publishes one (ENA, NCBI).

DOI-driven dataset discovery

kira-bm can resolve a DOI into public repository identifiers (GEO/SRA/BioProject/Assembly/PDB/UniProt) using structured metadata from Crossref, then hydrate and download the resolved datasets.

What it does:

• Resolves Crossref metadata (title/abstract/references/links).
• Extracts known identifiers via strict regex matching (PDB candidates must be labelled or upper-case and unit-like tokens such as 10mM are ignored).
• Validates identifiers using public APIs; service errors are reported, never silently treated as "not found".
• Hydrates hierarchies (GSE -> SRR via batched E-link, falling back to GSM walks; BioProject -> SRR/assemblies).
• Writes doi_resolution.json provenance to the project store.

What it does NOT do:

• No PDF parsing.
• No publisher HTML scraping.
• No fuzzy matching or probabilistic inference.

Example:

kira-bm fetch doi:10.1038/s41586-020-2649-2

Storage layout

Project store (./.kira-bm/) and global cache share the same layout:

proteins/<ID>/<ID>.<ext>            + metadata.json, metadata.raw.json
genomes/<ACCESSION>/...             (NCBI Datasets archive, md5-verified)
srr/<RUN>/reads.fastq.gz            or reads_1.fastq.gz, reads_2.fastq.gz[, reads_unpaired.fastq.gz]
srr/<RUN>/metadata.json
uniprot/<ACCESSION>/<ACCESSION>.fasta, metadata.json, raw.json
alphafold/<ACCESSION>/AF-<ACC>-F<n>-model_v<k>.{cif,pdb,pae.json}, metadata.json
ensembl/<species>@<release>/genome.fa.gz, annotation.gtf.gz, metadata.json
pfam/<PFxxxxx>/<PFxxxxx>.json, <PFxxxxx>.hmm.gz
expression/<GSE>/...                + metadata/metadata.json, metadata/geo_soft.txt
expression10x/<GSE>/...             (10x bundles preserved)
knowledge/go/go-basic.obo           (or knowledge/go@YYYY-MM-DD/)
knowledge/kegg/..., knowledge/reactome/...
doi/<ENCODED_DOI>/doi_resolution.json   (project only)
metadata/<TYPE>/<ID>[.<format>].json    (index records; paths relative to the store root)

Output contracts

--non-interactive mode:

• list and info output JSON to stdout.
• fetch and clear output JSON summaries; failed items carry action: "error".
• Errors go to stderr with non-zero exit codes: 1 usage/local error, 2 not found / nothing to do, 3 remote service or external tool failure (retry later).

Optional external tools

kira-bm may optionally invoke externally installed third-party tools (e.g. NCBI SRA Toolkit https://github.com/ncbi/sra-tools ) for runs that ENA does not mirror. These tools are not bundled, not distributed, and are subject to their own licenses. Users are responsible for installing and complying with those licenses.