set maximal python version to 3.13 for pyo3 deps, and add automatic click command to markdown generation

Author: Uri Neri

CONTRIBUTING.md

@@ -73,6 +73,31 @@ Check out our [project roadmap and TODO list](https://docs.google.com/spreadshee
 2. **Benchmarking**:
    - Use `/usr/bin/time` for resource monitoring. Alternatively, hyperfine is great too but. Ideallt - use SLURM and keep track of the job IDs for later analysis with seff/pyseff.
 
+## Documentation workflow
+
+- Docs source pages are in `docs/mkdocs_docs/`.
+- Docs site navigation is configured in `docs/mkdocs.yml` (`nav:` section).
+- Command docs are under `docs/mkdocs_docs/commands/`.
+- Keep command links in `README.md` aligned with pages listed in `docs/mkdocs.yml`.
+
+Use pixi docs tasks:
+- Serve locally (live reload): `pixi run -e dev docs-serve`
+- Build static docs: `pixi run -e dev docs-build`
+- Auto-generate command help pages:
+   - create missing pages: `pixi run -e dev python src/setup/export_command_help_to_docs.py`
+   - refresh existing auto-generated pages: `pixi run -e dev python src/setup/export_command_help_to_docs.py --overwrite`
+
+For command pages that need rich/static sections (mermaid, tables, links), add a
+per-command scaffold at:
+- `src/setup/help_export_scaffolds/<command_name>.md`
+
+The exporter injects scaffold content into generated pages under **Pinned Sections**.
+
+When adding a new command page:
+1. Add the markdown page in `docs/mkdocs_docs/commands/`.
+2. Add it to `nav` in `docs/mkdocs.yml`.
+3. Add/update links in `README.md` and `docs/mkdocs_docs/commands/index.md`.
+
 ## PyPI / TestPyPI release automation
 
 Releases are automated via GitHub Actions using trusted publishing (OIDC), with this flow:

README.md

@@ -1,13 +1,15 @@
-![RolyPoly Logo](https://code.jgi.doe.gov/rolypoly/docs/-/raw/main/docs/rolypoly_logo.png?ref_type=heads)
+![RolyPoly Logo](https://raw.githubusercontent.com/UriNeri/rolypoly/main/docs/rolypoly_logo.png)
 
 # RolyPoly
 
+[![PyPI version](https://img.shields.io/pypi/v/rolypoly-tk.svg?cacheSeconds=300)](https://pypi.org/project/rolypoly-tk/) [![Python versions](https://img.shields.io/pypi/pyversions/rolypoly-tk.svg?cacheSeconds=300)](https://pypi.org/project/rolypoly-tk/) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/rolypoly-tk?period=monthly&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=Downloads+%28month%29)](https://pepy.tech/projects/rolypoly-tk) [![License](https://img.shields.io/github/license/UriNeri/rolypoly.svg)](LICENSE) [![Docs](https://img.shields.io/badge/docs-urineri.github.io%2Frolypoly-blue)](https://urineri.github.io/rolypoly/)
+
 RolyPoly is an RNA virus analysis toolkit, meant to be a "swiss-army knife" for RNA virus discovery and characterization by including a variety of commands, wrappers, parsers, automations, and some "quality of life" features for any many of a virus investigation process (from raw read processing to genome annotation). While it includes an "end-2-end" command that employs an entire pipeline, the main goals of rolypoly are:
 - Help non-computational researchers take a deep dive into their data without compromising on using tools that are non-techie friendly.  
 - Help (software) developers of virus analysis pipeline "plug" holes missing from their framework, by using specific RolyPoly commands to add features to their existing code base.
 
 ## Note - Rolypoly is still under development (contributions welcome!)
-RolyPoly is an open, still in progress project - I aim to summarise the main functionality into a manuscript ~early 2026. Pull requests and contributions are welcome and will be considered (see [CONTRIBUTING.md](CONTRIBUTING.md)).  
+RolyPoly is an open, still in progress project - I aim to summarise the main functionality into a manuscript ~mid 2026. Pull requests and contributions are welcome and will be considered (see [CONTRIBUTING.md](CONTRIBUTING.md)).  
 This also means that there are bugs, verbose logging even for non debug mode, and some place holders and TODOs here and there.
 
 ## Installation
@@ -16,10 +18,10 @@ This also means that there are bugs, verbose logging even for non debug mode, an
 **Recommended for most users** who want a "just works" solution and primarily intend to use rolypoly as a CLI tool in an independent environment.
 
 We hope to have rolypoly available from bioconda in the near future.  
-In the meantime, it can be installed with the [`quick_setup.sh`](https://code.jgi.doe.gov/rolypoly/rolypoly/-/raw/main/src/setup/quick_setup.sh) script, which will also fetch the pre-generated data rolypoly requires.
+In the meantime, it can be installed with the [`quick_setup.sh`](https://raw.githubusercontent.com/UriNeri/rolypoly/main/src/setup/quick_setup.sh) script, which will also fetch the pre-generated data rolypoly requires.
 
 ```bash
-curl -O https://code.jgi.doe.gov/rolypoly/rolypoly/-/raw/main/src/setup/quick_setup.sh && \
+curl -O https://raw.githubusercontent.com/UriNeri/rolypoly/main/src/setup/quick_setup.sh && \
 bash quick_setup.sh 
 ```
 
@@ -44,7 +46,7 @@ By default if no positional arguments are supplied, rolypoly is installed into t
 curl -fsSL https://pixi.sh/install.sh | bash
 
 # Clone the repository
-git clone https://code.jgi.doe.gov/rolypoly/rolypoly.git
+git clone https://github.com/UriNeri/rolypoly.git
 cd rolypoly
 
 # Install for specific functionality (examples):
@@ -84,11 +86,11 @@ Legend:
 
 #### Raw-Reads
 - ✅ [`filter-reads`](https://urineri.github.io/rolypoly/commands/read_processing) — Host/rRNA/adapters/artifact filtering and QC (bbmap, falco, etc.)
-- ✅ [`shrink-reads`](https://urineri.github.io/rolypoly/commands/shrink_reads) — Downsample or subsample reads. Useful for testing or normalizing coverage across samples.
-- ✅ [`mask-dna`](https://urineri.github.io/rolypoly/commands/mask_dna) — Mask DNA regions in RNA-seq reads (bbmap, seqkit). Useful for avoiding mis-filtering of RNA virus reads in because of potential matches to EVEs.
+- ✅ [`shrink-reads`](https://urineri.github.io/rolypoly/commands/read_processing) — Downsample or subsample reads. Useful for testing or normalizing coverage across samples.
+- ✅ [`mask-dna`](https://urineri.github.io/rolypoly/commands/read_processing) — Mask DNA regions in RNA-seq reads (bbmap, seqkit). Useful for avoiding mis-filtering of RNA virus reads in because of potential matches to EVEs.
 
 #### Annotation
-- ✅ [`annotate`](https://urineri.github.io/rolypoly/commands/annotate) — Genome feature annotation (wraps the rna and prot commands)
+- ✅ [`annotate`](https://urineri.github.io/rolypoly/commands/annotate_rna/#annotate-rna) — Genome feature annotation (wraps the rna and prot commands)
 - ✅ [`annotate-rna`](https://urineri.github.io/rolypoly/commands/annotate_rna) — RNA secondary structure labelling and ribozyme detection (Infernal, ViennaRNA/linearfold, cmsearch on Rfam...)
 - 🧪 [`annotate-prot`](https://urineri.github.io/rolypoly/commands/annotate_prot) — Gene calling and Protein domain annotation and functional prediction (HMMER, Pfam, custom).
 
@@ -99,22 +101,22 @@ Legend:
 #### RNA Virus Identification
 - ✅ [`marker-search`](https://urineri.github.io/rolypoly/commands/marker_search) — Search for viral markers (mainly RdRps, genomad VVs, or user-provided), using profile-based methods (HMMER / MMseqs2). 
 - ✅ [`virus-mapping`](https://urineri.github.io/rolypoly/commands/search_viruses) — Map and identify viruses using nucleic acid search (MMseqs2).
-- ✅ `rdrp-motif-search` — Search RdRp motifs (A/B/C/D) in nucleotide or amino acid sequences.
+- ✅ [`rdrp-motif-search`](https://urineri.github.io/rolypoly/commands/rdrp_motif_search) — Search RdRp motifs (A/B/C/D) in nucleotide or amino acid sequences.
 
 #### Bining / Clustering
-- 🧪 `cluster` — Average Nucleic identity (ANI) based contig gropuing. Supports several common backends and methods.
-- 🧪 `extend` — Extend sequences by pile-up/assembly. Useful for combining assemblies of with low abundance viruses, or those with high microdiversity, at the cost of worse strain/sub-species resolution (i.e. can condense to a consenus). 
-- 🧪 `termini` — Shared termini grouping and motif reporting. Writes assignments + groups tables (TSV/CSV/Parquet/JSONL) and motif FASTA by default.
-- 🧪 `correlate` — group contigs based on co-occurence, co-abundance, minimal correlation (Spearman's) of these, or both.
-- 🤔 `binit` — Combines the above commands with sample information and genome attributes (e.g. require a shared termini AND protein complementarity, like CP + RdRp). See [notebooks/Exprimental/partiti_usecase/partiti_segment_workflow_experimental.ipynb](notebooks/Exprimental/partiti_usecase/partiti_segment_workflow_experimental.ipynb) for candidate workflow.
+- 🧪 [`cluster`](https://urineri.github.io/rolypoly/commands/cluster) — Average Nucleic identity (ANI) based contig grouping. Supports several common backends and methods.
+- 🧪 [`extend`](https://urineri.github.io/rolypoly/commands/extend) — Extend sequences by pile-up/assembly. Useful for combining assemblies of with low abundance viruses, or those with high microdiversity, at the cost of worse strain/sub-species resolution (i.e. can condense to a consensus).
+- 🧪 [`termini`](https://urineri.github.io/rolypoly/commands/binning_termini) — Shared termini grouping and motif reporting. Writes assignments + groups tables (TSV/CSV/Parquet/JSONL) and motif FASTA by default.
+- 🧪 [`correlate`](https://urineri.github.io/rolypoly/commands/binning_correlate) — Group contigs based on co-occurrence, co-abundance, minimal correlation (Spearman's) of these, or both.
+- 🤔 [`binit`](https://urineri.github.io/rolypoly/commands/binit) — Combines the above commands with sample information and genome attributes (e.g. require a shared termini AND protein complementarity, like CP + RdRp). See [notebooks/Exprimental/partiti_usecase/partiti_segment_workflow_experimental.ipynb](notebooks/Exprimental/partiti_usecase/partiti_segment_workflow_experimental.ipynb) for candidate workflow.
 
 #### Miscellaneous
-- ✅ `roll` — Run an end-to-end pipeline (before v0.7.1, named `end2end`).
-- ✅ `fetch-sra` — Download SRA fastq files (from ENA)
-- ✅ `fastx-calc` — Calculate per-sequence metrics (length, GC content, hash, ...)
-- ✅ `fastx-stats` — Calculate (-->aggregate) statistics for sequences (min, max, mean, median, ...) (input is file/s)
-- ✅ `rename-seqs` — Rename sequences (add a prefix, suffix, hash, running number, etc.)
-- 🚧 `quick-taxonomy` — Quick taxonomy assignment. Candidate workflows are [github.com/UriNeri/ictv-mmseqs2-protein-database](https://github.com/UriNeri/ictv-mmseqs2-protein-database) and [github.com/apcamargo/ictv-mmseqs2-protein-database](https://github.com/apcamargo/ictv-mmseqs2-protein-database) 
+- ✅ [`roll`](https://urineri.github.io/rolypoly/commands/end_to_end) — Run an end-to-end pipeline (before v0.7.1, named `end2end`).
+- ✅ [`fetch-sra`](https://urineri.github.io/rolypoly/commands/misc) — Download SRA fastq files (from ENA)
+- ✅ [`fastx-calc`](https://urineri.github.io/rolypoly/commands/misc) — Calculate per-sequence metrics (length, GC content, hash, ...)
+- ✅ [`fastx-stats`](https://urineri.github.io/rolypoly/commands/misc) — Calculate (-->aggregate) statistics for sequences (min, max, mean, median, ...) (input is file/s)
+- ✅ [`rename-seqs`](https://urineri.github.io/rolypoly/commands/misc) — Rename sequences (add a prefix, suffix, hash, running number, etc.)
+- 🚧 [`quick-taxonomy`](https://urineri.github.io/rolypoly/commands/misc) — Quick taxonomy assignment. Candidate workflows are [github.com/UriNeri/ictv-mmseqs2-protein-database](https://github.com/UriNeri/ictv-mmseqs2-protein-database) and [github.com/apcamargo/ictv-mmseqs2-protein-database](https://github.com/apcamargo/ictv-mmseqs2-protein-database) 
 - 🤔 support for [genotate](https://github.com/deprekate/genotate) for gene prediction.
 - 🤔 Genome refinement / strain de-entalgement / variant calling?
 - 🤔 Virus feature prediction (+/-ssRNA/dsRNA, circular/linear, mono/poly-segmented, capsid type, etc.)
@@ -125,7 +127,7 @@ If you have suggestions for additional commands or features, or want to implemen
 
 ## Dependencies
 
-**📦 Modular Installation Available**: RolyPoly supports both quick setup (one environment with all dependecies for all commands) and modular installation (command-specific environments). The modular approach is particularly useful for software developers who want to integrate specific rolypoly features with minimal dependency conflicts. See the [installation documentation](./docs/docs/mkdocs_docs/installation.md) for details.
+**📦 Modular Installation Available**: RolyPoly supports both quick setup (one environment with all dependecies for all commands) and modular installation (command-specific environments). The modular approach is particularly useful for software developers who want to integrate specific rolypoly features with minimal dependency conflicts. See the [installation documentation](https://urineri.github.io/rolypoly/installation/) for details.
 
 Not all 3rd party software is used by all the different commands. RolyPoly includes a "citation reminder" that will try to list all the external software used by a command. The "reminded citations" are pretty printed to console (stdout) and to a logfile. To shut off the terminal citation reminder printing, set `ROLYPOLY_REMIND_CITATIONS` to false in your `rpconfig.json` file.
 

docs/mkdocs.yml

@@ -99,17 +99,38 @@ nav:
   - Commands:
     - Overview: commands/index.md
     - External Data: commands/prepare_external_data.md
+    - Get Data (Auto Help): commands/get_data.md
+    - Version (Auto Help): commands/version.md
     - End to End: commands/end_to_end.md
+    - Roll (Auto Help): commands/roll.md
     - Read Processing: commands/read_processing.md
+    - Filter Reads (Auto Help): commands/filter_reads.md
+    - Shrink Reads (Auto Help): commands/shrink_reads.md
+    - Mask DNA (Auto Help): commands/mask_dna.md
     - Assembly: commands/assembly.md
+    - Assemble (Auto Help): commands/assemble.md
+    - Extend (Auto Help): commands/extend.md
     - Marker Gene Search: commands/marker_search.md
+    - RdRp Motif Search (Auto Help): commands/rdrp_motif_search.md
     - Assembly Filtering: commands/filter_assembly.md
+    - Filter Contigs (Auto Help): commands/filter_contigs.md
     - Virus Search: commands/search_viruses.md
+    - Virus Mapping (Auto Help): commands/virus_mapping.md
+    - Annotate (Auto Help): commands/annotate.md
     - RNA Annotation: commands/annotate_rna.md
     - Protein Annotation: commands/annotate_prot.md
     - Host Classification: commands/host_classify.md
     - Binning:
       - Termini Analysis: commands/binning_termini.md
       - Correlation Analysis: commands/binning_correlate.md
+      - Correlate (Auto Help): commands/correlate.md
+      - Termini (Auto Help): commands/termini.md
+      - Cluster (Auto Help): commands/cluster.md
+      - Binit (Auto Help): commands/binit.md
     - Miscellaneous: commands/misc.md
+    - FASTX Calc (Auto Help): commands/fastx_calc.md
+    - FASTX Stats (Auto Help): commands/fastx_stats.md
+    - Fetch SRA (Auto Help): commands/fetch_sra.md
+    - Rename Seqs (Auto Help): commands/rename_seqs.md
+    - Quick Taxonomy (Auto Help): commands/quick_taxonomy.md
 

docs/mkdocs_docs/commands/annotate.md

@@ -0,0 +1,46 @@
+# Annotate
+
+> Auto-generated draft from CLI metadata for `rolypoly annotate`.
+> Expand this page with command-specific context, examples, and citations.
+
+## Summary
+
+Run combined RNA + protein annotation on nucleotide viral sequences.
+
+## Description
+
+This command orchestrates `annotate-rna` and `annotate-prot` into a single
+    workflow and writes results into `rna_annotation/` and
+    `protein_annotation/` subdirectories under the selected output path.
+
+    Use `--skip-steps` to disable specific stages and `--override-parameters`
+    to forward JSON overrides to sub-tools.
+
+## Usage
+
+```bash
+rolypoly annotate [OPTIONS]
+```
+
+## Options
+
+- `-i`, `--input`: Input nucleotide sequence file (fasta, fna, fa, or faa) (type: `PATH`; required; default: `Sentinel.UNSET`)
+- `-o`, `--output`: Output file location. (type: `TEXT`; default: `rolypoly_annotation`)
+- `-t`, `--threads`: Number of threads (type: `INTEGER`; default: `1`)
+- `-g`, `--log-file`: Path to log file (type: `TEXT`; default: `/clusterfs/jgi/scratch/science/metagen/neri/code/rolypoly/annotate_logfile.txt`)
+- `-M`, `--memory`: Memory in GB. Example: -M 8gb (type: `TEXT`; default: `6gb`)
+- `--override-parameters`: JSON-like string of parameters to override. Example: --override-parameters '{"RNAfold": {"temperature": 37}, "ORFfinder": {"minimum_length": 150}}' (type: `TEXT`; default: `{}`)
+- `--skip-steps`: Comma-separated list of steps to skip. Example: --skip-steps RNA_annotation,protein_annotation or --skip-steps IRESfinder,RNAMotif or --skip-steps ORFfinder,hmmsearch (type: `TEXT`; default: ``)
+- `--secondary-structure-tool`: Tool for secondary structure prediction (type: `CHOICE`; default: `LinearFold`)
+- `--ires-tool`: Tool for IRES identification (type: `CHOICE`; default: `IRESfinder`)
+- `--trna-tool`: Tool for tRNA identification (type: `CHOICE`; default: `tRNAscan-SE`)
+- `--rnamotif-tool`: Tool for RNA sequence motif identification (type: `CHOICE`; default: `lightmotif`)
+- `--gene-prediction-tool`: Tool for gene prediction (type: `CHOICE`; default: `pyrodigal`)
+- `--domain-db`: Database for domain detection (NOTE: currently packaged with rolypoly data: Pfam, genomad, RVMT) (type: `CHOICE`; default: `Pfam`)
+- `--custom-domain-db`: Path to a custom domain database in HMM format (for use with --domain-db custom) (type: `TEXT`; default: ``)
+- `--min-orf-length`: Minimum ORF length for gene prediction (type: `INTEGER`; default: `30`)
+- `--search-tool`: Tool/command for protein domain detection. hmmer commands are via pyhmmer bindings. (type: `CHOICE`; default: `hmmsearch`)
+
+
+
+