revamp gtars docs

Author: nsheff

docs/gtars/README.md

@@ -4,10 +4,83 @@
 
 
 <p align="center">
-<a href="https://pypi.org/project/gtars"><img src="https://img.shields.io/pypi/v/geniml" alt=""></a>
+<a href="https://pypi.org/project/gtars"><img src="https://img.shields.io/pypi/v/gtars" alt=""></a>
+<a href="https://crates.io/crates/gtars"><img src="https://img.shields.io/crates/v/gtars?&logo=rust" alt="crates.io"></a>
 <a href="https://github.com/databio/gtars"><img src="https://img.shields.io/badge/source-github-354a75?logo=github"></a>
 </p>
 
-Gtars is a Rust package with Python bindings for genomic interval analysis.
 
-Coming soon!
+
+## Introduction
+
+`gtars` is a high-performance toolkit for *genomic tools and algorithms in Rust*. Built with Rust for speed and reliability, gtars provides core utilies for machine learning on genomic intervals for the [geniml](https://github.com/databio/geniml) Python package. It also provides lots of utility as a standalone library for alternative downstream use cases.
+
+## Installation
+
+### Rust Library
+
+Gtars uses a feature-flag system to allow you to include only the modules you need. Add to your `Cargo.toml`:
+
+```toml
+[dependencies]
+# Install specific features
+gtars = { version = "0.5", features = ["overlaprs", "tokenizers"] }
+
+# Or install from GitHub
+gtars = { git = "https://github.com/databio/gtars", features = ["overlaprs", "tokenizers"] }
+```
+
+Modules:
+
+- `core` - Core functionality and data structures
+- `tokenizers` - Genomic region tokenizers
+- `io` - I/O utilities
+- `refget` - Reference sequence access
+- `overlaprs` - Overlap operations
+- `uniwig` - Coverage computation
+- `igd` - Interval search
+- `bbcache` - BED file caching
+- `scoring` - Fragment scoring
+- `fragsplit` - Fragment splitting
+
+
+Example combinations:
+
+```toml
+# For machine learning tasks
+gtars = { version = "0.5", features = ["tokenizers", "core"] }
+
+# For genomic analysis
+gtars = { version = "0.5", features = ["overlaprs", "uniwig", "scoring"] }
+
+# For data access
+gtars = { version = "0.5", features = ["refget", "bbcache", "io"] }
+```
+
+### Python Package
+
+```bash
+pip install gtars
+```
+
+See further documentation under Python bindings.
+
+### Command-Line Interface
+
+Install from source:
+```bash
+git clone https://github.com/databio/gtars
+cd gtars
+cargo install --path gtars-cli
+```
+
+Or download precompiled binaries from the [releases page](https://github.com/databio/gtars/releases).
+
+
+## Development
+
+Run tests with `cargo test` from the workspace root. Please see [CONTRIBUTING.md](https://github.com/databio/gtars/blob/master/CONTRIBUTING.md) for development guidelines.
+
+## Module organization
+
+`gtars` is organized into modules. The modules section gives an [overview of each module](modules.md).

docs/gtars/changelog.md

@@ -5,36 +5,36 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
-## [0.3.0]
+## [0.3.0] -- 2023-07-30
 - move digests functionality to refget
 - add RefgetStore to refget and its associated python bindings
 - integrate support for `bits` + backend types for tokenizers (AIList or BITS)
 - reworked the tokenization CLI to support the new `bits` and `backend` options
 
-## [0.2.5]
+## [0.2.5] -- 2023-04-06
 - Rework tokenizer API to be more consistent with the HuggingFace tokenizers API.
 - Updates to `RegionSet` to improve performance and usability.
 - Added file_digest function to RegionSet struct
 - Fixed reqwest error in R bindings
 - Fixed [#107](https://github.com/databio/gtars/issues/107)
 
-## [0.2.4]
+## [0.2.4] -- 2023-03-05
 - Attempt to fix failing python bindings in CI linux [#104](https://github.com/databio/gtars/issues/104)
 
-## [0.2.3]
+## [0.2.3] -- 2023-03-05
 - Improved RegionSet, by adding a multiple new methods: `to_bed`, `to_bed_gz`, `to_bigbed`, `identifier()`, and others.
 - Fixed allowed `fasta_digest` to accept `Path` or `bytes` [#93](https://github.com/databio/gtars/issues/93)
 
-## [0.2.2]
+## [0.2.2] -- 2023-02-18
 - fix [#90](https://github.com/databio/gtars/issues/90)
 - fix [#89](https://github.com/databio/gtars/issues/89)
 
 
-## [0.2.1] 
+## [0.2.1] -- 2024-02-11 
 - allow comments at the beginning of fragment files
 - bump bigtools to 0.5.5, fixing [#74](https://github.com/databio/gtars/issues/74) and [#77](https://github.com/databio/gtars/issues/77)
 
-## [0.2.0] 
+## [0.2.0] -- 2024-01-13 
 - add position shift workflow for bam to bw (was previously added for bam to bed)
 - add scaling argument for bam to bw workflow [#53](https://github.com/databio/gtars/issues/53)
 - fix accumulation issue for bam workflow [#56](https://github.com/databio/gtars/issues/56)
@@ -58,10 +58,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - fix IGD overlap issue [#45](https://github.com/databio/gtars/issues/45)
 - add ga4gh refget digest functionality [#58](https://github.com/databio/gtars/pull/58)
 
-## [0.1.1]
+## [0.1.1] -- 2023-12-03
 - hot fix for broken python bindings; remove IGD from the python bindings for now
 
-## [0.1.0]
+## [0.1.0] -- 2023-12-03
 - Rust implementation of `uniwig` that expands on the C++ version
   - Uniwig now accepts a single sorted  `.bed` file, `.narrowPeak` file, or `.bam` file.
   - Outputs now include  `.wig`, `.npy`, `.bedGraph`, and `.bw`
@@ -70,64 +70,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Region scoring matrix calculation for region clustering
 - Fragment file splitter for pseudobulking
 
-## [0.0.15]
+## [0.0.15] -- 2023-07-29
 -  added meta tokenization tools and a new `MetaTokenizer` struct that can be used to tokenize regions using the meta-token strategy.
 -  added some annotations to the `pyo3` `#[pyclass]` and `#[pymethods]` attributes to make the python bindings more readable.
 
-## [0.0.14]
+## [0.0.14] -- 2023-06-11
 - renamed repository to `gtars` to better reflect the project's goals.
 
-## [0.0.13]
+## [0.0.13] -- 2023-06-03
 - implemented a fragment file tokenizer that will generate `.gtok` files directly from `fragments.tsv.gz` files.
 - fix an off-by-one error in the `region-to-id` maps in the `Universe` structs. This was leading to critical bugs in our models.
 
-## [0.0.12]
+## [0.0.12] -- 2023-05-28
 - optimize creation of `PyRegionSet` to reduce expensive cloning of `Universe` structs.
 
-## [0.0.11]
+## [0.0.11] -- 2023-05-22
 - redesigned API for the tokenizers to better emulate the huggingface tokenizers API.
 - implemented new traits for tokenizers to allow for more flexibility when creating new tokenizers.
 - bumped the version `pyo3` to `0.21.0`
 - added `rust-numpy` dependency to the python bindings for exporting tokenized regions as numpy arrays.
 - overall stability improvements to the tokenizers and the python bindings.
 
-## [0.0.10]
+## [0.0.10] -- 2024-01-24
 - update file format specifications
 
-## [0.0.9]
+## [0.0.9] -- 2024-01-22
 - start working on the concept of a `.gtok` file-format to store tokenized regions
 - added basic readers and writers for this format
 
-## [0.0.8]
+## [0.0.8] -- 2024-01-17
 - add a new `ids_as_strs` getter to the `TokenizedRegionSet` struct so that we can get the ids as strings quickly, this is meant mostly for interface with geniml.
 
-## [0.0.7]
+## [0.0.7] -- 2023-11-30
 - move things around based on rust club feedback
 
-## [0.0.6]
+## [0.0.6] -- 2024-02-20
 - update python bindings to support the module/submodule structure (https://github.com/PyO3/pyo3/issues/759#issuecomment-1828431711)
 - change name of some submodules
 - remove `consts` submodule, just add to base
 - expose a `__version__` attribute in the python bindings
 
-## [0.0.5]
+## [0.0.5] -- 2024-02-19
 - add many "core utils"
 - move `gtokenizers` into this package inside `gtars::tokenizers`
 - create `tokenize` cli
 - add tests for core utils and tokenizers
 - RegionSet is now backed by a polars DataFrame
 - new python bindings for core utils and tokenizers
 
-## [0.0.4]
+## [0.0.4] -- 2023-11-06
 - add type annotations to the python bindings
 
-## [0.0.3]
+## [0.0.3] -- 2023-11-06
 - work on python bindings initialization
 
-## [0.0.2]
+## [0.0.2] -- 2023-09-20
 - prepare for first release
 
-## [0.0.1]
+## [0.0.1] -- 2023-08-15
 - initial setup of repository
 - two main wrappers: 1) wrapper binary crate, and 2) wrapper library crate
 - `gtars` can be used as a library crate. or as a command line tool

docs/gtars/cli.md

@@ -0,0 +1,71 @@
+# gtars-cli
+
+Command-line interface for gtars tools.
+
+## Installation
+
+```bash
+# From source
+cd /path/to/gtars
+cargo install --path gtars-cli
+
+# Or download pre-built binary from releases page
+```
+
+## Available Commands
+
+The CLI provides the following subcommands (availability depends on features enabled during compilation):
+
+### igd
+Build and query IGD (Integrated Genome Database) indexes:
+```bash
+gtars igd create --input regions.bed --output index.igd
+gtars igd query --database index.igd --query chr1:1000-2000
+```
+
+### overlaprs
+Compute overlaps between genomic intervals:
+```bash
+gtars overlaprs --input1 regions1.bed --input2 regions2.bed
+```
+
+### uniwig
+Generate coverage tracks from BED/BAM files:
+```bash
+gtars uniwig --input reads.bam --output coverage.bw
+```
+
+### bbcache
+Cache and manage BED files from bedbase.org:
+```bash
+gtars bbcache get --id GSM123456
+gtars bbcache list
+```
+
+### scoring
+Score fragment overlaps against a reference:
+```bash
+gtars scoring --fragments frags.tsv.gz --universe peaks.bed --output scores.txt
+```
+
+### fragsplit
+Split fragment files by cell barcodes or clusters:
+```bash
+gtars fragsplit --fragments frags.tsv.gz --barcodes clusters.csv --output-dir splits/
+```
+
+
+## Global Options
+
+```bash
+gtars --help              # Show help
+gtars --version           # Show version
+gtars <command> --help    # Show command-specific help
+```
+
+## Building with Specific Features
+
+To build the CLI with specific tools:
+```bash
+cargo build --release --features "uniwig,overlaprs,igd"
+```

docs/gtars/core.md

@@ -0,0 +1,60 @@
+# gtars-core
+
+Core library providing fundamental data structures and utilities for genomic interval operations. This is the foundation that all other gtars modules build upon.
+
+## Features
+
+- Common genomic data structures (Region, RegionSet)
+- BED file parsing utilities
+- Shared constants and helper functions
+- Foundation for all gtars modules
+
+## Core Data Types
+
+### Region
+Represents a genomic interval with chromosome, start, and end coordinates:
+```rust
+use gtars_core::models::Region;
+
+// Create a region
+let region = Region::new("chr1", 1000, 2000);
+
+// Access properties
+println!("Chr: {}", region.chr);
+println!("Start: {}", region.start);
+println!("End: {}", region.end);
+```
+
+### RegionSet
+Collection of genomic regions:
+```rust
+use gtars_core::models::RegionSet;
+use std::path::Path;
+
+// Load from BED file
+let rs = RegionSet::try_from(Path::new("peaks.bed"))?;
+
+// Access regions
+println!("Number of regions: {}", rs.regions.len());
+
+// Iterate over regions
+for region in &rs.regions {
+    println!("{}: {}-{}", region.chr, region.start, region.end);
+}
+```
+
+## Available Modules
+
+- `models` - Core data structures (Region, RegionSet)
+- `utils` - Utility functions for file handling and parsing
+- `consts` - Shared constants
+
+## Dependencies
+
+Minimal external dependencies:
+
+- `anyhow` - Error handling
+- `flate2` - Gzip compression support
+- Other standard bioinformatics libraries
+
+This module serves as the foundation for all other gtars modules and maintains backward compatibility within major versions.