feat: Add vital articles wiki example and trainer tooling

* Update examples/wiki.rs so the crawler seeds come from Wikipedia’s
  Level 1–5 Vital Articles lists instead of a few topical pages, improving
  coverage when running the example.
* Introduce a new wiki_trainer/ Python project (Typer CLI, README,
  pyproject.toml, uv.lock, artifacts scaffolding) that can 1) filter
  Fastcrawl’s chunk JSONL into shuffled train/eval splits and 2) fine-tune
  Hugging Face causal LMs with configurable hyperparameters via prepare-
  data and train commands.
* Implement dataset cleaning/splitting helpers (data.py), structured
  config objects (config.py), and a transformers-based training pipeline
  (training.py) that handles tokenization, batching, evaluation/save
  cadence, and tokenizer/model persistence.
* Note: the new directory currently also tracks __pycache__/ and .pyc
  artifacts under wiki_trainer/src/wiki_trainer/, which will end up in the
  commit unless they’re removed or gitignored.

Author: aaronlifton

examples/wiki.rs

@@ -4,10 +4,15 @@ use std::sync::Arc;
 use url::Url;
 
 const SEEDS: &[&str] = &[
-    "https://en.wikipedia.org/wiki/Web_crawler",
-    "https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol",
-    "https://en.wikipedia.org/wiki/Capybara",
-    "https://en.wikipedia.org/wiki/Cat",
+    // "https://en.wikipedia.org/wiki/Web_crawler",
+    // "https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol",
+    // "https://en.wikipedia.org/wiki/Capybara",
+    // "https://en.wikipedia.org/wiki/Cat",
+    "https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Level/1",
+    "https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Level/2",
+    "https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Level/3",
+    "https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Level/4",
+    "https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Level/5",
 ];
 
 fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {

wiki_trainer/.gitignore

@@ -0,0 +1 @@
+artifacts/

wiki_trainer/.python-version

@@ -0,0 +1 @@
+3.13

wiki_trainer/README.md

@@ -0,0 +1,66 @@
+# Wiki Trainer
+
+Utilities for turning Fastcrawl's Wikipedia chunks into Hugging Face datasets and fine-tuning a causal language model using `transformers` + `uv`.
+
+## Prerequisites
+
+- Python 3.13 (already provided by the `uv` shim installed at repository root)
+- `uv` >= 0.9 for dependency + virtualenv management
+- GPU drivers/tooling that can run PyTorch (install CUDA/cuDNN or use CPU for smoke tests)
+- A local snapshot of chunks, e.g. `data/wiki_embeddings.jsonl` produced by Fastcrawl's embedder pipeline
+
+## Setup
+
+```sh
+cd wiki_trainer
+UV_CACHE_DIR=../.cache/uv uv sync  # creates .venv and installs dependencies (torch via the `training` extra)
+source .venv/bin/activate
+```
+
+`uv sync` respects the `pyproject.toml` optional dependency group named `training`, so PyTorch + bitsandbytes are installed automatically. Adjust `UV_CACHE_DIR` if you keep cache files elsewhere (the repo root already has `.cache/uv`).
+
+## Converting chunks to train/eval JSONL
+
+Run the `prepare-data` subcommand to down-select and split the chunk corpus. By default it expects OpenAI-style embedding JSONL rows (with `text`, `url`, etc.), but it also works with normalized Fastcrawl pages that include `body_text` or `chunks[].text`.
+
+```sh
+uv run wiki-trainer prepare-data \
+  ../data/wiki_embeddings.jsonl \
+  --output-dir artifacts/datasets \
+  --min-chars 200 \
+  --max-chars 1600 \
+  --max-chunks 50000 \
+  --eval-ratio 0.02
+```
+
+The command writes `train.jsonl` and `eval.jsonl` into `artifacts/datasets`. Each row keeps the original text plus metadata columns (`source_url`, `chunk_id`, `section_path`) so you can trace model behavior back to specific chunks.
+
+## Fine-tuning a model
+
+Once the dataset exists, call `wiki-trainer train` with your preferred Hugging Face checkpoint. The defaults target `distilgpt2`, but you can swap in any causal LM (TinyLlama, Mistral, etc.) so long as it fits on your hardware.
+
+```sh
+uv run wiki-trainer train \
+  artifacts/datasets \
+  --model-name TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
+  --output-dir artifacts/checkpoints/tinyllama \
+  --context-length 1024 \
+  --epochs 2 \
+  --batch-size 1 \
+  --grad-accum 16 \
+  --learning-rate 1e-4 \
+  --eval-steps 100
+```
+
+The CLI wraps Hugging Face's `Trainer` so standard knobs (batch size, gradient accumulation, precision flags) are exposed. Logs/checkpoints land under `artifacts/checkpoints/...` by default.
+
+## Tips
+
+- **Filtering.** Increase `--min-chars` to drop stubby chunks or pass `--max-chunks`/`--eval-ratio` to control dataset size.
+- **Precision.** Use `--bf16` or `--fp16` once your hardware + drivers support it; otherwise leave them disabled for CPU proof-of-life runs.
+- **Custom schedules.** Edit `wiki_trainer/config.py` to add weight-decay or warmup strategies, then re-export the CLI arguments if you need more control.
+- **Streaming/large corpora.** `prepare-data` currently loads the filtered samples into memory before shuffling. For multi-million chunk runs consider chunked pre-processing or swapping the implementation for a disk-backed shuffle buffer.
+
+## Repository integration
+
+The project stays isolated inside `wiki_trainer/` so it can evolve independently of the Rust crawler. Use `uv run wiki-trainer --help` to see every flag, and keep data artifacts under `wiki_trainer/artifacts/` (already referenced in the defaults) so they stay out of the Rust workspace.

wiki_trainer/pyproject.toml

@@ -0,0 +1,34 @@
+[project]
+name = "wiki-trainer"
+version = "0.1.0"
+description = "Train custom language models on Fastcrawl Wikipedia chunks"
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "typer[all]>=0.12.5",
+    "datasets>=2.19.1",
+    "transformers>=4.45.0",
+    "accelerate>=0.34.0",
+    "sentencepiece>=0.2.0",
+    "tqdm>=4.66.0",
+    "numpy>=1.26.0",
+]
+
+[project.optional-dependencies]
+training = ["torch>=2.4.1", "bitsandbytes>=0.43.1"]
+
+[dependency-groups]
+training = ["torch>=2.4.1", "bitsandbytes>=0.43.1"]
+
+[project.scripts]
+wiki-trainer = "wiki_trainer.cli:app"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/wiki_trainer"]
+
+[tool.uv]
+default-groups = ["training"]

wiki_trainer/src/wiki_trainer/__init__.py

@@ -0,0 +1,7 @@
+"""Utilities for preparing Wikipedia chunks and fine-tuning local language models."""
+
+from .config import DatasetConfig, TrainingConfig
+from .data import prepare_dataset
+from .training import train_model
+
+__all__ = ["DatasetConfig", "TrainingConfig", "prepare_dataset", "train_model"]