🔬 PyWatson - A Python scientific project management tool

A Python scientific project management tool inspired by DrWatson.jl. Scaffolds reproducible scientific projects with modern Python tooling (uv + pytest + ruff) and PyWatson path management, HDF5 data handling, and smart caching.

✨ Features

• 📁 Project scaffolding — three types (default, minimal, full) with DrWatson.jl directory layout; author info auto-filled from git config
• 🔧 Modern tooling — uv package manager, ruff for linting, pytest pre-configured
• 💾 Multiple data formats — HDF5, NumPy NPZ, Zarr via unified save_*/load_* API
• 📝 Parameter naming — savename()/parse_savename() for reproducible filenames from parameter dicts
• 📊 Parameter grids — dict_list() → Cartesian products; pywatson sweep previews without code
• ⚡ Smart caching — produce_or_load() skips recomputation; returns Path
• 🛡️ Atomic saves — safesave() (crash-safe) + tmpsave() (auto-cleanup)
• 📈 Result aggregation — collect_results() → pandas DataFrame
• 🎯 Reproducibility — snapshot_environment(), set_random_seed()
• 📊 Dashboard — pywatson status shows dirs, data, git state
• 📦 Adopt existing — pywatson adopt scans & reorganizes messy projects (see docs/ADOPT_GUIDE.md)
• 🐳 Docker & Zenodo — --docker adds reproducibility bundle (see docs/DOCKER_GUIDE.md)

🚀 Quick Start

Install

# Install uv (one-time)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install pywatson
uv tool install git+https://github.com/isaac-tes/pywatson.git

Create a project

# Interactive wizard
pywatson init

# Or non-interactive with flags
pywatson --project-name my-analysis \
  --author-name "Jane Doe" --author-email "jane@uni.edu" \
  --description "My research"

cd my-analysis
uv sync        # install dependencies
uv run pytest  # run tests

Adopt an existing project

pywatson adopt /path/to/messy-code \
  --author-name "Jane" --author-email "jane@uni.edu"
# Scans, classifies files, organizes into pywatson structure

See docs/ADOPT_GUIDE.md for details.

📚 uv Essentials

For complete docs: uv documentation

Command	Purpose
uv sync	Install all dependencies (create/update .venv)
uv run pytest	Run tests in project environment
uv add PACKAGE	Add a dependency
uv remove PACKAGE	Remove a dependency
uv pip list	List installed packages
uv tool install TOOL	Install CLI tool globally

Developer workflow: Clone repo → uv sync → uv run pywatson instead of pywatson

📂 Project Types

Type	Best for	Includes
default	Standard research	data/{sims,exp_raw,exp_pro}, notebooks, _research
minimal	Lightweight	src, data, scripts, tests, docs
full	Publication-ready	everything + config, Makefile, CI, docs

Default project structure

my-project/
├── src/my_project/
│   ├── __init__.py         # Public API (PyWatson helpers re-exported)
│   ├── core.py             # Your analysis code
│   ├── pywatson_utils.py   # Self-contained PyWatson utilities
├── scripts/
│   ├── generate_data.py
│   └── analyze_data.py
├── notebooks/
├── tests/
├── data/
│   ├── sims/
│   ├── exp_raw/
│   └── exp_pro/
├── plots/
├── _research/tmp/
├── pyproject.toml
└── README.md

pywatson_utils.py is copied verbatim into every generated project so projects are fully self-contained and do not depend on pywatson at runtime.

Usage at a Glance

from my_project import (
    datadir, savename, parse_savename, dict_list,
    save_data, load_data, tagsave, produce_or_load,
    collect_results, snapshot_environment, set_random_seed,
)

# Paths (work from any subdirectory)
datadir("sims")                       # Path(".../data/sims")

# Parameter filenames
params = {"N": 1000, "beta": 0.44}
savename(params)                      # "N=1000_beta=0.44.h5"
parse_savename("N=1000_beta=0.44.h5") # {"N": 1000, "beta": 0.44}

# Parameter grid → Cartesian product
for p in dict_list({"N": [100, 1000], "beta": [0.3, 0.44]}):
    save_data(run(p), savename(p), subdir="sims")

# Smart caching — run once, load thereafter
data, path = produce_or_load(savename(params), run_simulation, **params)

# Aggregate all results into a DataFrame
df = collect_results(subdir="sims", as_dataframe=True)

For the complete API, see docs/UTILITIES.md.

💻 CLI

pywatson --project-name PROJ_NAME  # create new project
pywatson adopt /path/to/code     # adopt existing project
pywatson status                  # dashboard (dirs, data, git)
pywatson sweep K=V1,V2 ...      # preview parameter-sweep names
pywatson summary                 # list HDF5 files + keys

Full reference: docs/CLI.md

📖 Documentation

Guide	What
QUICKSTART.md	End-to-end workflow
UTILITIES.md	API reference
CLI.md	CLI commands
ADOPT_GUIDE.md	Adopting existing projects
DOCKER_GUIDE.md	Docker reproducibility
ZENODO.md	Zenodo deposit

🛠️ Development

uv sync                          # install + dev deps
uv run pytest                    # run all tests
uv run ruff check src/ tests/    # lint
uv run ruff format src/ tests/   # format
uv build                         # package

Architecture

Component	Description
src/pywatson/core.py	ProjectScaffolder + Click CLI
src/pywatson/utils.py	PyWatson utilities (copied into generated projects)
src/pywatson/templates/	Jinja2 templates for generated project files
tests/	pytest suite (220+ tests)

🔄 vs DrWatson.jl

Feature	DrWatson.jl	PyWatson
Language	Julia	Python 3.12+
Package manager	Pkg.jl	uv
Project templates	1	3 (default, minimal, full)
Adopt existing projects	—	✅
Parameter naming	savename	savename + parse_savename
Parameter grid expansion	dict_list	dict_list
Smart caching	produce_or_load → (data, file)	produce_or_load → (data, Path)
Data formats	JLD2 (default), BSON, CSV via FileIO.jl	HDF5 (h5py), NumPy NPZ, Zarr, CSV/pandas
Git tagging	tagsave + tag! + git diff patch	tagsave
Atomic / safe saves	safesave (backup numbering)	safesave + tmpsave (auto-cleanup)
Collect results	collect_results → DataFrame	collect_results → pandas DataFrame
CLI	—	✅ (init, adopt, status, sweep, summary)
Docker integration	—	✅
Zenodo integration	—	✅

📄 License

MIT — see LICENSE.

🙏 Built with

• DrWatson.jl (inspiration)
• uv (package management)
• Click (CLI)
• Rich (terminal UI)