Adopt uv for Python package management across SmartEM ecosystem

[vredchenko]

Summary

Evaluate and implement uv as the primary Python package manager for SmartEM repositories. uv is a fast Python package installer written in Rust by Astral (creators of Ruff), offering 10-100x speed improvements over pip.

Current State

smartem-decisions

Aspect	Current Status
uv.lock	✅ Present (1913 lines, actively maintained)
CI/CD	❌ Uses pip (pip install -e .[all])
Documentation	❌ References pip/venv
Pre-commit	❌ Uses language: system
Local dev	✅ uv.lock updated regularly

smartem-devtools

Aspect	Current Status
.python-version	✅ Present (3.12)
uv.lock	❌ Not present

fandanGO-cryoem-dls

Aspect	Current Status
Package format	Legacy setup.py + requirements.txt
Ownership	Shared with FragmentScreen/FandanGO ecosystem
uv compatibility	Would require coordination with upstream maintainers

FragmentScreen repos (read-only)

• Use conda for environment management
• External ownership - not candidates for unilateral changes

---

Benefits of Full uv Adoption

Benefit	Impact
CI Speed	10-100x faster dependency installation in GitHub Actions
Reproducibility	uv.lock ensures identical environments across dev/CI/prod
Unified tooling	Replace pip + pip-tools + virtualenv + pyenv with single tool
Python management	uv python install 3.12 - no need for pyenv
Security	Built-in vulnerability scanning with uv audit
Developer experience	uvx for running tools without installing (like npx)

---

Migration Plan

Phase 1: smartem-decisions (High Priority)

Already has uv.lock - needs CI and docs alignment.

1.1 Update CI Workflows

Current .github/actions/install_requirements/action.yml:

- name: Setup python
  uses: actions/setup-python@v5
  with:
    python-version: ${{ env.PYTHON_VERSION }}

- name: Install packages
  run: pip install ${{ inputs.pip-install }}

Proposed (with uv):

- name: Install uv
  uses: astral-sh/setup-uv@v4
  with:
    enable-cache: true

- name: Setup Python
  run: uv python install ${{ env.PYTHON_VERSION }}

- name: Install packages
  run: uv sync --all-extras

1.2 Update _test.yml

Current:

- name: Install tox
  run: pip install tox tox-direct

- name: Run tests
  run: tox -e tests

Proposed:

- name: Run tests
  run: uv run pytest

1.3 Update _dist.yml

Current:

- name: Build sdist and wheel
  run: pipx run build

Proposed:

- name: Build sdist and wheel
  run: uv build

1.4 Update Documentation

Update docs/tutorials/installation.md:

## Development Installation (Recommended)

We recommend using [uv](https://docs.astral.sh/uv/) for fast, reproducible installations:

\`\`\`bash
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone and install
git clone https://github.com/DiamondLightSource/smartem-decisions.git
cd smartem-decisions
uv sync --all-extras

# Run commands in the environment
uv run pytest
uv run pyright src tests
\`\`\`

## Alternative: pip Installation

\`\`\`bash
pip install -e .[all]
\`\`\`

1.5 Update Pre-commit Config

Current .pre-commit-config.yaml:

- repo: local
  hooks:
    - id: ruff
      language: system
      entry: ruff check --force-exclude

Proposed (use ruff's official pre-commit hook):

- repo: https://github.com/astral-sh/ruff-pre-commit
  rev: v0.8.4
  hooks:
    - id: ruff
    - id: ruff-format

---

Phase 2: smartem-devtools (Medium Priority)

2.1 Add uv.lock

cd repos/DiamondLightSource/smartem-devtools
uv lock

2.2 Update Python scripts

Ensure any Python tooling uses uv run pattern.

---

Phase 3: fandanGO-cryoem-dls (Requires Coordination)

Note: This repo has shared ownership with the FragmentScreen/FandanGO ecosystem. Any structural changes (e.g., migrating from setup.py to pyproject.toml) require coordination with upstream maintainers.

Options:

1. Propose upstream migration - Suggest pyproject.toml adoption to FandanGO maintainers
2. Local uv usage only - Use uv locally without changing package structure
3. Wait for ecosystem alignment - Adopt uv when FandanGO ecosystem modernises

For now, this phase is deferred pending discussion with FandanGO maintainers.

---

Repos NOT Recommended for uv

Repo	Reason
FragmentScreen/*	Read-only, external ownership, conda-based
smartem-frontend	Node.js project (uses npm)
GitlabAriaPHP/*	PHP projects

---

Common uv Commands Reference

# Project setup
uv init                     # Create new project
uv sync                     # Install from lockfile
uv sync --all-extras        # Install with all optional deps

# Dependency management  
uv add requests             # Add dependency
uv add --dev pytest         # Add dev dependency
uv remove requests          # Remove dependency
uv lock                     # Update lockfile

# Running commands
uv run pytest               # Run in project environment
uv run python script.py     # Run script
uvx ruff check .            # Run tool without installing

# Python version management
uv python install 3.12      # Install Python version
uv python list              # List available versions

# Security
uv audit                    # Check for vulnerabilities

---

Tasks

• Phase 1.1: Update smartem-decisions CI workflows to use uv
• Phase 1.2: Update smartem-decisions installation documentation
• Phase 1.3: Update smartem-decisions pre-commit to use ruff-pre-commit
• Phase 1.4: Update REPO-GUIDELINES.md with uv commands
• Phase 2.1: Add uv.lock to smartem-devtools
• Phase 3: Discuss uv/pyproject.toml adoption with FandanGO maintainers (deferred)

---

References

• uv documentation
• astral-sh/setup-uv GitHub Action
• uv vs pip comparison
• Migrating from pip to uv