Restructure `_dev_utils` for better separation of dev tooling from package code

[thorwhalen]

Restructure _dev_utils for better separation of dev tooling from package code

Background

The cosmograph package bridges JavaScript (Cosmograph/Cosmos libraries) and Python, requiring alignment of:

• Argument names and signatures
• Type annotations
• Default values
• Documentation/descriptions

Ideal state: Single Source of Truth (SSOT) where one canonical definition drives both JS and Python interfaces.

Current reality: Multiple sources exist (TypeScript files, markdown docs, Python traitlets), creating potential misalignment.

Purpose of _dev_utils

The _dev_utils module mitigates the lack of SSOT by providing diagnostic and repair tools for two audiences:

1. Developers (manual use):

   • Analyze differences between JS/TS sources and Python implementations
   • Generate specification files (params_ssot.json, etc.) that align interfaces
   • Repair misalignments by updating signatures/docs

2. CI (automated checks):

   • Diagnose misalignments before publishing
   • Fail builds when JS/Python interfaces drift
   • Ensure published package maintains alignment

Problem

_dev_utils currently lives inside cosmograph/ package directory, causing issues:

• ❌ Extra dependencies (ju, tabled, hubcap) required only for dev tools
• ❌ CI linting/testing trips on missing dependencies
• ❌ Requires CI exclusions/workarounds
• ❌ Dev tools mixed with user-facing package code

Proposed Solutions

Option 1: Scripts directory at repo root (RECOMMENDED)

Move to dev_tools/ or scripts/ outside the package.

Pros:

• ✅ Standard practice, clear separation
• ✅ No CI complexity
• ✅ Simple path adjustments for file generation
• ✅ Still in-repo for easy access

Cons:

• ❌ Minor path updates needed

---

Option 2: Separate dev package (monorepo)

Create packages/cosmograph-dev/ alongside packages/cosmograph/.

Pros:

• ✅ Clean package-level separation
• ✅ Independent dependency management
• ✅ Can publish as separate tool if useful

Cons:

• ❌ More complex repo structure
• ❌ Overkill for current needs

---

Option 3: Separate repository

New repo: cosmograph-ssot-tools or similar.

Pros:

• ✅ Complete isolation
• ✅ Independent versioning

Cons:

• ❌ Cross-repo file generation complexity
• ❌ Harder to keep in sync
• ❌ Extra maintenance overhead

---

Option 4: Keep with conditional imports

Current approach with CI exclusions.

Pros:

• ✅ No code changes needed

Cons:

• ❌ Ongoing CI maintenance
• ❌ Not conventional
• ❌ Dev deps mixed with package

Recommended Solution: Option 1 (Scripts Directory)

Move _dev_utils → dev_tools at repo root. This is the standard Python practice for development utilities.

Implementation Steps

1. Restructure repository:

py_cosmograph/
├── cosmograph/               # Package code (unchanged)
│   ├── __init__.py
│   ├── base.py
│   ├── util.py
│   ├── widget/
│   └── data/                 # Generated specs remain here
│       └── params_ssot.json
├── dev_tools/                # Moved from cosmograph/_dev_utils/
│   ├── __init__.py
│   ├── _resources.py         # SSOT extraction/generation
│   ├── _code_sync.py         # Signature alignment tools
│   ├── _color_util.py
│   ├── _traitlets_util.py
│   ├── requirements.txt      # ju, tabled, hubcap, etc.
│   └── data/                 # Prep/cache files
├── tests/
└── .github/workflows/
    └── ci.yml

2. Update dev_tools/requirements.txt:

ju
tabled  
hubcap
# ... other dev-only deps

3. Update path references in dev_tools/:

Since tools now live outside package, adjust data file paths:

# Before (in cosmograph/_dev_utils/_resources.py):
from cosmograph.util import files
data_dir = files / "_dev_utils" / "data"

# After (in dev_tools/_resources.py):
from pathlib import Path
DEV_TOOLS_DIR = Path(__file__).parent
data_dir = DEV_TOOLS_DIR / "data"

# For writing to package data:
PACKAGE_DIR = DEV_TOOLS_DIR.parent / "cosmograph"
PACKAGE_DATA_DIR = PACKAGE_DIR / "data"

4. Update cosmograph/util.py:

Remove any imports from _dev_utils. The package should only read generated files:

# cosmograph/util.py
PARAMS_SSOT_PATH = data_dir / "params_ssot.json"

def _params_ssot(param_names=None):
    params_ssot = json.loads(PARAMS_SSOT_PATH.read_text())
    # ... rest unchanged

5. Update CI workflow (.github/workflows/ci.yml):

Remove any exclusions for _dev_utils:

# No changes needed - dev_tools/ outside package naturally excluded
- name: Run Tests
  run: hatch run test:pytest

- name: Ruff Validation  
  run: ruff check cosmograph  # Only checks package code

6. Add CI alignment check job (optional but recommended):

alignment-check:
  name: Check JS/Python Alignment
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    
    - name: Set up Python 3.10
      uses: actions/setup-python@v4
      with:
        python-version: "3.10"
    
    - name: Install dev tools dependencies
      run: pip install -r dev_tools/requirements.txt
    
    - name: Check alignment
      run: |
        cd dev_tools
        python -c "
        from _resources import ConfigsDacc
        dacc = ConfigsDacc()
        issues = list(dacc.diagnosis_items())
        if issues:
            for title, data in issues:
                print(f'MISALIGNMENT: {title}')
                print(data)
            exit(1)
        "

7. Developer workflow documentation:

Add dev_tools/README.md:

# Cosmograph Development Tools

Tools for maintaining JS/Python alignment.

## Setup
```bash
pip install -r requirements.txt

Usage

Diagnose alignment:

from dev_tools._resources import ConfigsDacc
dacc = ConfigsDacc()
dacc.print_diagnosis()

Regenerate specs:

# Updates cosmograph/data/params_ssot.json
dacc.cosmograph_base_params_json()

8. Update package manifest (MANIFEST.in or pyproject.toml):

Ensure dev_tools/ excluded from distribution:

# pyproject.toml
[tool.hatch.build.targets.wheel]
exclude = [
  "dev_tools",
  "tests",
]

---

Alternative Implementation Notes

If choosing Option 2 (Monorepo):

1. Use packages/ directory:

   packages/
   ├── cosmograph/      # Main package
   └── cosmograph-dev/  # Dev tools package
   

2. Update CI to install both:

   - name: Install packages
     run: |
       pip install -e packages/cosmograph
       pip install -e packages/cosmograph-dev

3. Dev tools import as: from cosmograph_dev._resources import ...

If choosing Option 3 (Separate repo):

1. Create cosmograph-ssot-tools repo
2. Add as git submodule or document manual sync process
3. Use GitHub Actions artifact upload/download for generated specs
4. Requires workflow to commit generated files back to main repo

---

Migration Checklist

• Create dev_tools/ directory at repo root
• Move cosmograph/_dev_utils/* → dev_tools/
• Create dev_tools/requirements.txt
• Update path references in dev_tools/*.py
• Remove _dev_utils imports from cosmograph/ package code
• Update .github/workflows/ci.yml (remove exclusions)
• Add alignment check CI job (optional)
• Test dev tools still generate files correctly
• Test package installs without dev dependencies
• Update documentation
• Clean up cosmograph/_dev_utils/ directory