|
| 1 | +# PyTensor Copilot Instructions |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +**PyTensor**: Python library for defining, optimizing, and evaluating mathematical expressions with multi-dimensional arrays. Focus on hackable graph analysis and manipulation. Supports C, JAX, and Numba compilation backends. ~27MB, 492 Python files, Python support as per numpy NEP 29, uses NumPy, SciPy, pytest. |
| 6 | + |
| 7 | +## PyTensor Design Principles |
| 8 | + |
| 9 | +Graph manipulation in Python, graph evaluation out of Python. |
| 10 | +Emulate NumPy user-facing API as much as possible. |
| 11 | + |
| 12 | +### API Differences from NumPy |
| 13 | + |
| 14 | +1. **Lazy evaluation**: Expressions are symbolic until `pytensor.function()` compiles or `.eval()` evaluates |
| 15 | +2. **Pure semantics**: `new_x = x[idx].set(y)` instead of `x[idx] = y` |
| 16 | +3. **Immutable/hashable**: PyTensor variables are hashable. `a == b` tests identity (`a is b`), not elementwise equality. |
| 17 | +4. **Static shapes**: Broadcasting requires static shape of 1. Valid: `pt.add(pt.vector("x", shape=(1,)), pt.vector("y"))`. Invalid: `pt.add(pt.vector("x", shape=(None,)), pt.vector("y"))` with x.shape=1. |
| 18 | +5. **Static rank and type**. PyTensor functions accepts variables with a specific dtype and number of dimensions. Length of each dimension can be static or dynamic. |
| 19 | + |
| 20 | +## Code Style |
| 21 | + |
| 22 | +**Uses pre-commit with ruff** |
| 23 | + |
| 24 | +**Performance** |
| 25 | +* Could should be performant |
| 26 | +* Avoid expensive work in hot loops |
| 27 | +* Avoid redundant checks. Let errors raise naturally |
| 28 | +* In contrast, silent errors should be prevented |
| 29 | + |
| 30 | +**Comments**: Should be used sparingly, only for complex logic |
| 31 | + |
| 32 | +**Testing**: Should be succinct |
| 33 | + - Prefer `tests.unittest_tools.assert_equal_computations` over numerical evaluation |
| 34 | + - Test multiple inputs on one compiled function vs multiple compilations |
| 35 | + - Minimize test conditions. Be smart, not fearful |
| 36 | + - Integrate with similar existing tests |
| 37 | + |
| 38 | +## Repository Structure |
| 39 | + |
| 40 | +### Root |
| 41 | +- `.github/` (workflows), |
| 42 | +- `doc/` (docs) |
| 43 | +- `pyproject.toml` (config), |
| 44 | +- `setup.py` (Cython build), |
| 45 | +- `conftest.py` (pytest config), |
| 46 | +- `environment.yml` (conda env) |
| 47 | + |
| 48 | +### Source (`pytensor/`) |
| 49 | +- `configdefaults.py`: Config system (floatX, mode) |
| 50 | +- `gradient.py`: Auto-differentiation |
| 51 | +- `compile/`: Function compilation |
| 52 | +- `graph/`: IR and optimization (`graph/rewriting/`) |
| 53 | +- `link/`: Backends (`c/`, `jax/`, `numba/`, `mlx/`, `pytorch/`) |
| 54 | +- `tensor/`: Tensor ops (largest module, subdirs: `random/`, `rewriting/`, `conv/`) |
| 55 | +- `scalar/`: Scalar ops |
| 56 | +- `scan/`: Loop operations (`scan_perform.pyx` Cython) |
| 57 | +- `sparse/`: Sparse tensors |
| 58 | +- `xtensor/` Tensor Ops with dimensions (lowers to Tensor ops) |
| 59 | + |
| 60 | +### Tests (`tests/`) |
| 61 | +Mirrors source structure. `unittest_tools.py` has testing utilities. |
| 62 | + |
| 63 | +## Critical: Environment & Commands |
| 64 | + |
| 65 | +**ALWAYS use micromamba environment**: PyTensor is pre-installed as editable in `.github/workflows/copilot-setup-steps.yml`. |
| 66 | + |
| 67 | +All commands MUST use: `micromamba run -n pytensor-test <command>` |
| 68 | + |
| 69 | +Example: `micromamba run -n pytensor-test python -c 'import pytensor; print(pytensor.__version__)'` |
| 70 | + |
| 71 | +## Testing & Building |
| 72 | + |
| 73 | +### Running Tests (ALWAYS use micromamba) |
| 74 | + |
| 75 | +```bash |
| 76 | +micromamba run -n pytensor-test python -m pytest tests/ # All tests |
| 77 | +micromamba run -n pytensor-test python -m pytest tests/test_updates.py -v # Single file |
| 78 | +micromamba run -n pytensor-test python -m pytest tests/ --runslow # Include slow tests |
| 79 | +``` |
| 80 | + |
| 81 | +Tests are run with `config.floatX == "float32"` and `config.mode = "FAST_COMPILE"`. If needed: |
| 82 | +- Cast numerical values `test_value.astype(symbolic_var.type.dtype)` |
| 83 | +- Use custom function mode `get_default_mode().excluding("fusion")` or skip tests in `FAST_COMPILE` if they are not directly relevant to the mode. |
| 84 | + |
| 85 | +Alternative backends (JAX, NUMBA, ...) are optional. Use `pytest.importorskip` to fail gracefully. |
| 86 | + |
| 87 | +### Pre-commit |
| 88 | + |
| 89 | +```bash |
| 90 | +micromamba run -n pytensor-test pre-commit |
| 91 | +``` |
| 92 | + |
| 93 | +### MyPy |
| 94 | + |
| 95 | +```bash |
| 96 | +micromamba run -n pytensor-test python ./scripts/run_mypy.py --verbose |
| 97 | +``` |
| 98 | +**PyTensor incompatible with strict mypy**. Type-hints are for users/developers not to appease mypy. Liberal `type: ignore[rule]` and file exclusions are acceptable. |
| 99 | + |
| 100 | +### Documentation |
| 101 | + |
| 102 | +```bash |
| 103 | +micromamba run -n pytensor-test python -m sphinx -b html ./doc ./html # Build docs (2-3 min) |
| 104 | +``` |
| 105 | +**Never commit `html` directory**. |
| 106 | + |
| 107 | + |
| 108 | +## CI/CD Pipeline |
| 109 | + |
| 110 | +### Workflows (`.github/workflows/`) |
| 111 | +1. **test.yml**: Main suite - Several Python versions, fast-compile (0/1), float32 (0/1), 7 test parts + backend jobs (numba, jax, torch) |
| 112 | +2. **mypy.yml**: Type checking |
| 113 | +3. **copilot-setup-steps.yml**: Environment setup |
| 114 | + |
| 115 | + |
| 116 | +## Trust These Instructions |
| 117 | +These instructions are comprehensive and tested. Only search for additional information if: |
| 118 | +1. Instructions are incomplete for your specific task |
| 119 | +2. Instructions are found to be incorrect |
| 120 | +3. You need deeper understanding of an implementation detail |
| 121 | + |
| 122 | + For most coding tasks, these instructions provide everything needed to build, test, and validate changes efficiently. |
0 commit comments