From 9bd87348ef4ef682caed41048ea50dda2c42d7e6 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Oct 2025 08:55:08 +0000 Subject: [PATCH 1/4] Initial plan From fbca3d6d32fceefe13f10c93fc594d0ee64f41a1 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 9 Oct 2025 09:07:14 +0000 Subject: [PATCH 2/4] Create comprehensive copilot-instructions.md file Co-authored-by: ricardoV94 <28983449+ricardoV94@users.noreply.github.com> --- .github/copilot-instructions.md | 119 ++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 .github/copilot-instructions.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000000..d2c8400246 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,119 @@ +# PyTensor Copilot Instructions + +## Overview + +**PyTensor**: Python library for defining, optimizing, and evaluating mathematical expressions with multi-dimensional arrays. Computational backend for PyMC with extensible graph framework. Supports C, JAX, and Numba compilation backends. ~27MB, 492 Python files, Python 3.11-3.13, uses NumPy, SciPy, pytest, Sphinx. + +## Critical: Environment & Commands + +**ALWAYS use micromamba environment**: PyTensor is pre-installed in `.github/workflows/copilot-setup-steps.yml`. + +All commands MUST use: `micromamba run -n pytensor-test ` + +Example: `micromamba run -n pytensor-test python -m pytest tests/` + +## Testing & Building + +### Running Tests (ALWAYS use micromamba) + +```bash +micromamba run -n pytensor-test python -m pytest tests/ # All tests (10-20 min) +micromamba run -n pytensor-test python -m pytest tests/test_updates.py -v # Single file (<2 min) +micromamba run -n pytensor-test python -m pytest tests/ --runslow # Include slow tests +``` + +### Code Style + +**DO NOT run pre-commit/ruff locally** - they're not in copilot environment. CI handles style validation via `.github/workflows/test.yml`. + +### Documentation + +```bash +micromamba run -n pytensor-test python -m sphinx -b html ./doc ./html # Build docs (2-3 min) +``` +**Never commit `html` directory**. + +### MyPy + +```bash +micromamba run -n pytensor-test python ./scripts/run_mypy.py --verbose +``` +**PyTensor incompatible with strict mypy**. Liberal `type: ignore[rule]` and file exclusions are acceptable. + +## PyTensor Design Principles + +### API Differences from NumPy + +1. **Lazy evaluation**: Expressions are symbolic until `pytensor.function()` compiles or `.eval()` evaluates +2. **Pure semantics**: Use `new_x = x[idx].set(y)` instead of `x[idx] = y` +3. **Immutable/hashable**: `a == b` tests identity (`a is b`), not equality +4. **Static shapes**: Broadcasting requires static shape of 1. Valid: `pt.add(pt.vector("x", shape=(1,)), pt.vector("y"))`. Invalid: `pt.add(pt.vector("x", shape=(None,)), pt.vector("y"))` with x.shape=1. + +### Config Flags + +Tests run with `config.floatX == "float32"` and `config.mode = "FAST_COMPILE"`: +- Cast values: `test_value.astype(symbolic_var.type.dtype)` +- Set custom mode or skip tests in `FAST_COMPILE` if needed + +## Code Style + +- **Comments**: Very sparingly, only for complex logic +- **Testing**: Very succinct + - Prefer `tests.unittest_tools.assert_equal_computations` over numerical evaluation + - Test multiple inputs on one compiled function vs multiple compilations + - Minimize conditions; if compiled, always evaluate once + - Integrate with existing tests via parametrization + +## Repository Structure + +### Root +`.github/` (workflows), `pyproject.toml` (config), `setup.py` (Cython build), `conftest.py` (pytest config), `environment.yml` (conda env) + +### Source (`pytensor/`) +- `configdefaults.py`: Config system (floatX, mode) +- `gradient.py`: Auto-differentiation +- `compile/`: Function compilation +- `graph/`: IR and optimization (`graph/rewriting/`) +- `link/`: Backends (`c/`, `jax/`, `numba/`, `pytorch/`) +- `tensor/`: Tensor ops (largest module, subdirs: `random/`, `rewriting/`, `conv/`) +- `scalar/`: Scalar ops +- `scan/`: Loop operations (`scan_perform.pyx` Cython) +- `sparse/`: Sparse tensors + +### Tests (`tests/`) +Mirrors source structure. `unittest_tools.py` has testing utilities. + +## CI/CD Pipeline + +### Workflows (`.github/workflows/`) +1. **test.yml**: Main suite - Python 3.11/3.12/3.13, fast-compile (0/1), float32 (0/1), 7 test parts + backend jobs (numba, jax, torch) +2. **mypy.yml**: Type checking +3. **copilot-setup-steps.yml**: Environment setup + +### PR Requirements +1. Style check (pre-commit: ruff, debug statements) +2. All test matrix configurations pass +3. MyPy passes (with exclusions) + +Test partitioning: 7 parts for parallelization (general, scan, tensor splits) + +## Common Issues + +### Import Errors +Always use: `micromamba run -n pytensor-test python ` + +### Cython Rebuild +If modifying `scan_perform.pyx`: `micromamba run -n pytensor-test pip install -e .` + +### BLAS Verification +```bash +micromamba run -n pytensor-test python -c 'import pytensor; assert pytensor.config.blas__ldflags != ""' +``` + +### Config Test Failures +- Cast dtypes: `value.astype(var.type.dtype)` +- Skip FAST_COMPILE if test depends on optimizations + +## Trust These Instructions + +Comprehensive and tested. Search only if: (1) incomplete, (2) incorrect, or (3) need deeper implementation details. From 002b34035f10a8ecc0c8939c545761d8072f566c Mon Sep 17 00:00:00 2001 From: Ricardo Vieira <28983449+ricardoV94@users.noreply.github.com> Date: Thu, 9 Oct 2025 11:41:27 +0200 Subject: [PATCH 3/4] Revise PyTensor Copilot instructions Updated PyTensor instructions to clarify environment usage, testing commands, and design principles. --- .github/copilot-instructions.md | 143 ++++++++++++++++---------------- 1 file changed, 70 insertions(+), 73 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index d2c8400246..b205571610 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -2,72 +2,42 @@ ## Overview -**PyTensor**: Python library for defining, optimizing, and evaluating mathematical expressions with multi-dimensional arrays. Computational backend for PyMC with extensible graph framework. Supports C, JAX, and Numba compilation backends. ~27MB, 492 Python files, Python 3.11-3.13, uses NumPy, SciPy, pytest, Sphinx. - -## Critical: Environment & Commands - -**ALWAYS use micromamba environment**: PyTensor is pre-installed in `.github/workflows/copilot-setup-steps.yml`. - -All commands MUST use: `micromamba run -n pytensor-test ` - -Example: `micromamba run -n pytensor-test python -m pytest tests/` - -## Testing & Building - -### Running Tests (ALWAYS use micromamba) - -```bash -micromamba run -n pytensor-test python -m pytest tests/ # All tests (10-20 min) -micromamba run -n pytensor-test python -m pytest tests/test_updates.py -v # Single file (<2 min) -micromamba run -n pytensor-test python -m pytest tests/ --runslow # Include slow tests -``` - -### Code Style - -**DO NOT run pre-commit/ruff locally** - they're not in copilot environment. CI handles style validation via `.github/workflows/test.yml`. - -### Documentation - -```bash -micromamba run -n pytensor-test python -m sphinx -b html ./doc ./html # Build docs (2-3 min) -``` -**Never commit `html` directory**. - -### MyPy - -```bash -micromamba run -n pytensor-test python ./scripts/run_mypy.py --verbose -``` -**PyTensor incompatible with strict mypy**. Liberal `type: ignore[rule]` and file exclusions are acceptable. +**PyTensor**: Python library for defining, optimizing, and evaluating mathematical expressions with multi-dimensional arrays. Focus on hackable graph analysis and manipulation. Supports C, JAX, and Numba compilation backends. ~27MB, 492 Python files, Python support as per numpy NEP 29, uses NumPy, SciPy, pytest. ## PyTensor Design Principles +Graph manipulation in Python, graph evaluation out of Python. +Emulate NumPy user-facing API as much as possible. + ### API Differences from NumPy 1. **Lazy evaluation**: Expressions are symbolic until `pytensor.function()` compiles or `.eval()` evaluates -2. **Pure semantics**: Use `new_x = x[idx].set(y)` instead of `x[idx] = y` -3. **Immutable/hashable**: `a == b` tests identity (`a is b`), not equality +2. **Pure semantics**: `new_x = x[idx].set(y)` instead of `x[idx] = y` +3. **Immutable/hashable**: PyTensor variables are hashable. `a == b` tests identity (`a is b`), not elemntwise equality. 4. **Static shapes**: Broadcasting requires static shape of 1. Valid: `pt.add(pt.vector("x", shape=(1,)), pt.vector("y"))`. Invalid: `pt.add(pt.vector("x", shape=(None,)), pt.vector("y"))` with x.shape=1. +5. **Static rank and type**. PyTensor functions accepts variables with a specfic dtype and number of dimensions. Length of each dimension can be static or dynamic. -### Config Flags +## Code Style -Tests run with `config.floatX == "float32"` and `config.mode = "FAST_COMPILE"`: -- Cast values: `test_value.astype(symbolic_var.type.dtype)` -- Set custom mode or skip tests in `FAST_COMPILE` if needed +**Uses pre-commit with ruff** -## Code Style +**Comments**: Should be used sparingly, only for complex logic -- **Comments**: Very sparingly, only for complex logic -- **Testing**: Very succinct - - Prefer `tests.unittest_tools.assert_equal_computations` over numerical evaluation - - Test multiple inputs on one compiled function vs multiple compilations - - Minimize conditions; if compiled, always evaluate once - - Integrate with existing tests via parametrization +**Testing**: Should be succinct + - Prefer `tests.unittest_tools.assert_equal_computations` over numerical evaluation + - Test multiple inputs on one compiled function vs multiple compilations + - Minimize test conditions. Be smart, not fearful + - Integrate with similar existing tests ## Repository Structure ### Root -`.github/` (workflows), `pyproject.toml` (config), `setup.py` (Cython build), `conftest.py` (pytest config), `environment.yml` (conda env) +- `.github/` (workflows), +- `doc/` (docs) +- `pyproject.toml` (config), +- `setup.py` (Cython build), +- `conftest.py` (pytest config), +- `environment.yml` (conda env) ### Source (`pytensor/`) - `configdefaults.py`: Config system (floatX, mode) @@ -79,41 +49,68 @@ Tests run with `config.floatX == "float32"` and `config.mode = "FAST_COMPILE"`: - `scalar/`: Scalar ops - `scan/`: Loop operations (`scan_perform.pyx` Cython) - `sparse/`: Sparse tensors +- `xtensor/` Tensor Ops with dimensions (lowers to Tensor ops) ### Tests (`tests/`) Mirrors source structure. `unittest_tools.py` has testing utilities. -## CI/CD Pipeline +## Critical: Environment & Commands -### Workflows (`.github/workflows/`) -1. **test.yml**: Main suite - Python 3.11/3.12/3.13, fast-compile (0/1), float32 (0/1), 7 test parts + backend jobs (numba, jax, torch) -2. **mypy.yml**: Type checking -3. **copilot-setup-steps.yml**: Environment setup +**ALWAYS use micromamba environment**: PyTensor is pre-installed as editable in `.github/workflows/copilot-setup-steps.yml`. -### PR Requirements -1. Style check (pre-commit: ruff, debug statements) -2. All test matrix configurations pass -3. MyPy passes (with exclusions) +All commands MUST use: `micromamba run -n pytensor-test ` -Test partitioning: 7 parts for parallelization (general, scan, tensor splits) +Example: `micromamba run -n pytensor-test python -c 'import pytensor; print(pytensor.__version__)'` -## Common Issues +## Testing & Building -### Import Errors -Always use: `micromamba run -n pytensor-test python ` +### Running Tests (ALWAYS use micromamba) -### Cython Rebuild -If modifying `scan_perform.pyx`: `micromamba run -n pytensor-test pip install -e .` +```bash +micromamba run -n pytensor-test python -m pytest tests/ # All tests +micromamba run -n pytensor-test python -m pytest tests/test_updates.py -v # Single file +micromamba run -n pytensor-test python -m pytest tests/ --runslow # Include slow tests +``` + +Tests are run with `config.floatX == "float32"` and `config.mode = "FAST_COMPILE"`. If needed: +- Cast numerical values `test_value.astype(symbolic_var.type.dtype)` +- Use custom function mode `get_default_mode().excluding("fusion")` or skip tests in `FAST_COMPILE` + +Alternative backends (JAX, NUMBA, ...) are optional. Use `pytest.importorskip` to fail gracefully. + +### Pre-commit -### BLAS Verification ```bash -micromamba run -n pytensor-test python -c 'import pytensor; assert pytensor.config.blas__ldflags != ""' +micromamba run -n pytensor-test pre-commit ``` -### Config Test Failures -- Cast dtypes: `value.astype(var.type.dtype)` -- Skip FAST_COMPILE if test depends on optimizations +### MyPy -## Trust These Instructions +```bash +micromamba run -n pytensor-test python ./scripts/run_mypy.py --verbose +``` +**PyTensor incompatible with strict mypy**. Type-hints are for users/developers not to appease mypy. Liberal `type: ignore[rule]` and file exclusions are acceptable. -Comprehensive and tested. Search only if: (1) incomplete, (2) incorrect, or (3) need deeper implementation details. +### Documentation + +```bash +micromamba run -n pytensor-test python -m sphinx -b html ./doc ./html # Build docs (2-3 min) +``` +**Never commit `html` directory**. + + +## CI/CD Pipeline + +### Workflows (`.github/workflows/`) +1. **test.yml**: Main suite - Several Python versions, fast-compile (0/1), float32 (0/1), 7 test parts + backend jobs (numba, jax, torch) +2. **mypy.yml**: Type checking +3. **copilot-setup-steps.yml**: Environment setup + + +## Trust These Instructions +These instructions are comprehensive and tested. Only search for additional information if: +1. Instructions are incomplete for your specific task +2. Instructions are found to be incorrect +3. You need deeper understanding of an implementation detail + + For most coding tasks, these instructions provide everything needed to build, test, and validate changes efficiently. From 124cb3a49912ab39887af112368e46875f856000 Mon Sep 17 00:00:00 2001 From: Ricardo Vieira <28983449+ricardoV94@users.noreply.github.com> Date: Thu, 9 Oct 2025 11:49:37 +0200 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index b205571610..367267d978 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -13,9 +13,9 @@ Emulate NumPy user-facing API as much as possible. 1. **Lazy evaluation**: Expressions are symbolic until `pytensor.function()` compiles or `.eval()` evaluates 2. **Pure semantics**: `new_x = x[idx].set(y)` instead of `x[idx] = y` -3. **Immutable/hashable**: PyTensor variables are hashable. `a == b` tests identity (`a is b`), not elemntwise equality. +3. **Immutable/hashable**: PyTensor variables are hashable. `a == b` tests identity (`a is b`), not elementwise equality. 4. **Static shapes**: Broadcasting requires static shape of 1. Valid: `pt.add(pt.vector("x", shape=(1,)), pt.vector("y"))`. Invalid: `pt.add(pt.vector("x", shape=(None,)), pt.vector("y"))` with x.shape=1. -5. **Static rank and type**. PyTensor functions accepts variables with a specfic dtype and number of dimensions. Length of each dimension can be static or dynamic. +5. **Static rank and type**. PyTensor functions accepts variables with a specific dtype and number of dimensions. Length of each dimension can be static or dynamic. ## Code Style