Skip to content

Commit 2684b86

Browse files
stanmartMarcAntoineSchmidtQCjtilly
authored
Switch to pixi (#802)
* Create draft of pixi.toml * Modify contributing docs to reflect new install process * Auto-update lockfile * Make CI unit tests work with pixi * Add lockfile * Add psutil to test deps * Add oldies tests * Fix test environments * Mske pre-commit hooks use pixi environments * Only use -nauto in CI * Add doctest * Remove update section from pixi.toml * Change ci job name * Remove multiple postinstalls * Pixify daily tests * Find minimum versions that actually work * Make lint environment leaner * Add formlulaic minimum version to oldies * Only run pre-commit on changes in CI * Pin pixi-setup to sha * Fix deps * Minor fixes * Use intel mkl in benchmarks * Try to make docs work * Make installing nigthlies a pixi task (for local use, too) * Fix pyarrow nighly installation * Add test feature to nightly environment * Add environment for simultaneous editable tabmat and glum install * Update dev dependencies * fix mypy quirk * fix mypy typing errors * Address @pavelzw's comments * Remove reference to conda/mamba from contributing instructions Co-authored-by: Jan Tilly <jan.tilly@quantco.com> --------- Co-authored-by: Marc-Antoine Schmidt <marc-antoine.schmidt@quantco.com> Co-authored-by: Jan Tilly <jan.tilly@quantco.com>
1 parent 817ac80 commit 2684b86

20 files changed

+25019
-218
lines changed

‎.gitattributes‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# GitHub syntax highlighting
2+
pixi.lock linguist-language=YAML
3+

‎.github/workflows/ci.yml‎

Lines changed: 22 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -3,75 +3,54 @@ on: [push]
33

44
jobs:
55
pre-commit-checks:
6-
name: Linux - pre-commit checks - Python 3.11
6+
name: Linux - pre-commit checks
77
timeout-minutes: 30
88
runs-on: ubuntu-latest
99
env:
1010
PRE_COMMIT_USE_MICROMAMBA: 1
1111
steps:
1212
- name: Checkout branch
1313
uses: actions/checkout@v4
14-
- name: Set up micromamba
15-
uses: mamba-org/setup-micromamba@f8b8a1e23a26f60a44c853292711bacfd3eac822
16-
- name: Add micromamba to GITHUB_PATH
17-
run: echo "${HOME}/micromamba-bin" >> "$GITHUB_PATH"
18-
- name: Install Python 3.11
19-
uses: actions/setup-python@v5
14+
- name: Set up pixi
15+
uses: prefix-dev/setup-pixi@v0.8.0
2016
with:
21-
python-version: "3.11"
22-
- name: Run pre-commit checks
23-
uses: pre-commit/action@v3.0.1
17+
environments: lint default
18+
- name: pre-commit
19+
run: pixi run -e lint pre-commit-run --color=always --show-diff-on-failure
2420

2521
unit-tests:
26-
name: Tests - ${{ matrix.os }} - Py${{ matrix.python-version }}
22+
name: Tests
2723
runs-on: ${{ matrix.os }}
2824
env:
2925
CI: True
3026
strategy:
3127
fail-fast: true
3228
matrix:
3329
include:
34-
- { os: ubuntu-latest, python-version: '3.9' }
35-
- { os: ubuntu-latest, python-version: '3.10' }
36-
- { os: ubuntu-latest, python-version: '3.11' }
37-
- { os: ubuntu-latest, python-version: '3.12' }
38-
- { os: macos-latest, python-version: '3.12' }
39-
- { os: windows-latest, python-version: '3.12' }
30+
- { os: ubuntu-latest, environment: 'py39' }
31+
- { os: ubuntu-latest, environment: 'py310' }
32+
- { os: ubuntu-latest, environment: 'py311' }
33+
- { os: ubuntu-latest, environment: 'py312' }
34+
- { os: ubuntu-latest, environment: 'oldies' }
35+
- { os: windows-latest, environment: 'py312' }
36+
- { os: macos-latest, environment: 'py312' }
4037
steps:
4138
- name: Checkout branch
4239
uses: actions/checkout@v4
43-
- name: Set up conda env
44-
uses: mamba-org/setup-micromamba@f8b8a1e23a26f60a44c853292711bacfd3eac822
40+
- name: Set up pixi
41+
uses: prefix-dev/setup-pixi@ba3bb36eb2066252b2363392b7739741bb777659
4542
with:
46-
environment-file: environment.yml
47-
init-shell: ${{ matrix.os == 'windows-latest' && 'powershell' || 'bash' }}
48-
cache-environment: true
49-
create-args: >-
50-
python=${{ matrix.python-version }}
51-
- name: Install repository (unix)
52-
if: matrix.os != 'windows-latest'
53-
shell: bash -el {0}
54-
run: pip install --no-use-pep517 --no-deps --disable-pip-version-check -e .
55-
- name: Install repository (windows)
56-
if: matrix.os == 'windows-latest'
57-
shell: powershell
58-
run: pip install --no-use-pep517 --no-deps --disable-pip-version-check -e .
59-
- name: Run pytest (unix)
60-
if: matrix.os != 'windows-latest'
61-
shell: bash -el {0}
62-
run: pytest -nauto tests/glm --doctest-modules src/glum
63-
- name: Run pytest (windows)
64-
if: matrix.os == 'windows-latest'
65-
shell: powershell
66-
run: pytest -nauto .\tests\glm --doctest-modules .\src\glum
43+
environments: ${{ matrix.environment }}
44+
- name: Install repository
45+
run: pixi run -e ${{ matrix.environment }} postinstall
46+
- name: Run pytest
47+
run: pixi run -e ${{ matrix.environment }} test -nauto
6748
- name: Run doctest
68-
if: matrix.os != 'windows-latest'
69-
shell: bash -el {0}
7049
# Check that the readme example will work by running via doctest.
7150
# We run outside the repo to make the test a bit more similar to
7251
# a user running after installing with conda.
7352
run: |
7453
mkdir ../temp
7554
cp README.md ../temp
7655
cd ../temp
77-
python -m doctest -v README.md
56+
pixi run --manifest-path ../glum/pixi.toml -e ${{ matrix.environment }} python -m doctest -v README.md

‎.github/workflows/daily.yml‎

Lines changed: 12 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -9,62 +9,42 @@ on:
99

1010
jobs:
1111
linux-daily-unittests:
12-
name: "Linux - daily unit tests - Python ${{ matrix.PYTHON_VERSION}} - ${{ matrix.NOTE }}"
12+
name: "Linux - daily unit tests - ${{ matrix.NOTE }}"
1313
runs-on: ubuntu-latest
1414
env:
1515
CI: True
1616
strategy:
1717
fail-fast: false
1818
matrix:
1919
include:
20-
- PYTHON_VERSION: '3.10'
20+
- environment: 'nightly'
2121
NOTE: 'Nightly Builds' # run once with nightlies
22-
- PYTHON_VERSION: '3.10'
23-
NOTE: 'Default Builds' # run once with normal dependencies
22+
- environment: 'py312'
23+
NOTE: 'Python 3.12' # run once with normal dependencies
2424
steps:
2525
- name: Checkout branch
2626
uses: actions/checkout@v4
27-
- uses: mamba-org/setup-micromamba@f8b8a1e23a26f60a44c853292711bacfd3eac822
27+
- name: Set up pixi
28+
uses: prefix-dev/setup-pixi@ba3bb36eb2066252b2363392b7739741bb777659
2829
with:
29-
environment-file: environment.yml
30-
create-args: >-
31-
python=${{ matrix.PYTHON_VERSION }}
30+
environments: ${{ matrix.environment }}
3231
- name: Install nightlies
3332
if: matrix.NOTE == 'Nightly Builds'
34-
shell: bash -el {0}
35-
run: |
36-
# needed for tabmat
37-
echo "Install compilation dependencies"
38-
micromamba install -y c-compiler cxx-compiler 'cython!=3.0.4' jemalloc-local libgomp mako xsimd
39-
40-
PRE_WHEELS="https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/"
41-
for pkg in pandas scikit-learn scipy; do
42-
echo "Installing $pkg nightly"
43-
micromamba remove -y --force $pkg
44-
pip install --pre --no-deps --only-binary :all: --upgrade --timeout=60 -i $PRE_WHEELS $pkg
45-
done
46-
echo Install pyarrow nightly
47-
micromamba remove -y --force pyarrow
48-
pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ --prefer-binary --pre --no-deps pyarrow
49-
echo Install tabmat nightly
50-
micromamba remove -y --force tabmat
51-
pip install --no-use-pep517 --no-deps git+https://github.com/Quantco/tabmat
33+
run: pixi run -e ${{ matrix.environment }} install-nightlies
5234
- name: Install repository
53-
shell: bash -el {0}
54-
run: pip install --no-use-pep517 --no-deps --disable-pip-version-check -e .
35+
run: pixi run -e ${{ matrix.environment }} postinstall
5536
- name: Run pytest
56-
shell: bash -el {0}
57-
run: pytest -nauto tests/glm --doctest-modules src/glum/
37+
run: pixi run -e ${{ matrix.environment }} test -nauto
5838
- name: Run doctest
59-
shell: bash -el {0}
6039
# Check that the readme example will work by running via doctest.
6140
# We run outside the repo to make the test a bit more similar to
6241
# a user running after installing with conda.
6342
run: |
6443
mkdir ../temp
6544
cp README.md ../temp
6645
cd ../temp
67-
python -m doctest -v README.md
46+
pixi run --manifest-path ../glum/pixi.toml -e ${{ matrix.environment }} python -m doctest -v README.md
47+
6848
- name: Issue on failure
6949
uses: actions/github-script@v7
7050
if: ${{ failure() }}

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,3 +147,7 @@ pkgs/*
147147
*.c
148148
*.cpp
149149
!*tmpl.cpp
150+
# pixi environments
151+
.pixi
152+
*.egg-info
153+

‎.pre-commit-config.yaml‎

Lines changed: 44 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,47 @@
1+
exclude: ^.pixi$
12
repos:
2-
- repo: https://github.com/Quantco/pre-commit-mirrors-ruff
3-
rev: 0.4.7
3+
- repo: local
44
hooks:
5-
- id: ruff-conda
6-
exclude: ^src/glum_benchmarks/orig_sklearn_fork/
7-
- id: ruff-format-conda
8-
exclude: ^src/glum_benchmarks/orig_sklearn_fork/
9-
- repo: https://github.com/Quantco/pre-commit-mirrors-mypy
10-
rev: "1.10.0"
11-
hooks:
12-
- id: mypy-conda
13-
args:
14-
- --check-untyped-defs
15-
- --ignore-missing-imports
16-
- --namespace-packages
5+
# ensure pixi environments are up to date
6+
# workaround for https://github.com/prefix-dev/pixi/issues/1482
7+
- id: pixi-install
8+
name: pixi-install
9+
entry: pixi install -e default -e lint
10+
language: system
11+
always_run: true
12+
require_serial: true
13+
pass_filenames: false
14+
# ruff
15+
- id: ruff
16+
name: ruff
17+
entry: pixi run -e lint ruff check --fix --exit-non-zero-on-fix --force-exclude
18+
language: system
19+
types_or: [python, pyi]
20+
require_serial: true
21+
- id: ruff-format
22+
name: ruff-format
23+
entry: pixi run -e lint ruff format --force-exclude
24+
language: system
25+
types_or: [python, pyi]
26+
require_serial: true
27+
# cython-lint
28+
- id: cython-lint
29+
name: cython-lint
30+
entry: pixi run -e lint cython-lint --no-pycodestyle
31+
language: system
32+
types: [cython]
33+
require_serial: true
34+
- id: cython-double-quote
35+
name: cython-double-quote
36+
entry: pixi run -e lint double-quote-cython-strings
37+
language: system
38+
types: [cython]
39+
require_serial: true
40+
# mypy
41+
- id: mypy
42+
name: mypy
43+
entry: pixi run -e default mypy
1744
exclude: (^tests/|^src/glum_benchmarks/orig_sklearn_fork/)
18-
additional_dependencies: [-c, conda-forge, types-setuptools=67.5, attrs]
19-
- repo: https://github.com/Quantco/pre-commit-mirrors-cython-lint
20-
rev: 0.16.2
21-
hooks:
22-
- id: cython-lint-conda
23-
args: [--no-pycodestyle]
24-
- id: double-quote-cython-strings-conda
45+
language: system
46+
types: [python]
47+
require_serial: true

‎.readthedocs.yml‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,11 @@ build:
33
os: ubuntu-22.04
44
tools:
55
python: mambaforge-4.10
6+
commands:
7+
- mamba install -c conda-forge -c nodefaults pixi==0.22.0
8+
- pixi run -e docs postinstall
9+
- pixi run readthedocs
610
sphinx:
711
configuration: docs/conf.py
8-
python:
9-
install:
10-
- method: pip
11-
path: .
12-
conda:
13-
environment: environment.yml
1412
formats:
1513
- pdf

‎docs/contributing.rst‎

Lines changed: 27 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -31,39 +31,46 @@ Install for development
3131
--------------------------------------------------
3232

3333
The first step is to set up a conda environment and install glum in editable mode.
34-
We strongly suggest to use ``mamba`` instead of ``conda`` as this provides the same functionality at much greater speed.
34+
The project uses [pixi](https://pixi.sh/latest/) for managing the environment. If you don't have pixi installed, start by [installing it](https://pixi.sh/latest/#installation).
3535

3636
::
3737

38-
# First, make sure you have conda-forge as your primary conda channel:
39-
conda config --add channels conda-forge
40-
4138
# Clone the repository
4239
git clone git@github.com:Quantco/glum.git
4340
cd glum
4441

45-
# Set up a conda environment with name "glum"
46-
mamba env create
42+
# Install the pre-commit hooks
43+
pixi run pre-commit-install
44+
45+
# Install the dependencies, as well as and glum in editable mode
46+
pixi run postinstall
4747

4848
# If you want to install the dependencies necessary for benchmarking against other GLM packages:
49-
mamba env update -n glum --file environment-benchmark.yml
49+
pixi run -e benchmark postinstall
5050

51-
# If you want to work on the tutorial notebooks:
52-
mamba env update -n glum --file environment-tutorial.yml
51+
# If you want to work on the tutorial notebooks or the documentation:
52+
pixi run -e docs postinstall
5353

54-
# Activate the previously created conda environment
55-
conda activate glum
54+
# You can run any command in the pixi environment with `pixi run <command>`. For example:
55+
pixi run [-e ENVIRONMENT] ipython
5656

57-
# Set up our pre-commit hooks for black, mypy, isort and flake8.
58-
pre-commit install
57+
# Alternatively, you can create a shell with the pixi environment activated:
58+
pixi shell
5959

60-
# Install this package in editable mode.
61-
pip install --no-use-pep517 --disable-pip-version-check -e .
60+
# Alternatively, a number of pixi tasks are available for commonly used commands.
61+
# You can run them with `pixi run <task>`.
62+
# To get a list of available tasks, run:
63+
pixi task list
6264

6365

6466
Testing and continuous integration
6567
--------------------------------------------------
66-
The test suite is in ``tests/``.
68+
The test suite is in ``tests/``. A pixi task is available to run the tests:
69+
70+
::
71+
72+
pixi run test
73+
6774

6875
Golden master tests
6976
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -80,12 +87,12 @@ If you want to skip the slow tests, add the ``-m "not slow"`` flag to any pytest
8087
Storing and modifying
8188
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
8289

83-
To store the golden master results:
90+
To store the golden master results, use the following pixi tasks:
8491

8592
::
8693

87-
python tests/glm/test_golden_master.py
88-
python tests/glm/test_benchmark_golden_master.py
94+
pixi run store-golden-master
95+
pixi run store-benchmark-golden-master
8996

9097
Add the ``--overwrite`` flag if you want to overwrite already existing golden master results
9198

@@ -108,9 +115,7 @@ The documentation is built with a mix of Sphinx, autodoc, and nbsphinx. To devel
108115

109116
::
110117

111-
cd docs
112-
make html
113-
python -m http.server --directory _build/html
118+
pixi run serve-docs
114119

115120
Then, navigate to `<http://localhost:8000>`_ to view the documentation.
116121

‎environment-benchmark.yml‎

Lines changed: 0 additions & 7 deletions
This file was deleted.

‎environment-tutorial.yml‎

Lines changed: 0 additions & 10 deletions
This file was deleted.

0 commit comments

Comments
 (0)