-
Notifications
You must be signed in to change notification settings - Fork 26
Improved CI #2079
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Improved CI #2079
Changes from all commits
c857d37
bca8880
c4a0a10
301c4d6
c6c662d
cecd478
a70b977
1de3fb8
a293627
14d596b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,30 +1,56 @@ | ||
| version: 2 | ||
| updates: | ||
| - package-ecosystem: docker | ||
| directory: "/mpcontribs-kernel-gateway" | ||
| schedule: | ||
| interval: weekly | ||
| time: '08:00' | ||
| timezone: US/Pacific | ||
| open-pull-requests-limit: 10 | ||
| - package-ecosystem: npm | ||
| directory: "/mpcontribs-portal" | ||
| schedule: | ||
| interval: weekly | ||
| time: '08:00' | ||
| timezone: US/Pacific | ||
| open-pull-requests-limit: 10 | ||
| - package-ecosystem: docker | ||
| directory: "/mpcontribs-portal" | ||
| schedule: | ||
| interval: weekly | ||
| time: '08:00' | ||
| timezone: US/Pacific | ||
| open-pull-requests-limit: 10 | ||
| - package-ecosystem: docker | ||
| directory: "/mpcontribs-api" | ||
| schedule: | ||
| interval: weekly | ||
| time: '08:00' | ||
| timezone: US/Pacific | ||
| open-pull-requests-limit: 10 | ||
| # --- GitHub Actions: keep workflow action versions current --- | ||
| - package-ecosystem: github-actions | ||
| directory: "/" | ||
| schedule: | ||
| interval: weekly | ||
| time: "08:00" | ||
| timezone: US/Pacific | ||
| groups: | ||
| actions: | ||
| patterns: ["*"] | ||
| open-pull-requests-limit: 10 | ||
|
|
||
| # --- Python via uv: monitors pyproject.toml + uv.lock -------- | ||
| - package-ecosystem: uv | ||
| directory: "/mpcontribs-api" | ||
| schedule: | ||
| interval: weekly | ||
| time: "08:00" | ||
| timezone: US/Pacific | ||
| groups: | ||
| python-minor-patch: | ||
| patterns: ["*"] | ||
| update-types: ["minor", "patch"] | ||
| open-pull-requests-limit: 10 | ||
|
|
||
| # --- Existing entries ---------------------------------------- | ||
| - package-ecosystem: docker | ||
| directory: "/mpcontribs-kernel-gateway" | ||
| schedule: | ||
| interval: weekly | ||
| time: "08:00" | ||
| timezone: US/Pacific | ||
| open-pull-requests-limit: 10 | ||
| - package-ecosystem: npm | ||
| directory: "/mpcontribs-portal" | ||
| schedule: | ||
| interval: weekly | ||
| time: "08:00" | ||
| timezone: US/Pacific | ||
| open-pull-requests-limit: 10 | ||
| - package-ecosystem: docker | ||
| directory: "/mpcontribs-portal" | ||
| schedule: | ||
| interval: weekly | ||
| time: "08:00" | ||
| timezone: US/Pacific | ||
| open-pull-requests-limit: 10 | ||
| - package-ecosystem: docker | ||
| directory: "/mpcontribs-api" | ||
| schedule: | ||
| interval: weekly | ||
| time: "08:00" | ||
| timezone: US/Pacific | ||
| open-pull-requests-limit: 10 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| # Reusable CI runner | ||
| name: _python-ci | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| package: | ||
| description: "Package directory, e.g. mpcontribs-api" | ||
| required: true | ||
| type: string | ||
| python-version: | ||
| description: "Override Python version; empty = use the package's .python-version pin" | ||
| required: false | ||
| type: string | ||
| default: "" | ||
| coverage-artifact: | ||
| description: "Name for the uploaded coverage XML; empty = don't upload" | ||
| required: false | ||
| type: string | ||
| default: "" | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| lint: | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| working-directory: ${{ inputs.package }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: astral-sh/setup-uv@v3 | ||
| with: | ||
| enable-cache: true | ||
| cache-dependency-glob: "${{ inputs.package }}/uv.lock" | ||
| - uses: extractions/setup-just@v4 | ||
| - run: uv python install ${{ inputs.python-version }} | ||
| - run: just install | ||
| - run: just lint | ||
| - run: just format-check | ||
| - run: just lock-check | ||
|
|
||
| typecheck: | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| working-directory: ${{ inputs.package }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: astral-sh/setup-uv@v3 | ||
| with: | ||
| enable-cache: true | ||
| cache-dependency-glob: "${{ inputs.package }}/uv.lock" | ||
| - uses: extractions/setup-just@v4 | ||
| - run: uv python install ${{ inputs.python-version }} | ||
| - run: just install | ||
| - run: just typecheck | ||
|
|
||
| test: | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| working-directory: ${{ inputs.package }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: astral-sh/setup-uv@v3 | ||
| with: | ||
| enable-cache: true | ||
| cache-dependency-glob: "${{ inputs.package }}/uv.lock" | ||
| - uses: extractions/setup-just@v4 | ||
| - run: uv python install ${{ inputs.python-version }} | ||
| - run: just install | ||
| - run: just test ci | ||
| - name: Upload coverage | ||
| if: ${{ inputs.coverage-artifact != '' }} | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: ${{ inputs.coverage-artifact }} | ||
| path: ${{ inputs.package }}/coverage.xml | ||
| if-no-files-found: ignore |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| # Adding a package: once it has a uv-based justfile contract (install/lint/ | ||
| # format-check/typecheck/lock-check/test ci), add it to BOTH the paths-filter | ||
| # `filters` below AND the `config` map in the discover step. Until a package is | ||
| # migrated, leave it out - it stays covered by the legacy testing.yml instead. | ||
|
|
||
| name: ci | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: [master] | ||
| push: | ||
| branches: [master] | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| discover: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| packages: ${{ steps.build.outputs.packages }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: dorny/paths-filter@v3 | ||
| id: filter | ||
| with: | ||
| filters: | | ||
| mpcontribs-api: | ||
| - 'mpcontribs-api/**' | ||
| - '.github/workflows/ci.yml' | ||
| - '.github/workflows/_python_ci.yml' | ||
| - name: Build matrix of changed (migrated) packages | ||
| id: build | ||
| env: | ||
| CHANGES: ${{ toJSON(steps.filter.outputs) }} | ||
| run: | | ||
| # Map of migrated package -> optional python-version override | ||
| # ("" = use the package's .python-version pin). Only packages whose | ||
| # filter reported a change are emitted into the matrix. | ||
| python3 - "$CHANGES" <<'PY' >> "$GITHUB_OUTPUT" | ||
| import json, sys | ||
| changes = json.loads(sys.argv[1]) | ||
| config = { | ||
| "mpcontribs-api": {"python-version": ""}, | ||
| # "mpcontribs-client": {"python-version": ""}, # add when migrated | ||
| } | ||
| include = [ | ||
| {"package": pkg, **cfg} | ||
| for pkg, cfg in config.items() | ||
| if changes.get(pkg) == "true" | ||
| ] | ||
| print("packages=" + json.dumps(include)) | ||
| PY | ||
|
|
||
| ci: | ||
| needs: discover | ||
| if: ${{ needs.discover.outputs.packages != '[]' }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| include: ${{ fromJson(needs.discover.outputs.packages) }} | ||
| uses: ./.github/workflows/_python_ci.yml | ||
| with: | ||
| package: ${{ matrix.package }} | ||
| python-version: ${{ matrix.python-version }} | ||
| coverage-artifact: coverage-${{ matrix.package }} | ||
|
Comment on lines
+58
to
+70
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Runs a unified CI workflow for each discovered repo. Provides which package is being ran, which version of python to use (defaults to the repos specified version), and where to put the test coverage artifact |
||
|
|
||
| # Always-reporting aggregate gate. Make ONLY this a required status check | ||
| gate: | ||
| name: ci-success | ||
| if: always() | ||
| needs: [discover, ci] | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Verify CI result | ||
| run: | | ||
| result="${{ needs.ci.result }}" | ||
| # 'skipped' is success here: it means no migrated package needed checking. | ||
| if [ "$result" = "success" ] || [ "$result" = "skipped" ]; then | ||
| echo "CI gate passed (ci job: $result)" | ||
| exit 0 | ||
| fi | ||
| echo "CI gate failed (ci job: $result)" | ||
| exit 1 | ||
|
Comment on lines
+73
to
+88
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What actually gets depended on by the repo rules. Should require a status check where context = "ci-success" |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| set shell := ["bash", "-euo", "pipefail", "-c"] | ||
|
|
||
| # Importable package name for coverage (src/ layout -> underscore). | ||
| cov_pkg := "mpcontribs_api" | ||
|
|
||
| default: | ||
| @just --list | ||
|
|
||
| # --- CI gates (non-mutating) ------------------------------------------------- | ||
|
|
||
| # Sync deps into the project venv. | ||
| install: | ||
| uv sync --all-extras --dev | ||
|
|
||
| # Lint, no autofix. | ||
| lint: | ||
| uv run ruff check | ||
|
|
||
| # Style check only (does not rewrite files). | ||
| format-check: | ||
| uv run ruff format --check | ||
|
|
||
| # Static type checking. | ||
| typecheck: | ||
| uv run basedpyright | ||
|
|
||
| # Fail if uv.lock is out of sync with pyproject.toml. | ||
| lock-check: | ||
| uv lock --check | ||
|
|
||
| # --- Local dev --------------------------------------------------------------- | ||
|
|
||
| # Run the dev server (auto-reloads on file changes). | ||
| dev: | ||
| uv run fastapi dev src/mpcontribs_api/app.py | ||
|
|
||
| # Format, fix, and lint all source code (mutates files — not a CI gate). | ||
| fmt: | ||
| uv run ruff format | ||
| -uv run ruff check --fix | ||
| -uv run ruff check --fix --unsafe-fixes | ||
| -uv run pydocstringformatter --write | ||
|
|
||
| # --- Tests ------------------------------------------------------------------- | ||
| # Local suites run fast and coverage-free; the `ci` suite (called by the | ||
| # pipeline) adds coverage + an 80% gate and excludes db-marked tests. | ||
| # | ||
| # Run a suite: all/a (default), unit/u, integration/i, db/d, ci/c. | ||
| test suite="all": | ||
| #!/usr/bin/env bash | ||
| case "{{suite}}" in | ||
| all|a) uv run pytest tests/ ;; | ||
| integration|i) uv run pytest tests/integration/ -m "not db" ;; | ||
| unit|u) uv run pytest tests/unit/ ;; | ||
| db|d) uv run pytest tests/integration/db/ -m db ;; | ||
| ci|c) uv run pytest tests/unit tests/integration -m "not db" \ | ||
| --cov={{cov_pkg}} --cov-report=xml --cov-report=term-missing ;; | ||
| *) echo "unknown suite '{{suite}}' — use: all/a, integration/i, unit/u, db/d, ci/c" && exit 1 ;; | ||
| esac | ||
|
|
||
| # Everything CI runs, in CI order — run locally before pushing. | ||
| ci: install lock-check lint format-check typecheck (test "ci") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Provides packages on a "new" stack (uv, justfile) alongside a potential python version override to further CI steps