Skip to content

ENH: add ability to test examples PR against PR in bids-specification #515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

ENH: add ability to test examples PR against PR in bids-specification #515

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6ffaf06
ENH: allow to test against bids-spec PR schema.
yarikoptic Sep 18, 2025
9f49902
Add rudimentary pull_requests_template.md with syntax to list bids-sp…
yarikoptic Sep 18, 2025
3f3e71e
CI: trigger workflow on PR description edits
yarikoptic Sep 18, 2025
b83dbe5
CI: fix workflow syntax errors for bids-pr matrix
yarikoptic Sep 18, 2025
b065cf0
CI: fix PR body parsing to handle special characters
yarikoptic Sep 18, 2025
6f613fc
noop if no match
yarikoptic Sep 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/pull_request_template.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!-- Fill out or delete as appropriate -->

- Needs testing against a bids-specification-pr: NUMBER-or-URL
- TODOs:
- [ ] ...
Comment on lines +1 to +5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect a BEP PR to evolve, possibly starting with some examples before a spec or validator PR is written, but then adding them in as we get closer to completion. To that end, I would suggest leaving permanent fields with a default value of NONE.

Suggested change
<!-- Fill out or delete as appropriate -->
- Needs testing against a bids-specification-pr: NUMBER-or-URL
- TODOs:
- [ ] ...
<!-- New examples often relate to in-progress changes to the specification or validator. Link to the relevant PRs, if any. -->
- BIDS Specification PR: NONE
- BIDS Validator PR: NONE
- TODOs:
- [ ] ...


<!-- PLEASE READ AND DELETE THE TEXT BELOW BEFORE OPENING THE PULL REQUEST -->

See the [CONTRIBUTING](https://github.com/bids-standard/bids-examples/blob/master/CONTRIBUTING.md) guide. Specifically:

- Please keep the title of your Pull Request (PR) short but informative - it will appear in the changelog.

46 changes: 44 additions & 2 deletions .github/workflows/validate_datasets.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
push:
branches: ['**']
pull_request:
types: [opened, synchronize, reopened, edited]
branches: ['**']
create:
branches: [master]
Expand All @@ -20,16 +21,50 @@ defaults:
shell: bash

jobs:
# Prepare the matrix dynamically based on whether a bids-specification PR is referenced
prepare-matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
bids_pr: ${{ steps.find-pr.outputs.pr_number }}
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0 # Get full history for commit message scanning

- name: Check for bids-specification PR reference
id: find-pr
run: |
# Check PR description if this is a PR
if [ "${{ github.event_name }}" == "pull_request" ]; then
cat << 'EOF' > /tmp/pr_body.txt
${{ github.event.pull_request.body }}
EOF
PR_NUM=$(grep -oP 'bids-specification-pr:\s*(https://github\.com/bids-standard/bids-specification/pulls?/)*\K[0-9]+' /tmp/pr_body.txt | head -1 || true)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect people to use the <org>/<repo>#<pr> syntax in some cases. I'm not sure if GitHub automatically rewrites it as a URL for pull_request.body or not, but you may want to account for that.

[ -n "$PR_NUM" ] && echo "pr_number=$PR_NUM" >> $GITHUB_OUTPUT || :
fi

- name: Set matrix
id: set-matrix
run: |
EXTRA_ITEM=''
if [ -n "${{ steps.find-pr.outputs.pr_number }}" ]; then
EXTRA_ITEM=', "bids-pr"'
fi
echo "matrix=[\"stable\", \"main\", \"dev\", \"legacy\"${EXTRA_ITEM}]" >> $GITHUB_OUTPUT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would either deactivate dev or just update dev steps to use your environment variables.


build:
needs: prepare-matrix
strategy:
fail-fast: false
matrix:
platform: [ubuntu-latest, macos-latest, windows-latest]
bids-validator: [stable, main, dev, legacy]
bids-validator: ${{ fromJson(needs.prepare-matrix.outputs.matrix) }}

runs-on: ${{ matrix.platform }}

env:
BIDS_PR: ${{ needs.prepare-matrix.outputs.bids_pr }}
TZ: Europe/Berlin
FORCE_COLOR: 1

Expand Down Expand Up @@ -65,7 +100,7 @@ jobs:
deno install -Agf -n bids-validator jsr:@bids/validator

- name: Install BIDS validator (main)
if: matrix.bids-validator == 'main'
if: matrix.bids-validator == 'main' || matrix.bids-validator == 'bids-pr'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, dev tracks unreleased changes to the schema, while main tracks the latest released version of BIDS. PRs against the spec will mostly aim for dev.

run: |
# If unmerged validator PRs are needed for testing, you can use
# https://github.com/<FORK>/bids-validator/raw/<BRANCH>/bids-validator/src/bids-validator.ts
Expand Down Expand Up @@ -130,6 +165,13 @@ jobs:
# release of https://jsr.io/@bids/schema
run: echo BIDS_SCHEMA=https://bids-specification.readthedocs.io/en/latest/schema.json >> $GITHUB_ENV

- name: Set BIDS_SCHEMA variable for bids-pr version
if: matrix.bids-validator == 'bids-pr'
# Use the readthedocs PR preview build for the schema
run: |
echo "Using schema from bids-specification PR #${{ env.BIDS_PR }}"
echo BIDS_SCHEMA=https://bids-specification--${{ env.BIDS_PR }}.org.readthedocs.build/en/${{ env.BIDS_PR }}/schema.json >> $GITHUB_ENV

- name: Validate all BIDS datasets using bids-validator
run: |
cat ./run_tests.sh
Expand Down
Loading