Skip to content

Commit a9d2f6b

Browse files
jgchngithub-actions[bot]
jgchn
and
github-actions[bot]
authored
Introducing Run Examples check in PRs (#908)
* Pr pipeline 2 (#5) * Cleaning up run examples pipeline Signed-off-by: Jing Chen <[email protected]> Rm run examples composite Signed-off-by: Jing Chen <[email protected]> Halt early if no file patch Signed-off-by: Jing Chen <[email protected]> Better msg Signed-off-by: Jing Chen <[email protected]> Modify some PDL files Signed-off-by: Jing Chen <[email protected]> Revert revisions Signed-off-by: Jing Chen <[email protected]> Modify some PDL programs Signed-off-by: Jing Chen <[email protected]> Fix if Signed-off-by: Jing Chen <[email protected]> Composite run examples Signed-off-by: Jing Chen <[email protected]> Checkout before using composite Test if Test if again Rm bash test Fix unfound path Update if check Delete a folder and see if test runs Bring back deleted example. Expects test to run Test if examples config is patched Mv yaml patch to composite action Try another patch strategy Change some files Checkout again in run examples Modify files Typo in composite action Use json in check parse Try to escape json Use changes instead in yaml action Bring back Ollama set up Gather new results from GH env Clean up workflows Fix lint Modify chatbot example Fix run examples config file * Add possible results for react examples * Preserve trailing white space * See if a commit is pushed on my behalf * Rename config file * Diff pdl file * Check if commit happens despite pytet failures * Allow pytest failure and continue run * github-actions[bot]: Updated results file when running examples on Mon Apr 7 23:24:42 UTC 2025 Signed-off-by: github-actions[bot] <[email protected]> * Set max parallel to 1 to prevent race conditions * See if yaml action is triggered * Change master branch * Fix pytest * Set update results to true * Test yaml actio again * Update result to true * Reverting update_results to False. If you wish to update again, change locally and push with change. * WRite permissions * Reverting update_results to False. If you wish to update again, change locally and push with change. * Don't commit using yaml action * Set update results to true * github-actions[bot]: Updated results file when running examples on Tue Apr 8 20:11:04 UTC 2025 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Reverting update_results to False. Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Try pytest failure * Test pytest behavior again * Test set +e * Test again * Set update to true to see if changed-files can capture results * PR test * github-actions[bot]: Updated results file when running examples Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * github-actions[bot]: Reverting update_result to false Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Test only with PR, not push * github-actions[bot]: Updated results file when running examples Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * github-actions[bot]: Reverting update_result to false Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * No pdl file change * Clean up and fix run examples nightly Signed-off-by: Jing Chen <[email protected]> * Fix empty parse * Fix * Bring back more models on ollama set up * Contrib readme updates --------- Signed-off-by: github-actions[bot] <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Jing Chen <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <75508867+github-actions[bot][email protected]> Get new results for failing progs github-actions[bot]: Run examples: updated result files on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> github-actions[bot]: Run examples: reverting update_result to false on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> See if ollama is cached Try caching ollama Set up Ollama outside composite action Revert changes Update runexamples config Signed-off-by: Jing Chen <[email protected]> * Update contrib readme Signed-off-by: Jing Chen <[email protected]> * Get test results for nightly run Signed-off-by: Jing Chen <[email protected]> * Update results on all files github-actions[bot]: Run examples: reverting update_result to false on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> github-actions[bot]: Run examples: updated result files on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Revert patches Signed-off-by: Jing Chen <[email protected]> * Small fix Signed-off-by: Jing Chen <[email protected]> * Rm dependency on yaml actions * Add repo to checkout * Update checkout ref * Dont check out again * Rm another checkout * Rm checkout refs * Fix yq * Configure git and try again * Fix source repo name * Checkout fix * Fix ref * github-actions[bot]: Run examples: reverting update_result to false on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Try push again * Fix update results bool * Try another push * Try using secrets * github-actions[bot]: Run examples: reverting update_result to false on your behalf Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Don't use composite action in pr * Refine bool comparison * Don't cache ollama * Add patch yaml step * Test if PAT works * Add checkout step * Checkout modification * Update with feedback Signed-off-by: Jing Chen <[email protected]> * Fix checkout Signed-off-by: Jing Chen <[email protected]> * Checkout fix Signed-off-by: Jing Chen <[email protected]> * Checkout fix 2 Signed-off-by: Jing Chen <[email protected]> * Checkout fix 3 Signed-off-by: Jing Chen <[email protected]> * Update code Signed-off-by: Jing Chen <[email protected]> * Update results Signed-off-by: Jing Chen <[email protected]> * Enable parallel runs Signed-off-by: Jing Chen <[email protected]> --------- Signed-off-by: Jing Chen <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent b0c37c5 commit a9d2f6b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+904
-357
lines changed

‎.github/actions/ollama/action.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: 'Ollama Setup'
2+
description: 'Composte action for Ollama set up in GH environment'
3+
runs:
4+
using: 'composite'
5+
steps:
6+
- name: Remove unnecessary files
7+
shell: bash
8+
run: |
9+
sudo rm -rf /usr/share/dotnet
10+
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
11+
12+
# Set up Ollama
13+
- name: Install Ollama and start server
14+
shell: bash
15+
run: |
16+
curl -fsSL https://ollama.com/install.sh | sudo -E sh
17+
18+
- name: Pull models in examples/
19+
shell: bash
20+
run: |
21+
ollama pull granite3.2:2b
22+
ollama pull granite3.2:8b
23+
ollama pull mxbai-embed-large
24+
ollama list
25+
26+
- name: Check that all required models are available
27+
shell: bash
28+
run: |
29+
models=("mxbai-embed-large" "granite3.2:2b" "granite3.2:8b")
30+
missing=0
31+
for model in "${models[@]}"; do
32+
if ! ollama list | awk 'NR>1 {print $1}' | grep -q "$model"; then
33+
echo "❌ Model $model is missing!"
34+
missing=1
35+
fi
36+
done
37+
38+
if [ "$missing" -eq 1 ]; then
39+
exit 1
40+
else
41+
echo "✅ All expected models are available."
42+
fi
43+
44+
- name: Wait for Ollama server
45+
shell: bash
46+
run: |
47+
sleep 5
48+
time curl -i http://localhost:11434

‎.github/actions/run-examples/action.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
name: 'Ollama Setup'
2+
description: 'Composte action to set up Run Examples'
3+
inputs:
4+
python-version:
5+
description: 'Python version'
6+
required: true
7+
default: '3.11'
8+
runner-os:
9+
description: 'Runner OS'
10+
required: true
11+
repository:
12+
description: 'Repository name this pull request is initiated from'
13+
required: false
14+
head-ref:
15+
description: 'Head ref of the repo'
16+
required: false
17+
default: 'main'
18+
token:
19+
description: 'Github token'
20+
required: false
21+
update-results:
22+
description: 'Whether to update the results for this run. Must be false for nightly runs'
23+
required: true
24+
check:
25+
description: 'Files to patch tests/test_examples_run.yaml with. These are the PDL files that the test will run against. Defaults to all PDL files.'
26+
required: false
27+
default: '[]'
28+
runs:
29+
using: 'composite'
30+
steps:
31+
# # Set up Ollama
32+
- uses: ./.github/actions/ollama
33+
34+
# Configure Run Examples environment
35+
- uses: actions/checkout@v4
36+
with:
37+
token: ${{ inputs.token }}
38+
ref: ${{ inputs.head-ref }}
39+
repository: ${{ inputs.repository }}
40+
fetch-depth: 0
41+
42+
- name: Patch tests/test_examples_run.yaml check with modified files
43+
shell: bash
44+
run: |
45+
yq -i '.check = (${{ inputs.check }})' tests/test_examples_run.yaml
46+
47+
- name: Print test Run Examples config
48+
shell: bash
49+
run: cat tests/test_examples_run.yaml
50+
51+
# Run tests
52+
- name: Set up Python ${{ inputs.python-version }}
53+
uses: actions/setup-python@v5
54+
with:
55+
python-version: ${{ inputs.python-version }}
56+
- name: Cache pip
57+
uses: actions/cache@v4
58+
with:
59+
# This path is specific to Ubuntu
60+
path: ${{ env.pythonLocation }}
61+
# Look to see if there is a cache hit for the setup file
62+
key: ${{ inputs.runner-os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
63+
restore-keys: |
64+
${{ inputs.runner-os }}-pip-new3
65+
${{ inputs.runner-os }}-new3
66+
- name: Install dependencies
67+
shell: bash
68+
run: pip install --upgrade --upgrade-strategy eager .[all]
69+
- name: Pip list packages
70+
shell: bash
71+
run: pip list
72+
- name: Run Pytest
73+
shell: bash
74+
run: |
75+
cat tests/test_examples_run.yaml
76+
(
77+
set +e
78+
py.test -v --capture=tee-sys -rfE -s tests/test_examples_run.py --disable-pytest-warnings
79+
EXIT_CODE=$?
80+
81+
if [ $EXIT_CODE -eq 0 ]; then
82+
echo "TEST_RESULT=PASSED" >> $GITHUB_ENV
83+
else
84+
echo "TEST_RESULT=FAILED" >> $GITHUB_ENV
85+
fi
86+
)
87+
88+
# Commit the results if update results
89+
- name: Push new results to branch
90+
shell: bash
91+
if: ${{ inputs.update-results == 'true' }}
92+
run: |
93+
git config --local user.name github-actions[bot]
94+
git config --local user.email 41898282+github-actions[bot]@users.noreply.github.com
95+
git status
96+
git pull origin ${{ inputs.head-ref }}
97+
git add tests/results/
98+
git diff --cached --quiet || git commit -s -m "github-actions[bot]: Run examples: updated result files on your behalf"
99+
# git push origin ${{ inputs.head-ref }}
100+
git push https://x-access-token:${{ inputs.token }}@github.com/${{ inputs.repository }} HEAD:${{ inputs.head-ref }}
101+
102+
- name: Check if pytest passed
103+
shell: bash
104+
run: |
105+
if [ "$TEST_RESULT" == "PASSED" ]; then
106+
exit 0
107+
else
108+
exit 1
109+
fi

‎.github/workflows/run-examples-prep.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
name: Run examples on modified PDL files
3+
on: [pull_request]
4+
jobs:
5+
tests:
6+
name: Execution tests
7+
runs-on: ubuntu-latest
8+
strategy:
9+
fail-fast: false
10+
matrix:
11+
python-version: ['3.11', '3.12', '3.13']
12+
13+
steps:
14+
# Detect modified PDL files, includes Add, Modified, but not Deleted
15+
- uses: actions/checkout@v4
16+
with:
17+
fetch-depth: 0
18+
- name: Detect all PDL files that were changed or added
19+
id: changed-pdl-files
20+
uses: tj-actions/changed-files@6cb76d07bee4c9772c6882c06c37837bf82a04d3 # v46
21+
with:
22+
files: |
23+
**.pdl
24+
json: "true"
25+
escape_json: "false"
26+
- name: List PDL files that were modified or added and append to test_examples_run.yaml
27+
run: |
28+
echo ${{ steps.changed-pdl-files.outputs.all_changed_files }}
29+
if [ ${{ steps.changed-pdl-files.outputs.all_changed_files_count }} -gt 0 ]; then
30+
echo "early-stop=false" >> $GITHUB_ENV
31+
else
32+
echo "No file need to be checked, skipping all subsequent tests."
33+
echo "early-stop=true" >> $GITHUB_ENV
34+
fi
35+
36+
- name: Run examples
37+
uses: ./.github/actions/run-examples
38+
# Only run if there are modified PDL files
39+
if: env.early-stop == 'false'
40+
with:
41+
python-version: ${{ matrix.python-version }}
42+
runner-os: ${{ runner.os }}
43+
repository: ${{ github.event.pull_request.head.repo.full_name }}
44+
head-ref: ${{ github.event.pull_request.head.ref }}
45+
token: ${{ github.token }}
46+
update-results: 'false'
47+
check: ${{ steps.changed-pdl-files.outputs.all_changed_files }}

‎.github/workflows/run-examples.yml‎

Lines changed: 13 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ on:
66
- cron: '0 1 * * *'
77
workflow_dispatch:
88

9-
109
jobs:
1110
tests:
1211
name: Execution tests
@@ -15,93 +14,17 @@ jobs:
1514
fail-fast: false
1615
matrix:
1716
python-version: ['3.11', '3.12', '3.13']
18-
1917
steps:
20-
21-
# Free up some disk space
22-
- name: Remove unnecessary files
23-
run: |
24-
sudo rm -rf /usr/share/dotnet
25-
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
26-
27-
# Set up Ollama
28-
- name: Install Ollama and start server
29-
shell: bash
30-
run: |
31-
curl -fsSL https://ollama.com/install.sh | sudo -E sh
32-
33-
- name: Pull models in examples/
34-
shell: bash
35-
run: |
36-
ollama pull granite3.2:2b
37-
ollama pull granite3.2:8b
38-
ollama pull mxbai-embed-large
39-
ollama list
40-
41-
- name: Check that all required models are available
42-
shell: bash
43-
run: |
44-
models=("mxbai-embed-large" "granite3.2:2b" "granite3.2:8b")
45-
missing=0
46-
for model in "${models[@]}"; do
47-
if ! ollama list | awk 'NR>1 {print $1}' | grep -q "$model"; then
48-
echo "❌ Model $model (or substring) is missing!"
49-
missing=1
50-
fi
51-
done
52-
53-
if [ "$missing" -eq 1 ]; then
54-
exit 1
55-
else
56-
echo "✅ All expected models are available."
57-
fi
58-
59-
- name: Wait for Ollama server
60-
shell: bash
61-
run: |
62-
sleep 10
63-
time curl -i http://localhost:11434
64-
65-
# Run tests
66-
- uses: actions/checkout@v4
67-
with:
68-
ref: ${{ github.head_ref }}
69-
- name: Set up Python ${{ matrix.python-version }}
70-
uses: actions/setup-python@v5
71-
with:
72-
python-version: ${{ matrix.python-version }}
73-
- name: Cache pip
74-
uses: actions/cache@v4
75-
with:
76-
# This path is specific to Ubuntu
77-
path: ${{ env.pythonLocation }}
78-
# Look to see if there is a cache hit for the setup file
79-
key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
80-
restore-keys: |
81-
${{ runner.os }}-pip-new3
82-
${{ runner.os }}-new3
83-
- name: Install dependencies
84-
run: pip install --upgrade --upgrade-strategy eager .[all]
85-
- name: pip list packages
86-
run: pip list
87-
- name: show pip dependencies
88-
run: |
89-
pip install pipdeptree
90-
pipdeptree -fl
91-
- name: run tests
92-
env:
93-
WATSONX_PROJECT_ID: ${{ secrets.WATSONX_PROJECT_ID }}
94-
WATSONX_APIKEY: ${{ secrets.WATSONX_APIKEY }}
95-
WATSONX_URL: ${{ secrets.WATSONX_URL }}
96-
REPLICATE_API_TOKEN: ${{ secrets.REPLICATE_API_TOKEN }}
97-
OLLAMA_GHACTIONS_RESULTS: true
98-
run: py.test -v --capture=tee-sys -rfE -s tests/test_examples_run.py
99-
- name: Update example result files (if any) generated from Ollama running on GH Actions
100-
if: matrix.python-version == '3.11'
101-
run: |
102-
git config --local user.name github-actions[bot]
103-
git config --local user.email "${{ github.actor_id }}+${{ github.actor }}@users.noreply.github.com"
104-
git status
105-
git add tests/results/
106-
git diff --cached --quiet || git commit -S -s -m "github-actions[bot]: Updated results file when running examples on $(date)"
107-
git push
18+
- uses: actions/checkout@v4
19+
with:
20+
ref: ${{ github.head_ref }}
21+
- uses: ./.github/actions/run-examples
22+
with:
23+
python-version: ${{ matrix.python-version }}
24+
runner-os: ${{ runner.os }}
25+
repository: ${{ github.repository }}
26+
head-ref: ${{ github.head_ref }}
27+
token: ${{ github.token }}
28+
update-results: 'false'
29+
check: '[]' # Empty list means run against all PDL programs
30+

0 commit comments

Comments
 (0)