Skip to content

Commit 5e7c225

Browse files
ashwinbashwinb
authored
test(recording): add a script to schedule recording workflow (llamastack#3170)
See comment here: llamastack#3162 (comment) -- TL;DR it is quite complex to invoke the recording workflow correctly for an end developer writing tests. This script simplifies the work. No more manual GitHub UI navigation! ## Script Functionality - Auto-detects your current branch and associated PR - Finds the right repository context (works from forks!) - Runs the workflow where it can actually commit back - Validates prerequisites and provides helpful error messages ## How to Use First ensure you are on the branch which introduced a new test and want it recorded. **Make sure you have pushed this branch remotely, easiest is to create a PR.** ``` # Record tests for current branch ./scripts/github/schedule-record-workflow.sh # Record specific test subdirectories ./scripts/github/schedule-record-workflow.sh --test-subdirs "agents,inference" # Record with vision tests enabled ./scripts/github/schedule-record-workflow.sh --run-vision-tests # Record tests matching a pattern ./scripts/github/schedule-record-workflow.sh --test-pattern "test_streaming" ``` ## Test Plan Ran `./scripts/github/schedule-record-workflow.sh -s inference -k tool_choice` which started https://github.com/ashwinb/llama-stack/actions/runs/17001534248/job/48204093292 which successfully committed recorded outputs.
1 parent 914c7be commit 5e7c225

File tree

4 files changed

+329
-2
lines changed

4 files changed

+329
-2
lines changed

‎.github/workflows/record-integration-tests.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,16 @@ jobs:
3535
contents: write
3636

3737
steps:
38+
- name: Echo workflow inputs
39+
run: |
40+
echo "::group::Workflow Inputs"
41+
echo "test-subdirs: ${{ inputs.test-subdirs }}"
42+
echo "test-provider: ${{ inputs.test-provider }}"
43+
echo "run-vision-tests: ${{ inputs.run-vision-tests }}"
44+
echo "test-pattern: ${{ inputs.test-pattern }}"
45+
echo "branch: ${{ github.ref_name }}"
46+
echo "::endgroup::"
47+
3848
- name: Checkout repository
3949
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
4050
with:

‎scripts/github/schedule-record-workflow.sh‎

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,279 @@
1+
#!/bin/bash
2+
3+
# Copyright (c) Meta Platforms, Inc. and affiliates.
4+
# All rights reserved.
5+
#
6+
# This source code is licensed under the terms described in the LICENSE file in
7+
# the root directory of this source tree.
8+
9+
# Script to easily trigger the integration test recording workflow
10+
# Usage: ./scripts/github/schedule-record-workflow.sh [options]
11+
12+
set -euo pipefail
13+
14+
# Default values
15+
BRANCH=""
16+
TEST_SUBDIRS=""
17+
TEST_PROVIDER="ollama"
18+
RUN_VISION_TESTS=false
19+
TEST_PATTERN=""
20+
21+
# Help function
22+
show_help() {
23+
cat << EOF
24+
Usage: $0 [OPTIONS]
25+
26+
Trigger the integration test recording workflow remotely. This way you do not need to have Ollama running locally.
27+
28+
OPTIONS:
29+
-b, --branch BRANCH Branch to run the workflow on (defaults to current branch)
30+
-s, --test-subdirs DIRS Comma-separated list of test subdirectories to run (REQUIRED)
31+
-p, --test-provider PROVIDER Test provider to use: vllm or ollama (default: ollama)
32+
-v, --run-vision-tests Include vision tests in the recording
33+
-k, --test-pattern PATTERN Regex pattern to pass to pytest -k
34+
-h, --help Show this help message
35+
36+
EXAMPLES:
37+
# Record tests for current branch with agents subdirectory
38+
$0 --test-subdirs "agents"
39+
40+
# Record tests for specific branch with vision tests
41+
$0 -b my-feature-branch --test-subdirs "inference" --run-vision-tests
42+
43+
# Record multiple test subdirectories with specific provider
44+
$0 --test-subdirs "agents,inference" --test-provider vllm
45+
46+
# Record tests matching a specific pattern
47+
$0 --test-subdirs "inference" --test-pattern "test_streaming"
48+
49+
EOF
50+
}
51+
52+
# PREREQUISITES:
53+
# - GitHub CLI (gh) must be installed and authenticated
54+
# - jq must be installed for JSON parsing
55+
# - You must be in a git repository that is a fork or clone of llamastack/llama-stack
56+
# - The branch must exist on the remote repository where you want to run the workflow
57+
# - You must specify test subdirectories to run with -s/--test-subdirs
58+
59+
# Parse command line arguments
60+
while [[ $# -gt 0 ]]; do
61+
case $1 in
62+
-b|--branch)
63+
BRANCH="$2"
64+
shift 2
65+
;;
66+
-s|--test-subdirs)
67+
TEST_SUBDIRS="$2"
68+
shift 2
69+
;;
70+
-p|--test-provider)
71+
TEST_PROVIDER="$2"
72+
shift 2
73+
;;
74+
-v|--run-vision-tests)
75+
RUN_VISION_TESTS=true
76+
shift
77+
;;
78+
-k|--test-pattern)
79+
TEST_PATTERN="$2"
80+
shift 2
81+
;;
82+
-h|--help)
83+
show_help
84+
exit 0
85+
;;
86+
*)
87+
echo "Unknown option: $1"
88+
show_help
89+
exit 1
90+
;;
91+
esac
92+
done
93+
94+
# Validate required parameters
95+
if [[ -z "$TEST_SUBDIRS" ]]; then
96+
echo "Error: --test-subdirs is required"
97+
echo "Please specify which test subdirectories to run, e.g.:"
98+
echo " $0 --test-subdirs \"agents,inference\""
99+
echo " $0 --test-subdirs \"inference\" --run-vision-tests"
100+
echo ""
101+
exit 1
102+
fi
103+
104+
# Validate test provider
105+
if [[ "$TEST_PROVIDER" != "vllm" && "$TEST_PROVIDER" != "ollama" ]]; then
106+
echo "❌ Error: Invalid test provider '$TEST_PROVIDER'"
107+
echo " Supported providers: vllm, ollama"
108+
echo " Example: $0 --test-subdirs \"agents\" --test-provider vllm"
109+
exit 1
110+
fi
111+
112+
# Check if required tools are installed
113+
if ! command -v gh &> /dev/null; then
114+
echo "Error: GitHub CLI (gh) is not installed. Please install it from https://cli.github.com/"
115+
exit 1
116+
fi
117+
118+
if ! gh auth status &> /dev/null; then
119+
echo "Error: GitHub CLI is not authenticated. Please run 'gh auth login'"
120+
exit 1
121+
fi
122+
123+
# If no branch specified, use current branch
124+
if [[ -z "$BRANCH" ]]; then
125+
BRANCH=$(git branch --show-current)
126+
echo "No branch specified, using current branch: $BRANCH"
127+
128+
# Optionally look for associated PR for context (not required)
129+
echo "Looking for associated PR..."
130+
131+
# Search for PRs in the main repo that might match this branch
132+
# This searches llamastack/llama-stack for any PR with this head branch name
133+
if PR_INFO=$(gh pr list --repo llamastack/llama-stack --head "$BRANCH" --json number,headRefName,headRepository,headRepositoryOwner,url,state --limit 1 2>/dev/null) && [[ "$PR_INFO" != "[]" ]]; then
134+
# Parse PR info using jq
135+
PR_NUMBER=$(echo "$PR_INFO" | jq -r '.[0].number')
136+
PR_HEAD_REPO=$(echo "$PR_INFO" | jq -r '.[0].headRepositoryOwner.login // "llamastack"')
137+
PR_URL=$(echo "$PR_INFO" | jq -r '.[0].url')
138+
PR_STATE=$(echo "$PR_INFO" | jq -r '.[0].state')
139+
140+
if [[ -n "$PR_NUMBER" && -n "$PR_HEAD_REPO" ]]; then
141+
echo "✅ Found associated PR #$PR_NUMBER ($PR_STATE)"
142+
echo " URL: $PR_URL"
143+
echo " Head repository: $PR_HEAD_REPO/llama-stack"
144+
145+
# Check PR state and block if merged
146+
if [[ "$PR_STATE" == "CLOSED" ]]; then
147+
echo "ℹ️ Note: This PR is closed, but workflow can still run to update recordings."
148+
elif [[ "$PR_STATE" == "MERGED" ]]; then
149+
echo "❌ Error: This PR is already merged."
150+
echo " Cannot record tests for a merged PR since changes can't be committed back."
151+
echo " Create a new branch/PR if you need to record new tests."
152+
exit 1
153+
fi
154+
fi
155+
else
156+
echo "ℹ️ No associated PR found for branch '$BRANCH'"
157+
echo "That's fine - the workflow just needs a pushed branch to run."
158+
fi
159+
echo ""
160+
fi
161+
162+
# Determine the target repository for workflow dispatch based on where the branch actually exists
163+
# We need to find which remote has the branch we want to run the workflow on
164+
165+
echo "Determining target repository for workflow..."
166+
167+
# Check if we have PR info with head repository
168+
if [[ -n "$PR_HEAD_REPO" ]]; then
169+
# Use the repository from the PR head
170+
TARGET_REPO="$PR_HEAD_REPO/llama-stack"
171+
echo "📍 Using PR head repository: $TARGET_REPO"
172+
173+
if [[ "$PR_HEAD_REPO" == "llamastack" ]]; then
174+
REPO_CONTEXT=""
175+
else
176+
REPO_CONTEXT="--repo $TARGET_REPO"
177+
fi
178+
else
179+
# Fallback: find which remote has the branch
180+
BRANCH_REMOTE=""
181+
for remote in $(git remote); do
182+
if git ls-remote --heads "$remote" "$BRANCH" | grep -q "$BRANCH"; then
183+
REMOTE_URL=$(git remote get-url "$remote")
184+
if [[ "$REMOTE_URL" == *"/llama-stack"* ]]; then
185+
REPO_OWNER=$(echo "$REMOTE_URL" | sed -n 's/.*[:/]\([^/]*\)\/llama-stack.*/\1/p')
186+
echo "📍 Found branch '$BRANCH' on remote '$remote' ($REPO_OWNER/llama-stack)"
187+
TARGET_REPO="$REPO_OWNER/llama-stack"
188+
BRANCH_REMOTE="$remote"
189+
break
190+
fi
191+
fi
192+
done
193+
194+
if [[ -z "$BRANCH_REMOTE" ]]; then
195+
echo "Error: Could not find branch '$BRANCH' on any llama-stack remote"
196+
echo ""
197+
echo "This could mean:"
198+
echo " - The branch doesn't exist on any remote yet (push it first)"
199+
echo " - The branch name is misspelled"
200+
echo " - No llama-stack remotes are configured"
201+
echo ""
202+
echo "Available remotes:"
203+
git remote -v
204+
echo ""
205+
echo "To push your branch: git push <remote> $BRANCH"
206+
echo "Common remotes to try: origin, upstream, your-username"
207+
exit 1
208+
fi
209+
210+
if [[ "$TARGET_REPO" == "llamastack/llama-stack" ]]; then
211+
REPO_CONTEXT=""
212+
else
213+
REPO_CONTEXT="--repo $TARGET_REPO"
214+
fi
215+
fi
216+
217+
echo " Workflow will run on: $TARGET_REPO"
218+
219+
# Verify the target repository has the workflow file
220+
echo "Verifying workflow exists on target repository..."
221+
if ! gh api "repos/$TARGET_REPO/contents/.github/workflows/record-integration-tests.yml" &>/dev/null; then
222+
echo "Error: The recording workflow does not exist on $TARGET_REPO"
223+
echo "This could mean:"
224+
echo " - The fork doesn't have the latest workflow file"
225+
echo " - The workflow file was renamed or moved"
226+
echo ""
227+
if [[ "$TARGET_REPO" != "llamastack/llama-stack" ]]; then
228+
echo "Try syncing your fork with upstream:"
229+
echo " git fetch upstream"
230+
echo " git checkout main"
231+
echo " git merge upstream/main"
232+
echo " git push origin main"
233+
fi
234+
exit 1
235+
fi
236+
237+
# Build the workflow dispatch command
238+
echo "Triggering integration test recording workflow..."
239+
echo "Branch: $BRANCH"
240+
echo "Test provider: $TEST_PROVIDER"
241+
echo "Test subdirs: $TEST_SUBDIRS"
242+
echo "Run vision tests: $RUN_VISION_TESTS"
243+
echo "Test pattern: ${TEST_PATTERN:-"(none)"}"
244+
echo ""
245+
246+
# Prepare inputs for gh workflow run
247+
INPUTS="-f test-subdirs='$TEST_SUBDIRS'"
248+
if [[ -n "$TEST_PROVIDER" ]]; then
249+
INPUTS="$INPUTS -f test-provider='$TEST_PROVIDER'"
250+
fi
251+
if [[ "$RUN_VISION_TESTS" == "true" ]]; then
252+
INPUTS="$INPUTS -f run-vision-tests=true"
253+
fi
254+
if [[ -n "$TEST_PATTERN" ]]; then
255+
INPUTS="$INPUTS -f test-pattern='$TEST_PATTERN'"
256+
fi
257+
258+
# Run the workflow
259+
WORKFLOW_CMD="gh workflow run record-integration-tests.yml --ref $BRANCH $REPO_CONTEXT $INPUTS"
260+
echo "Running: $WORKFLOW_CMD"
261+
echo ""
262+
263+
if eval "$WORKFLOW_CMD"; then
264+
echo "✅ Workflow triggered successfully!"
265+
echo ""
266+
echo "You can monitor the workflow run at:"
267+
echo "https://github.com/$TARGET_REPO/actions/workflows/record-integration-tests.yml"
268+
echo ""
269+
if [[ -n "$REPO_CONTEXT" ]]; then
270+
echo "Or use: gh run list --workflow=record-integration-tests.yml $REPO_CONTEXT"
271+
echo "And then: gh run watch <RUN_ID> $REPO_CONTEXT"
272+
else
273+
echo "Or use: gh run list --workflow=record-integration-tests.yml"
274+
echo "And then: gh run watch <RUN_ID>"
275+
fi
276+
else
277+
echo "❌ Failed to trigger workflow"
278+
exit 1
279+
fi

‎tests/README.md‎

Lines changed: 31 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,9 @@ FIREWORKS_API_KEY=your_key pytest -sv tests/integration/inference --stack-config
6060

6161
### Re-recording tests
6262

63-
If you want to re-record tests, you can do so with:
63+
#### Local Re-recording (Manual Setup Required)
64+
65+
If you want to re-record tests locally, you can do so with:
6466

6567
```bash
6668
LLAMA_STACK_TEST_INFERENCE_MODE=record \
@@ -71,14 +73,41 @@ LLAMA_STACK_TEST_INFERENCE_MODE=record \
7173

7274
This will record new API responses and overwrite the existing recordings.
7375

74-
7576
```{warning}
7677
7778
You must be careful when re-recording. CI workflows assume a specific setup for running the replay-mode tests. You must re-record the tests in the same way as the CI workflows. This means
7879
- you need Ollama running and serving some specific models.
7980
- you are using the `starter` distribution.
8081
```
8182

83+
#### Remote Re-recording (Recommended)
84+
85+
**For easier re-recording without local setup**, use the automated recording workflow:
86+
87+
```bash
88+
# Record tests for specific test subdirectories
89+
./scripts/github/schedule-record-workflow.sh --test-subdirs "agents,inference"
90+
91+
# Record with vision tests enabled
92+
./scripts/github/schedule-record-workflow.sh --test-subdirs "inference" --run-vision-tests
93+
94+
# Record with specific provider
95+
./scripts/github/schedule-record-workflow.sh --test-subdirs "agents" --test-provider vllm
96+
```
97+
98+
This script:
99+
- 🚀 **Runs in GitHub Actions** - no local Ollama setup required
100+
- 🔍 **Auto-detects your branch** and associated PR
101+
- 🍴 **Works from forks** - handles repository context automatically
102+
- ✅ **Commits recordings back** to your branch
103+
104+
**Prerequisites:**
105+
- GitHub CLI: `brew install gh && gh auth login`
106+
- jq: `brew install jq`
107+
- Your branch pushed to a remote
108+
109+
**Supported providers:** `vllm`, `ollama`
110+
82111

83112
### Next Steps
84113

‎tests/integration/README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,15 @@ cat recordings/responses/abc123.json | jq '.'
134134
```
135135

136136
### Re-recording Tests
137+
138+
#### Remote Re-recording (Recommended)
139+
Use the automated workflow script for easier re-recording:
140+
```bash
141+
./scripts/github/schedule-record-workflow.sh --test-subdirs "inference,agents"
142+
```
143+
See the [main testing guide](../README.md#remote-re-recording-recommended) for full details.
144+
145+
#### Local Re-recording
137146
```bash
138147
# Re-record specific tests
139148
LLAMA_STACK_TEST_INFERENCE_MODE=record \

0 commit comments

Comments
 (0)