Skip to content

Commit d2037cd

Browse files
cgraham-rscgraham-rs
authored
Add Connect integration deployment tests (#53)
* Add Connect integration test infrastructure * Use API key to enable successful content URL request * Test worfklow * Pass CONNECT_LICENSE in integration test workflow * extension.yml must pass secrets to called workflows * Add a reusable GitHub Action to execute Connect integration tests * Leverage the integration test Action, handle the complex matrix of inputs, test results * Add debug, ensure actions/checkout, allow Action to fail from calling job * Fix minor issues, cleanup CLI output * Use correct Action input from calling workflow * Debug license failure * Remove license debug * More matrix handling for test reporting * More result handling * More collect-results fixes and debug * More collect-results test result handling fixes * Trigger simple extension workflow * Trigger multiple simple extension workflows * Add dev preview to list of Connect versions to test against * Minor refactor to ensure we track matrix job success and failure * Use artifacts to collect all simple-extensions that were successfully packaged, other fixes * Fix collect-results to handle custom workflows correctly. * Fix integration-test-report artifact pattern * Ensure integration-test-report artifacts are unique for each workflow run * Fix integration test result publishing, add code comments, general cleanup * Remove unsupported Connect versions, remove temp app.py comments * Always checkout before running actions * Remove last temp code comment to trigger simple extension worflow * Address PR review items * Refactor to move everything into the integration folder, add test debug output * Temp code change to content to force CI to run * Add missing uv and setup from posit-sdk * Add path to setup-uv * Fix venv * Debug venv failures * More venv handling and debug * Set astral-sh/setup-uv to latest version * Ensure uv.lock is created, more debug, allow manual preview execution * Still fumbling with the container venv handling * Add simple integration README * Cleanup unused commands * Streamline collect-simple-extensions to use existing packaging artifacts * Set integration-tests collect-results job to run always() * Set simple-extension-release to run always() * Fix simple extension collection post-packaging * Collect results using the data in the XML test reports which we already have as artifacts * Add manifest.json to integration-session-manager * Remove comments from app.py to force CI
1 parent 681a7c7 commit d2037cd

File tree

14 files changed

+974
-12
lines changed

14 files changed

+974
-12
lines changed

‎.github/actions/connect-integration-test/action.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: 'Run Connect Integration Test'
2+
description: 'Run integration test for a single extension'
3+
4+
inputs:
5+
extension-name:
6+
description: 'Name of extension to test'
7+
required: true
8+
connect-version:
9+
description: 'Connect version to test against'
10+
required: true
11+
connect-license:
12+
description: 'Posit Connect license'
13+
required: true
14+
15+
outputs:
16+
test_status:
17+
description: 'Status of the integration test'
18+
value: ${{ steps.run-test.outputs.status }}
19+
reports_path:
20+
description: 'Path to test reports'
21+
value: ${{ steps.run-test.outputs.reports_path }}
22+
23+
runs:
24+
using: "composite"
25+
steps:
26+
- uses: actions/checkout@v4
27+
28+
# Setup Docker BuildX which is required for the integration tests
29+
- uses: docker/setup-buildx-action@v3
30+
31+
- name: Write Connect license
32+
shell: bash
33+
run: echo "${{ inputs.connect-license }}" > ./integration/license.lic
34+
35+
# Here we download the packaged extension artifact created from the calling workflow
36+
- uses: actions/download-artifact@v4
37+
with:
38+
name: ${{ inputs.extension-name }}.tar.gz
39+
path: integration/bundles
40+
41+
- uses: astral-sh/setup-uv@v5
42+
with:
43+
pyproject-file: "./integration/pyproject.toml"
44+
45+
- name: Install isolated Python with UV
46+
shell: bash
47+
working-directory: ./integration
48+
run: uv python install
49+
50+
# Run the test and capture the report path and status
51+
- shell: bash
52+
run: |
53+
make -C ./integration ${{ inputs.connect-version }} \
54+
EXTENSION_NAME=${{ inputs.extension-name }}
55+
echo "reports_path=$(pwd)/integration/reports" >> $GITHUB_OUTPUT
56+
echo "status=$?" >> $GITHUB_OUTPUT

‎.github/workflows/connect-integration-tests.yml‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
name: Connect Integration Tests
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
extensions:
7+
description: "JSON array of extension names to test"
8+
required: true
9+
type: string
10+
outputs:
11+
successful_extensions:
12+
description: "JSON array of extension names that have passed all tests"
13+
value: ${{ jobs.collect-results.outputs.successful_extensions }}
14+
secrets:
15+
CONNECT_LICENSE:
16+
required: true
17+
18+
jobs:
19+
# Determine the Connect versions to test against
20+
setup-integration-test:
21+
runs-on: ubuntu-latest
22+
timeout-minutes: 5
23+
outputs:
24+
versions: ${{ steps.versions.outputs.versions }}
25+
steps:
26+
- uses: actions/checkout@v4
27+
28+
- id: versions
29+
working-directory: ./integration
30+
# The `jq` command is "output compact, raw input, slurp, split on new lines, and remove the last (empty) element"
31+
# This results in a JSON array of Connect versions (e.g., ["2025.01.0", "2024.12.0"])
32+
run: |
33+
versions=$(make print-versions | jq -c -Rs 'split("\n") | .[:-1]')
34+
echo "Versions: $versions"
35+
echo "versions=$versions" >> "$GITHUB_OUTPUT"
36+
37+
# Run the Connect integration tests for each extension against each Connect version
38+
connect-integration-test:
39+
runs-on: ubuntu-latest
40+
timeout-minutes: 15 # Max time to run the integration tests
41+
needs: setup-integration-test
42+
strategy:
43+
# Do not fail fast so all extensions and Connect versions are processed
44+
fail-fast: false
45+
matrix:
46+
extension: ${{ fromJson(inputs.extensions) }}
47+
connect_version: ${{ fromJson(needs.setup-integration-test.outputs.versions) }}
48+
steps:
49+
- uses: actions/checkout@v4
50+
51+
- uses: ./.github/actions/connect-integration-test
52+
id: test
53+
with:
54+
extension-name: ${{ matrix.extension }}
55+
connect-version: ${{ matrix.connect_version }}
56+
connect-license: ${{ secrets.CONNECT_LICENSE }}
57+
58+
- uses: actions/upload-artifact@v4
59+
if: |
60+
always() &&
61+
steps.test.outcome != 'cancelled' &&
62+
steps.test.outcome != 'skipped'
63+
with:
64+
name: ${{ matrix.extension }}-${{ matrix.connect_version }}-test-report
65+
path: integration/reports/*.xml
66+
retention-days: 7
67+
68+
# Analyses the test result files and publishes the results on the GitHub Actions job summary page
69+
integration-test-report:
70+
needs: connect-integration-test
71+
runs-on: ubuntu-latest
72+
timeout-minutes: 5
73+
# Create a matrix so each extension gets its own test report
74+
strategy:
75+
fail-fast: false
76+
matrix:
77+
extension: ${{ fromJson(inputs.extensions) }}
78+
permissions:
79+
checks: write
80+
pull-requests: write
81+
# Only run if tests weren't skipped or cancelled
82+
if: |
83+
always() &&
84+
!contains(needs.connect-integration-test.result, 'skipped') &&
85+
!contains(needs.connect-integration-test.result, 'cancelled')
86+
steps:
87+
- uses: actions/download-artifact@v4
88+
id: download
89+
with:
90+
path: artifacts
91+
pattern: "${{ matrix.extension }}-*-test-report"
92+
93+
- uses: EnricoMi/publish-unit-test-result-action@v2
94+
if: ${{ steps.download.outputs.download-path != '' }}
95+
with:
96+
check_name: "Integration test results - ${{ matrix.extension }}"
97+
comment_mode: off
98+
files: "artifacts/**/*.xml"
99+
report_individual_runs: true
100+
101+
# Using the XML test reports provide a matrix of extensions that passed all of the Connect integration tests
102+
collect-results:
103+
needs: [connect-integration-test, setup-integration-test]
104+
runs-on: ubuntu-latest
105+
timeout-minutes: 5
106+
outputs:
107+
successful_extensions: ${{ steps.collect.outputs.successful_extensions }}
108+
if: always()
109+
steps:
110+
- uses: actions/download-artifact@v4
111+
id: download
112+
with:
113+
path: artifacts
114+
pattern: "*-test-report"
115+
116+
- id: collect
117+
run: |
118+
# Validate inputs first
119+
all_versions='${{ needs.setup-integration-test.outputs.versions }}'
120+
extensions='${{ inputs.extensions }}'
121+
122+
if [[ -z "$all_versions" || -z "$extensions" ]]; then
123+
echo "❌ Missing required inputs"
124+
exit 1
125+
fi
126+
127+
# Debug info
128+
echo "::group::Debug Inputs"
129+
echo "Extensions to check: $extensions"
130+
echo "Connect versions: $all_versions"
131+
echo "::endgroup::"
132+
133+
# Track extensions that passed ALL version tests
134+
success_list=()
135+
136+
for ext in $(echo "$extensions" | jq -r '.[]'); do
137+
all_passed=true
138+
echo "📦 Checking extension: $ext"
139+
140+
for version in $(echo "$all_versions" | jq -r '.[]'); do
141+
echo "🔎 Checking $ext @ $version"
142+
report_dir="artifacts/${ext}-${version}-test-report"
143+
144+
if [ ! -d "$report_dir" ]; then
145+
echo "❌ No test report for $ext @ $version"
146+
all_passed=false
147+
break
148+
fi
149+
150+
# Use grep to check for failures/errors using XML test report attributes
151+
failures=$(grep -o 'failures="[0-9]*"' "$report_dir"/*.xml | sed 's/failures="//g' | sed 's/"//g' | awk '{sum+=$1} END {print sum}' || echo "0")
152+
errors=$(grep -o 'errors="[0-9]*"' "$report_dir"/*.xml | sed 's/errors="//g' | sed 's/"//g' | awk '{sum+=$1} END {print sum}' || echo "0")
153+
154+
if [ "$failures" -gt 0 ] || [ "$errors" -gt 0 ]; then
155+
echo "❌ Found $failures failures, $errors errors in test suite attributes"
156+
# Extract and show some error details for debugging
157+
echo "::group::Error details"
158+
grep -r -A1 "<failure" "$report_dir" | head -10
159+
grep -r -A1 "<error" "$report_dir" | head -10
160+
echo "::endgroup::"
161+
all_passed=false
162+
break
163+
else
164+
echo "✅ Passed: $ext @ $version"
165+
fi
166+
done
167+
168+
if [ "$all_passed" = "true" ]; then
169+
success_list+=("$ext")
170+
echo "🎉 SUCCESS: $ext passed ALL versions"
171+
else
172+
echo "⚠️ FAILED: $ext failed one or more versions"
173+
fi
174+
done
175+
176+
# Format output and ensure we always have a valid JSON array
177+
if [ ${#success_list[@]} -eq 0 ]; then
178+
successful_extensions="[]"
179+
else
180+
successful_extensions=$(jq -n --arg arr "$(IFS=,; echo "${success_list[*]}")" \
181+
'$arr | split(",") | map(select(length > 0))' -c)
182+
fi
183+
184+
echo "successful_extensions=$successful_extensions" >> $GITHUB_OUTPUT

‎.github/workflows/extensions.yml‎

Lines changed: 78 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ jobs:
3333
integration-session-manager: extensions/integration-session-manager/**
3434
3535
# Runs for each extension that has changed from `simple-extension-changes`
36-
# Lints, packages, and releases the extension if the semver is updated.
36+
# Lints and packages in preparation for tests and and release.
3737
simple-extensions:
3838
needs: [simple-extension-changes]
3939
# Will only run if there are changes in the simple extensions
@@ -61,9 +61,80 @@ jobs:
6161
with:
6262
extension-name: ${{ matrix.extension }}
6363

64-
# Extensions are only released when this workflow triggers on `main`
65-
# otherwise, the release is skipped
66-
# See the action comments for more details
64+
# Uploading an artifact to reflect extensions that were successfully linted and packaged.
65+
# Using artifacts to avoid explicit output names for each simple extensions, requiring
66+
# less file changes when adding simple extensions to this workflow.
67+
# https://github.com/orgs/community/discussions/17245
68+
- name: Mark successful jobs
69+
run: echo "${{ matrix.extension }}" >> ${{ matrix.extension }}.packaged.successfully
70+
71+
- uses: actions/upload-artifact@v4
72+
with:
73+
name: ${{ matrix.extension }}.packaged.successfully
74+
path: ${{ matrix.extension }}.packaged.successfully
75+
76+
# Collect success results from the matrix simple-extensions jobs for later use
77+
collect-simple-extensions:
78+
needs: simple-extensions
79+
runs-on: ubuntu-latest
80+
outputs:
81+
successful_extensions: ${{ steps.collect.outputs.successful_extensions }}
82+
steps:
83+
- uses: actions/download-artifact@v4
84+
with:
85+
# Uses artifacts generated from package-extension which are only present if packaging was successful
86+
pattern: "*.packaged.successfully"
87+
path: packages
88+
merge-multiple: true
89+
90+
- id: collect
91+
run: |
92+
SUCCESS_LIST=()
93+
if [ -d "packages" ]; then
94+
for FILE in packages/*.packaged.successfully; do
95+
if [ -f "$FILE" ]; then
96+
# Extract extension name from filename (remove .packaged.successfully)
97+
EXT=$(basename "$FILE" .packaged.successfully)
98+
SUCCESS_LIST+=("$EXT")
99+
echo "Found successful extension package: $EXT"
100+
fi
101+
done
102+
fi
103+
104+
SUCCESSFUL_EXTENSIONS=$(jq -n --arg arr "$(IFS=,; echo "${SUCCESS_LIST[*]}")" \
105+
'$arr | split(",")' -c)
106+
echo "Successful extensions: $SUCCESSFUL_EXTENSIONS"
107+
echo "successful_extensions=$SUCCESSFUL_EXTENSIONS" >> "$GITHUB_OUTPUT"
108+
109+
110+
# Runs Connect integration tests for each extension that were successfully linted and packaged
111+
simple-extension-connect-integration-tests:
112+
needs: [collect-simple-extensions]
113+
uses: ./.github/workflows/connect-integration-tests.yml
114+
with:
115+
extensions: ${{ needs.collect-simple-extensions.outputs.successful_extensions }}
116+
if: fromJSON(needs.collect-simple-extensions.outputs.successful_extensions) != '[]'
117+
secrets: inherit
118+
119+
# Runs the release process for each extension that was packaged, passed all tests, if the semver is updated, on main
120+
simple-extension-release:
121+
runs-on: ubuntu-latest
122+
needs: [simple-extension-connect-integration-tests]
123+
if: always()
124+
env:
125+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
126+
strategy:
127+
# Do not fail fast so all extensions are processed
128+
fail-fast: false
129+
matrix:
130+
# Ensure the matrix is set to only process extensions that passed ALL integration tests
131+
extension: ${{ fromJSON(needs.simple-extension-connect-integration-tests.outputs.successful_extensions) }}
132+
# Extensions are only released when this workflow triggers on `main`
133+
# otherwise, the release is skipped
134+
# See the action comments for more details
135+
steps:
136+
- uses: actions/checkout@v4
137+
67138
- uses: ./.github/actions/release-extension
68139
with:
69140
extension-name: ${{ matrix.extension }}
@@ -104,10 +175,10 @@ jobs:
104175
# publisher-command-center extension directory
105176
if: ${{ needs.complex-extension-changes.outputs.publisher-command-center == 'true' }}
106177
uses: ./.github/workflows/publisher-command-center.yml
107-
178+
secrets: inherit
108179

109180
# All extensions have been linted, packaged, and released, if necessary
110-
# Continuing to update the extension list with the latest release data
181+
# Continuing to update the extension list with the latest release data
111182

112183
# Gathers all release data from GitHub releases triggered by this workflow
113184
# For use in the `update-extension-list` job
@@ -116,7 +187,7 @@ jobs:
116187
runs-on: ubuntu-latest
117188
# Requires that the `simple-extensions` and all custom workflow jobs are
118189
# completed before running this job
119-
needs: [simple-extensions, publisher-command-center]
190+
needs: [simple-extension-release, publisher-command-center]
120191
if: ${{ always() }}
121192
outputs:
122193
releases: ${{ steps.fetch-releases.outputs.releases }}

‎.github/workflows/publisher-command-center.yml‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -59,9 +59,23 @@ jobs:
5959
extension-name: ${{ env.EXTENSION_NAME }}
6060
artifact-name: ${{ env.EXTENSION_NAME }}
6161

62-
# Release the extension using the release-extension action
63-
# Will only create a GitHub release if merged to `main` and the semver
64-
# version has been updated
62+
connect-integration-tests:
63+
needs: extension
64+
uses: ./.github/workflows/connect-integration-tests.yml
65+
secrets: inherit
66+
with:
67+
extensions: '["publisher-command-center"]' # JSON array format to match the workflow input schema
68+
69+
release:
70+
runs-on: ubuntu-latest
71+
needs: [extension, connect-integration-tests]
72+
# Release the extension using the release-extension action
73+
# Will only create a GitHub release if merged to `main` and the semver
74+
# version has been updated
75+
steps:
76+
# Checkout the repository so the rest of the actions can run with no issue
77+
- uses: actions/checkout@v4
78+
6579
- uses: ./.github/actions/release-extension
6680
with:
6781
extension-name: ${{ env.EXTENSION_NAME }}

0 commit comments

Comments
 (0)