Skip to content

Commit 0ae7a19

Browse files
cgraham-rsdotNomad
cgraham-rs
and
dotNomad
authored
Integration tests should honor minimumConnectVersion (#267)
* Only run integrations tests if the minimum Connect version is met * Temporarily change reaper extension's minimumConnectVersion to invoke integration tests * Have Claude setup a complex test matrix for running integration tests on different extensions and Connect versions * Temporarily change Content Health Monitor's minimumConnectVersion to invoke integration tests * Fix matrix JSON formatting issues * Fail setup-extension-versions if the minimumConnectVersion is not in CONNECT_VERSIONS * Fix collect-results step to handle failed setup * Add 2025.07.0 to CONNECT_VERSIONS * Revert "Temporarily change reaper extension's minimumConnectVersion to invoke integration tests" This reverts commit 26b299d. * Revert "Temporarily change Content Health Monitor's minimumConnectVersion to invoke integration tests" This reverts commit b1448c7. * Update .github/workflows/connect-integration-tests.yml Co-authored-by: Jordan Jensen <[email protected]> * Temporarily change reaper extension's minimumConnectVersion to invoke integration tests * Temporarily change Content Health Monitor's minimumConnectVersion to invoke integration tests * Revert "Temporarily change reaper extension's minimumConnectVersion to invoke integration tests" This reverts commit b2809e3. * Revert "Temporarily change Content Health Monitor's minimumConnectVersion to invoke integration tests" This reverts commit cb3911a. --------- Co-authored-by: Jordan Jensen <[email protected]>
1 parent 00c7c92 commit 0ae7a19

File tree

2 files changed

+151
-17
lines changed

2 files changed

+151
-17
lines changed

‎.github/workflows/connect-integration-tests.yml‎

Lines changed: 150 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,18 @@
11
name: Connect Integration Tests
22

3+
# This workflow tests Connect extensions against compatible versions of Posit Connect.
4+
# It determines compatibility based on each extension's minimumConnectVersion property,
5+
# which is required in all extension manifests.
6+
#
7+
# Key workflow features:
8+
# 1. Determines all available Connect versions from integration/Makefile
9+
# 2. Reads each extension's minimumConnectVersion from its manifest
10+
# 3. Creates a dynamic test matrix that only includes compatible combinations:
11+
# - All extensions are ALWAYS tested with the preview version
12+
# - Extensions are ONLY tested with Connect versions that meet or exceed their minimumConnectVersion
13+
# 4. Runs integration tests only for compatible combinations (instead of skipped jobs polluting results)
14+
# 5. Collects and analyzes test results to determine which extensions passed all their tests
15+
316
on:
417
workflow_call:
518
inputs:
@@ -34,20 +47,121 @@ jobs:
3447
echo "Versions: $versions"
3548
echo "versions=$versions" >> "$GITHUB_OUTPUT"
3649
37-
# Run the Connect integration tests for each extension against each Connect version
50+
# Determine compatible Connect versions for each extension based on minimumConnectVersion
51+
setup-extension-versions:
52+
runs-on: ubuntu-latest
53+
timeout-minutes: 5
54+
needs: setup-test
55+
outputs:
56+
matrix: ${{ steps.generate_matrix.outputs.matrix }}
57+
steps:
58+
- uses: actions/checkout@v5
59+
60+
# This step generates a matrix of compatible extension/version combinations
61+
# by checking each extension's minimumConnectVersion against available Connect versions
62+
- id: generate_matrix
63+
name: Generate extension-version compatibility matrix
64+
run: |
65+
# Get the list of all Connect versions from the setup-test job
66+
ALL_VERSIONS=$(echo '${{ needs.setup-test.outputs.versions }}')
67+
# Get the list of extensions we need to test from workflow inputs
68+
EXTENSIONS=$(echo '${{ inputs.extensions }}')
69+
70+
echo "::group::Generating compatibility matrix for extensions"
71+
echo "All available Connect versions: $ALL_VERSIONS"
72+
echo "Extensions to test: $EXTENSIONS"
73+
74+
# Extract versions list for direct checking
75+
CONNECT_VERSIONS=$(echo "$ALL_VERSIONS" | jq -r '.[]')
76+
echo "::endgroup::"
77+
78+
# This array will hold all the valid extension+version combinations
79+
# Each entry will be a JSON object: {"extension": "ext-name", "connect_version": "ver"}
80+
matrix_pairs=()
81+
82+
# Process each extension separately to find compatible versions
83+
for ext in $(echo "$EXTENSIONS" | jq -r '.[]'); do
84+
echo "::group::Processing extension: $ext"
85+
86+
# Get minimumConnectVersion from the extension's manifest.json
87+
MANIFEST_FILE="extensions/$ext/manifest.json"
88+
89+
# Extract the minimumConnectVersion (linting ensures this exists)
90+
MIN_VERSION=$(jq -r '.extension.minimumConnectVersion' "$MANIFEST_FILE")
91+
echo "Extension $ext requires minimum Connect version: $MIN_VERSION"
92+
93+
# VALIDATION: Check if minimumConnectVersion exists in CONNECT_VERSIONS list
94+
if ! echo "$CONNECT_VERSIONS" | grep -q "^$MIN_VERSION$"; then
95+
echo "❌ ERROR: Extension $ext requires Connect version $MIN_VERSION which is not in CONNECT_VERSIONS list!"
96+
echo "❌ Please add $MIN_VERSION to CONNECT_VERSIONS in integration/Makefile"
97+
exit 1
98+
fi
99+
100+
# RULE 1: Always include the preview version for all extensions
101+
# This ensures we always test with the latest development version
102+
matrix_pairs+=("{\"extension\":\"$ext\",\"connect_version\":\"connect-preview\"}")
103+
echo "✅ Always adding connect-preview for $ext"
104+
105+
# Process each Connect version for this extension
106+
for version in $(echo "$ALL_VERSIONS" | jq -r '.[]'); do
107+
# Skip the preview version (already added above)
108+
if [[ "$version" == "connect-preview" ]]; then
109+
continue
110+
fi
111+
112+
# RULE 2: Only include Connect versions that meet or exceed minimumConnectVersion
113+
# Using sort -V for semantic version comparison (standard in GNU sort)
114+
# The head -n1 returns the lower version after sorting
115+
# If the current version is the minimum or higher, it's compatible
116+
if [[ "$(printf '%s\n' "$MIN_VERSION" "$version" | sort -V | head -n1)" == "$MIN_VERSION" ||
117+
"$MIN_VERSION" == "$version" ]]; then
118+
matrix_pairs+=("{\"extension\":\"$ext\",\"connect_version\":\"$version\"}")
119+
echo "✅ Adding $version for $ext (meets minimum version $MIN_VERSION)"
120+
else
121+
# Skip incompatible versions (lower than minimum requirement)
122+
echo "⏭️ Excluding $version for $ext (below minimum version $MIN_VERSION)"
123+
fi
124+
done
125+
126+
echo "::endgroup::"
127+
done
128+
129+
# Convert the array of JSON strings into a proper JSON array
130+
# This creates the final matrix to be used by the test job
131+
if [ ${#matrix_pairs[@]} -eq 0 ]; then
132+
matrix_json="[]"
133+
else
134+
# Use jq with compact output (-c) for GitHub Actions compatibility
135+
matrix_json=$(printf '%s\n' "${matrix_pairs[@]}" | jq -c -s .)
136+
fi
137+
138+
# Set the output that will be used by downstream jobs - no indentation
139+
echo "matrix=${matrix_json}" >> $GITHUB_OUTPUT
140+
141+
# Output matrix for debugging
142+
echo "::group::Generated compatibility matrix"
143+
echo "Generated $(echo "$matrix_json" | jq 'length') combinations"
144+
# Print raw (no indentation, compact format)
145+
echo "$matrix_json"
146+
echo "::endgroup::"
147+
148+
# Run the Connect integration tests for each extension against compatible Connect versions only
38149
test:
39150
runs-on: ubuntu-latest
40151
timeout-minutes: 15 # Max time to run the integration tests
41-
needs: setup-test
152+
needs: [setup-test, setup-extension-versions]
42153
strategy:
43154
# Do not fail fast so all extensions and Connect versions are processed
44155
fail-fast: false
156+
# Use the dynamically generated matrix from setup-extension-versions
157+
# This will only create jobs for extension/version combinations that are compatible
158+
# based on each extension's minimumConnectVersion requirement
45159
matrix:
46-
extension: ${{ fromJson(inputs.extensions) }}
47-
connect_version: ${{ fromJson(needs.setup-test.outputs.versions) }}
160+
include: ${{ fromJson(needs.setup-extension-versions.outputs.matrix) }}
48161
steps:
49162
- uses: actions/checkout@v4
50163

164+
# Run the integration test for this extension/version combination
51165
- uses: ./.github/actions/connect-integration-test
52166
id: test
53167
with:
@@ -57,18 +171,15 @@ jobs:
57171

58172
# Upload the test report XML files as artifacts for use by downstream jobs
59173
- uses: actions/upload-artifact@v4
60-
if: |
61-
always() &&
62-
steps.test.outcome != 'cancelled' &&
63-
steps.test.outcome != 'skipped'
174+
if: always() && steps.test.outcome != 'cancelled' && steps.test.outcome != 'skipped'
64175
with:
65176
name: ${{ matrix.extension }}-${{ matrix.connect_version }}-test-report
66177
path: integration/reports/*.xml
67178
retention-days: 7
68179

69-
# Using the XML test reports provide a matrix of extensions that passed all of the Connect integration tests
180+
# Aggregate test results to determine which extensions passed all their tests
70181
collect-results:
71-
needs: [test, setup-test]
182+
needs: [test, setup-extension-versions]
72183
runs-on: ubuntu-latest
73184
timeout-minutes: 5
74185
outputs:
@@ -81,44 +192,64 @@ jobs:
81192
path: artifacts
82193
pattern: "*-test-report"
83194

195+
# Collect and analyze results from test reports
84196
- id: collect
197+
name: Collect and analyze test results
85198
run: |
86199
# Validate inputs first
87-
all_versions='${{ needs.setup-test.outputs.versions }}'
88200
extensions='${{ inputs.extensions }}'
201+
matrix='${{ needs.setup-extension-versions.outputs.matrix }}'
202+
203+
# Check if setup-extension-versions failed (which would be due to version validation)
204+
if [[ "${{ needs.setup-extension-versions.result }}" == "failure" ]]; then
205+
echo "⚠️ setup-extension-versions job failed - likely due to version validation"
206+
echo "⚠️ No extensions passed validation, returning empty result"
207+
echo "successful_extensions=[]" >> $GITHUB_OUTPUT
208+
exit 0
209+
fi
89210
90-
if [[ -z "$all_versions" || -z "$extensions" ]]; then
211+
if [[ -z "$matrix" || -z "$extensions" ]]; then
91212
echo "❌ Missing required inputs"
92-
exit 1
213+
echo "successful_extensions=[]" >> $GITHUB_OUTPUT
214+
exit 0
93215
fi
94216
95217
# Debug info
96218
echo "::group::Debug Inputs"
97219
echo "Extensions to check: $extensions"
98-
echo "Connect versions: $all_versions"
220+
echo "Test matrix: $matrix"
99221
echo "::endgroup::"
100222
101223
# Track extensions that passed ALL version tests
102224
success_list=()
103225
226+
# Process each extension to determine if it passed all its tests
104227
for ext in $(echo "$extensions" | jq -r '.[]'); do
105228
all_passed=true
106229
echo "📦 Checking extension: $ext"
107230
108-
for version in $(echo "$all_versions" | jq -r '.[]'); do
231+
# Get the list of versions that should have been tested for this extension
232+
# This uses the filtered matrix so we only check versions that were actually tested
233+
versions_for_extension=$(echo "$matrix" | jq -r --arg ext "$ext" '[.[] | select(.extension == $ext) | .connect_version]')
234+
echo "Versions for $ext: $versions_for_extension"
235+
236+
# Check each version that was tested for this extension
237+
for version in $(echo "$versions_for_extension" | jq -r '.[]'); do
109238
echo "🔎 Checking $ext @ $version"
110239
report_dir="artifacts/${ext}-${version}-test-report"
111240
241+
# Check if we have test results for this combination
112242
if [ ! -d "$report_dir" ]; then
113243
echo "❌ No test report for $ext @ $version"
114244
all_passed=false
115245
break
116246
fi
117247
118-
# Use grep to check for failures/errors using XML test report attributes
248+
# Check for test failures/errors in XML report attributes
119249
failures=$(grep -o 'failures="[0-9]*"' "$report_dir"/*.xml | sed 's/failures="//g' | sed 's/"//g' | awk '{sum+=$1} END {print sum}' || echo "0")
120250
errors=$(grep -o 'errors="[0-9]*"' "$report_dir"/*.xml | sed 's/errors="//g' | sed 's/"//g' | awk '{sum+=$1} END {print sum}' || echo "0")
121251
252+
# If there were failures or errors, mark this extension as failed
122253
if [ "$failures" -gt 0 ] || [ "$errors" -gt 0 ]; then
123254
echo "❌ Found $failures failures, $errors errors in test suite attributes"
124255
# Extract and show some error details for debugging
@@ -133,6 +264,7 @@ jobs:
133264
fi
134265
done
135266
267+
# Add to success list if all version tests passed
136268
if [ "$all_passed" = "true" ]; then
137269
success_list+=("$ext")
138270
echo "🎉 SUCCESS: $ext passed ALL versions"
@@ -141,7 +273,8 @@ jobs:
141273
fi
142274
done
143275
144-
# Format output and ensure we always have a valid JSON array
276+
# Format output as a valid JSON array
277+
# Handle the case where no extensions passed
145278
if [ ${#success_list[@]} -eq 0 ]; then
146279
successful_extensions="[]"
147280
else

‎integration/Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ PYTEST_ARGS ?= "-s"
6868
# possible versions to test against in CI or when using the `all` target.
6969
CONNECT_VERSIONS := \
7070
connect-preview \
71+
2025.07.0 \
7172
2025.06.0 \
7273
2025.05.0 \
7374
2025.04.0

0 commit comments

Comments
 (0)