Skip to content

Enable test retry for daily builds #13086

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jan 24, 2025
Merged

Enable test retry for daily builds #13086

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
dbab955
Enable test retry for daily builds
laurit Jan 22, 2025
7bc8830
try to fix
laurit Jan 22, 2025
bd4d68a
run job url task for failed jobs
laurit Jan 22, 2025
a156772
make test less flaky
laurit Jan 22, 2025
b302fa5
remove debugging
laurit Jan 22, 2025
07d4052
add flaky test report to indy and latest deps workflows
laurit Jan 22, 2025
ae0a6f1
delete flaky test
laurit Jan 22, 2025
a8d4f26
try a different way for getting the job url
laurit Jan 23, 2025
4512896
hope this works
laurit Jan 23, 2025
23f6073
hope this works
laurit Jan 23, 2025
b21badf
rework get job url in indy and latest tests
laurit Jan 24, 2025
f7dac2c
remove debug
laurit Jan 24, 2025
79a0354
spotless
laurit Jan 24, 2025
705a62d
Merge branch 'main' into flaky-test-reporter
laurit Jan 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/repository-settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ for [`dependabot/**/**`](https://github.com/open-telemetry/community/blob/main/d
- Key is associated with [@trask](https://github.com/trask)'s gmail address
- `SONATYPE_KEY` - owned by [@trask](https://github.com/trask)
- `SONATYPE_USER` - owned by [@trask](https://github.com/trask)
- `FLAKY_TEST_REPORTER_ACCESS_KEY` - owned by [@laurit](https://github.com/laurit)

### Organization secrets

Expand Down
19 changes: 19 additions & 0 deletions .github/workflows/build-common.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,25 @@ jobs:
if: ${{ !cancelled() && hashFiles('build-scan.txt') != '' }}
run: cat build-scan.txt

- name: Get current job url
uses: Tiryoh/gha-jobid-action@be260d8673c9211a84cdcf37794ebd654ba81eef # v1.4.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have security worries about using unpopular external actions, is there another way? would it be ok to just link to the run? https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced that action with a different solution that uses actions/github-script

id: jobs
if: ${{ !cancelled() }}
with:
job_name: "common / test${{ matrix.test-partition }} (${{ matrix.test-java-version }}, ${{ matrix.vm }})"
per_page: 100 # input matrix size here if it is larger than 30

- name: Flaky test report
if: ${{ !cancelled() }}
env:
FLAKY_TEST_REPORTER_ACCESS_KEY: ${{ secrets.FLAKY_TEST_REPORTER_ACCESS_KEY }}
JOB_URL: ${{ steps.jobs.outputs.html_url }}
run: |
if [ -s build-scan.txt ]; then
export BUILD_SCAN_URL=$(cat build-scan.txt)
fi
./gradlew :test-report:reportFlakyTests

- name: Upload deadlock detector artifacts if any
if: failure()
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
Expand Down
3 changes: 0 additions & 3 deletions .github/workflows/build-daily-no-build-cache.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,16 @@ jobs:
common:
uses: ./.github/workflows/build-common.yml
with:
max-test-retries: 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would think if we wanted to enable retries we would need to increase this value instead of remove it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See

opentelemetry-java-instrumentation/conventions/src/main/kotlin/otel.java-conventions.gradle.kts

Lines 369 to 370 in edbd9aa

val defaultMaxRetries = if (System.getenv().containsKey("CI")) 5 else 0
val maxTestRetries = gradle.startParameter.projectProperties["maxTestRetries"]?.toInt() ?: defaultMaxRetries

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for pointing that out

no-build-cache: true

test-latest-deps:
uses: ./.github/workflows/reusable-test-latest-deps.yml
with:
max-test-retries: 0
no-build-cache: true

test-indy:
uses: ./.github/workflows/reusable-test-indy.yml
with:
max-test-retries: 0
no-build-cache: true

# muzzle is not included here because it doesn't use gradle cache anyway and so is already covered
Expand Down
6 changes: 0 additions & 6 deletions .github/workflows/build-daily.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,12 @@ on:
jobs:
common:
uses: ./.github/workflows/build-common.yml
with:
max-test-retries: 0

test-latest-deps:
uses: ./.github/workflows/reusable-test-latest-deps.yml
with:
max-test-retries: 0

test-indy:
uses: ./.github/workflows/reusable-test-indy.yml
with:
max-test-retries: 0

muzzle:
uses: ./.github/workflows/reusable-muzzle.yml
Expand Down
19 changes: 19 additions & 0 deletions .github/workflows/reusable-test-indy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,22 @@ jobs:
- name: Build scan
if: ${{ !cancelled() && hashFiles('build-scan.txt') != '' }}
run: cat build-scan.txt

- name: Get current job url
uses: Tiryoh/gha-jobid-action@be260d8673c9211a84cdcf37794ebd654ba81eef # v1.4.0
id: jobs
if: ${{ !cancelled() }}
with:
job_name: "test-indy / testIndy${{ matrix.test-partition }}"
per_page: 100 # input matrix size here if it is larger than 30

- name: Flaky test report
if: ${{ !cancelled() }}
env:
FLAKY_TEST_REPORTER_ACCESS_KEY: ${{ secrets.FLAKY_TEST_REPORTER_ACCESS_KEY }}
JOB_URL: ${{ steps.jobs.outputs.html_url }}
run: |
if [ -s build-scan.txt ]; then
export BUILD_SCAN_URL=$(cat build-scan.txt)
fi
./gradlew :test-report:reportFlakyTests
19 changes: 19 additions & 0 deletions .github/workflows/reusable-test-latest-deps.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,25 @@ jobs:
if: ${{ !cancelled() && hashFiles('build-scan.txt') != '' }}
run: cat build-scan.txt

- name: Get current job url
uses: Tiryoh/gha-jobid-action@be260d8673c9211a84cdcf37794ebd654ba81eef # v1.4.0
id: jobs
if: ${{ !cancelled() }}
with:
job_name: "test-latest-deps / testLatestDeps${{ matrix.test-partition }}"
per_page: 100 # input matrix size here if it is larger than 30

- name: Flaky test report
if: ${{ !cancelled() }}
env:
FLAKY_TEST_REPORTER_ACCESS_KEY: ${{ secrets.FLAKY_TEST_REPORTER_ACCESS_KEY }}
JOB_URL: ${{ steps.jobs.outputs.html_url }}
run: |
if [ -s build-scan.txt ]; then
export BUILD_SCAN_URL=$(cat build-scan.txt)
fi
./gradlew :test-report:reportFlakyTests

- name: Upload deadlock detector artifacts if any
if: failure()
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
Expand Down
1 change: 1 addition & 0 deletions settings.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ include(":instrumentation-annotations-support-testing")

// misc
include(":dependencyManagement")
include(":test-report")
include(":testing:agent-exporter")
include(":testing:agent-for-testing")
include(":testing:armeria-shaded-for-testing")
Expand Down
27 changes: 27 additions & 0 deletions test-report/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
plugins {
id("otel.java-conventions")
}

dependencies {
implementation("com.google.api-client:google-api-client:2.7.1")
implementation("com.google.apis:google-api-services-sheets:v4-rev20250106-2.0.0")
implementation("com.google.auth:google-auth-library-oauth2-http:1.30.1")
}

otelJava {
minJavaVersionSupported.set(JavaVersion.VERSION_17)
}

tasks {
val reportFlakyTests by registering(JavaExec::class) {
dependsOn(classes)

mainClass.set("io.opentelemetry.instrumentation.testreport.FlakyTestReporter")
classpath(sourceSets["main"].runtimeClasspath)

systemProperty("scanPath", project.rootDir)
systemProperty("googleSheetsAccessKey", System.getenv("FLAKY_TEST_REPORTER_ACCESS_KEY"))
systemProperty("buildScanUrl", System.getenv("BUILD_SCAN_URL"))
systemProperty("jobUrl", System.getenv("JOB_URL"))
}
}
Loading
Loading