Skip to content

Fix flaky OpenSearchTestBasePluginFuncTest (#20955)#20975

Merged
andrross merged 2 commits intoopensearch-project:mainfrom
psychbot:fix-flaky-OpenSearchTestBasePluginFuncTest
Mar 25, 2026
Merged

Fix flaky OpenSearchTestBasePluginFuncTest (#20955)#20975
andrross merged 2 commits intoopensearch-project:mainfrom
psychbot:fix-flaky-OpenSearchTestBasePluginFuncTest

Conversation

@psychbot
Copy link
Member

@psychbot psychbot commented Mar 23, 2026

Description

When integTest runs with maxParallelForks > 1, all parallel test JVMs share the same default .gradle-test-kit directory. This can causes a race condition where concurrent tests simultaneously read and write the Groovy DSL compiled script cache (cp_settings), may result in a NoSuchFileException.

Fixing by assigning each test method its own isolated TestKit directory via withTestKitDir() using a subdirectory within the already-unique temporary folder.

Related Issues

Should resolve #20955

java.nio.file.NoSuchFileException: /var/jenkins/workspace/gradle-check/search/buildSrc/build/tmp/integTest/work/.gradle-test-kit/caches/9.4.0/groovy-dsl/02bba72833c5af3120d1ca946c9f6b22/instrumented/cp_settings

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added >test-failure Test failure from CI, local build, etc. autocut flaky-test Random test failure that succeeds on second run labels Mar 23, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Mar 23, 2026

PR Reviewer Guide 🔍

(Review updated until commit d58a480)

Here are some key observations to aid the review process:

🧪 No relevant tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ No major issues detected

@github-actions
Copy link
Contributor

github-actions bot commented Mar 23, 2026

PR Code Suggestions ✨

Latest suggestions up to d58a480
Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Ensure unique test kit directory per test

The testKitDir is created once in setup() and shared across all test methods, but if
multiple tests run in parallel or the same directory is reused across test runs, it
could cause conflicts. Consider using a unique subdirectory per test (e.g.,
incorporating the spec name or a UUID) to ensure isolation between test executions.

buildSrc/src/integTest/groovy/org/opensearch/gradle/fixtures/AbstractGradleFuncTest.groovy [54]

-testKitDir = testProjectDir.newFolder('.gradle-test-kit')
+testKitDir = testProjectDir.newFolder('.gradle-test-kit-' + UUID.randomUUID().toString())
Suggestion importance[1-10]: 3

__

Why: The testProjectDir is already a unique TemporaryFolder per test (created by the @Rule annotation), so each test already gets its own isolated directory. Adding a UUID suffix would be redundant and adds unnecessary complexity, making the suggestion of low value.

Low

Previous suggestions

Suggestions up to commit 6a6dd43
CategorySuggestion                                                                                                                                    Impact
General
Avoid shared test kit directory conflicts

The testKitDir is created once in setup() and shared across all test methods via
gradleRunner. If multiple gradleRunner calls occur in parallel or across tests
sharing state, they may conflict over the same directory. Consider using a unique
subdirectory per test or per runner invocation to avoid potential flakiness from
shared state.

buildSrc/src/integTest/groovy/org/opensearch/gradle/fixtures/AbstractGradleFuncTest.groovy [54]

-testKitDir = testProjectDir.newFolder('.gradle-test-kit')
+testKitDir = testProjectDir.newFolder('.gradle-test-kit-' + UUID.randomUUID().toString())
Suggestion importance[1-10]: 3

__

Why: The testProjectDir is a @Rule TemporaryFolder which creates a fresh directory per test, so testKitDir is already unique per test. The suggestion addresses a concern that doesn't apply in this context, making it a low-impact change.

Low

When integTest runs with maxParallelForks > 1, all parallel test JVMs
share the same default .gradle-test-kit directory. This causes a race
condition in Gradle 9.4 where concurrent tests simultaneously read and
write the Groovy DSL compiled script cache (cp_settings), resulting in
a NoSuchFileException and UnexpectedBuildFailure.

Fix by assigning each test method its own isolated TestKit directory
via withTestKitDir() using a subdirectory within the already-unique
TemporaryFolder. This eliminates cross-test cache contention while
preserving daemon and task history reuse within the same test method
(required for UP_TO_DATE assertions).

Signed-off-by: Dharmesh <dharmesh.singh@uber.com>
@psychbot psychbot force-pushed the fix-flaky-OpenSearchTestBasePluginFuncTest branch from 6a6dd43 to d58a480 Compare March 23, 2026 22:20
@github-actions
Copy link
Contributor

Persistent review updated to latest commit d58a480

@github-actions
Copy link
Contributor

❌ Gradle check result for d58a480: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

Failed to generate code suggestions for PR

@github-actions
Copy link
Contributor

❌ Gradle check result for d3e110b: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

✅ Gradle check result for d3e110b: SUCCESS

@codecov
Copy link

codecov bot commented Mar 25, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.25%. Comparing base (85113a4) to head (d3e110b).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #20975      +/-   ##
============================================
- Coverage     73.31%   73.25%   -0.06%     
+ Complexity    72544    72526      -18     
============================================
  Files          5819     5819              
  Lines        331399   331399              
  Branches      47887    47887              
============================================
- Hits         242955   242768     -187     
- Misses        68935    69112     +177     
- Partials      19509    19519      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@andrross andrross merged commit a00bcc4 into opensearch-project:main Mar 25, 2026
38 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autocut flaky-test Random test failure that succeeds on second run skip-changelog >test-failure Test failure from CI, local build, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[AUTOCUT] Gradle Check Flaky Test Report for OpenSearchTestBasePluginFuncTest

2 participants