Skip to content

Fix flaky GCS ADC fallback test by isolating in its own JVM#146013

Open
jdconrad wants to merge 11 commits intoelastic:mainfrom
jdconrad:gcs-adc-test-v2
Open

Fix flaky GCS ADC fallback test by isolating in its own JVM#146013
jdconrad wants to merge 11 commits intoelastic:mainfrom
jdconrad:gcs-adc-test-v2

Conversation

@jdconrad
Copy link
Copy Markdown
Contributor

@jdconrad jdconrad commented Apr 9, 2026

  • Move testApplicationDefaultCredentialsFallbackToComputeEngine from GoogleCloudStorageServiceTests to a new GoogleCloudStorageAdcTests class
  • Run it via a dedicated gcsAdcTest Gradle task in its own JVM, ensuring the DefaultCredentialsProvider singleton starts with clean state (cachedCredentials, checkedComputeEngine)
  • Add auth validation and metadata request handling to the GCS test fixture

The test was failing ~38% of the time (#145689) because Google's DefaultCredentialsProvider singleton caches credentials and GCE detection state. When another test in the same JVM primed the singleton before this test ran, the credential discovery flow (including the File.isFile() entitlement check) was bypassed entirely.

Closes #145689

jdconrad and others added 7 commits April 2, 2026 14:55
When no credentials_file is configured, GoogleCloudStorageService calls
GoogleCredentials.getApplicationDefault(), which uses DefaultCredentialsProvider
to discover credentials. If credential files are inaccessible (e.g. due to
entitlement checks returning false from File.isFile()), it falls through to
the GCE metadata server.

This test exercises that code path using a mock HttpTransport that simulates
the GCE metadata server, asserting that credentials are successfully obtained
when no explicit credentials_file is set.

Relates: elastic#145626

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
File.isFile() returns false when entitlement is denied (regardless of
whether GOOGLE_APPLICATION_CREDENTIALS is set), so DefaultCredentialsProvider
always falls through to the mock metadata server. The guard was redundant.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The testApplicationDefaultCredentialsFallbackToComputeEngine test was
failing ~38% of the time because Google's DefaultCredentialsProvider
singleton caches credentials and GCE detection state across tests in
the same JVM. When another test primed the singleton before this test
ran, the credential discovery flow was bypassed entirely.

Move the test to a dedicated GoogleCloudStorageAdcTests class run by a
separate gcsAdcTest Gradle task, guaranteeing a clean singleton state.
Also add auth validation and metadata request handling to the GCS test
fixture.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@elasticsearchmachine elasticsearchmachine added v9.5.0 needs:triage Requires assignment of a team area label labels Apr 9, 2026
@jdconrad jdconrad added >test Issues or PRs that are addressing/adding tests :Core/Infra/Entitlements Entitlements infrastructure branch:9.2 branch:8.19 branch:9.3 branch:9.4 labels Apr 9, 2026
@elasticsearchmachine elasticsearchmachine added Team:Core/Infra Meta label for core/infra team v9.2.9 v8.19.15 v9.3.4 v9.4.1 and removed needs:triage Requires assignment of a team area label labels Apr 9, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 9, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 3bd7be9d-c3c7-438f-a4df-142b0186022b

📥 Commits

Reviewing files that changed from the base of the PR and between 44a079d and 337f119.

📒 Files selected for processing (2)
  • modules/repository-gcs/build.gradle
  • modules/repository-gcs/src/test/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageAdcTests.java
🚧 Files skipped from review as they are similar to previous changes (2)
  • modules/repository-gcs/src/test/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageAdcTests.java
  • modules/repository-gcs/build.gradle

📝 Walkthrough

Walkthrough

This change addresses test flakiness by restructuring how the Application Default Credentials fallback test executes. The test method testApplicationDefaultCredentialsFallbackToComputeEngine() was moved from GoogleCloudStorageServiceTests to a new dedicated test class GoogleCloudStorageAdcTests. Correspondingly, the Gradle build configuration was updated to create an isolated gcsAdcTest task that runs only this test while explicitly removing the GOOGLE_APPLICATION_CREDENTIALS environment variable. The default test task excludes this test, and the check task now depends on both the existing gcsThirdPartyTest and the new gcsAdcTest.

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed PR successfully isolates the flaky ADC test into a dedicated JVM via gcsAdcTest task, directly addressing issue #145689's root cause (DefaultCredentialsProvider singleton caching).
Out of Scope Changes check ✅ Passed All changes directly support the objective: test relocation, Gradle task isolation, and removal of the original test from GoogleCloudStorageServiceTests.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • 🛠️ Update Documentation: Commit on current branch
  • 🛠️ Update Documentation: Create PR

Comment @coderabbitai help to get the list of available commands and usage tips.

jdconrad and others added 3 commits April 9, 2026 12:37
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Core/Infra/Entitlements Entitlements infrastructure Team:Core/Infra Meta label for core/infra team >test Issues or PRs that are addressing/adding tests v8.19.15 v9.2.9 v9.3.4 v9.4.1 v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] GoogleCloudStorageServiceTests testApplicationDefaultCredentialsFallbackToComputeEngine failing

2 participants