Add retries to GCS sink healthcheck#4
Open
alexander-jiang wants to merge 2 commits intodata_infra_vector_stablefrom
Open
Add retries to GCS sink healthcheck#4alexander-jiang wants to merge 2 commits intodata_infra_vector_stablefrom
alexander-jiang wants to merge 2 commits intodata_infra_vector_stablefrom
Conversation
…heck response to logs when the healthcheck attempts fail
| @@ -1,5 +1,5 @@ | |||
| diff --git a/src/gcp.rs b/src/gcp.rs | |||
| index bfc486f92..148fa9dec 100644 | |||
| index bfc486f92..baa8e143d 100644 | |||
Check warning
Code scanning / check-spelling
Candidate Pattern
| @@ -1,5 +1,5 @@ | |||
| diff --git a/src/gcp.rs b/src/gcp.rs | |||
| index bfc486f92..148fa9dec 100644 | |||
| index bfc486f92..baa8e143d 100644 | |||
Check failure
Code scanning / check-spelling
Unrecognized Spelling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Request to update the data_infra_vector_stable branch, which is used to build the discord_data_vector_base image.
Summary
We've noticed that GCS sink healthcheck in Vector will sometimes fail temporarily (due to timeout) and then will self-recover. The Data Infra team hasn't found a clear reason as to why the healthchecks time-out (e.g. I haven't been able to replicate the healthcheck timeouts in staging), and we want to reduce the impact of ephemeral/noisy healthcheck failures or timeouts on the Vector deployments which are part of our critical event ingestion pipeline. At the same time, we shouldn't completely ignore healthcheck failures.
This PR makes the following changes:
Documentation update:
The PR also updates the
patches/README.mdfile: our Vector build pipeline no longer uses the*.patchfiles to apply patches onto a commit from the vector repository, but instead builds from a Discord-owned fork of the Vector repository.