release-26.1: roachtest: add Datadog integration for test log ingestion #160566
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, test.log artifacts were only available on TeamCity's UI. This change adds a datadog package that uploads test.log files to Datadog during test cleanup on master and release branches. The implementation scans the log file serially and uses a worker pool to upload log entries in batches of 1000 using the Datadog API client. Each log entry is tagged with test metadata (test name, owner, cloud, platform, version) and includes attributes for higher cardinality fields (cluster name, build number, result, duration). This enables full-text search across roachtest runs and improves observability for test triage and failure analysis. Epic: None Release note: None
blathers-crl
bot
added
blathers-backport
blathers-crl
bot
requested review from
golgeek,
herkolategan and
shailendra-patel
williamchoe3
requested review from
nameisbhaskar
and removed request for
a team
|
Thanks for opening a backport. Before merging, please confirm that the change does not break backwards compatibility and otherwise complies with the backport policy. Include a brief release justification in the PR description explaining why the backport is appropriate. All backports must be reviewed by the TL for the owning area. While the stricter LTS policy does not yet apply, please exercise judgment and consider gating non-critical changes behind a disabled-by-default feature flag when appropriate. |
blathers-crl
bot
added
backport
Member
|
This change is |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 1/1 commits from #158528 on behalf of @williamchoe3.
Previously, we didn't have an easy way to do full text search across CI test runs across different TC Build Configurations and branches. To do that, you would have to download the artifacts for what you wanted to search for.
This change adds a datadog package that uploads test.log files to Datadog during test cleanup on master and release branches. The implementation scans the log file serially and uses a worker pool to upload log entries in batches of 1000 using the Datadog API client. Each log entry is tagged with test metadata (test name, owner, cloud, platform, version) and includes attributes for higher cardinality fields (cluster name, build number, result, duration). See comments for more details.
The entry point for datadog upload in
roachtestwill be during the post step test intest_runner.go.Added a new roachtest flagdatadog-always-uploadfor e2e testing on a non release branch. Modified TC build scripts to pass new env vars and the teamcity build properties file.Release justification: Test only change