diff --git a/internal/civisibility/CIVISIBILITY_OVERVIEW.md b/internal/civisibility/CIVISIBILITY_OVERVIEW.md new file mode 100644 index 0000000000..d981a666b5 --- /dev/null +++ b/internal/civisibility/CIVISIBILITY_OVERVIEW.md @@ -0,0 +1,90 @@ +# `internal/civisibility` Walkthrough + +## High-Level Purpose +- Implements Datadog Test Optimization for Go: bootstraps tracing/log streaming, exposes manual test lifecycle APIs, and auto-instruments `testing`. +- Coordinates feature toggles such as Intelligent Test Runner (ITR), early flake detection, flaky retries, impacted tests, test management, subtest-level directives, code coverage, and CI logs. +- Normalizes CI/git metadata, fetches repository deltas, and communicates with backend settings/coverage/logs endpoints through a configurable client layer. +- Houses telemetry hooks to measure git command usage, HTTP behavior, and test instrumentation statistics. + +## Top-Level Layout +- `civisibility.go` – atomic state/test-mode switches shared across integrations. +- `constants/` – tag keys, span types, environment variable names, capability flags, span metadata helpers. +- `integrations/` – tracer bootstrapping, feature negotiation, manual test lifecycle API, Go `testing` instrumentation (including subtest orchestration), log streaming. +- `integrations/gotesting/subtests/` – mock backend + scenario harness exercising parent/subtest directive matrices and retry ownership rules. +- `utils/` – CI provider discovery, git utilities, code owners lookup, network clients, telemetry plumbing, name canonicalization, impacted test logic, fixtures. + +## Core Components + +### Root State Management +`civisibility.go` stores a process-wide `status` (`StateUninitialized` through `StateExited`) and a `isTestMode` flag using `atomic` types. Integrations set these to coordinate tracer startup/teardown, and tests can toggle “test mode” for mock tracer usage. + +### Constants Package +- `ci.go`, `env.go`, `git.go`, `os.go`, `runtime.go` declare string keys for CI metadata, git fields, OS/runtime descriptors, and env var names driving feature toggles (`DD_CIVISIBILITY_*`, `CIVisibility*`). +- `tags.go`, `test_tags.go`, `span_types.go` centralize span tag names, capability markers, and status/type enumerations used by integrations and utils. +- `test_tags.go` captures nuanced flags (`test.itr.forced_run`, quarantine/disable toggles, retry reasons, coverage toggles) ensuring consistent tagging between retries, EFD, ITR, and test management flows. + +### Integrations Package +- `civisibility.go` handles one-time tracer initialization: sets `DD_CIVISIBILITY_ENABLED`, forces tracing sample rate to 1, preloads CI metadata/code owners, optionally sets service name from repo URL, and registers signal handlers that call `ExitCiVisibility`. Close actions queue via `PushCiVisibilityCloseAction`, running in LIFO order during shutdown. +- `civisibility_features.go` orchestrates backend settings negotiation. It spawns asynchronous git pack uploads, retries settings fetch if backend needs git data, applies env-var kill switches for features (flaky retries, impacted tests, test management, subtest directives), and lazily loads supplementary data (known tests, skippables, impacted tests analyzer). Settings and HTTP client live in package-level vars protected by `sync.Once`. +- `manual_api*.go` files expose strongly-typed interfaces for user-driven test lifecycle: `TestSession`, `TestModule`, `TestSuite`, `Test`, along with option structs for command, framework, working directory, start/finish timestamps, and error reporting via `ErrorOption`. Variants (`manual_api_ddtest*.go`) adapt to `ddtest` helper semantics, and `manual_api_mocktracer_test.go` validates API behavior under mock tracer mode. +- `gotesting/` auto-instruments `testing.M`, `testing.T`, and `testing.B`. + - `testing.go`, `testingT.go`, `testingB.go`, and related files manage session/module/suite creation, attach tags (including module/suite counters), handle chatty output, skip logic, coverage capture, log streaming, and telemetry emission. They integrate with `integrations.GetSettings()`, `net` clients, and `logs`. + - Hierarchical identity plumbing: `testIdentity` (module, suite, base name, full name, path segments) plus `matchTestManagementData` allow subtests to resolve directives like `TestParent/SubChild`. `getTestManagementData` reports whether a directive was an exact match or inherited from an ancestor. + - `instrumentation.go` wires wrappers around test functions, stores execution metadata (retries, new/modified flags, quarantined/disabled states, attempt-to-fix ownership), and coordinates with backend settings for retries/EFD/ITR. It leverages `unsafe` pointers and reflective lookup to map `testing` internals, guarded by `sync` primitives. Subtests consult the parent execution metadata to decide whether they should wrap themselves or defer to the parent-run retry loop. + - Attempt-to-fix ownership rules: + 1. Parent-only directives orchestrate retries and tag success/failure; children inherit attempt-to-fix tagging but emit no retry spans. + 2. Child-only directives wrap the subtest locally (when feature flag enabled and exact match present) while leaving the parent neutral. + 3. Parent and child requesting attempt-to-fix results in the parent winning; subtests receive tags but do not run retries. + 4. Parent quarantine + attempt-to-fix keeps the parent as the retry owner while subtests inherit quarantine tags; a quarantined parent without attempt-to-fix leaves children free to execute their own retries if explicitly configured. + - Feature gating: `DD_CIVISIBILITY_SUBTEST_FEATURES_ENABLED` enables subtest directives. Standard debug logs capture identity/ownership traces when enabled. + - `instrumentation_orchestrion.go` and `orchestrion.yml` support bytecode rewriting via Orchestrion for transparent instrumentation in user code. The orchestrion path computes subtest identities, inspects parent metadata, and applies the same ownership logic as the manual wrappers. + - `coverage/` builds code coverage payloads, writes them via `coverage_writer`, and includes an auto-generated `test_coverage_msgp.go` for MsgPack encoding. + - `reflections.go` / `_test.go` ensure compatibility with `go test` internal structures across versions; helper routines detect struct offsets, function pointers, and maintain compatibility with new Go releases. +- `logs/` encapsulates CI log forwarding: gating via `DD_CIVISIBILITY_LOGS_ENABLED` stable config, packaging log entries with consistent tags, buffering/flush policies, payload formatting (`logs_payload.go`) and writer lifecycle (`logs_writer.go`). + +### Subtest Scenario Harness (`integrations/gotesting/subtests/`) +- Provides an executable matrix covering parent/subtest directive permutations. `main_test.go` enables the subtest feature flag and spawns subprocesses per scenario using `SUBTEST_MATRIX_SCENARIO`. +- `subtestcontroller_test.go` spins up a mock backend (`startSubtestServer`) that surfaces settings, test-management payloads, and stubbed endpoints for logs/git to keep tests hermetic. +- Scenarios assert span counts, `test.status`, `test.is_quarantined`, `test.is_disabled`, retry metadata (`test.is_retry`, `test.retry_reason`), and ownership of attempt-to-fix success tags. Parallel subtests and custom retry budgets are part of the matrix. +- Utilities (`scenarioContext`, `setParentDirective`, `setSubDirective`) help craft hierarchical directives; helper assertions document expected tagging for collaborators extending the matrix. + +### Utils Package +- `ci_providers.go` detects CI metadata across numerous providers (AppVeyor, Azure Pipelines, GitHub Actions, Jenkins, etc.), normalizes refs/URLs, removes secrets, supports user overrides through `DD_GIT_*` env vars, and logs detected provider. Fixtures under `testdata/fixtures/{providers}` supply provider-specific JSON. +- `environmentTags.go` maintains cached CI tags/metrics with thread-safe mutation (`AddCITags*`, `ResetCITags*`), expands `~`, computes relative paths, and augments CPU metrics (logical cores). +- `git.go` performs git command execution with telemetry instrumentation, synchronized access (`gitCommandMutex`), shallow clone detection/unshallow, pack-file generation (`MaxPackFileSizeInMb`, `CreatePackfiles`), base branch discovery, and sensitive info filtering. Interacts with `utils/telemetry` enums to classify commands/errors. Backed by tests covering command paths and error handling. +- `file_environmental_data.go` and `_test.go` collect file-level metadata (size, permissions, hash) referenced by impacted tests. `filebitmap/` stores efficient bitmap representation of file coverage. +- `impactedtests/` implements incremental test selection. `algorithm.md` documents the base branch detection heuristic (with 2025 updates) and ties closely to git utilities; `impacted_tests.go` consumes backend responses to track new/modified tests. +- `codeowners.go` parses CODEOWNERS files with caching and fallback to repo root; fixtures for GitHub/GitLab located under `testdata/fixtures/codeowners`. +- `names.go` normalizes module/suite names via runtime function lookup and heuristics, ensuring consistent tagging even with nested/subtests; tests validate complex name resolution. +- `home.go` and `file_environmental_data.go` handle home directory discovery with consideration for CI sandboxes and Windows drive letters. +- `net/` houses HTTP client logic: + - `client.go` builds agent or agentless clients, selects base URL/subdomain, attaches tags/headers, and exposes methods for settings, known tests, pack files, coverage, logs, skippables, and test management APIs. Incorporates retry/backoff (`math/rand/v2` jitter), compression awareness, telemetry hooks, and optional EVP proxy over Unix sockets. + - `http.go`, `coverage.go`, `logs_api.go`, `settings_api.go`, etc., serialize network payloads, set proper endpoints, compress payloads, and capture request/response telemetry (status codes, compression flags, payload sizes). + - `skippable.go`, `known_tests_api.go`, `test_management_tests_api.go` parse backend responses into typed structs for downstream integrations. +- `telemetry/` defines dimensional labels for events (framework identifiers, CI provider tags, error types, git command categories) used throughout the package to emit consistent metrics. +- `names_test.go`, `git_test.go`, `codeowners_test.go`, `ci_providers_test.go`, `net/*_test.go`, etc., provide extensive coverage, often using fixtures to simulate CI environments and network responses. + +## Testing, Fixtures, and Tooling +- Extensive `_test.go` coverage in integrations (`manual_api`, `gotesting`, `logs`) and utils ensures feature toggles, retries, coverage serialization, and network clients behave as expected. +- Subtest matrix harness (`integrations/gotesting/subtests`) runs under `go test` and exercises the parent/subtest permutations needed to guard subtest-specific instrumentation changes. Enable debug logging to surface per-scenario diagnostics. +- `integrations/gotesting/testcontroller_test.go` retains historical scenarios for flaky retries, EFD, ITR, and impacted tests; it coexists with the new subtest harness to avoid regression gaps. +- `utils/testdata/fixtures/providers/*.json` mimics CI payloads; `github-event.json` supports webhook parsing tests. +- Generated assets: `coverage/test_coverage_msgp.go` (MsgPack via `go:generate`), with tests to ensure deterministic encoding. +- `integrations/gotesting/reflections_test.go` safeguards reflection-based hooks against Go runtime changes. +- Mock tracer support via `mocktracer` allows unit tests to assert spans without real agent connectivity. + +## Notable Nuances & Design Choices +- Heavy use of `sync.Once`, `atomic`, and mutexes to guard global state, ensuring idempotent initialization even under concurrent instrumentation hooks. +- Feature toggles honor both backend settings and local env overrides, often logging when overrides disable capabilities to aid troubleshooting. Subtest features default off unless an env var enables them, allowing gradual rollout. +- Subtest wrappers strictly require an exact directive match before wrapping to avoid unnecessary allocations when hierarchy lookups fall back to parent configuration. +- Git operations are serialized to avoid repository lock contention, and telemetry logs command timings plus categorized exit codes to monitor flaky git environments. +- Instrumentation leans on `unsafe.Pointer` and reflection to interpose on testing internals, a delicate strategy mitigated by fallback logic and version checks. Helper utilities (`reflections.go`) centralize offsets so new Go releases require updates in a single place. +- Coverage and impacted test features rely on asynchronous git uploads; close actions ensure goroutines finish before process exit. +- Network layer supports agentless uploads with API key validation and on-the-fly compression, while also accommodating Datadog agent EVP proxy over HTTP or Unix sockets. +- `orchestrion.yml` indicates support for compile-time rewriting, hinting at hybrid instrumentation strategies (manual wrappers plus bytecode injection). +- Logging pipeline mirrors test span IDs and includes service/host tags, but is guarded behind stable-config flag to avoid unexpected log emission. + +## Getting Involved +- When touching `integrations/gotesting`, run both the legacy controller suite and the subtest matrix (`go test ./internal/civisibility/integrations/gotesting/...`). Many scenarios spawn subprocesses; enable debug logging for verbose traces. +- Any change to retry ownership or metadata propagation should be mirrored in the harness scenarios and in `docs/SUBTEST_FEATURE_IMPLEMENTATION.md` to keep documentation synchronized. +- Utility changes often require updating fixtures or provider expectations; leverage the existing test suites instead of ad-hoc scripts. diff --git a/internal/civisibility/constants/env.go b/internal/civisibility/constants/env.go index ad6e485a48..869e476dd3 100644 --- a/internal/civisibility/constants/env.go +++ b/internal/civisibility/constants/env.go @@ -55,4 +55,7 @@ const ( // CIVisibilityInternalParallelEarlyFlakeDetectionEnabled indicates if the internal parallel early flake detection feature is enabled. CIVisibilityInternalParallelEarlyFlakeDetectionEnabled = "DD_CIVISIBILITY_INTERNAL_PARALLEL_EARLY_FLAKE_DETECTION_ENABLED" + + // CIVisibilitySubtestFeaturesEnabled indicates if subtest-specific management and retry features are enabled. + CIVisibilitySubtestFeaturesEnabled = "DD_CIVISIBILITY_SUBTEST_FEATURES_ENABLED" ) diff --git a/internal/civisibility/docs/SUBTEST_FEATURE_IMPLEMENTATION.md b/internal/civisibility/docs/SUBTEST_FEATURE_IMPLEMENTATION.md new file mode 100644 index 0000000000..2d8de82b28 --- /dev/null +++ b/internal/civisibility/docs/SUBTEST_FEATURE_IMPLEMENTATION.md @@ -0,0 +1,151 @@ +# Subtest Test Management & Retry Enablement + +## Goals + +- Honour Datadog Test Management directives (disable, quarantine, attempt-to-fix) for Go + subtests created with `t.Run`. +- Allow subtests to orchestrate their own retry loops while keeping parent behaviour intact. +- Preserve existing semantics for flaky retries, coverage, logging, telemetry, and testify support. +- Provide deterministic integration tests that exercise the parent/subtest matrix. + +This document walks through the implementation added since commit +`a722e23890a1f7306af5ac6bf9060e4846dbb895`, highlighting how the new pieces work together and the +nuances reviewers should keep in mind. + +## Architecture Overview + +The feature builds on three pillars: + +1. **Identity plumbing** – every test/subtest now has a `testIdentity` with module, suite, base name, + full name, and hierarchical segments (see `newTestIdentity`). This identity is threaded through + `applyAdditionalFeaturesToTestFunc`, the orchestrion wrapper, and tests so configuration lookups + can fall back to parent segments when a subtest lacks explicit settings. + +2. **Enhanced instrumentation** – `applyAdditionalFeaturesToTestFunc` and + `instrumentTestingTFunc` gained subtest-aware logic. Parents still own global wrappers, but + subtests may now wrap themselves when the feature flag (`DD_CIVISIBILITY_SUBTEST_FEATURES_ENABLED`) is + on and an exact directive exists. Attempt-to-fix orchestration honours a single “owner” to avoid + double retries. + +3. **Scenario harness** – the new `integrations/gotesting/subtests` package spins up a mock backend + and walks through the parent/subtest matrix. Each scenario asserts span counts and tags, including + retry telemetry (`test.is_retry`, `test.retry_reason`), so regressions surface immediately. + +## Feature Gating & Settings Bootstrap + +- `integrations/civisibility_features.go` honours `DD_CIVISIBILITY_SUBTEST_FEATURES_ENABLED` so + environments can opt into subtest directives explicitly. +- `ciSettings.SubtestFeaturesEnabled` guards *every* subtest-specific branch. When the flag is off, + behaviour is identical to pre-feature builds. + +## Identity & Management Lookup + +- `integrations/gotesting/testing.go` + - Introduces `testIdentity` and `newTestIdentity`. Segments allow lookups like + `TestParent/Sub1/Sub2`, falling back to `TestParent/Sub1`, then `TestParent`. + - `commonInfo` now carries a pointer to the identity so wrappers can reuse it. + - Internal test/benchmark instrumentation stores the identity and threads it into + `applyAdditionalFeaturesToTestFunc`. +- `getTestManagementData` returns both the matched properties and `matchKind` + (`Exact`, `Ancestor`, or `None`) so subtests can tell whether they should run the additional + wrapper or inherit parent behaviour. +- The subtest matrix exercises identity matching directly; no extra test-only helpers are required. + +## Instrumentation Enhancements + +### `applyAdditionalFeaturesToTestFunc` + +- Accepts an optional `parentExecMeta` so subtests know if their parent already owns attempt-to-fix + retries. +- Builds a metadata struct with explicit flags (`hasExplicitAttemptToFix`, etc.) and the match kind. +- For subtests: + - Requires the feature flag plus an **exact** match before wrapping. + - Disables EFD/flaky auto retries to avoid double wrapping. + - Only orchestrates attempt-to-fix locally if the subtest explicitly asked for it and the parent + isn’t already handling retries. +- During execution: + - Propagates explicit directives before OR-ing inherited values, ensuring targeted overrides win. + - Emits debug logs to help trace scenario execution when verbose logging is enabled. + - Only the orchestrator in charge emits attempt-to-fix retry logs and success tags. + +### `instrumentation_orchestrion.go` + +- On entry, computes the subtest identity and inspects parent metadata when available. +- Short-circuits in two cases: + - Feature flag disabled or no directives present (harness mode short-circuits to defer to the scenario driver). + - Parent already handles everything (legacy behaviour). +- For subtests needing instrumentation: + - Calls `applyAdditionalFeaturesToTestFunc` with parent metadata so ownership stays consistent. + - Handles panic, fail, skip, and pass paths, tagging attempt-to-fix success/failure and retry + exhaustion where appropriate. + - Writes verbose debug messages when debug logging is enabled to aid troubleshooting. + +## Attempt-to-Fix Ownership Rules + +1. **Parent only** – Parent orchestrates retries, subtests inherit attempt-to-fix tagging but never + claim success. Child spans have `test.is_retry=false`. +2. **Subtest only** – Parent remains neutral, child wraps itself and emits retry spans. +3. **Parent & subtest** – Parent wins; child spans show attempt-to-fix tagging with zero retry tags, + ensuring telemetry is counted once. +4. **Parent quarantine + attempt-to-fix** – When the parent carries both directives it remains the + retry owner; children inherit the quarantine tag but do not emit retry spans. A quarantined parent + without attempt-to-fix leaves subtests free to orchestrate their own retries if requested. + +These rules are codified in `parentAttemptFixScenario`, +`subAttemptFixOnlyScenario`, `parentAndSubAttemptFixScenario`, and +`parentQuarantinedAttemptFixScenario`. + +## Scenario Harness (`integrations/gotesting/subtests`) + +- `TestMain` iterates scenarios by spawning subprocesses to keep environment state isolated. +- `startSubtestServer` mimics Datadog APIs (settings, test-management, logs, git endpoints). Every + branch is commented and safe under the sandbox. +- Each scenario uses `scenarioContext` helpers to build a mock payload, including parent and subtest + directives. +- `subtestcontroller_test.go` validates: + - Span counts per resource. + - Passage/failure status tags. + - Attempt-to-fix telemetry (`test.is_retry`, `test.retry_reason`, `test.test_management.attempt_to_fix_passed`). + - Quarantine/disable tags for parent/child combinations. + - Parallel subtests (`SubAttemptFixParallel`) behave identically to sequential ones. +- Debug logging provides detailed trace output, including identity matches, + retry decisions, and span metadata. + +### Scenarios Covered + +- Baseline (no directives) +- `sub_disabled` +- `sub_quarantined` +- `parent_quarantined` +- `parent_quarantined_attempt_to_fix` +- `parent_attempt_to_fix` +- `sub_attempt_to_fix_only` +- `sub_attempt_to_fix_custom_retries` +- `sub_attempt_to_fix_parallel` +- `parent_and_sub_attempt_to_fix` + +Each scenario is thoroughly documented inline so future contributors can extend the matrix. + +## Testing Metadata & Utilities + +- Subtest scenarios interact with the existing instrumentation helpers (`propagateTestExecutionMetadataFlags`, + `setTestTagsFromExecutionMetadata`, etc.), demonstrating metadata propagation without introducing additional + white-box-only harnesses. +- Ancestor fallback is exercised through the `subtests` matrix scenarios, removing the need for exported helper wrappers and keeping white-box tests out of the hot path. + +## Feature Flags & Environment Variables + +- `DD_CIVISIBILITY_SUBTEST_FEATURES_ENABLED` – enable subtest-specific behaviour. +- `CIVisibilityTestManagementAttemptToFixRetries` – global attempt-to-fix retry budget (already + supported but exercised by new scenarios). + +## Summary + +- Subtests now honour Datadog Test Management directives without breaking parent behaviour, testify + support, or legacy flows. +- Attempt-to-fix orchestration chooses a single owner (parent by default) to prevent double retries + and conflicting telemetry. +- The integration harness thoroughly exercises the directive matrix, including parallel subtests, + custom retry budgets, and quarantine inheritance. +- Extensive inline documentation across instrumentation and tests explains each branch and decision, + easing PR review and future maintenance. diff --git a/internal/civisibility/integrations/civisibility_features.go b/internal/civisibility/integrations/civisibility_features.go index dc469af623..0fa909421a 100644 --- a/internal/civisibility/integrations/civisibility_features.go +++ b/internal/civisibility/integrations/civisibility_features.go @@ -147,6 +147,13 @@ func ensureSettingsInitialization(serviceName string) { ciSettings.TestManagement.AttemptToFixRetries = testManagementAttemptToFixRetriesEnv } + // determine if subtest-specific features are enabled via environment variables + subtestFeaturesEnabled := internal.BoolEnv(constants.CIVisibilitySubtestFeaturesEnabled, true) + if !subtestFeaturesEnabled { + log.Debug("civisibility: subtest test management features disabled by environment variable") + } + ciSettings.SubtestFeaturesEnabled = subtestFeaturesEnabled + // check if we need to wait for the upload to finish before continuing if ciSettings.ImpactedTestsEnabled { log.Debug("civisibility: impacted tests is enabled we need to wait for the upload to finish (for the unshallow process)") diff --git a/internal/civisibility/integrations/gotesting/instrumentation.go b/internal/civisibility/integrations/gotesting/instrumentation.go index 8eaa4c4f62..eef3eeed68 100644 --- a/internal/civisibility/integrations/gotesting/instrumentation.go +++ b/internal/civisibility/integrations/gotesting/instrumentation.go @@ -47,6 +47,10 @@ type ( allAttemptsPassed bool // flag to check if all attempts passed for a test marked as attempt to fix allRetriesFailed bool // flag to check if all retries failed for a test hasAdditionalFeatureWrapper bool // flag to check if the current execution is part of an additional feature wrapper + identity *testIdentity // identity of the current execution (test or subtest) + hasExplicitQuarantined bool // flag to mark if quarantine state comes from explicit configuration + hasExplicitDisabled bool // flag to mark if disabled state comes from explicit configuration + hasExplicitAttemptToFix bool // flag to mark if attempt-to-fix state comes from explicit configuration } // runTestWithRetryOptions contains the options for calling runTestWithRetry function @@ -185,8 +189,9 @@ func checkIfCIVisibilityExitIsRequiredByPanic() bool { return !settings.FlakyTestRetriesEnabled && !settings.EarlyFlakeDetection.Enabled } -// applyAdditionalFeaturesToTestFunc applies all the additional features as wrapper of a func(*testing.T) -func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo) func(*testing.T) { +// applyAdditionalFeaturesToTestFunc applies all the additional features as wrapper of a func(*testing.T). +// parentExecMeta is optional and allows subtests to inherit behaviour from their parent test when needed. +func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo, parentExecMeta *testExecutionMetadata) func(*testing.T) { // Apply additional features settings := integrations.GetSettings() @@ -198,18 +203,32 @@ func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo) return f } + identity := testInfo.identity + if identity == nil { + // Derive an identity for tests that did not populate it (such as subtests discovered at runtime). + identity = newTestIdentity(testInfo.moduleName, testInfo.suiteName, testInfo.testName) + } + isSubtest := len(identity.Segments) > 1 + var meta struct { - isTestManagementEnabled bool - isEarlyFlakeDetectionEnabled bool - isFlakyTestRetriesEnabled bool - isQuarantined bool - isDisabled bool - isAttemptToFix bool - isNew bool - isModified bool + identity *testIdentity + isTestManagementEnabled bool + isEarlyFlakeDetectionEnabled bool + isFlakyTestRetriesEnabled bool + isQuarantined bool + isDisabled bool + isAttemptToFix bool + isNew bool + isModified bool + hasExplicitQuarantined bool + hasExplicitDisabled bool + hasExplicitAttemptToFix bool + managementMatchKind testManagementMatchKind + shouldOrchestrateAttemptToFix bool } // init metadata + meta.identity = identity meta.isTestManagementEnabled = settings.TestManagement.Enabled meta.isEarlyFlakeDetectionEnabled = settings.EarlyFlakeDetection.Enabled meta.isFlakyTestRetriesEnabled = settings.FlakyTestRetriesEnabled @@ -218,18 +237,57 @@ func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo) meta.isAttemptToFix = false meta.isNew = false meta.isModified = false + meta.hasExplicitQuarantined = false + meta.hasExplicitDisabled = false + meta.hasExplicitAttemptToFix = false + meta.managementMatchKind = testManagementMatchNone + meta.shouldOrchestrateAttemptToFix = false // Test Management feature if meta.isTestManagementEnabled { - if data, ok := getTestManagementData(testInfo); ok && data != nil { + // Pull the most specific directives available for the current identity. + if data, matchKind, ok := getTestManagementData(identity); ok && data != nil { + meta.managementMatchKind = matchKind meta.isQuarantined = data.Quarantined meta.isDisabled = data.Disabled meta.isAttemptToFix = data.AttemptToFix + if matchKind == testManagementMatchExact { + meta.hasExplicitQuarantined = true + meta.hasExplicitDisabled = true + meta.hasExplicitAttemptToFix = true + } } } + // determine whether attempt-to-fix retries should be orchestrated at this level + meta.shouldOrchestrateAttemptToFix = meta.isAttemptToFix + if parentExecMeta != nil && parentExecMeta.isAttemptToFix { + // The parent already controls the attempt-to-fix loop; subtests should only orchestrate if explicitly requested. + meta.shouldOrchestrateAttemptToFix = meta.hasExplicitAttemptToFix && meta.isAttemptToFix && !parentExecMeta.isAttemptToFix + } + + if isSubtest { + if !settings.SubtestFeaturesEnabled { + // Feature gate keeps legacy behaviour when subtests support is disabled. + return f + } + // Require an exact match before applying subtest-specific directives; fallbacks remain parent-scoped. + if meta.managementMatchKind != testManagementMatchExact { + return f + } + shouldWrap := meta.isQuarantined || meta.isDisabled || meta.isAttemptToFix || + meta.hasExplicitQuarantined || meta.hasExplicitDisabled || meta.hasExplicitAttemptToFix + if !shouldWrap { + return f + } + // Subtests currently inherit parent EFD/flaky retry behaviour; disable here to avoid double wrapping. + meta.isEarlyFlakeDetectionEnabled = false + meta.isFlakyTestRetriesEnabled = false + } + // Early Flake Detection feature if meta.isEarlyFlakeDetectionEnabled { + // Record whether the test is new so we can surface it in spans later. isKnown, hasKnownData := isKnownTest(testInfo) meta.isNew = hasKnownData && !isKnown } @@ -255,9 +313,31 @@ func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo) preExecMetaAdjust: func(execMeta *testExecutionMetadata, _ int) { // Synchronize the test execution metadata with the original test execution metadata. - execMeta.isQuarantined = execMeta.isQuarantined || ptrMeta.isQuarantined - execMeta.isDisabled = execMeta.isDisabled || ptrMeta.isDisabled - execMeta.isAttemptToFix = execMeta.isAttemptToFix || ptrMeta.isAttemptToFix + execMeta.identity = ptrMeta.identity + if ptrMeta.hasExplicitQuarantined { + // Honour the explicitly requested quarantine flag for this execution. + execMeta.isQuarantined = ptrMeta.isQuarantined + execMeta.hasExplicitQuarantined = true + } else { + // Otherwise accumulate quarantine state from earlier runs. + execMeta.isQuarantined = execMeta.isQuarantined || ptrMeta.isQuarantined + } + if ptrMeta.hasExplicitDisabled { + // Apply the disabled directive exactly as configured. + execMeta.isDisabled = ptrMeta.isDisabled + execMeta.hasExplicitDisabled = true + } else { + // Merge prior disabled state from parent/wrappers. + execMeta.isDisabled = execMeta.isDisabled || ptrMeta.isDisabled + } + if ptrMeta.hasExplicitAttemptToFix { + // Only explicit attempt-to-fix should override propagated state. + execMeta.isAttemptToFix = ptrMeta.isAttemptToFix + execMeta.hasExplicitAttemptToFix = true + } else { + // Otherwise inherit whether previous owners already requested attempt-to-fix. + execMeta.isAttemptToFix = execMeta.isAttemptToFix || ptrMeta.isAttemptToFix + } execMeta.isEarlyFlakeDetectionEnabled = execMeta.isEarlyFlakeDetectionEnabled || ptrMeta.isEarlyFlakeDetectionEnabled execMeta.isFlakyTestRetriesEnabled = execMeta.isFlakyTestRetriesEnabled || ptrMeta.isFlakyTestRetriesEnabled execMeta.allAttemptsPassed = atomic.LoadInt32(&allAttemptsPassed) == 1 @@ -268,20 +348,29 @@ func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo) // Propagate flags from the original test metadata. propagateTestExecutionMetadataFlags(execMeta, originalExecMeta) + ptrMeta.identity = execMeta.identity ptrMeta.isQuarantined = execMeta.isQuarantined ptrMeta.isDisabled = execMeta.isDisabled ptrMeta.isAttemptToFix = execMeta.isAttemptToFix + ptrMeta.hasExplicitQuarantined = execMeta.hasExplicitQuarantined + ptrMeta.hasExplicitDisabled = execMeta.hasExplicitDisabled + ptrMeta.hasExplicitAttemptToFix = execMeta.hasExplicitAttemptToFix ptrMeta.isEarlyFlakeDetectionEnabled = execMeta.isEarlyFlakeDetectionEnabled ptrMeta.isFlakyTestRetriesEnabled = execMeta.isFlakyTestRetriesEnabled ptrMeta.isNew = execMeta.isANewTest ptrMeta.isModified = execMeta.isAModifiedTest }, preIsLastRetry: func(execMeta *testExecutionMetadata, _ int, remainingRetries int64) bool { - if execMeta.isAttemptToFix || isAnEfdExecution(execMeta) { + if execMeta.isAttemptToFix && ptrMeta.shouldOrchestrateAttemptToFix { // For attempt-to-fix tests and EFD, the last retry is when remaining retries == 1. return remainingRetries == 1 } + if isAnEfdExecution(execMeta) { + // For EFD, the last retry is when remaining retries == 1. + return remainingRetries == 1 + } + // FlakyTestRetries also considers the global remaining retry count. if execMeta.isFlakyTestRetriesEnabled { return remainingRetries == 1 || atomic.LoadInt64(&integrations.GetFlakyRetriesSettings().RemainingTotalRetryCount) == 1 @@ -293,7 +382,10 @@ func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo) // adjust retry count only runs after the first run // Attempt To Fix retries are always set to the configured value. - if execMeta.isAttemptToFix { + if execMeta.isAttemptToFix && ptrMeta.shouldOrchestrateAttemptToFix { + if execMeta.identity != nil && len(execMeta.identity.Segments) > 1 { + log.Debug("postAdjustRetryCount attempt_to_fix identity=%s setting=%d", execMeta.identity.FullName, settings.TestManagement.AttemptToFixRetries) + } return int64(settings.TestManagement.AttemptToFixRetries) } @@ -339,8 +431,16 @@ func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo) } else if skipped { status = "SKIP" } + if execMeta.identity != nil && len(execMeta.identity.Segments) > 1 { + log.Debug("postPerExecution attempt_to_fix identity=%s orchestrate=%t run=%d status=%s", execMeta.identity.FullName, ptrMeta.shouldOrchestrateAttemptToFix, executionIndex, status) + } - ptrToLocalT.Logf(" [attempt to fix retry: %d (%s)]", executionIndex+1, status) + if ptrMeta.shouldOrchestrateAttemptToFix { + isSubtest := execMeta.identity != nil && len(execMeta.identity.Segments) > 1 + if !isSubtest { + ptrToLocalT.Logf(" [attempt to fix retry: %d (%s)]", executionIndex+1, status) + } + } return } @@ -366,7 +466,7 @@ func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo) } }, postShouldRetry: func(ptrToLocalT *testing.T, execMeta *testExecutionMetadata, _ int, remainingRetries int64) bool { - if execMeta.isAttemptToFix { + if execMeta.isAttemptToFix && ptrMeta.shouldOrchestrateAttemptToFix { // For attempt-to-fix tests, retry if remaining retries > 0. return remainingRetries > 0 } @@ -540,7 +640,8 @@ func runTestWithRetry(options *runTestWithRetryOptions) { } } -// executeTestIteration executes a single test iteration and handles retries. +// executeTestIteration runs a single attempt of the test (or subtest), recording metadata and +// ensuring the retry orchestration has the latest execution context. func executeTestIteration(execOpts *executionOptions) bool { // Iteration lock execOpts.mutex.Lock() @@ -571,7 +672,9 @@ func executeTestIteration(execOpts *executionOptions) bool { if localTPrivateFields.parent == nil { panic("parent of the test is nil") } - *localTPrivateFields.parent = unsafe.Pointer(&testing.T{}) + dummyParent := &testing.T{} + copyTestWithoutParent(execOpts.options.t, dummyParent) + *localTPrivateFields.parent = unsafe.Pointer(dummyParent) // Create an execution metadata instance execMeta := createTestMetadata(ptrToLocalT, execOpts.options.t) @@ -643,7 +746,13 @@ func executeTestIteration(execOpts *executionOptions) bool { } // Extract module and suite if present - currentSuite := execMeta.test.Suite() + if execMeta.test == nil && execMeta.identity != nil { + log.Debug("execMeta.test nil for %s", execMeta.identity.FullName) + } + var currentSuite integrations.TestSuite + if execMeta.test != nil { + currentSuite = execMeta.test.Suite() + } if execOpts.suite == nil && currentSuite != nil { execOpts.suite = currentSuite } @@ -697,7 +806,10 @@ func propagateTestExecutionMetadataFlags(execMeta *testExecutionMetadata, origin execMeta.isFlakyTestRetriesEnabled = execMeta.isFlakyTestRetriesEnabled || originalExecMeta.isFlakyTestRetriesEnabled execMeta.isQuarantined = execMeta.isQuarantined || originalExecMeta.isQuarantined execMeta.isDisabled = execMeta.isDisabled || originalExecMeta.isDisabled - execMeta.isAttemptToFix = execMeta.isAttemptToFix || originalExecMeta.isAttemptToFix + if !execMeta.hasExplicitAttemptToFix && originalExecMeta.isAttemptToFix { + // Preserve attempt-to-fix inheritance only when the child didn't explicitly override it. + execMeta.isAttemptToFix = true + } } // isAnEfdExecution checks if the current test execution is an Early Flake Detection execution. diff --git a/internal/civisibility/integrations/gotesting/instrumentation_orchestrion.go b/internal/civisibility/integrations/gotesting/instrumentation_orchestrion.go index e36edbe61e..e96ea3efd2 100644 --- a/internal/civisibility/integrations/gotesting/instrumentation_orchestrion.go +++ b/internal/civisibility/integrations/gotesting/instrumentation_orchestrion.go @@ -140,106 +140,157 @@ func instrumentTestingTFunc(f func(*testing.T)) func(*testing.T) { suiteName = testifyData.suiteName } - // Increment the test count in the module. - addModulesCounters(moduleName, 1) + subtestIdentity := newTestIdentity(moduleName, suiteName, t.Name()) + isSubtest := len(subtestIdentity.Segments) > 1 - // Increment the test count in the suite. - addSuitesCounters(suiteName, 1) + var testPrivateFields *commonPrivateFields + var parentExecMeta *testExecutionMetadata - // Create or retrieve the module, suite, and test for CI visibility. - module := session.GetOrCreateModule(moduleName) - suite := module.GetOrCreateSuite(suiteName) - test := suite.CreateTest(t.Name()) + if isSubtest { + testPrivateFields = getTestPrivateFields(t) + if testPrivateFields != nil && testPrivateFields.parent != nil { + parentExecMeta = getTestMetadataFromPointer(*testPrivateFields.parent) + } - // If we have testify data we use the method function from testify so the test source is properly set - if testifyData != nil { - test.SetTestFunc(testifyData.methodFunc) - } else { - // If not, let's set the original function - test.SetTestFunc(originalFunc) - } + settings := integrations.GetSettings() + shouldInstrument := settings != nil && settings.SubtestFeaturesEnabled + hasDirective := false - // Get the metadata regarding the execution (in case is already created from the additional features) - execMeta := getTestMetadata(t) - if execMeta == nil { - // in case there's no additional features then we create the metadata for this execution and defer the disposal - execMeta = createTestMetadata(t, nil) - defer deleteTestMetadata(t) - } + log.Debug("subtest gating module=%s suite=%s identity=%s", moduleName, suiteName, subtestIdentity.FullName) + + if parentExecMeta != nil { + if parentExecMeta.isAttemptToFix || parentExecMeta.isDisabled || parentExecMeta.isQuarantined { + hasDirective = true + log.Debug("subtest gating parent directive for %s: attempt_to_fix=%t disabled=%t quarantined=%t", + subtestIdentity.FullName, parentExecMeta.isAttemptToFix, parentExecMeta.isDisabled, parentExecMeta.isQuarantined) + } + } - // Because this is a subtest let's propagate some execution metadata from the parent test - testPrivateFields := getTestPrivateFields(t) - if testPrivateFields != nil && testPrivateFields.parent != nil { - parentExecMeta := getTestMetadataFromPointer(*testPrivateFields.parent) - propagateTestExecutionMetadataFlags(execMeta, parentExecMeta) + if !hasDirective && shouldInstrument { + if data, matchKind, hasData := getTestManagementData(subtestIdentity); hasData && matchKind == testManagementMatchExact && data != nil { + if data.Disabled || data.Quarantined || data.AttemptToFix { + hasDirective = true + log.Debug("subtest gating exact match for %s: disabled=%t quarantined=%t attempt_to_fix=%t", + subtestIdentity.FullName, data.Disabled, data.Quarantined, data.AttemptToFix) + } + } else { + log.Debug("subtest gating no exact match for %s (hasData=%t matchKind=%d)", subtestIdentity.FullName, hasData, matchKind) + } + } } - // Set some required tags from the execution metadata - cancelExecution := setTestTagsFromExecutionMetadata(test, execMeta) - if cancelExecution { - checkModuleAndSuite(module, suite) - return + subtestInfo := &commonInfo{ + moduleName: moduleName, + suiteName: suiteName, + testName: subtestIdentity.FullName, + identity: subtestIdentity, } - defer func() { - // Collect and write logs - collectAndWriteLogs(t, test) + runSubtest := func(currentT *testing.T) { + localIdentity := subtestIdentity + if currentT.Name() != subtestIdentity.FullName { + // Nested subtests have their own full identity path. + localIdentity = newTestIdentity(moduleName, suiteName, currentT.Name()) + } - if r := recover(); r != nil { - // Handle panic and set error information. - if execMeta.isARetry && execMeta.isLastRetry { - if execMeta.allRetriesFailed { - test.SetTag(constants.TestHasFailedAllRetries, "true") - } - if execMeta.isAttemptToFix { - test.SetTag(constants.TestAttemptToFixPassed, "false") - } - } - test.SetError(integrations.WithErrorInfo("panic", fmt.Sprint(r), utils.GetStacktrace(1))) - test.Close(integrations.ResultStatusFail) + addModulesCounters(moduleName, 1) + addSuitesCounters(suiteName, 1) + + log.Debug("instrumentTestingTFunc: creating test span for %s", currentT.Name()) + + module := session.GetOrCreateModule(moduleName) + suite := module.GetOrCreateSuite(suiteName) + test := suite.CreateTest(currentT.Name()) + + if testifyData != nil { + // Testify-based suites expose the original method so we should record that. + test.SetTestFunc(testifyData.methodFunc) + } else { + // Otherwise fall back to the standard testing function pointer. + test.SetTestFunc(originalFunc) + } + + execMeta := getTestMetadata(currentT) + if execMeta == nil { + // Create fresh metadata when additional-feature wrappers were not executed above us. + execMeta = createTestMetadata(currentT, nil) + defer deleteTestMetadata(currentT) + } + execMeta.identity = localIdentity + + currentPrivates := getTestPrivateFields(currentT) + if currentPrivates != nil && currentPrivates.parent != nil { + parentFromCurrent := getTestMetadataFromPointer(*currentPrivates.parent) + propagateTestExecutionMetadataFlags(execMeta, parentFromCurrent) + } + + cancelExecution := setTestTagsFromExecutionMetadata(test, execMeta) + if cancelExecution { checkModuleAndSuite(module, suite) - // this is not an internal test. Retries are not applied to subtest (because the parent internal test is going to be retried) - // so for this case we avoid closing CI Visibility, but we don't stop the panic from happening. - // it will be handled by `t.Run` - if checkIfCIVisibilityExitIsRequiredByPanic() && !execMeta.isAttemptToFix { - integrations.ExitCiVisibility() - } - panic(r) + return } - // Normal finalization: determine the test result based on its state. - if t.Failed() { - if execMeta.isARetry && execMeta.isLastRetry { - if execMeta.allRetriesFailed { - test.SetTag(constants.TestHasFailedAllRetries, "true") + + defer func() { + collectAndWriteLogs(currentT, test) + + if r := recover(); r != nil { + // Set failure metadata before rethrowing the panic. + if execMeta.isARetry && execMeta.isLastRetry { + if execMeta.allRetriesFailed { + test.SetTag(constants.TestHasFailedAllRetries, "true") + } + if execMeta.isAttemptToFix { + test.SetTag(constants.TestAttemptToFixPassed, "false") + } } - if execMeta.isAttemptToFix { - test.SetTag(constants.TestAttemptToFixPassed, "false") + test.SetError(integrations.WithErrorInfo("panic", fmt.Sprint(r), utils.GetStacktrace(1))) + test.Close(integrations.ResultStatusFail) + checkModuleAndSuite(module, suite) + if checkIfCIVisibilityExitIsRequiredByPanic() && !execMeta.isAttemptToFix { + integrations.ExitCiVisibility() } + panic(r) } - test.SetTag(ext.Error, true) - suite.SetTag(ext.Error, true) - module.SetTag(ext.Error, true) - test.Close(integrations.ResultStatusFail) - } else if t.Skipped() { - if execMeta.isAttemptToFix && execMeta.isARetry && execMeta.isLastRetry { - test.SetTag(constants.TestAttemptToFixPassed, "false") - } - test.Close(integrations.ResultStatusSkip) - } else { - if execMeta.isAttemptToFix && execMeta.isARetry && execMeta.isLastRetry { - if execMeta.allAttemptsPassed { - test.SetTag(constants.TestAttemptToFixPassed, "true") - } else { + + if currentT.Failed() { + // Failure path: bubble up retry metadata so spans reflect attempts. + if execMeta.isARetry && execMeta.isLastRetry { + if execMeta.allRetriesFailed { + test.SetTag(constants.TestHasFailedAllRetries, "true") + } + if execMeta.isAttemptToFix { + test.SetTag(constants.TestAttemptToFixPassed, "false") + } + } + test.SetTag(ext.Error, true) + suite.SetTag(ext.Error, true) + module.SetTag(ext.Error, true) + test.Close(integrations.ResultStatusFail) + } else if currentT.Skipped() { + // Skip path still needs to communicate attempt-to-fix failure on the final run. + if execMeta.isAttemptToFix && execMeta.isARetry && execMeta.isLastRetry { test.SetTag(constants.TestAttemptToFixPassed, "false") } + test.Close(integrations.ResultStatusSkip) + } else { + // Success path: tag attempt-to-fix success only if all retries eventually passed. + if execMeta.isAttemptToFix && execMeta.isARetry && execMeta.isLastRetry { + if execMeta.allAttemptsPassed { + test.SetTag(constants.TestAttemptToFixPassed, "true") + } else { + test.SetTag(constants.TestAttemptToFixPassed, "false") + } + } + test.Close(integrations.ResultStatusPass) } - test.Close(integrations.ResultStatusPass) - } - checkModuleAndSuite(module, suite) - }() + checkModuleAndSuite(module, suite) + }() + + f(currentT) + } - // Execute the original test function. - f(t) + wrappedFunc := applyAdditionalFeaturesToTestFunc(runSubtest, subtestInfo, parentExecMeta) + wrappedFunc(t) } setInstrumentationMetadata(runtime.FuncForPC(reflect.Indirect(reflect.ValueOf(instrumentedFn)).Pointer()), &instrumentationMetadata{IsInternal: true}) diff --git a/internal/civisibility/integrations/gotesting/subtests/fixtures_test.go b/internal/civisibility/integrations/gotesting/subtests/fixtures_test.go new file mode 100644 index 0000000000..1eb553ea95 --- /dev/null +++ b/internal/civisibility/integrations/gotesting/subtests/fixtures_test.go @@ -0,0 +1,46 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2025 Datadog, Inc. + +package subtests + +import ( + "os" + "testing" + + gotesting "github.com/DataDog/dd-trace-go/v2/internal/civisibility/integrations/gotesting" +) + +// TestSubtestManagement exercises multiple subtests so the Datadog Go testing +// instrumentation can attach management directives at different hierarchy levels. +func TestSubtestManagement(t *testing.T) { + gt := gotesting.GetTest(t) + + gt.Run("SubDisabled", func(t *testing.T) { + t.Log("subtest intentionally disabled by management directives") + }) + + gt.Run("SubQuarantined", func(t *testing.T) { + t.Log("subtest intentionally quarantined by management directives") + }) + + gt.Run("SubAttemptFix", func(t *testing.T) { + }) + + gt.Run("SubAttemptFixParallel", func(t *testing.T) { + // Run this attempt-to-fix wrapper in parallel when the scenario requests it. + if os.Getenv(parallelToggleEnv) == "1" { + t.Parallel() + } + }) +} + +// TestParentDisabled validates fallback behaviour when only the parent test is +// configured by management data. The child subtest should inherit the disabled +// directive even without an explicit entry. +func TestParentDisabled(t *testing.T) { + gotesting.GetTest(t).Run("Child", func(t *testing.T) { + t.Log("child inherits disabled directive from parent") + }) +} diff --git a/internal/civisibility/integrations/gotesting/subtests/main_test.go b/internal/civisibility/integrations/gotesting/subtests/main_test.go new file mode 100644 index 0000000000..4736f1b7fb --- /dev/null +++ b/internal/civisibility/integrations/gotesting/subtests/main_test.go @@ -0,0 +1,76 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2025 Datadog, Inc. + +package subtests + +import ( + "bytes" + "fmt" + "os" + "os/exec" + "testing" + + "github.com/DataDog/dd-trace-go/v2/internal/civisibility/constants" + "github.com/DataDog/dd-trace-go/v2/internal/log" +) + +const ( + subtestScenarioEnv = "SUBTEST_MATRIX_SCENARIO" +) + +// TestMain forces subtest-specific features on and orchestrates scenario subprocesses so this suite exercises the flag-enabled paths. +func TestMain(m *testing.M) { + prevDD, hadDD := os.LookupEnv(constants.CIVisibilitySubtestFeaturesEnabled) + + if scenario := os.Getenv(subtestScenarioEnv); scenario != "" { + code := runMatrixScenario(m, scenario) + restoreEnv(constants.CIVisibilitySubtestFeaturesEnabled, prevDD, hadDD) + os.Exit(code) + } + + for _, scenario := range matrixScenarioNames() { + cmd := exec.Command(os.Args[0], os.Args[1:]...) + var buffer bytes.Buffer + cmd.Stdout = &buffer + cmd.Stderr = &buffer + if log.DebugEnabled() { + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + } + cmd.Env = append(os.Environ(), fmt.Sprintf("%s=%s", subtestScenarioEnv, scenario)) + + fmt.Printf("\n**** [RUNNING SUBTEST SCENARIO: %s]\n", scenario) + if err := cmd.Run(); err != nil { + if exitErr, ok := err.(*exec.ExitError); ok { + fmt.Printf("\n**** [SCENARIO %s FAILED WITH EXIT CODE: %d]\n", scenario, exitErr.ExitCode()) + fmt.Printf("**** [SCENARIO %s OUTPUT]\n%s\n", scenario, buffer.String()) + restoreEnv(constants.CIVisibilitySubtestFeaturesEnabled, prevDD, hadDD) + os.Exit(exitErr.ExitCode()) + } + fmt.Printf("failed to run scenario %s: %v\n", scenario, err) + restoreEnv(constants.CIVisibilitySubtestFeaturesEnabled, prevDD, hadDD) + os.Exit(1) + } + fmt.Printf("**** [SCENARIO %s COMPLETED]\n", scenario) + } + + code := m.Run() + + restoreEnv(constants.CIVisibilitySubtestFeaturesEnabled, prevDD, hadDD) + + os.Exit(code) +} + +func restoreEnv(key, value string, had bool) { + if !had { + if err := os.Unsetenv(key); err != nil { + panic(err) + } + return + } + if err := os.Setenv(key, value); err != nil { + panic(err) + } +} diff --git a/internal/civisibility/integrations/gotesting/subtests/subtestcontroller_test.go b/internal/civisibility/integrations/gotesting/subtests/subtestcontroller_test.go new file mode 100644 index 0000000000..8916960461 --- /dev/null +++ b/internal/civisibility/integrations/gotesting/subtests/subtestcontroller_test.go @@ -0,0 +1,856 @@ +// Unless explicitly stated otherwise all files in this repository are licensed +// under the Apache License Version 2.0. +// This product includes software developed at Datadog (https://www.datadoghq.com/). +// Copyright 2025 Datadog, Inc. + +package subtests + +import ( + "compress/gzip" + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "os" + "reflect" + "sort" + "strconv" + "testing" + + "github.com/DataDog/dd-trace-go/v2/ddtrace/ext" + "github.com/DataDog/dd-trace-go/v2/ddtrace/mocktracer" + "github.com/DataDog/dd-trace-go/v2/internal/civisibility/constants" + "github.com/DataDog/dd-trace-go/v2/internal/civisibility/integrations" + gotesting "github.com/DataDog/dd-trace-go/v2/internal/civisibility/integrations/gotesting" + "github.com/DataDog/dd-trace-go/v2/internal/civisibility/utils" + "github.com/DataDog/dd-trace-go/v2/internal/civisibility/utils/net" + "github.com/DataDog/dd-trace-go/v2/internal/log" +) + +const ( + moduleUnderTest = "github.com/DataDog/dd-trace-go/v2/internal/civisibility/integrations/gotesting/subtests" + suiteUnderTest = "fixtures_test.go" + parentTestName = "TestSubtestManagement" + parallelToggleEnv = "SUBTEST_MATRIX_PARALLEL" +) + +var ( + availableScenarios = []*matrixScenario{ + baselineScenario(), + subDisabledScenario(), + subQuarantinedScenario(), + parentQuarantinedScenario(), + parentQuarantinedAttemptFixScenario(), + parentAttemptFixScenario(), + subAttemptFixOnlyScenario(), + subAttemptFixCustomRetriesScenario(), + subAttemptFixParallelScenario(), + parentAndSubAttemptFixScenario(), + } + scenarioByName = func() map[string]*matrixScenario { + m := make(map[string]*matrixScenario, len(availableScenarios)) + for _, sc := range availableScenarios { + m[sc.name] = sc + } + return m + }() +) + +type directive struct { + disabled bool + quarantined bool + attemptToFix bool +} + +type matrixScenario struct { + name string + configure func(*scenarioContext) + validate func([]*mocktracer.Span) +} + +type scenarioContext struct { + data *net.TestManagementTestsResponseDataModules + attemptToFixRetries int + env map[string]string +} + +// newScenarioContext prepares an empty scenario scaffold with storage for module directives. +func newScenarioContext() *scenarioContext { + return &scenarioContext{ + data: &net.TestManagementTestsResponseDataModules{ + Modules: make(map[string]net.TestManagementTestsResponseDataSuites), + }, + attemptToFixRetries: 0, + env: make(map[string]string), + } +} + +// setParentDirective records backend directives for the parent test in the scenario payload. +func (ctx *scenarioContext) setParentDirective(dir directive) { + module := ctx.data.Modules[moduleUnderTest] + // Lazily allocate the suites map so directives can be written. + if module.Suites == nil { + module.Suites = make(map[string]net.TestManagementTestsResponseDataTests) + } + suite := module.Suites[suiteUnderTest] + // Lazily allocate the test map for the suite. + if suite.Tests == nil { + suite.Tests = make(map[string]net.TestManagementTestsResponseDataTestProperties) + } + props := net.TestManagementTestsResponseDataTestPropertiesAttributes{ + Disabled: dir.disabled, + Quarantined: dir.quarantined, + AttemptToFix: dir.attemptToFix, + } + suite.Tests[parentTestName] = net.TestManagementTestsResponseDataTestProperties{Properties: props} + module.Suites[suiteUnderTest] = suite + ctx.data.Modules[moduleUnderTest] = module +} + +// setSubDirective associates directives with a specific subtest within the scenario payload. +func (ctx *scenarioContext) setSubDirective(subName string, dir directive) { + module := ctx.data.Modules[moduleUnderTest] + // Ensure the suite map exists before inserting the subtest directive. + if module.Suites == nil { + module.Suites = make(map[string]net.TestManagementTestsResponseDataTests) + } + suite := module.Suites[suiteUnderTest] + // Initialise per-test properties map when missing. + if suite.Tests == nil { + suite.Tests = make(map[string]net.TestManagementTestsResponseDataTestProperties) + } + fullName := fmt.Sprintf("%s/%s", parentTestName, subName) + props := net.TestManagementTestsResponseDataTestPropertiesAttributes{ + Disabled: dir.disabled, + Quarantined: dir.quarantined, + AttemptToFix: dir.attemptToFix, + } + suite.Tests[fullName] = net.TestManagementTestsResponseDataTestProperties{Properties: props} + module.Suites[suiteUnderTest] = suite + ctx.data.Modules[moduleUnderTest] = module +} + +// ensureSuite creates suite entries in the mock payload when missing so directives can be attached. +func (ctx *scenarioContext) ensureSuite() { + module := ctx.data.Modules[moduleUnderTest] + // Avoid nil maps so later writes succeed. + if module.Suites == nil { + module.Suites = make(map[string]net.TestManagementTestsResponseDataTests) + } + suite := module.Suites[suiteUnderTest] + // Guarantee there is at least an empty tests map for the suite. + if suite.Tests == nil { + suite.Tests = make(map[string]net.TestManagementTestsResponseDataTestProperties) + } + module.Suites[suiteUnderTest] = suite + ctx.data.Modules[moduleUnderTest] = module +} + +// setEnv records an environment variable override to be applied during scenario execution. +func (ctx *scenarioContext) setEnv(key, value string) { + if ctx.env == nil { + ctx.env = make(map[string]string) + } + ctx.env[key] = value +} + +// baselineScenario captures the control case where no directives are present, ensuring +// that subtests execute normally without management tags or retry orchestration. +func baselineScenario() *matrixScenario { + return &matrixScenario{ + name: "baseline", + configure: func(ctx *scenarioContext) { + // Ensure the suite exists so subsequent lookups succeed. + ctx.ensureSuite() + // Explicitly reset the attempt-to-fix retry budget for the identity scenario. + ctx.attemptToFixRetries = 0 + module, suite := utils.GetModuleAndSuiteName(reflect.ValueOf(TestSubtestManagement).Pointer()) + debugMatrixf("baseline identity module=%s suite=%s", module, suite) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + debugMatrixf("baseline captured %d test spans", len(testSpans)) + for _, span := range testSpans { + // Skip nil spans because they do not carry any metadata. + if span == nil { + continue + } + // Log each resource to help diagnose unexpected spans during debugging. + if resource, ok := span.Tag(ext.ResourceName).(string); ok { + debugMatrixf(" - resource: %s", resource) + } + } + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 1, "parent baseline") + + assertTagEquals(parentSpans[0], constants.TestStatus, constants.TestStatusPass, "parent baseline status") + assertTagNotTrue(parentSpans[0], constants.TestIsDisabled, "parent baseline disabled") + assertTagNotTrue(parentSpans[0], constants.TestIsQuarantined, "parent baseline quarantined") + assertTagNotTrue(parentSpans[0], constants.TestIsAttempToFix, "parent baseline attempt_to_fix") + + for _, sub := range []string{"SubDisabled", "SubQuarantined", "SubAttemptFix", "SubAttemptFixParallel"} { + resource := fmt.Sprintf("%s/%s", parentResource, sub) + subSpans := spansByResource(testSpans, resource) + requireSpanCount(subSpans, 1, fmt.Sprintf("subtest %s baseline count", sub)) + assertTagEquals(subSpans[0], constants.TestStatus, constants.TestStatusPass, fmt.Sprintf("subtest %s baseline status", sub)) + assertTagNotTrue(subSpans[0], constants.TestIsDisabled, fmt.Sprintf("subtest %s baseline disabled", sub)) + assertTagNotTrue(subSpans[0], constants.TestIsQuarantined, fmt.Sprintf("subtest %s baseline quarantined", sub)) + assertTagNotTrue(subSpans[0], constants.TestIsAttempToFix, fmt.Sprintf("subtest %s baseline attempt_to_fix", sub)) + } + }, + } +} + +// subAttemptFixOnlyScenario verifies that only the child subtest orchestrates attempt-to-fix retries +// while the parent remains neutral. +func subAttemptFixOnlyScenario() *matrixScenario { + return &matrixScenario{ + name: "sub_attempt_to_fix_only", + configure: func(ctx *scenarioContext) { + // Initialise the suite and make the retry budget available to the subtest. + ctx.ensureSuite() + ctx.attemptToFixRetries = 3 + ctx.setSubDirective("SubAttemptFix", directive{attemptToFix: true}) + module, suite := utils.GetModuleAndSuiteName(reflect.ValueOf(TestSubtestManagement).Pointer()) + debugMatrixf("sub_attempt_to_fix_only identity module=%s suite=%s", module, suite) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + debugMatrixf("sub_attempt_to_fix_only captured %d test spans", len(testSpans)) + for _, span := range testSpans { + // Guard against nil entries to avoid panics when introspecting tags. + if span == nil { + continue + } + // Provide verbose details when debugging span resources. + if resource, ok := span.Tag(ext.ResourceName).(string); ok { + debugMatrixf(" - resource: %s", resource) + } + } + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 1, "sub attempt-to-fix-only parent count") + assertTagNotTrue(parentSpans[0], constants.TestIsAttempToFix, "sub attempt-to-fix-only parent tag") + + subResource := fmt.Sprintf("%s/%s", parentResource, "SubAttemptFix") + subSpans := spansByResource(testSpans, subResource) + requireSpanCount(subSpans, 3, "sub attempt-to-fix-only span count") + sort.Slice(subSpans, func(i, j int) bool { + // Order spans chronologically so comments and assertions match the retry lifecycle. + return subSpans[i].StartTime().Before(subSpans[j].StartTime()) + }) + for idx, span := range subSpans { + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("sub attempt-to-fix-only tag span %d", idx)) + } + lastSpan := subSpans[len(subSpans)-1] + assertTagEquals(lastSpan, constants.TestAttemptToFixPassed, "true", "sub attempt-to-fix-only success") + assertTagEquals(lastSpan, constants.TestStatus, constants.TestStatusPass, "sub attempt-to-fix-only final status") + assertTagCount(subSpans, constants.TestIsRetry, "true", 2, "sub attempt-to-fix-only retry tag count") + assertTagCount(subSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 2, "sub attempt-to-fix-only retry reason count") + }, + } +} + +// subAttemptFixCustomRetriesScenario demonstrates that a child can request a larger retry +// budget without involving the parent, ensuring the additional attempts are tagged correctly. +func subAttemptFixCustomRetriesScenario() *matrixScenario { + return &matrixScenario{ + name: "sub_attempt_to_fix_custom_retries", + configure: func(ctx *scenarioContext) { + // Initialise suite storage so directives can be attached safely. + ctx.ensureSuite() + ctx.attemptToFixRetries = 5 + ctx.setSubDirective("SubAttemptFix", directive{attemptToFix: true}) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 1, "sub attempt-to-fix custom parent count") + assertTagNotTrue(parentSpans[0], constants.TestIsAttempToFix, "sub attempt-to-fix custom parent tag") + + subResource := fmt.Sprintf("%s/%s", parentResource, "SubAttemptFix") + subSpans := spansByResource(testSpans, subResource) + requireSpanCount(subSpans, 5, "sub attempt-to-fix custom child span count") + sort.Slice(subSpans, func(i, j int) bool { + // Sort to make reasoning about retries deterministic. + return subSpans[i].StartTime().Before(subSpans[j].StartTime()) + }) + for idx, span := range subSpans { + // Each retry should still carry the attempt-to-fix tag for visibility. + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("sub attempt-to-fix custom tag span %d", idx)) + } + subFinal := subSpans[len(subSpans)-1] + assertTagEquals(subFinal, constants.TestAttemptToFixPassed, "true", "sub attempt-to-fix custom success") + assertTagEquals(subFinal, constants.TestStatus, constants.TestStatusPass, "sub attempt-to-fix custom final status") + assertTagCount(subSpans, constants.TestIsRetry, "true", 4, "sub attempt-to-fix custom retry tag count") + assertTagCount(subSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 4, "sub attempt-to-fix custom retry reason count") + }, + } +} + +// subAttemptFixParallelScenario asserts that parallel subtests inherit attempt-to-fix behaviour +// without conflicting with the sequential sibling. +func subAttemptFixParallelScenario() *matrixScenario { + return &matrixScenario{ + name: "sub_attempt_to_fix_parallel", + configure: func(ctx *scenarioContext) { + // Prepare suite metadata before writing directives. + ctx.ensureSuite() + ctx.attemptToFixRetries = 3 + ctx.setSubDirective("SubAttemptFix", directive{attemptToFix: true}) + ctx.setSubDirective("SubAttemptFixParallel", directive{attemptToFix: true}) + ctx.setEnv(parallelToggleEnv, "1") + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 1, "sub attempt-to-fix parallel parent count") + assertTagNotTrue(parentSpans[0], constants.TestIsAttempToFix, "sub attempt-to-fix parallel parent tag") + + checkParallelChild := func(child string) { + // Focus validations on a single subtest resource at a time. + resource := fmt.Sprintf("%s/%s", parentResource, child) + childSpans := spansByResource(testSpans, resource) + requireSpanCount(childSpans, 3, fmt.Sprintf("%s attempt-to-fix parallel span count", child)) + sort.Slice(childSpans, func(i, j int) bool { + // Sort to match retry order regardless of goroutine scheduling. + return childSpans[i].StartTime().Before(childSpans[j].StartTime()) + }) + for idx, span := range childSpans { + // Confirm each execution is correctly tagged as part of the attempt-to-fix flow. + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("%s attempt-to-fix parallel tag span %d", child, idx)) + } + final := childSpans[len(childSpans)-1] + assertTagEquals(final, constants.TestAttemptToFixPassed, "true", fmt.Sprintf("%s attempt-to-fix parallel success", child)) + assertTagEquals(final, constants.TestStatus, constants.TestStatusPass, fmt.Sprintf("%s attempt-to-fix parallel status", child)) + assertTagCount(childSpans, constants.TestIsRetry, "true", 2, fmt.Sprintf("%s attempt-to-fix parallel retry tag count", child)) + assertTagCount(childSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 2, fmt.Sprintf("%s attempt-to-fix parallel retry reason count", child)) + } + + checkParallelChild("SubAttemptFix") + checkParallelChild("SubAttemptFixParallel") + }, + } +} + +// subDisabledScenario asserts that a subtest disabled directive skips the child while +// leaving the parent untouched. +func subDisabledScenario() *matrixScenario { + return &matrixScenario{ + name: "sub_disabled", + configure: func(ctx *scenarioContext) { + ctx.ensureSuite() + ctx.setParentDirective(directive{}) + ctx.setSubDirective("SubDisabled", directive{disabled: true}) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 1, "sub disabled parent count") + assertTagNotTrue(parentSpans[0], constants.TestIsDisabled, "parent disabled tag") + + subResource := fmt.Sprintf("%s/%s", parentResource, "SubDisabled") + subSpans := spansByResource(testSpans, subResource) + requireSpanCount(subSpans, 1, "sub disabled span count") + assertTagEquals(subSpans[0], constants.TestIsDisabled, "true", "sub disabled tag") + assertTagEquals(subSpans[0], constants.TestStatus, constants.TestStatusSkip, "sub disabled status") + }, + } +} + +// subQuarantinedScenario ensures a child-only quarantine directive reports the subtest as quarantined. +func subQuarantinedScenario() *matrixScenario { + return &matrixScenario{ + name: "sub_quarantined", + configure: func(ctx *scenarioContext) { + ctx.ensureSuite() + ctx.setParentDirective(directive{}) + ctx.setSubDirective("SubQuarantined", directive{quarantined: true}) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 1, "sub quarantined parent count") + assertTagNotTrue(parentSpans[0], constants.TestIsQuarantined, "parent quarantined tag") + + subResource := fmt.Sprintf("%s/%s", parentResource, "SubQuarantined") + subSpans := spansByResource(testSpans, subResource) + requireSpanCount(subSpans, 1, "sub quarantined span count") + assertTagEquals(subSpans[0], constants.TestIsQuarantined, "true", "sub quarantined tag") + assertTagEquals(subSpans[0], constants.TestStatus, constants.TestStatusPass, "sub quarantined status") + }, + } +} + +// parentQuarantinedScenario shows that a quarantined parent automatically propagates the tag to its child. +func parentQuarantinedScenario() *matrixScenario { + return &matrixScenario{ + name: "parent_quarantined", + configure: func(ctx *scenarioContext) { + ctx.ensureSuite() + ctx.setParentDirective(directive{quarantined: true}) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 1, "parent quarantined span count") + assertTagEquals(parentSpans[0], constants.TestIsQuarantined, "true", "parent quarantined tag") + assertTagEquals(parentSpans[0], constants.TestStatus, constants.TestStatusPass, "parent quarantined status") + + subResource := fmt.Sprintf("%s/%s", parentResource, "SubQuarantined") + subSpans := spansByResource(testSpans, subResource) + requireSpanCount(subSpans, 1, "parent quarantined child span count") + assertTagEquals(subSpans[0], constants.TestIsQuarantined, "true", "parent quarantined child tag") + assertTagEquals(subSpans[0], constants.TestStatus, constants.TestStatusPass, "parent quarantined child status") + }, + } +} + +// parentQuarantinedAttemptFixScenario validates that a quarantined parent orchestrating retries +// propagates quarantine tags to the child while keeping ownership of the attempt-to-fix lifecycle. +func parentQuarantinedAttemptFixScenario() *matrixScenario { + return &matrixScenario{ + name: "parent_quarantined_attempt_to_fix", + configure: func(ctx *scenarioContext) { + ctx.ensureSuite() + ctx.attemptToFixRetries = 3 + ctx.setParentDirective(directive{quarantined: true, attemptToFix: true}) + ctx.setSubDirective("SubAttemptFix", directive{attemptToFix: true, quarantined: true}) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 3, "parent quarantine attempt-to-fix span count") + // Confirm every parent retry is both quarantined and marked as attempt-to-fix. + for idx, span := range parentSpans { + assertTagEquals(span, constants.TestIsQuarantined, "true", fmt.Sprintf("parent quarantine attempt-to-fix quarantined tag span %d", idx)) + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("parent quarantine attempt-to-fix attempt tag span %d", idx)) + } + parentFinal := parentSpans[len(parentSpans)-1] + assertTagEquals(parentFinal, constants.TestAttemptToFixPassed, "true", "parent quarantine attempt-to-fix success") + assertTagEquals(parentFinal, constants.TestStatus, constants.TestStatusPass, "parent quarantine attempt-to-fix status") + assertTagCount(parentSpans, constants.TestIsRetry, "true", 2, "parent quarantine attempt-to-fix retry tag count") + assertTagCount(parentSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 2, "parent quarantine attempt-to-fix retry reason count") + + subResource := fmt.Sprintf("%s/%s", parentResource, "SubAttemptFix") + subSpans := spansByResource(testSpans, subResource) + requireSpanCount(subSpans, 3, "parent quarantine attempt-to-fix child span count") + // Each child execution must inherit the quarantined + attempt-to-fix state. + for idx, span := range subSpans { + assertTagEquals(span, constants.TestIsQuarantined, "true", fmt.Sprintf("child quarantine attempt-to-fix quarantined tag span %d", idx)) + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("child quarantine attempt-to-fix attempt tag span %d", idx)) + } + childFinal := subSpans[len(subSpans)-1] + assertTagNotTrue(childFinal, constants.TestAttemptToFixPassed, "child quarantine attempt-to-fix success tag ownership") + assertTagEquals(childFinal, constants.TestStatus, constants.TestStatusPass, "child quarantine attempt-to-fix status") + assertTagCount(subSpans, constants.TestIsRetry, "true", 0, "child quarantine attempt-to-fix retry tag count") + assertTagCount(subSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 0, "child quarantine attempt-to-fix retry reason count") + }, + } +} + +// parentAttemptFixScenario checks that a parent-level attempt-to-fix directive wraps both +// parent and child executions with consistent retry tagging. +func parentAttemptFixScenario() *matrixScenario { + return &matrixScenario{ + name: "parent_attempt_to_fix", + configure: func(ctx *scenarioContext) { + ctx.ensureSuite() + ctx.attemptToFixRetries = 3 + ctx.setParentDirective(directive{attemptToFix: true}) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 3, "parent attempt-to-fix span count") + // Each parent span should reflect that attempt-to-fix logic is active. + for idx, span := range parentSpans { + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("parent attempt-to-fix tag span %d", idx)) + } + assertTagCount(parentSpans, constants.TestIsRetry, "true", 2, "parent attempt-to-fix retry tag count") + assertTagCount(parentSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 2, "parent attempt-to-fix retry reason count") + + subResource := fmt.Sprintf("%s/%s", parentResource, "SubAttemptFix") + subSpans := spansByResource(testSpans, subResource) + requireSpanCount(subSpans, 3, "sub attempt-to-fix inherited span count") + // The child inherits the attempt-to-fix directive and should pass under retry pressure. + for idx, span := range subSpans { + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("sub inherited attempt-to-fix tag span %d", idx)) + assertTagEquals(span, constants.TestStatus, constants.TestStatusPass, fmt.Sprintf("sub inherited attempt-to-fix status span %d", idx)) + } + assertTagCount(subSpans, constants.TestIsRetry, "true", 2, "sub inherited attempt-to-fix retry tag count") + assertTagCount(subSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 2, "sub inherited attempt-to-fix retry reason count") + }, + } +} + +// parentAndSubAttemptFixScenario makes sure that when both parent and child request +// attempt-to-fix behaviour the parent retains ownership of success tagging. +func parentAndSubAttemptFixScenario() *matrixScenario { + return &matrixScenario{ + name: "parent_and_sub_attempt_to_fix", + configure: func(ctx *scenarioContext) { + ctx.ensureSuite() + ctx.attemptToFixRetries = 3 + ctx.setParentDirective(directive{attemptToFix: true}) + ctx.setSubDirective("SubAttemptFix", directive{attemptToFix: true}) + }, + validate: func(spans []*mocktracer.Span) { + testSpans := filterTestSpans(spans) + + parentResource := fmt.Sprintf("%s.%s", suiteUnderTest, parentTestName) + parentSpans := spansByResource(testSpans, parentResource) + requireSpanCount(parentSpans, 3, "parent/sub attempt-to-fix parent span count") + // Validate every parent execution reflects the attempt-to-fix directive. + for idx, span := range parentSpans { + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("parent/sub attempt-to-fix parent tag span %d", idx)) + } + parentFinal := parentSpans[len(parentSpans)-1] + assertTagEquals(parentFinal, constants.TestAttemptToFixPassed, "true", "parent/sub attempt-to-fix parent success") + assertTagEquals(parentFinal, constants.TestStatus, constants.TestStatusPass, "parent/sub attempt-to-fix parent final status") + assertTagCount(parentSpans, constants.TestIsRetry, "true", 2, "parent/sub attempt-to-fix parent retry tag count") + assertTagCount(parentSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 2, "parent/sub attempt-to-fix parent retry reason count") + + subResource := fmt.Sprintf("%s/%s", parentResource, "SubAttemptFix") + subSpans := spansByResource(testSpans, subResource) + requireSpanCount(subSpans, 3, "parent/sub attempt-to-fix child span count") + // Even though the child has its own directive, it should not claim retry success. + for idx, span := range subSpans { + assertTagEquals(span, constants.TestIsAttempToFix, "true", fmt.Sprintf("parent/sub attempt-to-fix child tag span %d", idx)) + } + subFinal := subSpans[len(subSpans)-1] + assertTagNotTrue(subFinal, constants.TestAttemptToFixPassed, "parent/sub attempt-to-fix child success tag ownership") + assertTagEquals(subFinal, constants.TestStatus, constants.TestStatusPass, "parent/sub attempt-to-fix child final status") + assertTagCount(subSpans, constants.TestIsRetry, "true", 0, "parent/sub attempt-to-fix child retry tag count") + assertTagCount(subSpans, constants.TestRetryReason, constants.AttemptToFixRetryReason, 0, "parent/sub attempt-to-fix child retry reason count") + }, + } +} + +// filterTestSpans keeps only test spans so scenario assertions ignore suite/module noise. +func filterTestSpans(spans []*mocktracer.Span) []*mocktracer.Span { + var out []*mocktracer.Span + for _, span := range spans { + // Ignore placeholders without data. + if span == nil { + continue + } + // Only keep spans that represent tests, not suites or infrastructure. + if tag := span.Tag(ext.SpanType); tag == constants.SpanTypeTest { + out = append(out, span) + } + } + return out +} + +// spansByResource returns the subset of spans whose resource matches the provided identifier. +func spansByResource(spans []*mocktracer.Span, resource string) []*mocktracer.Span { + var out []*mocktracer.Span + for _, span := range spans { + // Skip nil entries that cannot hold tags. + if span == nil { + continue + } + // Collect spans that target the requested resource name. + if value, ok := span.Tag(ext.ResourceName).(string); ok && value == resource { + out = append(out, span) + } + } + return out +} + +// requireSpanCount asserts that a span slice contains exactly the amount expected for the scenario. +func requireSpanCount(spans []*mocktracer.Span, expected int, label string) { + // Panic with context when the scenario produced an unexpected number of spans. + if len(spans) != expected { + panic(fmt.Sprintf("%s: expected %d spans, got %d", label, expected, len(spans))) + } +} + +// assertTagCount enforces that a precise number of spans expose a tag/value pair, mirroring +// telemetry expectations such as retry counters. +func assertTagCount(spans []*mocktracer.Span, key string, value string, expected int, label string) { + var count int + for _, span := range spans { + if span == nil { + continue + } + if tag, ok := span.Tag(key).(string); ok && tag == value { + count++ + } + } + debugMatrixf("%s: observed %d spans with %s=%q", label, count, key, value) + if count != expected { + panic(fmt.Sprintf("%s: expected %d spans with tag %s=%q, got %d", label, expected, key, value, count)) + } +} + +// assertTagEquals verifies that a span tag matches the desired value and fails fast otherwise. +func assertTagEquals(span *mocktracer.Span, key string, want string, label string) { + if span == nil { + panic(fmt.Sprintf("%s: span is nil", label)) + } + if value, _ := span.Tag(key).(string); value != want { + panic(fmt.Sprintf("%s: expected tag %s=%q, got %q", label, key, want, value)) + } +} + +// assertTagNotTrue ensures a boolean-like tag is either absent or false, useful when testing inheritance. +func assertTagNotTrue(span *mocktracer.Span, key string, label string) { + if span == nil { + return + } + if value, ok := span.Tag(key).(string); ok && value == "true" { + panic(fmt.Sprintf("%s: expected tag %s to be absent/false, got true", label, key)) + } +} + +// matrixScenarioNames returns the list of scenario identifiers executed by TestMain. +func matrixScenarioNames() []string { + names := make([]string, 0, len(availableScenarios)) + for _, sc := range availableScenarios { + // Skip placeholder slots that may be left empty in future expansions. + if sc == nil { + continue + } + names = append(names, sc.name) + } + return names +} + +// runMatrixScenario executes a specific scenario in isolation by configuring the mock backend, +// running the go test harness, and validating the resulting spans. +func runMatrixScenario(m *testing.M, scenario string) int { + sc, ok := scenarioByName[scenario] + // Abort quickly when the requested scenario does not exist. + if !ok { + fmt.Printf("unknown subtest matrix scenario: %s\n", scenario) + return 1 + } + + ctx := newScenarioContext() + sc.configure(ctx) + debugMatrixf("scenario %s management data: %+v", scenario, ctx.data) + + var envSnapshots []envSnapshot + for key, value := range ctx.env { + envSnapshots = append(envSnapshots, setEnv(key, value)) + } + defer func() { + for i := len(envSnapshots) - 1; i >= 0; i-- { + envSnapshots[i].restore() + } + }() + + _, restore := startSubtestServer(subtestServerConfig{ + managementData: ctx.data, + attemptToFixRetries: ctx.attemptToFixRetries, + }) + defer restore() + + settings := integrations.GetSettings() + if settings != nil { + settings.SubtestFeaturesEnabled = true + debugMatrixf("subtest matrix: settings.SubtestFeaturesEnabled=%t", settings.SubtestFeaturesEnabled) + } else { + debugMatrixf("subtest matrix: settings unavailable") + } + + tracer := integrations.InitializeCIVisibilityMock() + + exitCode := gotesting.RunM(m) + // When the run fails, dump span resources for easier diagnosis. + if exitCode != 0 { + finished := tracer.FinishedSpans() + debugMatrixf("scenario %s exit code %d with %d spans", scenario, exitCode, len(finished)) + for i, span := range finished { + // Skip nil entries yet keep the loop for consistent indices. + if span == nil { + continue + } + // Provide per-span resource names to speed up debugging. + if resource, ok := span.Tag(ext.ResourceName).(string); ok { + debugMatrixf(" span[%d] resource=%s status=%v", i, resource, span.Tag(constants.TestStatus)) + } + } + return exitCode + } + + sc.validate(tracer.FinishedSpans()) + + return 0 +} + +// debugMatrixf emits scenario-scoped diagnostics using the package logger. +func debugMatrixf(format string, args ...interface{}) { + log.Debug(format, args...) +} + +type envSnapshot struct { + key string + value string + had bool +} + +// setEnv overrides an environment variable and returns a snapshot that can restore it. +func setEnv(key, value string) envSnapshot { + prev, had := os.LookupEnv(key) + if err := os.Setenv(key, value); err != nil { + panic(err) + } + return envSnapshot{key: key, value: prev, had: had} +} + +func (s envSnapshot) restore() { + var err error + if s.had { + err = os.Setenv(s.key, s.value) + } else { + err = os.Unsetenv(s.key) + } + if err != nil { + panic(err) + } +} + +type subtestServerConfig struct { + managementData *net.TestManagementTestsResponseDataModules + attemptToFixRetries int +} + +// startSubtestServer spins up the mock backend that feeds settings and management payloads to the harness. +func startSubtestServer(cfg subtestServerConfig) (*httptest.Server, func()) { + if cfg.managementData == nil { + cfg.managementData = &net.TestManagementTestsResponseDataModules{ + Modules: make(map[string]net.TestManagementTestsResponseDataSuites), + } + } + + handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + // Route each request to a stub mirroring the Datadog backend endpoints. + switch r.URL.Path { + case "/api/v2/libraries/tests/services/setting": + // Provide the library settings response required to enable management. + debugMatrixf("subtest server: settings request") + defer r.Body.Close() + w.Header().Set("Content-Type", "application/json") + var settings net.SettingsResponseData + settings.CodeCoverage = false + settings.FlakyTestRetriesEnabled = false + settings.ItrEnabled = false + settings.TestsSkipping = false + settings.KnownTestsEnabled = false + settings.ImpactedTestsEnabled = false + settings.EarlyFlakeDetection.Enabled = false + settings.TestManagement.Enabled = true + settings.TestManagement.AttemptToFixRetries = cfg.attemptToFixRetries + resp := struct { + Data struct { + ID string `json:"id"` + Type string `json:"type"` + Attributes net.SettingsResponseData `json:"attributes"` + } `json:"data"` + }{} + resp.Data.ID = "settings" + resp.Data.Type = "ci_app_libraries_tests_settings" + resp.Data.Attributes = settings + if err := json.NewEncoder(w).Encode(&resp); err != nil { + panic(err) + } + case "/api/v2/test/libraries/test-management/tests": + // Serve the management payload that drives scenario directives. + debugMatrixf("subtest server: test-management request") + defer r.Body.Close() + w.Header().Set("Content-Type", "application/json") + if payload, err := json.Marshal(cfg.managementData); err == nil { + debugMatrixf("subtest server: management payload %s", payload) + } + resp := struct { + Data struct { + ID string `json:"id"` + Type string `json:"type"` + Attributes net.TestManagementTestsResponseDataModules `json:"attributes"` + } `json:"data"` + }{} + resp.Data.ID = "test-management" + resp.Data.Type = "ci_app_libraries_tests" + resp.Data.Attributes = *cfg.managementData + if err := json.NewEncoder(w).Encode(&resp); err != nil { + panic(err) + } + case "/api/v2/ci/libraries/tests": + // Return an empty known-tests payload to satisfy the client. + debugMatrixf("subtest server: known-tests request") + defer r.Body.Close() + w.Header().Set("Content-Type", "application/json") + _, _ = io.Copy(io.Discard, r.Body) + w.Write([]byte(`{"data":{"attributes":{"tests":{}}}}`)) + case "/api/v2/git/repository/search_commits": + // Stub git search commits used during CI Visibility bootstrap. + debugMatrixf("subtest server: search-commits request") + defer r.Body.Close() + _, _ = io.Copy(io.Discard, r.Body) + w.WriteHeader(http.StatusOK) + w.Write([]byte(`{}`)) + case "/api/v2/git/repository/packfile": + // Accept packfile uploads even though the sandbox blocks writes. + debugMatrixf("subtest server: packfile request") + defer r.Body.Close() + _, _ = io.Copy(io.Discard, r.Body) + w.WriteHeader(http.StatusAccepted) + case "/api/v2/logs": + // Consume CI Visibility logs (ignored during tests) to prevent backpressure. + debugMatrixf("subtest server: logs intake request") + defer r.Body.Close() + var reader io.Reader = r.Body + if r.Header.Get("Content-Encoding") == "gzip" { + gz, err := gzip.NewReader(r.Body) + if err == nil { + defer gz.Close() + reader = gz + } + } + _, _ = io.Copy(io.Discard, reader) + w.WriteHeader(http.StatusAccepted) + default: + http.NotFound(w, r) + } + }) + + server := httptest.NewServer(handler) + + snapshots := []envSnapshot{ + setEnv(constants.CIVisibilityEnabledEnvironmentVariable, "1"), + setEnv(constants.CIVisibilityAgentlessEnabledEnvironmentVariable, "1"), + setEnv(constants.CIVisibilityAgentlessURLEnvironmentVariable, server.URL), + setEnv(constants.APIKeyEnvironmentVariable, "test-api-key"), + setEnv(constants.CIVisibilityTestManagementAttemptToFixRetriesEnvironmentVariable, strconv.Itoa(cfg.attemptToFixRetries)), + } + + cleanup := func() { + for i := len(snapshots) - 1; i >= 0; i-- { + snapshots[i].restore() + } + server.Close() + } + + return server, cleanup +} diff --git a/internal/civisibility/integrations/gotesting/testing.go b/internal/civisibility/integrations/gotesting/testing.go index b7a6ccc29f..dc4f1cc4ed 100644 --- a/internal/civisibility/integrations/gotesting/testing.go +++ b/internal/civisibility/integrations/gotesting/testing.go @@ -25,6 +25,7 @@ import ( "github.com/DataDog/dd-trace-go/v2/internal/civisibility/utils" "github.com/DataDog/dd-trace-go/v2/internal/civisibility/utils/net" "github.com/DataDog/dd-trace-go/v2/internal/civisibility/utils/telemetry" + "github.com/DataDog/dd-trace-go/v2/internal/log" ) const ( @@ -65,11 +66,25 @@ var ( ) type ( + // testIdentity represents the fully-qualified identity of a Go test or subtest. + // It captures the module and suite where the test belongs, the base test name + // (top-level test), the full hierarchical name reported by Go (including subtests), + // and every individual path segment in order. This allows test management logic to + // resolve configuration at any depth while still falling back to parent segments. + testIdentity struct { + ModuleName string + SuiteName string + BaseName string + FullName string + Segments []string + } + // commonInfo holds common information about tests and benchmarks. commonInfo struct { moduleName string suiteName string testName string + identity *testIdentity } // testingTInfo holds information specific to tests. @@ -88,6 +103,33 @@ type ( M testing.M ) +// newTestIdentity builds a testIdentity instance for the provided module, suite, +// and fully-qualified Go test name. The base name corresponds to the first path +// segment (the top-level test declared via testing.T.Run). The Segments slice +// always contains at least one entry so consumers can traverse parent levels. +func newTestIdentity(moduleName, suiteName, fullName string) *testIdentity { + if fullName == "" { + fullName = "" + } + segments := strings.Split(fullName, "/") + baseName := segments[0] + return &testIdentity{ + ModuleName: moduleName, + SuiteName: suiteName, + BaseName: baseName, + FullName: fullName, + Segments: segments, + } +} + +type testManagementMatchKind int + +const ( + testManagementMatchNone testManagementMatchKind = iota + testManagementMatchExact + testManagementMatchAncestor +) + // Run initializes CI Visibility, instruments tests and benchmarks, and runs them. func (ddm *M) Run() int { m := (*testing.M)(ddm) @@ -140,12 +182,14 @@ func (ddm *M) instrumentInternalTests(internalTests *[]testing.InternalTest) { testInfos = make([]*testingTInfo, len(*internalTests)) for idx, test := range *internalTests { moduleName, suiteName := utils.GetModuleAndSuiteName(reflect.Indirect(reflect.ValueOf(test.F)).Pointer()) + identity := newTestIdentity(moduleName, suiteName, test.Name) testInfo := &testingTInfo{ originalFunc: test.F, commonInfo: commonInfo{ moduleName: moduleName, suiteName: suiteName, testName: test.Name, + identity: identity, }, } @@ -213,6 +257,7 @@ func (ddm *M) executeInternalTest(testInfo *testingTInfo) func(*testing.T) { execMeta = createTestMetadata(t, nil) defer deleteTestMetadata(t) } + execMeta.identity = testInfo.identity // Create or retrieve the module, suite, and test for CI visibility. module := session.GetOrCreateModule(testInfo.moduleName) @@ -374,7 +419,7 @@ func (ddm *M) executeInternalTest(testInfo *testingTInfo) func(*testing.T) { } // Get the additional feature wrapper - return applyAdditionalFeaturesToTestFunc(instrumentedFunc, &testInfo.commonInfo) + return applyAdditionalFeaturesToTestFunc(instrumentedFunc, &testInfo.commonInfo, nil) } // instrumentInternalBenchmarks instruments the internal benchmarks for CI visibility. @@ -387,12 +432,14 @@ func (ddm *M) instrumentInternalBenchmarks(internalBenchmarks *[]testing.Interna benchmarkInfos = make([]*testingBInfo, len(*internalBenchmarks)) for idx, benchmark := range *internalBenchmarks { moduleName, suiteName := utils.GetModuleAndSuiteName(reflect.Indirect(reflect.ValueOf(benchmark.F)).Pointer()) + identity := newTestIdentity(moduleName, suiteName, benchmark.Name) benchmarkInfo := &testingBInfo{ originalFunc: benchmark.F, commonInfo: commonInfo{ moduleName: moduleName, suiteName: suiteName, testName: benchmark.Name, + identity: identity, }, } @@ -618,23 +665,46 @@ func isKnownTest(testInfo *commonInfo) (isKnown bool, hasKnownData bool) { return false, false } -// getTestManagementData retrieves the test management data for a test -func getTestManagementData(testInfo *commonInfo) (data *net.TestManagementTestsResponseDataTestPropertiesAttributes, hasTestManagementData bool) { +// getTestManagementData retrieves the test management data for a test identity. +// It returns the matched properties, the type of match, and a flag indicating whether +// test-management data exists for the containing module/suite. +func getTestManagementData(identity *testIdentity) (data *net.TestManagementTestsResponseDataTestPropertiesAttributes, matchKind testManagementMatchKind, hasTestManagementData bool) { testManagementData := integrations.GetTestManagementTestsData() - if testManagementData != nil && len(testManagementData.Modules) > 0 { - // Check if the test is quarantined - if module, ok := testManagementData.Modules[testInfo.moduleName]; ok { - if suite, ok := module.Suites[testInfo.suiteName]; ok { - if test, ok := suite.Tests[testInfo.testName]; ok { - return &test.Properties, true - } + return matchTestManagementData(identity, testManagementData) +} + +// matchTestManagementData finds the best-matching test-management directive for a given identity within the provided dataset. +func matchTestManagementData(identity *testIdentity, modules *net.TestManagementTestsResponseDataModules) (data *net.TestManagementTestsResponseDataTestPropertiesAttributes, matchKind testManagementMatchKind, hasTestManagementData bool) { + if identity == nil || modules == nil || len(modules.Modules) == 0 { + return nil, testManagementMatchNone, false + } + + module, ok := modules.Modules[identity.ModuleName] + if !ok { + return nil, testManagementMatchNone, true + } + + suite, ok := module.Suites[identity.SuiteName] + if !ok { + return nil, testManagementMatchNone, true + } + + if len(suite.Tests) == 0 { + return nil, testManagementMatchNone, true + } + + for i := len(identity.Segments); i > 0; i-- { + candidate := strings.Join(identity.Segments[:i], "/") + if test, ok := suite.Tests[candidate]; ok { + kind := testManagementMatchExact + if candidate != identity.FullName { + kind = testManagementMatchAncestor } + return &test.Properties, kind, true } - - return nil, true } - return nil, false + return nil, testManagementMatchNone, true } // setTestTagsFromExecutionMetadata sets the test tags from the execution metadata. @@ -643,6 +713,9 @@ func setTestTagsFromExecutionMetadata(test integrations.Test, execMeta *testExec // Set the Test Optimization test to the execution metadata execMeta.test = test + if execMeta.identity != nil && len(execMeta.identity.Segments) > 1 { + log.Debug("setTestTagsFromExecutionMetadata assigned test for %s", execMeta.identity.FullName) + } // If the execution is for a new test we tag the test event as new if execMeta.isANewTest { diff --git a/internal/civisibility/utils/net/settings_api.go b/internal/civisibility/utils/net/settings_api.go index ee439a17dc..d8141699fd 100644 --- a/internal/civisibility/utils/net/settings_api.go +++ b/internal/civisibility/utils/net/settings_api.go @@ -68,6 +68,7 @@ type ( Enabled bool `json:"enabled"` AttemptToFixRetries int `json:"attempt_to_fix_retries"` } `json:"test_management"` + SubtestFeaturesEnabled bool `json:"-"` } ) diff --git a/internal/env/supported_configurations.gen.go b/internal/env/supported_configurations.gen.go index dc7edf5f87..c81a14fde3 100644 --- a/internal/env/supported_configurations.gen.go +++ b/internal/env/supported_configurations.gen.go @@ -42,6 +42,7 @@ var SupportedConfigurations = map[string]struct{}{ "DD_CIVISIBILITY_IMPACTED_TESTS_DETECTION_ENABLED": {}, "DD_CIVISIBILITY_INTERNAL_PARALLEL_EARLY_FLAKE_DETECTION_ENABLED": {}, "DD_CIVISIBILITY_LOGS_ENABLED": {}, + "DD_CIVISIBILITY_SUBTEST_FEATURES_ENABLED": {}, "DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT": {}, "DD_CUSTOM_TRACE_ID": {}, "DD_DATA_STREAMS_ENABLED": {}, diff --git a/internal/env/supported_configurations.json b/internal/env/supported_configurations.json index 857dd19f9b..5281253bee 100644 --- a/internal/env/supported_configurations.json +++ b/internal/env/supported_configurations.json @@ -99,6 +99,9 @@ "DD_CIVISIBILITY_LOGS_ENABLED": [ "A" ], + "DD_CIVISIBILITY_SUBTEST_FEATURES_ENABLED": [ + "A" + ], "DD_CIVISIBILITY_TOTAL_FLAKY_RETRY_COUNT": [ "A" ],