Refactors environment variables handling to make it testable. #9586
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AlexeyKuznetsov-DD
added
type: enhancement
AlexeyKuznetsov-DD
requested review from
a team,
PerfectSlayer,
bric3,
daniel-mohedano,
nikita-tkachenko-datadog,
sarahchen6 and
shatzi
and removed request for
a team and
PerfectSlayer
AlexeyKuznetsov-DD
changed the title
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 7 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.021 s) : 0, 1020606
Total [baseline] (10.728 s) : 0, 10728466
Agent [candidate] (1.019 s) : 0, 1018639
Total [candidate] (10.72 s) : 0, 10720487
section appsec
Agent [baseline] (1.2 s) : 0, 1200459
Total [baseline] (11.124 s) : 0, 11124409
Agent [candidate] (1.199 s) : 0, 1199012
Total [candidate] (11.029 s) : 0, 11029324
section iast
Agent [baseline] (1.158 s) : 0, 1158433
Total [baseline] (10.994 s) : 0, 10994376
Agent [candidate] (1.155 s) : 0, 1155275
Total [candidate] (11.003 s) : 0, 11002932
section profiling
Agent [baseline] (1.16 s) : 0, 1159735
Total [baseline] (11.063 s) : 0, 11062672
Agent [candidate] (1.181 s) : 0, 1181262
Total [candidate] (11.182 s) : 0, 11181872
gantt
title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.464 ms) : 0, 1464
crashtracking [candidate] (1.474 ms) : 0, 1474
BytebuddyAgent [baseline] (694.723 ms) : 0, 694723
BytebuddyAgent [candidate] (695.052 ms) : 0, 695052
GlobalTracer [baseline] (242.296 ms) : 0, 242296
GlobalTracer [candidate] (242.63 ms) : 0, 242630
AppSec [baseline] (32.516 ms) : 0, 32516
AppSec [candidate] (32.378 ms) : 0, 32378
Debugger [baseline] (6.406 ms) : 0, 6406
Debugger [candidate] (6.423 ms) : 0, 6423
Remote Config [baseline] (771.164 µs) : 0, 771
Remote Config [candidate] (713.206 µs) : 0, 713
Telemetry [baseline] (9.464 ms) : 0, 9464
Telemetry [candidate] (9.398 ms) : 0, 9398
Flare Poller [baseline] (11.7 ms) : 0, 11700
Flare Poller [candidate] (9.388 ms) : 0, 9388
section appsec
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.464 ms) : 0, 1464
BytebuddyAgent [baseline] (721.928 ms) : 0, 721928
BytebuddyAgent [candidate] (721.413 ms) : 0, 721413
GlobalTracer [baseline] (235.917 ms) : 0, 235917
GlobalTracer [candidate] (235.947 ms) : 0, 235947
IAST [baseline] (24.748 ms) : 0, 24748
IAST [candidate] (24.645 ms) : 0, 24645
AppSec [baseline] (175.999 ms) : 0, 175999
AppSec [candidate] (175.355 ms) : 0, 175355
Debugger [baseline] (6.127 ms) : 0, 6127
Debugger [candidate] (6.096 ms) : 0, 6096
Remote Config [baseline] (637.888 µs) : 0, 638
Remote Config [candidate] (626.732 µs) : 0, 627
Telemetry [baseline] (8.467 ms) : 0, 8467
Telemetry [candidate] (8.438 ms) : 0, 8438
Flare Poller [baseline] (3.976 ms) : 0, 3976
Flare Poller [candidate] (3.897 ms) : 0, 3897
section iast
crashtracking [baseline] (1.466 ms) : 0, 1466
crashtracking [candidate] (1.454 ms) : 0, 1454
BytebuddyAgent [baseline] (821.04 ms) : 0, 821040
BytebuddyAgent [candidate] (818.93 ms) : 0, 818930
GlobalTracer [baseline] (233.02 ms) : 0, 233020
GlobalTracer [candidate] (232.398 ms) : 0, 232398
IAST [baseline] (26.708 ms) : 0, 26708
IAST [candidate] (26.291 ms) : 0, 26291
AppSec [baseline] (35.134 ms) : 0, 35134
AppSec [candidate] (34.982 ms) : 0, 34982
Debugger [baseline] (6.179 ms) : 0, 6179
Debugger [candidate] (6.155 ms) : 0, 6155
Remote Config [baseline] (611.853 µs) : 0, 612
Remote Config [candidate] (621.797 µs) : 0, 622
Telemetry [baseline] (8.596 ms) : 0, 8596
Telemetry [candidate] (8.629 ms) : 0, 8629
Flare Poller [baseline] (4.219 ms) : 0, 4219
Flare Poller [candidate] (4.18 ms) : 0, 4180
section profiling
crashtracking [baseline] (1.437 ms) : 0, 1437
crashtracking [candidate] (1.473 ms) : 0, 1473
BytebuddyAgent [baseline] (720.528 ms) : 0, 720528
BytebuddyAgent [candidate] (732.558 ms) : 0, 732558
GlobalTracer [baseline] (217.125 ms) : 0, 217125
GlobalTracer [candidate] (221.051 ms) : 0, 221051
AppSec [baseline] (32.429 ms) : 0, 32429
AppSec [candidate] (32.792 ms) : 0, 32792
Debugger [baseline] (6.451 ms) : 0, 6451
Debugger [candidate] (8.237 ms) : 0, 8237
Remote Config [baseline] (874.507 µs) : 0, 875
Remote Config [candidate] (796.313 µs) : 0, 796
Telemetry [baseline] (16.116 ms) : 0, 16116
Telemetry [candidate] (14.714 ms) : 0, 14714
Flare Poller [baseline] (4.108 ms) : 0, 4108
Flare Poller [candidate] (4.282 ms) : 0, 4282
ProfilingAgent [baseline] (107.422 ms) : 0, 107422
ProfilingAgent [candidate] (111.339 ms) : 0, 111339
Profiling [baseline] (108.581 ms) : 0, 108581
Profiling [candidate] (112.49 ms) : 0, 112490
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.014 s) : 0, 1013786
Total [baseline] (8.701 s) : 0, 8700789
Agent [candidate] (1.022 s) : 0, 1022413
Total [candidate] (8.718 s) : 0, 8718341
section iast
Agent [baseline] (1.151 s) : 0, 1151068
Total [baseline] (9.348 s) : 0, 9347646
Agent [candidate] (1.151 s) : 0, 1150627
Total [candidate] (9.292 s) : 0, 9292093
gantt
title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.472 ms) : 0, 1472
BytebuddyAgent [baseline] (691.994 ms) : 0, 691994
BytebuddyAgent [candidate] (697.296 ms) : 0, 697296
GlobalTracer [baseline] (240.986 ms) : 0, 240986
GlobalTracer [candidate] (243.05 ms) : 0, 243050
AppSec [baseline] (32.231 ms) : 0, 32231
AppSec [candidate] (32.503 ms) : 0, 32503
Debugger [baseline] (6.379 ms) : 0, 6379
Debugger [candidate] (6.476 ms) : 0, 6476
Remote Config [baseline] (710.849 µs) : 0, 711
Remote Config [candidate] (715.085 µs) : 0, 715
Telemetry [baseline] (9.318 ms) : 0, 9318
Telemetry [candidate] (9.401 ms) : 0, 9401
Flare Poller [baseline] (9.552 ms) : 0, 9552
Flare Poller [candidate] (10.191 ms) : 0, 10191
section iast
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.487 ms) : 0, 1487
BytebuddyAgent [baseline] (815.646 ms) : 0, 815646
BytebuddyAgent [candidate] (815.198 ms) : 0, 815198
GlobalTracer [baseline] (231.707 ms) : 0, 231707
GlobalTracer [candidate] (231.252 ms) : 0, 231252
IAST [baseline] (26.3 ms) : 0, 26300
IAST [candidate] (26.352 ms) : 0, 26352
AppSec [baseline] (35.141 ms) : 0, 35141
AppSec [candidate] (35.294 ms) : 0, 35294
Debugger [baseline] (6.137 ms) : 0, 6137
Debugger [candidate] (6.14 ms) : 0, 6140
Remote Config [baseline] (612.473 µs) : 0, 612
Remote Config [candidate] (605.64 µs) : 0, 606
Telemetry [baseline] (8.624 ms) : 0, 8624
Telemetry [candidate] (8.606 ms) : 0, 8606
Flare Poller [baseline] (4.169 ms) : 0, 4169
Flare Poller [candidate] (4.238 ms) : 0, 4238
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 4 performance regressions! Performance is the same for 6 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
dateFormat X
axisFormat %s
section baseline
no_agent (38.265 ms) : 37964, 38565
. : milestone, 38265,
appsec (48.852 ms) : 48414, 49290
. : milestone, 48852,
code_origins (43.823 ms) : 43444, 44202
. : milestone, 43823,
iast (46.497 ms) : 46093, 46901
. : milestone, 46497,
profiling (50.143 ms) : 49641, 50645
. : milestone, 50143,
tracing (45.258 ms) : 44869, 45646
. : milestone, 45258,
section candidate
no_agent (35.27 ms) : 34986, 35554
. : milestone, 35270,
appsec (50.972 ms) : 50525, 51420
. : milestone, 50972,
code_origins (46.478 ms) : 46085, 46870
. : milestone, 46478,
iast (46.368 ms) : 45975, 46761
. : milestone, 46368,
profiling (48.335 ms) : 47897, 48772
. : milestone, 48335,
tracing (45.576 ms) : 45182, 45970
. : milestone, 45576,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
dateFormat X
axisFormat %s
section baseline
no_agent (4.385 ms) : 4335, 4434
. : milestone, 4385,
iast (9.307 ms) : 9144, 9470
. : milestone, 9307,
iast_FULL (13.82 ms) : 13548, 14093
. : milestone, 13820,
iast_GLOBAL (10.688 ms) : 10494, 10881
. : milestone, 10688,
profiling (8.472 ms) : 8340, 8604
. : milestone, 8472,
tracing (7.877 ms) : 7763, 7991
. : milestone, 7877,
section candidate
no_agent (4.327 ms) : 4273, 4382
. : milestone, 4327,
iast (9.59 ms) : 9413, 9767
. : milestone, 9590,
iast_FULL (14.706 ms) : 14415, 14998
. : milestone, 14706,
iast_GLOBAL (11.112 ms) : 10913, 11312
. : milestone, 11112,
profiling (8.927 ms) : 8771, 9083
. : milestone, 8927,
tracing (7.81 ms) : 7696, 7924
. : milestone, 7810,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
dateFormat X
axisFormat %s
section baseline
no_agent (14.901 s) : 14901000, 14901000
. : milestone, 14901000,
appsec (14.985 s) : 14985000, 14985000
. : milestone, 14985000,
iast (18.848 s) : 18848000, 18848000
. : milestone, 18848000,
iast_GLOBAL (17.822 s) : 17822000, 17822000
. : milestone, 17822000,
profiling (15.314 s) : 15314000, 15314000
. : milestone, 15314000,
tracing (15.108 s) : 15108000, 15108000
. : milestone, 15108000,
section candidate
no_agent (15.722 s) : 15722000, 15722000
. : milestone, 15722000,
appsec (15.057 s) : 15057000, 15057000
. : milestone, 15057000,
iast (18.64 s) : 18640000, 18640000
. : milestone, 18640000,
iast_GLOBAL (17.983 s) : 17983000, 17983000
. : milestone, 17983000,
profiling (15.261 s) : 15261000, 15261000
. : milestone, 15261000,
tracing (15.35 s) : 15350000, 15350000
. : milestone, 15350000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
dateFormat X
axisFormat %s
section baseline
no_agent (1.482 ms) : 1471, 1494
. : milestone, 1482,
appsec (3.683 ms) : 3469, 3897
. : milestone, 3683,
iast (2.227 ms) : 2164, 2291
. : milestone, 2227,
iast_GLOBAL (2.263 ms) : 2199, 2326
. : milestone, 2263,
profiling (2.051 ms) : 2000, 2102
. : milestone, 2051,
tracing (2.034 ms) : 1985, 2083
. : milestone, 2034,
section candidate
no_agent (1.48 ms) : 1469, 1492
. : milestone, 1480,
appsec (2.471 ms) : 2420, 2522
. : milestone, 2471,
iast (2.221 ms) : 2157, 2284
. : milestone, 2221,
iast_GLOBAL (2.265 ms) : 2201, 2329
. : milestone, 2265,
profiling (2.083 ms) : 2031, 2136
. : milestone, 2083,
tracing (2.022 ms) : 1973, 2072
. : milestone, 2022,
|
daniel-mohedano
approved these changes
Sep 23, 2025
Contributor
daniel-mohedano
left a comment
daniel-mohedano
left a comment
There was a problem hiding this comment.
LGTM, some small feedback regarding changes to CIVis tests
...t-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityInstrumentationTest.groovy
Outdated
Show resolved
Hide resolved
...bility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilitySmokeTest.groovy
Outdated
Show resolved
Hide resolved
...bility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilitySmokeTest.groovy
Outdated
Show resolved
Hide resolved
Contributor
Author
@PerfectSlayer That make sense, I will chat with @mhlidd and rework my PR. |
Contributor
Adding on, but Config Inversion will only restrict the usages of |
mhlidd
requested review from
manuel-alvarez-alvarez,
mtoffl01 and
smola
and removed request for
a team
mhlidd
force-pushed
the
alexeyk/refactored-env-vars-to-be-testable
branch
from
54f2e28 to
348f975
Compare
Debugger benchmarksParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics. See unchanged results
Request duration reports for reportsgantt
title reports - request duration [CI 0.99] : candidate=None, baseline=None
dateFormat X
axisFormat %s
section baseline
noprobe (326.107 µs) : 286, 366
. : milestone, 326,
basic (281.841 µs) : 275, 289
. : milestone, 282,
loop (8.968 ms) : 8964, 8972
. : milestone, 8968,
section candidate
noprobe (326.682 µs) : 299, 354
. : milestone, 327,
basic (279.222 µs) : 273, 285
. : milestone, 279,
loop (8.981 ms) : 8953, 9009
. : milestone, 8981,
|
|
🎯 Code Coverage 🔗 Commit SHA: 06b8d5d | Docs | Was this helpful? Give us feedback! |
...t-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityInstrumentationTest.groovy
Outdated
Show resolved
Hide resolved
Comment on lines
+181
to
+182
| env.set("DD_GIT_COMMIT_SHA", "sha1"); | ||
| env.set("DD_GIT_REPOSITORY_URL", "http://github.com"); |
Contributor
There was a problem hiding this comment.
🎯 suggestion: This is typically the kind of test that should be migrated to be tested using config mocked value, not environment variables.
Contributor
Author
|
@nikita-tkachenko-datadog, @PerfectSlayer, @bric3 - I've updated the code with your recommendations, please take a look |
bric3
approved these changes
Oct 7, 2025
| static Object checkStaticAssertions() { | ||
| assert System.getProperty("dd.trace.enabled") == null | ||
| assert System.getenv("DD_TRACE_ENABLED") == null | ||
| assert EnvironmentVariables.get("DD_TRACE_ENABLED") == null |
Contributor
There was a problem hiding this comment.
question: shouldn't we use ControllableEnvironmentVariables for all test instead ?
...aws-java/aws-java-lambda-handler-1.2/src/test/groovy/LambdaHandlerInstrumentationTest.groovy
Show resolved
Hide resolved
# Conflicts: # dd-java-agent/instrumentation/aws-java/aws-java-lambda-handler-1.2/build.gradle
bric3
approved these changes
Oct 10, 2025
AlexeyKuznetsov-DD
deleted the
alexeyk/refactored-env-vars-to-be-testable
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Refactors environment variables handling to make it testable.
Motivation
Relying directly on environment variables in tests is considered a bad practice.
@PerfectSlayer introduced a shared component,
EnvironmentVariables, which is already used across the codebase.This PR makes
EnvironmentVariablestestable by introducing a provider that allows configuring initial values for environment variables in tests.Additional Notes
In one case, I had to initialize environment variables in a static block, because the test expected instrumentation to be auto-enabled by a special variable before instrumentation was loaded.
If class loading order cannot be avoided, I added support for a special environment variable:
TEST_ENV_PROPAGATE_VARS— a comma-separated list of variables that should be propagated from the real environment into the test environment.This change also removed the need for the JUnit4
@RuleAPI and the legacysystem-ruleslibrary.