Implement Failed Test Replay #9214

daniel-mohedano · 2025-07-22T14:22:37Z

What Does This Do

Implements Test Optimization's Failed Test Replay using Live Debugger's Exception Replay. When the feature is enabled and a test is retried due to Auto Test Retries, Exception Replay's logic will create a probe for the exception thrown (in the case of the test probably an assertion error, but not limited to it). When the test is retried, the probe captures debugging information if the exception is encountered again, creating a snapshot of the variables. If the snapshot is captured, it is send as a log to Datadog. The following modifications were made to Exception Replay's original implementation:

Exception Replay is enabled if Failed Test Replay is enabled by the user.
- If the build system (Maven or Gradle) is instrumented, the property is propagated to the child process
- If running in headless mode, without build system instrumentation, Failed Test Replay is marked as active through Config. The enabling of Exception Replay now checks for either the property to be marked as enabled or Failed Test Replay being marked as active. This works due to CiVisibility's system being initialized before Live Debugger's.
DefaultExceptionDebugger was modified to support Failed Test Replay:
- Will not skip instrumenting errors if Failed Test Replay is marked as active, in order to avoid ignoring test assertion errors.
- If Failed Test Replay is active, the instrumentation will be applied synchronously. Failed test retries can be performed in rapid succession and the async approach to the instrumentation meant that most of the times the instrumentation was not performed before the next test failure.
Adds a product field to snapshots, populated with test_optimization if Failed Test Replay was marked as active. This allows us to have the option of not billing customers for logs generated by the product.
Removed Live Debugger's dependency on Remote Config being enabled for its configuration to be initialized.
Exception Replay now supports Agentless mode when Failed Test Replay is enabled. If DD_CIVISIBILITY_AGENTLESS_ENABLED is set, Live Debugger's logic for Exception Replay will use the logs API instead of the agent's.
If Failed Test Replay is enabled, a TestListener is registered to flush DebuggerSink on test suite end, to avoid unsent snapshots.

Additional changes:

Refactored BackendApiFactory.Intake to a standalone Intake, given that it is useful in order to compute agentless mode URLs.
Updated libraries capabilities to add failed_test_replay in test frameworks that support Auto Test Retries.
Other changes related to adding di_enabled to the Settings response and telemetry.

Validation:

MavenSmokeTest now has an additional test for Failed Test Replay, validating the feature when build system instrumentation is present.
Implemented JUnitConsoleSmokeTest to validate the feature in headless mode. This test should ensure that the ordering dependency between CiVisibility's system and Live Debugger's is always accounted for.

Motivation

Test Optimization wants to improve the support for Failed Test Replay, implementing it in additional languages apart from JS.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-2242

pr-commenter · 2025-07-23T14:41:30Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1755008672	1755009018
end_time	2025-08-12T14:25:53	2025-08-12T14:31:38
git_branch	master	daniel.mohedano/failed-test-replay
git_commit_sha	`ef2e9f0`	`ec9c54d`
start_time	2025-08-12T14:24:33	2025-08-12T14:30:19

See matching parameters

	Baseline	Candidate
ci_job_id	1076237414	1076237414
ci_pipeline_id	73422301	73422301
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1755007922	1755007922

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics.

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-34.065µs; +12.653µs] or [-11.993%; +4.455%]	unstable [-46.992µs; +20.376µs] or [-14.431%; +6.257%]	unstable [-60.348µs; +30.700µs] or [-17.690%; +8.999%]	unstable [-107.318µs; +97.110µs] or [-11.426%; +10.339%]	same
scenario:basic	same	same	same	unstable [-76.934µs; +174.582µs] or [-9.545%; +21.660%]	unstable [-184.594op/s; +184.594op/s] or [-7.015%; +7.015%]
scenario:loop	unsure [+3.330µs; +10.693µs] or [+0.038%; +0.121%]	same	same	same	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (325.633 µs) : 285, 367
.   : milestone, 326,
basic (279.407 µs) : 273, 286
.   : milestone, 279,
loop (8.966 ms) : 8961, 8971
.   : milestone, 8966,
section candidate
noprobe (312.325 µs) : 293, 332
.   : milestone, 312,
basic (282.331 µs) : 275, 289
.   : milestone, 282,
loop (8.968 ms) : 8964, 8973
.   : milestone, 8968,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	325.633 µs [284.618 µs, 366.648 µs]
basic	279.407 µs [272.973 µs, 285.841 µs]
loop	8.966 ms [8.961 ms, 8.971 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	312.325 µs [292.974 µs, 331.676 µs]
basic	282.331 µs [275.356 µs, 289.305 µs]
loop	8.968 ms [8.964 ms, 8.973 ms]

pr-commenter · 2025-07-23T14:55:48Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/failed-test-replay
git_commit_date	1755006834	1755007922
git_commit_sha	`ef2e9f0`	`ec9c54d`
release_version	1.53.0-SNAPSHOT~ef2e9f03e6	1.51.0-SNAPSHOT~ec9c54d63e

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1755009915	1755009915
ci_job_id	1076237407	1076237407
ci_pipeline_id	73422301	73422301
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-3blf4mjn 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-3blf4mjn 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 12 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~ec9c54d63e, baseline=1.53.0-SNAPSHOT~ef2e9f03e6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.047 s) : 0, 1047152
Total [baseline] (8.577 s) : 0, 8577299
Agent [candidate] (1.06 s) : 0, 1060331
Total [candidate] (8.673 s) : 0, 8672935
section iast
Agent [baseline] (1.175 s) : 0, 1175131
Total [baseline] (9.346 s) : 0, 9345578
Agent [candidate] (1.2 s) : 0, 1199871
Total [candidate] (9.406 s) : 0, 9406343

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	iast	1.175 s	127.98 ms (12.2%)
Total	tracing	8.577 s	-
Total	iast	9.346 s	768.28 ms (9.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.06 s	-
Agent	iast	1.2 s	139.54 ms (13.2%)
Total	tracing	8.673 s	-
Total	iast	9.406 s	733.408 ms (8.5%)

gantt
    title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~ec9c54d63e, baseline=1.53.0-SNAPSHOT~ef2e9f03e6

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.429 ms) : 0, 1429
crashtracking [candidate] (1.445 ms) : 0, 1445
BytebuddyAgent [baseline] (731.562 ms) : 0, 731562
BytebuddyAgent [candidate] (741.398 ms) : 0, 741398
GlobalTracer [baseline] (241.74 ms) : 0, 241740
GlobalTracer [candidate] (245.304 ms) : 0, 245304
AppSec [baseline] (29.896 ms) : 0, 29896
AppSec [candidate] (30.639 ms) : 0, 30639
Debugger [baseline] (6.044 ms) : 0, 6044
Debugger [candidate] (6.142 ms) : 0, 6142
Remote Config [baseline] (645.398 µs) : 0, 645
Remote Config [candidate] (670.308 µs) : 0, 670
Telemetry [baseline] (14.892 ms) : 0, 14892
Telemetry [candidate] (13.517 ms) : 0, 13517
section iast
crashtracking [baseline] (1.44 ms) : 0, 1440
crashtracking [candidate] (1.468 ms) : 0, 1468
BytebuddyAgent [baseline] (848.653 ms) : 0, 848653
BytebuddyAgent [candidate] (868.411 ms) : 0, 868411
GlobalTracer [baseline] (232.255 ms) : 0, 232255
GlobalTracer [candidate] (235.205 ms) : 0, 235205
IAST [baseline] (29.009 ms) : 0, 29009
IAST [candidate] (29.549 ms) : 0, 29549
AppSec [baseline] (27.412 ms) : 0, 27412
AppSec [candidate] (28.314 ms) : 0, 28314
Debugger [baseline] (6.58 ms) : 0, 6580
Debugger [candidate] (6.73 ms) : 0, 6730
Remote Config [baseline] (582.402 µs) : 0, 582
Remote Config [candidate] (600.692 µs) : 0, 601
Telemetry [baseline] (8.236 ms) : 0, 8236
Telemetry [candidate] (8.348 ms) : 0, 8348

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~ec9c54d63e, baseline=1.53.0-SNAPSHOT~ef2e9f03e6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.044 s) : 0, 1043706
Total [baseline] (10.725 s) : 0, 10724922
Agent [candidate] (1.054 s) : 0, 1053592
Total [candidate] (10.829 s) : 0, 10829419
section appsec
Agent [baseline] (1.239 s) : 0, 1238658
Total [baseline] (10.865 s) : 0, 10864799
Agent [candidate] (1.22 s) : 0, 1219781
Total [candidate] (10.819 s) : 0, 10819239
section iast
Agent [baseline] (1.176 s) : 0, 1175616
Total [baseline] (10.918 s) : 0, 10918061
Agent [candidate] (1.175 s) : 0, 1174540
Total [candidate] (11.019 s) : 0, 11019229
section profiling
Agent [baseline] (1.194 s) : 0, 1194498
Total [baseline] (10.883 s) : 0, 10882549
Agent [candidate] (1.202 s) : 0, 1201552
Total [candidate] (10.871 s) : 0, 10871450

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.044 s	-
Agent	appsec	1.239 s	194.952 ms (18.7%)
Agent	iast	1.176 s	131.91 ms (12.6%)
Agent	profiling	1.194 s	150.791 ms (14.4%)
Total	tracing	10.725 s	-
Total	appsec	10.865 s	139.877 ms (1.3%)
Total	iast	10.918 s	193.139 ms (1.8%)
Total	profiling	10.883 s	157.628 ms (1.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	appsec	1.22 s	166.189 ms (15.8%)
Agent	iast	1.175 s	120.948 ms (11.5%)
Agent	profiling	1.202 s	147.959 ms (14.0%)
Total	tracing	10.829 s	-
Total	appsec	10.819 s	-10.18 ms (-0.1%)
Total	iast	11.019 s	189.81 ms (1.8%)
Total	profiling	10.871 s	42.031 ms (0.4%)

gantt
    title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~ec9c54d63e, baseline=1.53.0-SNAPSHOT~ef2e9f03e6

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.419 ms) : 0, 1419
crashtracking [candidate] (1.439 ms) : 0, 1439
BytebuddyAgent [baseline] (729.332 ms) : 0, 729332
BytebuddyAgent [candidate] (737.346 ms) : 0, 737346
GlobalTracer [baseline] (241.455 ms) : 0, 241455
GlobalTracer [candidate] (243.689 ms) : 0, 243689
AppSec [baseline] (29.875 ms) : 0, 29875
AppSec [candidate] (30.359 ms) : 0, 30359
Debugger [baseline] (6.016 ms) : 0, 6016
Debugger [candidate] (6.077 ms) : 0, 6077
Remote Config [baseline] (654.301 µs) : 0, 654
Remote Config [candidate] (659.173 µs) : 0, 659
Telemetry [baseline] (14.013 ms) : 0, 14013
Telemetry [candidate] (12.872 ms) : 0, 12872
section appsec
crashtracking [baseline] (1.457 ms) : 0, 1457
crashtracking [candidate] (1.438 ms) : 0, 1438
BytebuddyAgent [baseline] (766.26 ms) : 0, 766260
BytebuddyAgent [candidate] (752.547 ms) : 0, 752547
GlobalTracer [baseline] (238.46 ms) : 0, 238460
GlobalTracer [candidate] (235.442 ms) : 0, 235442
AppSec [baseline] (170.047 ms) : 0, 170047
AppSec [candidate] (168.955 ms) : 0, 168955
Debugger [baseline] (7.321 ms) : 0, 7321
Debugger [candidate] (7.917 ms) : 0, 7917
Remote Config [baseline] (640.19 µs) : 0, 640
Remote Config [candidate] (624.134 µs) : 0, 624
Telemetry [baseline] (9.234 ms) : 0, 9234
Telemetry [candidate] (8.418 ms) : 0, 8418
IAST [baseline] (23.923 ms) : 0, 23923
IAST [candidate] (23.425 ms) : 0, 23425
section iast
crashtracking [baseline] (1.437 ms) : 0, 1437
crashtracking [candidate] (1.431 ms) : 0, 1431
BytebuddyAgent [baseline] (847.885 ms) : 0, 847885
BytebuddyAgent [candidate] (847.785 ms) : 0, 847785
GlobalTracer [baseline] (233.593 ms) : 0, 233593
GlobalTracer [candidate] (231.912 ms) : 0, 231912
AppSec [baseline] (25.964 ms) : 0, 25964
AppSec [candidate] (26.229 ms) : 0, 26229
Debugger [baseline] (7.514 ms) : 0, 7514
Debugger [candidate] (9.356 ms) : 0, 9356
Remote Config [baseline] (590.638 µs) : 0, 591
Remote Config [candidate] (595.248 µs) : 0, 595
Telemetry [baseline] (8.324 ms) : 0, 8324
Telemetry [candidate] (8.288 ms) : 0, 8288
IAST [baseline] (29.377 ms) : 0, 29377
IAST [candidate] (28.035 ms) : 0, 28035
section profiling
ProfilingAgent [baseline] (108.633 ms) : 0, 108633
ProfilingAgent [candidate] (109.5 ms) : 0, 109500
crashtracking [baseline] (1.4 ms) : 0, 1400
crashtracking [candidate] (1.418 ms) : 0, 1418
BytebuddyAgent [baseline] (761.228 ms) : 0, 761228
BytebuddyAgent [candidate] (765.633 ms) : 0, 765633
GlobalTracer [baseline] (221.662 ms) : 0, 221662
GlobalTracer [candidate] (222.614 ms) : 0, 222614
AppSec [baseline] (29.954 ms) : 0, 29954
AppSec [candidate] (30.252 ms) : 0, 30252
Debugger [baseline] (6.266 ms) : 0, 6266
Debugger [candidate] (6.345 ms) : 0, 6345
Remote Config [baseline] (704.526 µs) : 0, 705
Remote Config [candidate] (677.704 µs) : 0, 678
Telemetry [baseline] (15.106 ms) : 0, 15106
Telemetry [candidate] (15.276 ms) : 0, 15276
Profiling [baseline] (109.26 ms) : 0, 109260
Profiling [candidate] (110.151 ms) : 0, 110151

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/failed-test-replay
git_commit_date	1755006834	1755007922
git_commit_sha	`ef2e9f0`	`ec9c54d`
release_version	1.53.0-SNAPSHOT~ef2e9f03e6	1.51.0-SNAPSHOT~ec9c54d63e

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1755009586	1755009586
ci_job_id	1076237408	1076237408
ci_pipeline_id	73422301	73422301
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-wftuneha 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-wftuneha 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 12 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~ec9c54d63e, baseline=1.53.0-SNAPSHOT~ef2e9f03e6
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.367 ms) : 4310, 4424
.   : milestone, 4367,
iast (9.402 ms) : 9242, 9561
.   : milestone, 9402,
iast_FULL (13.776 ms) : 13508, 14044
.   : milestone, 13776,
iast_GLOBAL (10.578 ms) : 10390, 10765
.   : milestone, 10578,
profiling (8.793 ms) : 8649, 8936
.   : milestone, 8793,
tracing (7.504 ms) : 7392, 7617
.   : milestone, 7504,
section candidate
no_agent (4.487 ms) : 4435, 4539
.   : milestone, 4487,
iast (9.366 ms) : 9214, 9517
.   : milestone, 9366,
iast_FULL (13.623 ms) : 13355, 13891
.   : milestone, 13623,
iast_GLOBAL (10.523 ms) : 10335, 10712
.   : milestone, 10523,
profiling (8.932 ms) : 8772, 9092
.   : milestone, 8932,
tracing (7.659 ms) : 7549, 7769
.   : milestone, 7659,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.367 ms [4.31 ms, 4.424 ms]	-
iast	9.402 ms [9.242 ms, 9.561 ms]	5.035 ms (115.3%)
iast_FULL	13.776 ms [13.508 ms, 14.044 ms]	9.409 ms (215.5%)
iast_GLOBAL	10.578 ms [10.39 ms, 10.765 ms]	6.211 ms (142.2%)
profiling	8.793 ms [8.649 ms, 8.936 ms]	4.426 ms (101.4%)
tracing	7.504 ms [7.392 ms, 7.617 ms]	3.138 ms (71.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.487 ms [4.435 ms, 4.539 ms]	-
iast	9.366 ms [9.214 ms, 9.517 ms]	4.879 ms (108.7%)
iast_FULL	13.623 ms [13.355 ms, 13.891 ms]	9.136 ms (203.6%)
iast_GLOBAL	10.523 ms [10.335 ms, 10.712 ms]	6.036 ms (134.5%)
profiling	8.932 ms [8.772 ms, 9.092 ms]	4.445 ms (99.1%)
tracing	7.659 ms [7.549 ms, 7.769 ms]	3.172 ms (70.7%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~ec9c54d63e, baseline=1.53.0-SNAPSHOT~ef2e9f03e6
    dateFormat X
    axisFormat %s
section baseline
no_agent (37.058 ms) : 36759, 37358
.   : milestone, 37058,
appsec (46.84 ms) : 46423, 47257
.   : milestone, 46840,
code_origins (44.585 ms) : 44214, 44955
.   : milestone, 44585,
iast (44.276 ms) : 43875, 44678
.   : milestone, 44276,
profiling (49.829 ms) : 49347, 50310
.   : milestone, 49829,
tracing (44.88 ms) : 44491, 45268
.   : milestone, 44880,
section candidate
no_agent (36.233 ms) : 35946, 36519
.   : milestone, 36233,
appsec (47.49 ms) : 47064, 47916
.   : milestone, 47490,
code_origins (45.637 ms) : 45227, 46047
.   : milestone, 45637,
iast (45.529 ms) : 45122, 45935
.   : milestone, 45529,
profiling (49.068 ms) : 48618, 49518
.   : milestone, 49068,
tracing (43.769 ms) : 43375, 44164
.   : milestone, 43769,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.058 ms [36.759 ms, 37.358 ms]	-
appsec	46.84 ms [46.423 ms, 47.257 ms]	9.782 ms (26.4%)
code_origins	44.585 ms [44.214 ms, 44.955 ms]	7.526 ms (20.3%)
iast	44.276 ms [43.875 ms, 44.678 ms]	7.218 ms (19.5%)
profiling	49.829 ms [49.347 ms, 50.31 ms]	12.77 ms (34.5%)
tracing	44.88 ms [44.491 ms, 45.268 ms]	7.821 ms (21.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.233 ms [35.946 ms, 36.519 ms]	-
appsec	47.49 ms [47.064 ms, 47.916 ms]	11.257 ms (31.1%)
code_origins	45.637 ms [45.227 ms, 46.047 ms]	9.405 ms (26.0%)
iast	45.529 ms [45.122 ms, 45.935 ms]	9.296 ms (25.7%)
profiling	49.068 ms [48.618 ms, 49.518 ms]	12.835 ms (35.4%)
tracing	43.769 ms [43.375 ms, 44.164 ms]	7.537 ms (20.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/failed-test-replay
git_commit_date	1755006834	1755007922
git_commit_sha	`ef2e9f0`	`ec9c54d`
release_version	1.53.0-SNAPSHOT~ef2e9f03e6	1.51.0-SNAPSHOT~ec9c54d63e

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1755010120	1755010120
ci_job_id	1076237409	1076237409
ci_pipeline_id	73422301	73422301
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-qrb5d704 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-qrb5d704 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~ec9c54d63e, baseline=1.53.0-SNAPSHOT~ef2e9f03e6
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.293 s) : 15293000, 15293000
.   : milestone, 15293000,
appsec (14.81 s) : 14810000, 14810000
.   : milestone, 14810000,
iast (18.301 s) : 18301000, 18301000
.   : milestone, 18301000,
iast_GLOBAL (18.135 s) : 18135000, 18135000
.   : milestone, 18135000,
profiling (15.903 s) : 15903000, 15903000
.   : milestone, 15903000,
tracing (15.119 s) : 15119000, 15119000
.   : milestone, 15119000,
section candidate
no_agent (15.052 s) : 15052000, 15052000
.   : milestone, 15052000,
appsec (15.037 s) : 15037000, 15037000
.   : milestone, 15037000,
iast (18.974 s) : 18974000, 18974000
.   : milestone, 18974000,
iast_GLOBAL (18.125 s) : 18125000, 18125000
.   : milestone, 18125000,
profiling (15.826 s) : 15826000, 15826000
.   : milestone, 15826000,
tracing (14.886 s) : 14886000, 14886000
.   : milestone, 14886000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.293 s [15.293 s, 15.293 s]	-
appsec	14.81 s [14.81 s, 14.81 s]	-483.0 ms (-3.2%)
iast	18.301 s [18.301 s, 18.301 s]	3.008 s (19.7%)
iast_GLOBAL	18.135 s [18.135 s, 18.135 s]	2.842 s (18.6%)
profiling	15.903 s [15.903 s, 15.903 s]	610.0 ms (4.0%)
tracing	15.119 s [15.119 s, 15.119 s]	-174.0 ms (-1.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.052 s [15.052 s, 15.052 s]	-
appsec	15.037 s [15.037 s, 15.037 s]	-15.0 ms (-0.1%)
iast	18.974 s [18.974 s, 18.974 s]	3.922 s (26.1%)
iast_GLOBAL	18.125 s [18.125 s, 18.125 s]	3.073 s (20.4%)
profiling	15.826 s [15.826 s, 15.826 s]	774.0 ms (5.1%)
tracing	14.886 s [14.886 s, 14.886 s]	-166.0 ms (-1.1%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~ec9c54d63e, baseline=1.53.0-SNAPSHOT~ef2e9f03e6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.489 ms) : 1477, 1501
.   : milestone, 1489,
appsec (2.483 ms) : 2429, 2536
.   : milestone, 2483,
iast (2.216 ms) : 2153, 2279
.   : milestone, 2216,
iast_GLOBAL (2.254 ms) : 2191, 2317
.   : milestone, 2254,
profiling (2.076 ms) : 2024, 2128
.   : milestone, 2076,
tracing (2.022 ms) : 1973, 2070
.   : milestone, 2022,
section candidate
no_agent (1.482 ms) : 1471, 1494
.   : milestone, 1482,
appsec (2.428 ms) : 2378, 2478
.   : milestone, 2428,
iast (2.219 ms) : 2156, 2282
.   : milestone, 2219,
iast_GLOBAL (2.262 ms) : 2199, 2325
.   : milestone, 2262,
profiling (2.045 ms) : 1995, 2095
.   : milestone, 2045,
tracing (2.032 ms) : 1983, 2081
.   : milestone, 2032,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.489 ms [1.477 ms, 1.501 ms]	-
appsec	2.483 ms [2.429 ms, 2.536 ms]	993.828 µs (66.7%)
iast	2.216 ms [2.153 ms, 2.279 ms]	727.383 µs (48.8%)
iast_GLOBAL	2.254 ms [2.191 ms, 2.317 ms]	765.171 µs (51.4%)
profiling	2.076 ms [2.024 ms, 2.128 ms]	587.086 µs (39.4%)
tracing	2.022 ms [1.973 ms, 2.07 ms]	532.719 µs (35.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.482 ms [1.471 ms, 1.494 ms]	-
appsec	2.428 ms [2.378 ms, 2.478 ms]	945.825 µs (63.8%)
iast	2.219 ms [2.156 ms, 2.282 ms]	737.217 µs (49.7%)
iast_GLOBAL	2.262 ms [2.199 ms, 2.325 ms]	779.813 µs (52.6%)
profiling	2.045 ms [1.995 ms, 2.095 ms]	563.05 µs (38.0%)
tracing	2.032 ms [1.983 ms, 2.081 ms]	549.75 µs (37.1%)

datadog-official · 2025-08-11T09:23:09Z

Code coverage: total 57.25%, patch 45.45% (view details)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ec9c54d | Docs | Was this helpful? Give us feedback!}

daniel-mohedano added 4 commits July 4, 2025 12:22

add di_enabled to settings response

2c219fd

add FTR related metrics

a911ae0

add FTR to execution settings

d20159e

add basic exception replay integration in agent mode

c4c84d6

daniel-mohedano added type: enhancement Enhancements and improvements tag: do not merge Do not merge changes comp: ci visibility Continuous Integration Visibility comp: debugger Dynamic Instrumentation labels Jul 22, 2025

daniel-mohedano added 3 commits July 22, 2025 16:27

feat: headless and agentless changes

ae31670

Merge branch 'master' into daniel.mohedano/failed-test-replay

4f2d6a9

fix: tests

c7eeac8

daniel-mohedano added 8 commits August 4, 2025 10:21

fix: testng capabilities

14cf11c

feat: refactor agentless intakes

461fb1f

Merge branch 'master' into daniel.mohedano/failed-test-replay

14b1054

chore: update smoke test fixtures

a87eff0

test: add unit test for new Intake enum

19b4edf

test: remove ftr from instrumentation tests (not used)

9e81cc7

test: introduce FTR smoke tests for headfull and headless modes

4e3deb4

style: spotless and codenarc

f931654

daniel-mohedano added 3 commits August 11, 2025 12:31

feat: add test event finished FTR telemetry

eacde60

feat: add product field to snapshots

0228f71

feat: implement SuiteEnd listener for sink flushing

88fd665

daniel-mohedano changed the title ~~Failed Test Replay~~ Implement Failed Test Replay Aug 12, 2025

feat: introduce new config variables for debugger

ec9c54d

daniel-mohedano removed the tag: do not merge Do not merge changes label Aug 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement Failed Test Replay #9214

Implement Failed Test Replay #9214

Uh oh!

daniel-mohedano commented Jul 22, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jul 23, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jul 23, 2025 •

edited

Loading

Uh oh!

datadog-official bot commented Aug 11, 2025 •

edited by datadog-datadog-prod-us1 bot

Loading

Uh oh!

Uh oh!

Implement Failed Test Replay #9214

Are you sure you want to change the base?

Implement Failed Test Replay #9214

Uh oh!

Conversation

daniel-mohedano commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Contributor Checklist

Uh oh!

pr-commenter bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

pr-commenter bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

datadog-official bot commented Aug 11, 2025 • edited by datadog-datadog-prod-us1 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

daniel-mohedano commented Jul 22, 2025 •

edited

Loading

pr-commenter bot commented Jul 23, 2025 •

edited

Loading

pr-commenter bot commented Jul 23, 2025 •

edited

Loading

datadog-official bot commented Aug 11, 2025 •

edited by datadog-datadog-prod-us1 bot

Loading