Refactor exploration tests by jpbempel · Pull Request #8150 · DataDog/dd-trace-java

jpbempel · 2025-01-03T16:36:08Z

What Does This Do

reuse the library built by previous job
make a job per repo to run exploration tests
upload results logs as artifacts
clone a specific and stable branch of jackson
enable jackson core and databind repos
use a sdkmanrc file for java configuration
run exploration tests automatically when changes in debugger

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

- reuse the library built by previous job - make a job per repo to run exploration tests - upload results logs as artifacts - clone a specific and stable branch of jackson - enable jackson core and databind repos - use a sdkmanrc file for java configuration - run exploration tests automatically when changes in debugger

pr-commenter · 2025-01-03T16:58:08Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1735926361	1735926751
end_time	2025-01-03T17:47:18	2025-01-03T17:53:47
git_branch	master	jpbempel/exploration-tests
git_commit_sha	`a8b33d5`	`33f7db4`
start_time	2025-01-03T17:46:02	2025-01-03T17:52:32

See matching parameters

	Baseline	Candidate
ci_job_id	754199464	754199464
ci_pipeline_id	52125884	52125884
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1735925764	1735925764

Summary

Found 5 performance improvements and 0 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:loop	better [-643.900µs; -638.209µs] or [-5.934%; -5.882%]	better [-680.018µs; -623.549µs] or [-6.167%; -5.655%]	better [-690.262µs; -584.555µs] or [-6.217%; -5.265%]	better [-821.301µs; -370.536µs] or [-7.158%; -3.229%]	better [+4.634op/s; +6.470op/s] or [+5.167%; +7.214%]

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-26.931µs; +12.040µs] or [-10.036%; +4.487%]	unstable [-42.770µs; +19.544µs] or [-13.784%; +6.299%]	unstable [-53.098µs; +30.108µs] or [-16.401%; +9.300%]	unstable [-106.298µs; +81.266µs] or [-17.734%; +13.558%]	unstable [-96.441op/s; +164.741op/s] or [-3.713%; +6.343%]
scenario:basic	same	same	same	unstable [-44.426µs; +32.302µs] or [-8.085%; +5.879%]	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (310.291 µs) : 276, 345
.   : milestone, 310,
basic (305.327 µs) : 295, 315
.   : milestone, 305,
loop (11.027 ms) : 11005, 11048
.   : milestone, 11027,
section candidate
noprobe (298.678 µs) : 275, 322
.   : milestone, 299,
basic (309.085 µs) : 301, 317
.   : milestone, 309,
loop (10.375 ms) : 10345, 10405
.   : milestone, 10375,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	310.291 µs [275.681 µs, 344.901 µs]
basic	305.327 µs [295.428 µs, 315.225 µs]
loop	11.027 ms [11.005 ms, 11.048 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	298.678 µs [274.875 µs, 322.481 µs]
basic	309.085 µs [301.01 µs, 317.16 µs]
loop	10.375 ms [10.345 ms, 10.405 ms]

evanchooly

i'm not a gitlab guru so those parts i'm not 100% on but look reasonable to me. the rest looks good. i love sdkman for stuff like this.

evanchooly · 2025-01-03T16:57:10Z

dd-java-agent/agent-debugger/exploration-tests/Dockerfile.exploration-tests

-    yes | sdk install maven $MAVEN_VERSION && \
+    sdk env install && \
    rm -rf $HOME/.sdkman/archives/* && \
    rm -rf $HOME/.sdkman/tmp/*"


sdk flush will do this a bit more reliably.

pr-commenter · 2025-01-03T17:19:35Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/exploration-tests
git_commit_date	1735897629	1735925764
git_commit_sha	`a8b33d5`	`33f7db4`
release_version	1.45.0-SNAPSHOT~a8b33d5c49	1.45.0-SNAPSHOT~33f7db4692

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1735928200	1735928200
ci_job_id	754199458	754199458
ci_pipeline_id	52125884	52125884
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 58 metrics, 4 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:petclinic:profiling:Remote Config	worse [+16.838µs; +67.127µs] or [+2.585%; +10.304%]	693.470µs	651.487µs

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~33f7db4692, baseline=1.45.0-SNAPSHOT~a8b33d5c49

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1051186
Total [baseline] (10.421 s) : 0, 10420652
Agent [candidate] (1.056 s) : 0, 1056488
Total [candidate] (10.443 s) : 0, 10443428
section appsec
Agent [baseline] (1.189 s) : 0, 1189093
Total [baseline] (10.732 s) : 0, 10731845
Agent [candidate] (1.185 s) : 0, 1185497
Total [candidate] (10.694 s) : 0, 10694458
section iast
Agent [baseline] (1.183 s) : 0, 1183221
Total [baseline] (11.037 s) : 0, 11037277
Agent [candidate] (1.179 s) : 0, 1179003
Total [candidate] (10.948 s) : 0, 10947886
section profiling
Agent [baseline] (1.274 s) : 0, 1274411
Total [baseline] (10.82 s) : 0, 10819759
Agent [candidate] (1.279 s) : 0, 1278977
Total [candidate] (10.782 s) : 0, 10782065

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	appsec	1.189 s	137.907 ms (13.1%)
Agent	iast	1.183 s	132.035 ms (12.6%)
Agent	profiling	1.274 s	223.225 ms (21.2%)
Total	tracing	10.421 s	-
Total	appsec	10.732 s	311.193 ms (3.0%)
Total	iast	11.037 s	616.625 ms (5.9%)
Total	profiling	10.82 s	399.107 ms (3.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	appsec	1.185 s	129.008 ms (12.2%)
Agent	iast	1.179 s	122.515 ms (11.6%)
Agent	profiling	1.279 s	222.488 ms (21.1%)
Total	tracing	10.443 s	-
Total	appsec	10.694 s	251.03 ms (2.4%)
Total	iast	10.948 s	504.459 ms (4.8%)
Total	profiling	10.782 s	338.637 ms (3.2%)

gantt
    title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~33f7db4692, baseline=1.45.0-SNAPSHOT~a8b33d5c49

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (711.071 ms) : 0, 711071
BytebuddyAgent [candidate] (716.192 ms) : 0, 716192
GlobalTracer [baseline] (255.695 ms) : 0, 255695
GlobalTracer [candidate] (257.913 ms) : 0, 257913
AppSec [baseline] (55.194 ms) : 0, 55194
AppSec [candidate] (55.309 ms) : 0, 55309
Remote Config [baseline] (718.015 µs) : 0, 718
Remote Config [candidate] (715.716 µs) : 0, 716
Telemetry [baseline] (13.446 ms) : 0, 13446
Telemetry [candidate] (11.264 ms) : 0, 11264
section appsec
BytebuddyAgent [baseline] (731.006 ms) : 0, 731006
BytebuddyAgent [candidate] (727.904 ms) : 0, 727904
GlobalTracer [baseline] (253.649 ms) : 0, 253649
GlobalTracer [candidate] (253.273 ms) : 0, 253273
AppSec [baseline] (171.0 ms) : 0, 171000
AppSec [candidate] (170.496 ms) : 0, 170496
Remote Config [baseline] (657.188 µs) : 0, 657
Remote Config [candidate] (668.443 µs) : 0, 668
Telemetry [baseline] (8.075 ms) : 0, 8075
Telemetry [candidate] (8.408 ms) : 0, 8408
IAST [baseline] (19.335 ms) : 0, 19335
IAST [candidate] (19.42 ms) : 0, 19420
section iast
BytebuddyAgent [baseline] (830.907 ms) : 0, 830907
BytebuddyAgent [candidate] (828.309 ms) : 0, 828309
GlobalTracer [baseline] (248.109 ms) : 0, 248109
GlobalTracer [candidate] (246.73 ms) : 0, 246730
AppSec [baseline] (58.184 ms) : 0, 58184
AppSec [candidate] (58.258 ms) : 0, 58258
Remote Config [baseline] (657.674 µs) : 0, 658
Remote Config [candidate] (655.926 µs) : 0, 656
Telemetry [baseline] (8.623 ms) : 0, 8623
Telemetry [candidate] (8.615 ms) : 0, 8615
IAST [baseline] (21.693 ms) : 0, 21693
IAST [candidate] (21.367 ms) : 0, 21367
section profiling
BytebuddyAgent [baseline] (701.169 ms) : 0, 701169
BytebuddyAgent [candidate] (704.726 ms) : 0, 704726
GlobalTracer [baseline] (373.05 ms) : 0, 373050
GlobalTracer [candidate] (373.39 ms) : 0, 373390
AppSec [baseline] (54.017 ms) : 0, 54017
AppSec [candidate] (54.413 ms) : 0, 54413
Remote Config [baseline] (651.487 µs) : 0, 651
Remote Config [candidate] (693.47 µs) : 0, 693
Telemetry [baseline] (7.811 ms) : 0, 7811
Telemetry [candidate] (7.816 ms) : 0, 7816
ProfilingAgent [baseline] (96.027 ms) : 0, 96027
ProfilingAgent [candidate] (96.225 ms) : 0, 96225
Profiling [baseline] (96.051 ms) : 0, 96051
Profiling [candidate] (96.248 ms) : 0, 96248

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~33f7db4692, baseline=1.45.0-SNAPSHOT~a8b33d5c49

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058015
Total [baseline] (8.615 s) : 0, 8614832
Agent [candidate] (1.051 s) : 0, 1050964
Total [candidate] (8.584 s) : 0, 8584228
section iast
Agent [baseline] (1.186 s) : 0, 1186074
Total [baseline] (9.195 s) : 0, 9195187
Agent [candidate] (1.179 s) : 0, 1179035
Total [candidate] (9.189 s) : 0, 9189122
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.179 s) : 0, 1178920
Total [baseline] (9.144 s) : 0, 9144490
Agent [candidate] (1.178 s) : 0, 1177527
Total [candidate] (9.163 s) : 0, 9162584
section iast_TELEMETRY_OFF
Agent [baseline] (1.176 s) : 0, 1175925
Total [baseline] (9.184 s) : 0, 9184378
Agent [candidate] (1.19 s) : 0, 1190106
Total [candidate] (9.286 s) : 0, 9285578

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	iast	1.186 s	128.06 ms (12.1%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.179 s	120.905 ms (11.4%)
Agent	iast_TELEMETRY_OFF	1.176 s	117.91 ms (11.1%)
Total	tracing	8.615 s	-
Total	iast	9.195 s	580.354 ms (6.7%)
Total	iast_HARDCODED_SECRET_DISABLED	9.144 s	529.657 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.184 s	569.546 ms (6.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	iast	1.179 s	128.072 ms (12.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.178 s	126.563 ms (12.0%)
Agent	iast_TELEMETRY_OFF	1.19 s	139.143 ms (13.2%)
Total	tracing	8.584 s	-
Total	iast	9.189 s	604.893 ms (7.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.163 s	578.356 ms (6.7%)
Total	iast_TELEMETRY_OFF	9.286 s	701.35 ms (8.2%)

gantt
    title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~33f7db4692, baseline=1.45.0-SNAPSHOT~a8b33d5c49

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (716.697 ms) : 0, 716697
BytebuddyAgent [candidate] (710.648 ms) : 0, 710648
GlobalTracer [baseline] (257.717 ms) : 0, 257717
GlobalTracer [candidate] (256.179 ms) : 0, 256179
AppSec [baseline] (55.538 ms) : 0, 55538
AppSec [candidate] (55.012 ms) : 0, 55012
Remote Config [baseline] (740.063 µs) : 0, 740
Remote Config [candidate] (716.844 µs) : 0, 717
Telemetry [baseline] (12.185 ms) : 0, 12185
Telemetry [candidate] (13.348 ms) : 0, 13348
section iast
BytebuddyAgent [baseline] (834.243 ms) : 0, 834243
BytebuddyAgent [candidate] (829.255 ms) : 0, 829255
GlobalTracer [baseline] (247.884 ms) : 0, 247884
GlobalTracer [candidate] (246.444 ms) : 0, 246444
AppSec [baseline] (58.031 ms) : 0, 58031
AppSec [candidate] (58.053 ms) : 0, 58053
Remote Config [baseline] (641.106 µs) : 0, 641
Remote Config [candidate] (645.124 µs) : 0, 645
Telemetry [baseline] (8.559 ms) : 0, 8559
Telemetry [candidate] (8.446 ms) : 0, 8446
IAST [baseline] (21.638 ms) : 0, 21638
IAST [candidate] (21.084 ms) : 0, 21084
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (828.506 ms) : 0, 828506
BytebuddyAgent [candidate] (826.938 ms) : 0, 826938
GlobalTracer [baseline] (246.739 ms) : 0, 246739
GlobalTracer [candidate] (246.694 ms) : 0, 246694
AppSec [baseline] (58.031 ms) : 0, 58031
AppSec [candidate] (58.349 ms) : 0, 58349
Remote Config [baseline] (656.697 µs) : 0, 657
Remote Config [candidate] (655.005 µs) : 0, 655
Telemetry [baseline] (8.614 ms) : 0, 8614
Telemetry [candidate] (8.615 ms) : 0, 8615
IAST [baseline] (21.325 ms) : 0, 21325
IAST [candidate] (21.334 ms) : 0, 21334
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (826.502 ms) : 0, 826502
BytebuddyAgent [candidate] (836.582 ms) : 0, 836582
GlobalTracer [baseline] (246.515 ms) : 0, 246515
GlobalTracer [candidate] (249.377 ms) : 0, 249377
AppSec [baseline] (57.976 ms) : 0, 57976
AppSec [candidate] (58.335 ms) : 0, 58335
Remote Config [baseline] (647.846 µs) : 0, 648
Remote Config [candidate] (657.424 µs) : 0, 657
Telemetry [baseline] (8.444 ms) : 0, 8444
Telemetry [candidate] (8.575 ms) : 0, 8575
IAST [baseline] (20.821 ms) : 0, 20821
IAST [candidate] (21.395 ms) : 0, 21395

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-03T17:45:00	2025-01-03T17:51:57
git_branch	master	jpbempel/exploration-tests
git_commit_date	1735897629	1735925764
git_commit_sha	`a8b33d5`	`33f7db4`
release_version	1.45.0-SNAPSHOT~a8b33d5c49	1.45.0-SNAPSHOT~33f7db4692
start_time	2025-01-03T17:44:46	2025-01-03T17:51:44

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1735927070	1735927070
ci_job_id	754199459	754199459
ci_pipeline_id	52125884	52125884
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~33f7db4692, baseline=1.45.0-SNAPSHOT~a8b33d5c49
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.357 ms) : 1338, 1377
.   : milestone, 1357,
appsec (1.726 ms) : 1702, 1750
.   : milestone, 1726,
appsec_no_iast (1.749 ms) : 1724, 1775
.   : milestone, 1749,
iast (1.503 ms) : 1480, 1525
.   : milestone, 1503,
profiling (1.524 ms) : 1501, 1546
.   : milestone, 1524,
tracing (1.489 ms) : 1465, 1513
.   : milestone, 1489,
section candidate
no_agent (1.371 ms) : 1351, 1391
.   : milestone, 1371,
appsec (1.722 ms) : 1698, 1745
.   : milestone, 1722,
appsec_no_iast (1.771 ms) : 1747, 1795
.   : milestone, 1771,
iast (1.5 ms) : 1477, 1523
.   : milestone, 1500,
profiling (1.537 ms) : 1512, 1562
.   : milestone, 1537,
tracing (1.492 ms) : 1468, 1516
.   : milestone, 1492,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.357 ms [1.338 ms, 1.377 ms]	-
appsec	1.726 ms [1.702 ms, 1.75 ms]	368.686 µs (27.2%)
appsec_no_iast	1.749 ms [1.724 ms, 1.775 ms]	392.274 µs (28.9%)
iast	1.503 ms [1.48 ms, 1.525 ms]	145.324 µs (10.7%)
profiling	1.524 ms [1.501 ms, 1.546 ms]	166.334 µs (12.3%)
tracing	1.489 ms [1.465 ms, 1.513 ms]	131.412 µs (9.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.371 ms [1.351 ms, 1.391 ms]	-
appsec	1.722 ms [1.698 ms, 1.745 ms]	351.123 µs (25.6%)
appsec_no_iast	1.771 ms [1.747 ms, 1.795 ms]	400.216 µs (29.2%)
iast	1.5 ms [1.477 ms, 1.523 ms]	129.273 µs (9.4%)
profiling	1.537 ms [1.512 ms, 1.562 ms]	166.417 µs (12.1%)
tracing	1.492 ms [1.468 ms, 1.516 ms]	121.211 µs (8.8%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~33f7db4692, baseline=1.45.0-SNAPSHOT~a8b33d5c49
    dateFormat X
    axisFormat %s
section baseline
no_agent (371.577 µs) : 352, 391
.   : milestone, 372,
iast (497.438 µs) : 475, 519
.   : milestone, 497,
iast_FULL (649.088 µs) : 627, 671
.   : milestone, 649,
iast_GLOBAL (523.362 µs) : 501, 546
.   : milestone, 523,
iast_HARDCODED_SECRET_DISABLED (495.298 µs) : 474, 517
.   : milestone, 495,
iast_INACTIVE (454.027 µs) : 433, 475
.   : milestone, 454,
iast_TELEMETRY_OFF (477.71 µs) : 456, 499
.   : milestone, 478,
tracing (446.606 µs) : 426, 467
.   : milestone, 447,
section candidate
no_agent (373.617 µs) : 354, 394
.   : milestone, 374,
iast (490.729 µs) : 469, 512
.   : milestone, 491,
iast_FULL (653.123 µs) : 631, 675
.   : milestone, 653,
iast_GLOBAL (521.722 µs) : 500, 544
.   : milestone, 522,
iast_HARDCODED_SECRET_DISABLED (487.881 µs) : 466, 509
.   : milestone, 488,
iast_INACTIVE (452.061 µs) : 431, 473
.   : milestone, 452,
iast_TELEMETRY_OFF (478.487 µs) : 457, 500
.   : milestone, 478,
tracing (458.29 µs) : 437, 480
.   : milestone, 458,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	371.577 µs [351.959 µs, 391.196 µs]	-
iast	497.438 µs [475.458 µs, 519.417 µs]	125.86 µs (33.9%)
iast_FULL	649.088 µs [626.852 µs, 671.324 µs]	277.511 µs (74.7%)
iast_GLOBAL	523.362 µs [501.077 µs, 545.647 µs]	151.784 µs (40.8%)
iast_HARDCODED_SECRET_DISABLED	495.298 µs [473.582 µs, 517.014 µs]	123.721 µs (33.3%)
iast_INACTIVE	454.027 µs [432.988 µs, 475.066 µs]	82.449 µs (22.2%)
iast_TELEMETRY_OFF	477.71 µs [456.198 µs, 499.222 µs]	106.132 µs (28.6%)
tracing	446.606 µs [425.935 µs, 467.278 µs]	75.029 µs (20.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	373.617 µs [353.619 µs, 393.616 µs]	-
iast	490.729 µs [469.159 µs, 512.298 µs]	117.111 µs (31.3%)
iast_FULL	653.123 µs [631.439 µs, 674.807 µs]	279.506 µs (74.8%)
iast_GLOBAL	521.722 µs [499.639 µs, 543.804 µs]	148.104 µs (39.6%)
iast_HARDCODED_SECRET_DISABLED	487.881 µs [466.436 µs, 509.327 µs]	114.264 µs (30.6%)
iast_INACTIVE	452.061 µs [430.821 µs, 473.3 µs]	78.444 µs (21.0%)
iast_TELEMETRY_OFF	478.487 µs [457.202 µs, 499.771 µs]	104.869 µs (28.1%)
tracing	458.29 µs [436.578 µs, 480.002 µs]	84.673 µs (22.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/exploration-tests
git_commit_date	1735897629	1735925764
git_commit_sha	`a8b33d5`	`33f7db4`
release_version	1.45.0-SNAPSHOT~a8b33d5c49	1.45.0-SNAPSHOT~33f7db4692

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1735927752	1735927752
ci_job_id	754199460	754199460
ci_pipeline_id	52125884	52125884
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~33f7db4692, baseline=1.45.0-SNAPSHOT~a8b33d5c49
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.499 s) : 15499000, 15499000
.   : milestone, 15499000,
appsec (15.103 s) : 15103000, 15103000
.   : milestone, 15103000,
iast (18.911 s) : 18911000, 18911000
.   : milestone, 18911000,
iast_GLOBAL (17.818 s) : 17818000, 17818000
.   : milestone, 17818000,
profiling (15.628 s) : 15628000, 15628000
.   : milestone, 15628000,
tracing (15.039 s) : 15039000, 15039000
.   : milestone, 15039000,
section candidate
no_agent (14.854 s) : 14854000, 14854000
.   : milestone, 14854000,
appsec (15.024 s) : 15024000, 15024000
.   : milestone, 15024000,
iast (19.021 s) : 19021000, 19021000
.   : milestone, 19021000,
iast_GLOBAL (18.129 s) : 18129000, 18129000
.   : milestone, 18129000,
profiling (15.599 s) : 15599000, 15599000
.   : milestone, 15599000,
tracing (15.133 s) : 15133000, 15133000
.   : milestone, 15133000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.499 s [15.499 s, 15.499 s]	-
appsec	15.103 s [15.103 s, 15.103 s]	-396.0 ms (-2.6%)
iast	18.911 s [18.911 s, 18.911 s]	3.412 s (22.0%)
iast_GLOBAL	17.818 s [17.818 s, 17.818 s]	2.319 s (15.0%)
profiling	15.628 s [15.628 s, 15.628 s]	129.0 ms (0.8%)
tracing	15.039 s [15.039 s, 15.039 s]	-460.0 ms (-3.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.854 s [14.854 s, 14.854 s]	-
appsec	15.024 s [15.024 s, 15.024 s]	170.0 ms (1.1%)
iast	19.021 s [19.021 s, 19.021 s]	4.167 s (28.1%)
iast_GLOBAL	18.129 s [18.129 s, 18.129 s]	3.275 s (22.0%)
profiling	15.599 s [15.599 s, 15.599 s]	745.0 ms (5.0%)
tracing	15.133 s [15.133 s, 15.133 s]	279.0 ms (1.9%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~33f7db4692, baseline=1.45.0-SNAPSHOT~a8b33d5c49
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (2.352 ms) : 2310, 2394
.   : milestone, 2352,
iast (2.099 ms) : 2046, 2153
.   : milestone, 2099,
iast_GLOBAL (2.155 ms) : 2102, 2209
.   : milestone, 2155,
profiling (1.976 ms) : 1933, 2020
.   : milestone, 1976,
tracing (1.941 ms) : 1900, 1982
.   : milestone, 1941,
section candidate
no_agent (1.475 ms) : 1464, 1486
.   : milestone, 1475,
appsec (2.362 ms) : 2320, 2403
.   : milestone, 2362,
iast (2.113 ms) : 2060, 2167
.   : milestone, 2113,
iast_GLOBAL (2.15 ms) : 2096, 2204
.   : milestone, 2150,
profiling (2.46 ms) : 2272, 2649
.   : milestone, 2460,
tracing (1.947 ms) : 1906, 1988
.   : milestone, 1947,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.463 ms, 1.486 ms]	-
appsec	2.352 ms [2.31 ms, 2.394 ms]	877.749 µs (59.5%)
iast	2.099 ms [2.046 ms, 2.153 ms]	624.742 µs (42.4%)
iast_GLOBAL	2.155 ms [2.102 ms, 2.209 ms]	680.864 µs (46.2%)
profiling	1.976 ms [1.933 ms, 2.02 ms]	502.045 µs (34.1%)
tracing	1.941 ms [1.9 ms, 1.982 ms]	466.588 µs (31.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.475 ms [1.464 ms, 1.486 ms]	-
appsec	2.362 ms [2.32 ms, 2.403 ms]	886.535 µs (60.1%)
iast	2.113 ms [2.06 ms, 2.167 ms]	638.381 µs (43.3%)
iast_GLOBAL	2.15 ms [2.096 ms, 2.204 ms]	675.19 µs (45.8%)
profiling	2.46 ms [2.272 ms, 2.649 ms]	985.406 µs (66.8%)
tracing	1.947 ms [1.906 ms, 1.988 ms]	471.912 µs (32.0%)

jpbempel added type: enhancement Enhancements and improvements comp: testing Testing tag: no release notes Changes to exclude from release notes comp: debugger Dynamic Instrumentation labels Jan 3, 2025

jpbempel requested review from a team as code owners January 3, 2025 16:36

jpbempel requested review from evanchooly and smola and removed request for a team January 3, 2025 16:36

evanchooly approved these changes Jan 3, 2025

View reviewed changes

use sdk flush

33f7db4

jpbempel added comp: debugger Dynamic Instrumentation and removed comp: debugger Dynamic Instrumentation labels Jan 3, 2025

jpbempel merged commit 886169b into master Jan 3, 2025
8 checks passed

jpbempel deleted the jpbempel/exploration-tests branch January 3, 2025 21:56

github-actions bot added this to the 1.45.0 milestone Jan 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor exploration tests#8150

Refactor exploration tests#8150
jpbempel merged 2 commits intomasterfrom
jpbempel/exploration-tests

jpbempel commented Jan 3, 2025 •

edited by atlassian bot

Loading

Uh oh!

pr-commenter bot commented Jan 3, 2025 •

edited

Loading

Uh oh!

evanchooly left a comment

Uh oh!

evanchooly Jan 3, 2025

Uh oh!

jpbempel Jan 3, 2025

Uh oh!

pr-commenter bot commented Jan 3, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jpbempel commented Jan 3, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Jan 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

evanchooly left a comment

Choose a reason for hiding this comment

Uh oh!

evanchooly Jan 3, 2025

Choose a reason for hiding this comment

Uh oh!

jpbempel Jan 3, 2025

Choose a reason for hiding this comment

Uh oh!

pr-commenter bot commented Jan 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jpbempel commented Jan 3, 2025 •

edited by atlassian bot

Loading

pr-commenter bot commented Jan 3, 2025 •

edited

Loading

pr-commenter bot commented Jan 3, 2025 •

edited

Loading