Skip to content

Increase Load benchmarks warmup time #9704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sarahchen6 merged 5 commits into from
Oct 27, 2025
Merged

Increase Load benchmarks warmup time #9704

sarahchen6 merged 5 commits into from
Oct 27, 2025

Conversation

@sarahchen6
Copy link
Contributor

@sarahchen6 sarahchen6 commented Oct 8, 2025 •
edited
Loading

What Does This Do

This PR increases our high_load benchmark warmup time period to 165s and reduces our petclinic repetitions to 2 in order to keep the total CI runtime roughly the same.

Motivation

After experimenting with @igoragoli, his detailed analyses (TYSM 🙏 ), and building off of #9567, this warm up time has shown to improve benchmark stability without drastically increasing CI time. The coefficient of variation for agg_http_req_duration_p50 drops from consistently >10% on master right now to 2-5% on this branch. Results in the BP UI for this branch.

Let's try this out and adjust again if needed!

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@datadog-official
Copy link

datadog-official bot commented Oct 8, 2025 •
edited by datadog-datadog-prod-us1 bot
Loading

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 59.67%

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1ab64de | Docs | Was this helpful? Give us feedback!

@pr-commenter
Copy link

pr-commenter bot commented Oct 8, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sarahchen6/adjust-load-benchmarks
git_commit_date 1761247499 1761254932
git_commit_sha c0b9147 c5e6ecc
release_version 1.55.0-SNAPSHOT~c0b9147e6b 1.55.0-SNAPSHOT~c5e6eccd00
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1761256945 1761256945
ci_job_id 1195179284 1195179284
ci_pipeline_id 80159149 80159149
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-abbrpynp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-abbrpynp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 metrics, 11 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.55.0-SNAPSHOT~c5e6eccd00, baseline=1.55.0-SNAPSHOT~c0b9147e6b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.031 s) : 0, 1030755
Total [baseline] (10.774 s) : 0, 10773577
Agent [candidate] (1.035 s) : 0, 1035121
Total [candidate] (10.842 s) : 0, 10842397
section appsec
Agent [baseline] (1.207 s) : 0, 1206811
Total [baseline] (10.984 s) : 0, 10983928
Agent [candidate] (1.204 s) : 0, 1204184
Total [candidate] (10.926 s) : 0, 10925967
section iast
Agent [baseline] (1.17 s) : 0, 1170171
Total [baseline] (11.112 s) : 0, 11112253
Agent [candidate] (1.159 s) : 0, 1159322
Total [candidate] (11.001 s) : 0, 11000625
section profiling
Agent [baseline] (1.179 s) : 0, 1178935
Total [baseline] (10.868 s) : 0, 10867680
Agent [candidate] (1.179 s) : 0, 1179295
Total [candidate] (10.934 s) : 0, 10933930
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.031 s -
Agent appsec 1.207 s 176.056 ms (17.1%)
Agent iast 1.17 s 139.416 ms (13.5%)
Agent profiling 1.179 s 148.18 ms (14.4%)
Total tracing 10.774 s -
Total appsec 10.984 s 210.352 ms (2.0%)
Total iast 11.112 s 338.676 ms (3.1%)
Total profiling 10.868 s 94.103 ms (0.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.035 s -
Agent appsec 1.204 s 169.062 ms (16.3%)
Agent iast 1.159 s 124.201 ms (12.0%)
Agent profiling 1.179 s 144.173 ms (13.9%)
Total tracing 10.842 s -
Total appsec 10.926 s 83.57 ms (0.8%)
Total iast 11.001 s 158.228 ms (1.5%)
Total profiling 10.934 s 91.533 ms (0.8%) 
gantt
    title petclinic - break down per module: candidate=1.55.0-SNAPSHOT~c5e6eccd00, baseline=1.55.0-SNAPSHOT~c0b9147e6b

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.497 ms) : 0, 1497
crashtracking [candidate] (1.465 ms) : 0, 1465
BytebuddyAgent [baseline] (703.089 ms) : 0, 703089
BytebuddyAgent [candidate] (705.688 ms) : 0, 705688
GlobalTracer [baseline] (244.54 ms) : 0, 244540
GlobalTracer [candidate] (245.881 ms) : 0, 245881
AppSec [baseline] (32.24 ms) : 0, 32240
AppSec [candidate] (32.524 ms) : 0, 32524
Debugger [baseline] (6.435 ms) : 0, 6435
Debugger [candidate] (6.481 ms) : 0, 6481
Remote Config [baseline] (686.95 µs) : 0, 687
Remote Config [candidate] (689.49 µs) : 0, 689
Telemetry [baseline] (14.523 ms) : 0, 14523
Telemetry [candidate] (14.541 ms) : 0, 14541
Flare Poller [baseline] (6.484 ms) : 0, 6484
Flare Poller [candidate] (6.613 ms) : 0, 6613
section appsec
crashtracking [baseline] (1.491 ms) : 0, 1491
crashtracking [candidate] (1.462 ms) : 0, 1462
BytebuddyAgent [baseline] (729.716 ms) : 0, 729716
BytebuddyAgent [candidate] (727.075 ms) : 0, 727075
GlobalTracer [baseline] (236.713 ms) : 0, 236713
GlobalTracer [candidate] (236.492 ms) : 0, 236492
AppSec [baseline] (173.817 ms) : 0, 173817
AppSec [candidate] (174.092 ms) : 0, 174092
Debugger [baseline] (5.903 ms) : 0, 5903
Debugger [candidate] (5.939 ms) : 0, 5939
Remote Config [baseline] (631.09 µs) : 0, 631
Remote Config [candidate] (628.716 µs) : 0, 629
Telemetry [baseline] (8.398 ms) : 0, 8398
Telemetry [candidate] (8.416 ms) : 0, 8416
Flare Poller [baseline] (3.942 ms) : 0, 3942
Flare Poller [candidate] (3.915 ms) : 0, 3915
IAST [baseline] (24.966 ms) : 0, 24966
IAST [candidate] (25.029 ms) : 0, 25029
section iast
crashtracking [baseline] (1.476 ms) : 0, 1476
crashtracking [candidate] (1.461 ms) : 0, 1461
BytebuddyAgent [baseline] (829.906 ms) : 0, 829906
BytebuddyAgent [candidate] (822.388 ms) : 0, 822388
GlobalTracer [baseline] (234.903 ms) : 0, 234903
GlobalTracer [candidate] (233.349 ms) : 0, 233349
AppSec [baseline] (32.019 ms) : 0, 32019
AppSec [candidate] (29.775 ms) : 0, 29775
Debugger [baseline] (6.265 ms) : 0, 6265
Debugger [candidate] (6.138 ms) : 0, 6138
Remote Config [baseline] (621.195 µs) : 0, 621
Remote Config [candidate] (595.9 µs) : 0, 596
Telemetry [baseline] (8.785 ms) : 0, 8785
Telemetry [candidate] (8.432 ms) : 0, 8432
Flare Poller [baseline] (4.197 ms) : 0, 4197
Flare Poller [candidate] (4.079 ms) : 0, 4079
IAST [baseline] (30.499 ms) : 0, 30499
IAST [candidate] (31.855 ms) : 0, 31855
section profiling
ProfilingAgent [baseline] (110.13 ms) : 0, 110130
ProfilingAgent [candidate] (109.957 ms) : 0, 109957
crashtracking [baseline] (1.455 ms) : 0, 1455
crashtracking [candidate] (1.482 ms) : 0, 1482
BytebuddyAgent [baseline] (730.438 ms) : 0, 730438
BytebuddyAgent [candidate] (730.907 ms) : 0, 730907
GlobalTracer [baseline] (221.234 ms) : 0, 221234
GlobalTracer [candidate] (221.428 ms) : 0, 221428
AppSec [baseline] (32.562 ms) : 0, 32562
AppSec [candidate] (33.132 ms) : 0, 33132
Debugger [baseline] (13.871 ms) : 0, 13871
Debugger [candidate] (12.795 ms) : 0, 12795
Remote Config [baseline] (706.601 µs) : 0, 707
Remote Config [candidate] (681.307 µs) : 0, 681
Telemetry [baseline] (8.998 ms) : 0, 8998
Telemetry [candidate] (8.977 ms) : 0, 8977
Flare Poller [baseline] (4.173 ms) : 0, 4173
Flare Poller [candidate] (4.166 ms) : 0, 4166
Profiling [baseline] (110.788 ms) : 0, 110788
Profiling [candidate] (110.58 ms) : 0, 110580
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.55.0-SNAPSHOT~c5e6eccd00, baseline=1.55.0-SNAPSHOT~c0b9147e6b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.039 s) : 0, 1039128
Total [baseline] (8.662 s) : 0, 8661887
Agent [candidate] (1.033 s) : 0, 1033001
Total [candidate] (8.672 s) : 0, 8671583
section iast
Agent [baseline] (1.17 s) : 0, 1169549
Total [baseline] (9.342 s) : 0, 9341986
Agent [candidate] (1.16 s) : 0, 1160280
Total [candidate] (9.322 s) : 0, 9322210
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.039 s -
Agent iast 1.17 s 130.421 ms (12.6%)
Total tracing 8.662 s -
Total iast 9.342 s 680.099 ms (7.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.033 s -
Agent iast 1.16 s 127.278 ms (12.3%)
Total tracing 8.672 s -
Total iast 9.322 s 650.627 ms (7.5%) 
gantt
    title insecure-bank - break down per module: candidate=1.55.0-SNAPSHOT~c5e6eccd00, baseline=1.55.0-SNAPSHOT~c0b9147e6b

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.48 ms) : 0, 1480
crashtracking [candidate] (1.502 ms) : 0, 1502
BytebuddyAgent [baseline] (709.097 ms) : 0, 709097
BytebuddyAgent [candidate] (704.233 ms) : 0, 704233
GlobalTracer [baseline] (246.176 ms) : 0, 246176
GlobalTracer [candidate] (245.571 ms) : 0, 245571
AppSec [baseline] (32.794 ms) : 0, 32794
AppSec [candidate] (32.449 ms) : 0, 32449
Debugger [baseline] (6.486 ms) : 0, 6486
Debugger [candidate] (6.447 ms) : 0, 6447
Remote Config [baseline] (683.052 µs) : 0, 683
Remote Config [candidate] (671.61 µs) : 0, 672
Telemetry [baseline] (12.214 ms) : 0, 12214
Telemetry [candidate] (14.441 ms) : 0, 14441
Flare Poller [baseline] (8.803 ms) : 0, 8803
Flare Poller [candidate] (6.469 ms) : 0, 6469
section iast
crashtracking [baseline] (1.481 ms) : 0, 1481
crashtracking [candidate] (1.459 ms) : 0, 1459
BytebuddyAgent [baseline] (830.136 ms) : 0, 830136
BytebuddyAgent [candidate] (822.455 ms) : 0, 822455
GlobalTracer [baseline] (235.058 ms) : 0, 235058
GlobalTracer [candidate] (233.926 ms) : 0, 233926
AppSec [baseline] (31.442 ms) : 0, 31442
AppSec [candidate] (29.894 ms) : 0, 29894
Debugger [baseline] (6.126 ms) : 0, 6126
Debugger [candidate] (6.21 ms) : 0, 6210
Remote Config [baseline] (600.803 µs) : 0, 601
Remote Config [candidate] (612.388 µs) : 0, 612
Telemetry [baseline] (8.58 ms) : 0, 8580
Telemetry [candidate] (8.535 ms) : 0, 8535
Flare Poller [baseline] (4.189 ms) : 0, 4189
Flare Poller [candidate] (4.182 ms) : 0, 4182
IAST [baseline] (30.399 ms) : 0, 30399
IAST [candidate] (31.761 ms) : 0, 31761
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sarahchen6/adjust-load-benchmarks
git_commit_date 1761247499 1761254932
git_commit_sha c0b9147 c5e6ecc
release_version 1.55.0-SNAPSHOT~c0b9147e6b 1.55.0-SNAPSHOT~c5e6eccd00
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1761257435 1761257435
ci_job_id 1195179285 1195179285
ci_pipeline_id 80159149 80159149
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-341yb63g 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-341yb63g 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 5 performance improvements and 1 performance regressions! Performance is the same for 6 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:insecure-bank:iast:high_load better
[-212.560µs; -122.806µs] or [-6.489%; -3.749%]		 unstable
[-68.878op/s; +216.253op/s] or [-4.930%; +15.479%]		 3.108ms 1470.750op/s 3.276ms 1397.062op/s
scenario:load:insecure-bank:profiling:high_load better
[-77.097µs; -42.616µs] or [-3.967%; -2.193%]		 unstable
[-180.633op/s; +323.508op/s] or [-7.790%; +13.951%]		 1.884ms 2390.250op/s 1.944ms 2318.812op/s
scenario:load:petclinic:tracing:high_load worse
[+0.832ms; +1.243ms] or [+4.530%; +6.768%]		 unstable
[-40.440op/s; +13.440op/s] or [-15.921%; +5.292%]		 19.408ms 240.500op/s 18.370ms 254.000op/s
scenario:load:petclinic:no_agent:high_load better
[-985.459µs; -601.652µs] or [-5.515%; -3.367%]		 unstable
[-18.714op/s; +42.839op/s] or [-7.166%; +16.404%]		 17.075ms 273.219op/s 17.868ms 261.156op/s
scenario:load:petclinic:profiling:high_load better
[-1163.356µs; -742.259µs] or [-5.769%; -3.681%]		 unstable
[-14.141op/s; +36.891op/s] or [-6.107%; +15.931%]		 19.214ms 242.938op/s 20.167ms 231.562op/s
scenario:load:petclinic:iast:high_load better
[-1138.450µs; -723.242µs] or [-5.865%; -3.726%]		 unstable
[-14.450op/s; +38.575op/s] or [-6.010%; +16.044%]		 18.481ms 252.500op/s 19.412ms 240.438op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~c5e6eccd00, baseline=1.55.0-SNAPSHOT~c0b9147e6b
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.2 ms) : 1188, 1212
.   : milestone, 1200,
iast (3.276 ms) : 3231, 3320
.   : milestone, 3276,
iast_FULL (5.79 ms) : 5733, 5848
.   : milestone, 5790,
iast_GLOBAL (3.604 ms) : 3553, 3655
.   : milestone, 3604,
profiling (1.944 ms) : 1928, 1960
.   : milestone, 1944,
tracing (1.779 ms) : 1764, 1793
.   : milestone, 1779,
section candidate
no_agent (1.209 ms) : 1197, 1221
.   : milestone, 1209,
iast (3.108 ms) : 3069, 3147
.   : milestone, 3108,
iast_FULL (5.833 ms) : 5774, 5891
.   : milestone, 5833,
iast_GLOBAL (3.616 ms) : 3561, 3671
.   : milestone, 3616,
profiling (1.884 ms) : 1868, 1900
.   : milestone, 1884,
tracing (1.791 ms) : 1775, 1806
.   : milestone, 1791,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.2 ms [1.188 ms, 1.212 ms] -
iast 3.276 ms [3.231 ms, 3.32 ms] 2.076 ms (173.0%)
iast_FULL 5.79 ms [5.733 ms, 5.848 ms] 4.591 ms (382.6%)
iast_GLOBAL 3.604 ms [3.553 ms, 3.655 ms] 2.404 ms (200.4%)
profiling 1.944 ms [1.928 ms, 1.96 ms] 743.801 µs (62.0%)
tracing 1.779 ms [1.764 ms, 1.793 ms] 578.974 µs (48.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.209 ms [1.197 ms, 1.221 ms] -
iast 3.108 ms [3.069 ms, 3.147 ms] 1.899 ms (157.1%)
iast_FULL 5.833 ms [5.774 ms, 5.891 ms] 4.624 ms (382.6%)
iast_GLOBAL 3.616 ms [3.561 ms, 3.671 ms] 2.407 ms (199.1%)
profiling 1.884 ms [1.868 ms, 1.9 ms] 675.028 µs (55.8%)
tracing 1.791 ms [1.775 ms, 1.806 ms] 581.957 µs (48.1%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~c5e6eccd00, baseline=1.55.0-SNAPSHOT~c0b9147e6b
    dateFormat X
    axisFormat %s
section baseline
no_agent (17.868 ms) : 17685, 18051
.   : milestone, 17868,
appsec (18.351 ms) : 18161, 18540
.   : milestone, 18351,
code_origins (18.375 ms) : 18189, 18562
.   : milestone, 18375,
iast (19.412 ms) : 19213, 19610
.   : milestone, 19412,
profiling (20.167 ms) : 19968, 20366
.   : milestone, 20167,
tracing (18.37 ms) : 18183, 18557
.   : milestone, 18370,
section candidate
no_agent (17.075 ms) : 16901, 17248
.   : milestone, 17075,
appsec (18.45 ms) : 18259, 18641
.   : milestone, 18450,
code_origins (18.593 ms) : 18406, 18779
.   : milestone, 18593,
iast (18.481 ms) : 18294, 18668
.   : milestone, 18481,
profiling (19.214 ms) : 19022, 19406
.   : milestone, 19214,
tracing (19.408 ms) : 19213, 19603
.   : milestone, 19408,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 17.868 ms [17.685 ms, 18.051 ms] -
appsec 18.351 ms [18.161 ms, 18.54 ms] 482.606 µs (2.7%)
code_origins 18.375 ms [18.189 ms, 18.562 ms] 507.411 µs (2.8%)
iast 19.412 ms [19.213 ms, 19.61 ms] 1.544 ms (8.6%)
profiling 20.167 ms [19.968 ms, 20.366 ms] 2.299 ms (12.9%)
tracing 18.37 ms [18.183 ms, 18.557 ms] 502.297 µs (2.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 17.075 ms [16.901 ms, 17.248 ms] -
appsec 18.45 ms [18.259 ms, 18.641 ms] 1.376 ms (8.1%)
code_origins 18.593 ms [18.406 ms, 18.779 ms] 1.518 ms (8.9%)
iast 18.481 ms [18.294 ms, 18.668 ms] 1.406 ms (8.2%)
profiling 19.214 ms [19.022 ms, 19.406 ms] 2.139 ms (12.5%)
tracing 19.408 ms [19.213 ms, 19.603 ms] 2.334 ms (13.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sarahchen6/adjust-load-benchmarks
git_commit_date 1761247499 1761254932
git_commit_sha c0b9147 c5e6ecc
release_version 1.55.0-SNAPSHOT~c0b9147e6b 1.55.0-SNAPSHOT~c5e6eccd00
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1761257129 1761257129
ci_job_id 1195179286 1195179286
ci_pipeline_id 80159149 80159149
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-e29b6u46 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-e29b6u46 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~c5e6eccd00, baseline=1.55.0-SNAPSHOT~c0b9147e6b
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (3.671 ms) : 3457, 3886
.   : milestone, 3671,
iast (2.205 ms) : 2141, 2269
.   : milestone, 2205,
iast_GLOBAL (2.255 ms) : 2190, 2319
.   : milestone, 2255,
profiling (2.06 ms) : 2008, 2112
.   : milestone, 2060,
tracing (2.026 ms) : 1977, 2076
.   : milestone, 2026,
section candidate
no_agent (1.475 ms) : 1463, 1486
.   : milestone, 1475,
appsec (3.748 ms) : 3529, 3966
.   : milestone, 3748,
iast (2.216 ms) : 2152, 2280
.   : milestone, 2216,
iast_GLOBAL (2.238 ms) : 2174, 2302
.   : milestone, 2238,
profiling (2.079 ms) : 2025, 2132
.   : milestone, 2079,
tracing (2.033 ms) : 1983, 2083
.   : milestone, 2033,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.476 ms [1.465 ms, 1.488 ms] -
appsec 3.671 ms [3.457 ms, 3.886 ms] 2.195 ms (148.7%)
iast 2.205 ms [2.141 ms, 2.269 ms] 729.341 µs (49.4%)
iast_GLOBAL 2.255 ms [2.19 ms, 2.319 ms] 778.548 µs (52.7%)
profiling 2.06 ms [2.008 ms, 2.112 ms] 583.585 µs (39.5%)
tracing 2.026 ms [1.977 ms, 2.076 ms] 550.174 µs (37.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.475 ms [1.463 ms, 1.486 ms] -
appsec 3.748 ms [3.529 ms, 3.966 ms] 2.273 ms (154.1%)
iast 2.216 ms [2.152 ms, 2.28 ms] 741.255 µs (50.3%)
iast_GLOBAL 2.238 ms [2.174 ms, 2.302 ms] 763.427 µs (51.8%)
profiling 2.079 ms [2.025 ms, 2.132 ms] 604.012 µs (41.0%)
tracing 2.033 ms [1.983 ms, 2.083 ms] 558.59 µs (37.9%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~c5e6eccd00, baseline=1.55.0-SNAPSHOT~c0b9147e6b
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.792 s) : 14792000, 14792000
.   : milestone, 14792000,
appsec (14.685 s) : 14685000, 14685000
.   : milestone, 14685000,
iast (18.46 s) : 18460000, 18460000
.   : milestone, 18460000,
iast_GLOBAL (17.862 s) : 17862000, 17862000
.   : milestone, 17862000,
profiling (15.788 s) : 15788000, 15788000
.   : milestone, 15788000,
tracing (15.082 s) : 15082000, 15082000
.   : milestone, 15082000,
section candidate
no_agent (15.483 s) : 15483000, 15483000
.   : milestone, 15483000,
appsec (15.082 s) : 15082000, 15082000
.   : milestone, 15082000,
iast (18.712 s) : 18712000, 18712000
.   : milestone, 18712000,
iast_GLOBAL (18.275 s) : 18275000, 18275000
.   : milestone, 18275000,
profiling (15.607 s) : 15607000, 15607000
.   : milestone, 15607000,
tracing (15.271 s) : 15271000, 15271000
.   : milestone, 15271000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.792 s [14.792 s, 14.792 s] -
appsec 14.685 s [14.685 s, 14.685 s] -107.0 ms (-0.7%)
iast 18.46 s [18.46 s, 18.46 s] 3.668 s (24.8%)
iast_GLOBAL 17.862 s [17.862 s, 17.862 s] 3.07 s (20.8%)
profiling 15.788 s [15.788 s, 15.788 s] 996.0 ms (6.7%)
tracing 15.082 s [15.082 s, 15.082 s] 290.0 ms (2.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.483 s [15.483 s, 15.483 s] -
appsec 15.082 s [15.082 s, 15.082 s] -401.0 ms (-2.6%)
iast 18.712 s [18.712 s, 18.712 s] 3.229 s (20.9%)
iast_GLOBAL 18.275 s [18.275 s, 18.275 s] 2.792 s (18.0%)
profiling 15.607 s [15.607 s, 15.607 s] 124.0 ms (0.8%)
tracing 15.271 s [15.271 s, 15.271 s] -212.0 ms (-1.4%)

@sarahchen6 sarahchen6 force-pushed the sarahchen6/adjust-load-benchmarks branch from dfeae3d to c5e6ecc Compare October 23, 2025 21:28
@sarahchen6 sarahchen6 changed the title Experiment with load benchmarks Increase high_load benchmarks warmup time Oct 24, 2025
@sarahchen6 sarahchen6 changed the title Increase high_load benchmarks warmup time Increase Load benchmarks warmup time Oct 24, 2025
@sarahchen6 sarahchen6 added type: enhancement Enhancements and improvements tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Oct 24, 2025
@sarahchen6 sarahchen6 marked this pull request as ready for review October 24, 2025 15:48
@sarahchen6 sarahchen6 requested a review from a team as a code owner October 24, 2025 15:48
@sarahchen6 sarahchen6 requested review from PerfectSlayer and igoragoli and removed request for a team October 24, 2025 15:48
PerfectSlayer
PerfectSlayer approved these changes Oct 27, 2025
View reviewed changes
Copy link
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s try it then!

igoragoli
igoragoli approved these changes Oct 27, 2025
View reviewed changes
Copy link
Contributor

@igoragoli igoragoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drops from consistently >10% on master right now to 2-5% on this branch

🤩

I looked at some other petclinic scenario results, most have a low CV (<10%) as well.

@sarahchen6 sarahchen6 merged commit a632fe7 into master Oct 27, 2025
552 checks passed
@sarahchen6 sarahchen6 deleted the sarahchen6/adjust-load-benchmarks branch October 27, 2025 20:25
@github-actions github-actions bot added this to the 1.55.0 milestone Oct 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

@igoragoli igoragoli igoragoli approved these changes

Assignees

No one assigned

Labels

comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes type: enhancement Enhancements and improvements

Projects

None yet

Milestone

1.55.0

Development

Successfully merging this pull request may close these issues.

4 participants

@sarahchen6 @PerfectSlayer @igoragoli