Benchmark code changes from 1.49.0 to 1.50.0 #8968
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 3 performance improvements and 2 performance regressions! Performance is the same for 56 metrics, 10 unstable metrics.
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.029 s) : 0, 1028997
Total [baseline] (10.651 s) : 0, 10651010
Agent [candidate] (1.015 s) : 0, 1015254
Total [candidate] (10.578 s) : 0, 10577847
section appsec
Agent [baseline] (1.189 s) : 0, 1188907
Total [baseline] (10.696 s) : 0, 10695723
Agent [candidate] (1.167 s) : 0, 1166686
Total [candidate] (10.751 s) : 0, 10751085
section iast
Agent [baseline] (1.165 s) : 0, 1164863
Total [baseline] (10.878 s) : 0, 10878156
Agent [candidate] (1.143 s) : 0, 1143353
Total [candidate] (10.781 s) : 0, 10780534
section profiling
Agent [baseline] (1.285 s) : 0, 1285458
Total [baseline] (10.979 s) : 0, 10979367
Agent [candidate] (1.278 s) : 0, 1278004
Total [candidate] (10.877 s) : 0, 10877169
gantt
title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.484 ms) : 0, 684484
BytebuddyAgent [candidate] (672.58 ms) : 0, 672580
GlobalTracer [baseline] (242.631 ms) : 0, 242631
GlobalTracer [candidate] (240.942 ms) : 0, 240942
AppSec [baseline] (58.954 ms) : 0, 58954
AppSec [candidate] (57.798 ms) : 0, 57798
Debugger [baseline] (6.278 ms) : 0, 6278
Debugger [candidate] (6.242 ms) : 0, 6242
Remote Config [baseline] (760.716 µs) : 0, 761
Remote Config [candidate] (715.058 µs) : 0, 715
Telemetry [baseline] (12.299 ms) : 0, 12299
Telemetry [candidate] (13.547 ms) : 0, 13547
section appsec
BytebuddyAgent [baseline] (712.545 ms) : 0, 712545
BytebuddyAgent [candidate] (694.464 ms) : 0, 694464
GlobalTracer [baseline] (236.861 ms) : 0, 236861
GlobalTracer [candidate] (234.015 ms) : 0, 234015
IAST [baseline] (21.96 ms) : 0, 21960
IAST [candidate] (21.811 ms) : 0, 21811
AppSec [baseline] (179.916 ms) : 0, 179916
AppSec [candidate] (179.207 ms) : 0, 179207
Debugger [baseline] (5.919 ms) : 0, 5919
Debugger [candidate] (5.832 ms) : 0, 5832
Remote Config [baseline] (616.202 µs) : 0, 616
Remote Config [candidate] (612.406 µs) : 0, 612
Telemetry [baseline] (7.372 ms) : 0, 7372
Telemetry [candidate] (7.391 ms) : 0, 7391
section iast
BytebuddyAgent [baseline] (811.048 ms) : 0, 811048
BytebuddyAgent [candidate] (790.821 ms) : 0, 790821
GlobalTracer [baseline] (233.095 ms) : 0, 233095
GlobalTracer [candidate] (231.728 ms) : 0, 231728
IAST [baseline] (27.146 ms) : 0, 27146
IAST [candidate] (22.994 ms) : 0, 22994
AppSec [baseline] (55.217 ms) : 0, 55217
AppSec [candidate] (59.673 ms) : 0, 59673
Debugger [baseline] (6.065 ms) : 0, 6065
Debugger [candidate] (6.068 ms) : 0, 6068
Remote Config [baseline] (610.222 µs) : 0, 610
Remote Config [candidate] (608.828 µs) : 0, 609
Telemetry [baseline] (7.973 ms) : 0, 7973
Telemetry [candidate] (8.01 ms) : 0, 8010
section profiling
BytebuddyAgent [baseline] (681.922 ms) : 0, 681922
BytebuddyAgent [candidate] (669.237 ms) : 0, 669237
GlobalTracer [baseline] (363.12 ms) : 0, 363120
GlobalTracer [candidate] (382.963 ms) : 0, 382963
AppSec [baseline] (65.276 ms) : 0, 65276
AppSec [candidate] (56.635 ms) : 0, 56635
Debugger [baseline] (6.261 ms) : 0, 6261
Debugger [candidate] (6.21 ms) : 0, 6210
Remote Config [baseline] (671.756 µs) : 0, 672
Remote Config [candidate] (657.122 µs) : 0, 657
Telemetry [baseline] (8.334 ms) : 0, 8334
Telemetry [candidate] (8.184 ms) : 0, 8184
ProfilingAgent [baseline] (108.653 ms) : 0, 108653
ProfilingAgent [candidate] (103.554 ms) : 0, 103554
Profiling [baseline] (108.679 ms) : 0, 108679
Profiling [candidate] (103.58 ms) : 0, 103580
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.038 s) : 0, 1037677
Total [baseline] (8.57 s) : 0, 8570036
Agent [candidate] (1.011 s) : 0, 1010663
Total [candidate] (8.548 s) : 0, 8547576
section iast
Agent [baseline] (1.156 s) : 0, 1156041
Total [baseline] (9.215 s) : 0, 9214706
Agent [candidate] (1.152 s) : 0, 1152038
Total [candidate] (9.266 s) : 0, 9266399
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.155 s) : 0, 1154719
Total [baseline] (9.172 s) : 0, 9171820
Agent [candidate] (1.148 s) : 0, 1147510
Total [candidate] (9.213 s) : 0, 9212678
section iast_TELEMETRY_OFF
Agent [baseline] (1.161 s) : 0, 1160802
Total [baseline] (9.236 s) : 0, 9235622
Agent [candidate] (1.142 s) : 0, 1141686
Total [candidate] (9.228 s) : 0, 9227587
gantt
title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (690.71 ms) : 0, 690710
BytebuddyAgent [candidate] (671.164 ms) : 0, 671164
GlobalTracer [baseline] (243.185 ms) : 0, 243185
GlobalTracer [candidate] (240.424 ms) : 0, 240424
AppSec [baseline] (60.286 ms) : 0, 60286
AppSec [candidate] (57.538 ms) : 0, 57538
Debugger [baseline] (7.779 ms) : 0, 7779
Debugger [candidate] (6.167 ms) : 0, 6167
Remote Config [baseline] (753.161 µs) : 0, 753
Remote Config [candidate] (709.745 µs) : 0, 710
Telemetry [baseline] (11.263 ms) : 0, 11263
Telemetry [candidate] (11.223 ms) : 0, 11223
section iast
BytebuddyAgent [baseline] (804.508 ms) : 0, 804508
BytebuddyAgent [candidate] (798.536 ms) : 0, 798536
GlobalTracer [baseline] (231.651 ms) : 0, 231651
GlobalTracer [candidate] (232.429 ms) : 0, 232429
IAST [baseline] (25.319 ms) : 0, 25319
IAST [candidate] (23.085 ms) : 0, 23085
AppSec [baseline] (56.498 ms) : 0, 56498
AppSec [candidate] (59.795 ms) : 0, 59795
Debugger [baseline] (6.031 ms) : 0, 6031
Debugger [candidate] (6.014 ms) : 0, 6014
Remote Config [baseline] (596.455 µs) : 0, 596
Remote Config [candidate] (606.608 µs) : 0, 607
Telemetry [baseline] (7.903 ms) : 0, 7903
Telemetry [candidate] (7.986 ms) : 0, 7986
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (804.316 ms) : 0, 804316
BytebuddyAgent [candidate] (793.845 ms) : 0, 793845
GlobalTracer [baseline] (231.386 ms) : 0, 231386
GlobalTracer [candidate] (232.598 ms) : 0, 232598
IAST [baseline] (26.033 ms) : 0, 26033
IAST [candidate] (23.201 ms) : 0, 23201
AppSec [baseline] (55.135 ms) : 0, 55135
AppSec [candidate] (59.8 ms) : 0, 59800
Debugger [baseline] (5.924 ms) : 0, 5924
Debugger [candidate] (6.078 ms) : 0, 6078
Remote Config [baseline] (581.986 µs) : 0, 582
Remote Config [candidate] (591.634 µs) : 0, 592
Telemetry [baseline] (7.861 ms) : 0, 7861
Telemetry [candidate] (7.855 ms) : 0, 7855
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (808.05 ms) : 0, 808050
BytebuddyAgent [candidate] (790.144 ms) : 0, 790144
GlobalTracer [baseline] (232.921 ms) : 0, 232921
GlobalTracer [candidate] (231.696 ms) : 0, 231696
IAST [baseline] (29.136 ms) : 0, 29136
IAST [candidate] (22.593 ms) : 0, 22593
AppSec [baseline] (52.598 ms) : 0, 52598
AppSec [candidate] (59.395 ms) : 0, 59395
Debugger [baseline] (6.041 ms) : 0, 6041
Debugger [candidate] (6.027 ms) : 0, 6027
Remote Config [baseline] (608.361 µs) : 0, 608
Remote Config [candidate] (621.749 µs) : 0, 622
Telemetry [baseline] (7.867 ms) : 0, 7867
Telemetry [candidate] (7.82 ms) : 0, 7820
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 1 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
dateFormat X
axisFormat %s
section baseline
no_agent (38.696 ms) : 38392, 39001
. : milestone, 38696,
appsec (49.44 ms) : 48986, 49893
. : milestone, 49440,
code_origins (45.687 ms) : 45297, 46077
. : milestone, 45687,
iast (42.804 ms) : 42436, 43173
. : milestone, 42804,
profiling (49.532 ms) : 49091, 49973
. : milestone, 49532,
tracing (44.836 ms) : 44478, 45194
. : milestone, 44836,
section candidate
no_agent (38.609 ms) : 38302, 38917
. : milestone, 38609,
appsec (46.097 ms) : 45691, 46503
. : milestone, 46097,
code_origins (46.513 ms) : 46130, 46897
. : milestone, 46513,
iast (45.023 ms) : 44626, 45421
. : milestone, 45023,
profiling (49.102 ms) : 48674, 49530
. : milestone, 49102,
tracing (44.005 ms) : 43647, 44362
. : milestone, 44005,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
dateFormat X
axisFormat %s
section baseline
no_agent (4.35 ms) : 4298, 4402
. : milestone, 4350,
iast (9.361 ms) : 9201, 9522
. : milestone, 9361,
iast_FULL (14.046 ms) : 13767, 14326
. : milestone, 14046,
iast_GLOBAL (9.992 ms) : 9818, 10166
. : milestone, 9992,
profiling (8.794 ms) : 8659, 8928
. : milestone, 8794,
tracing (7.708 ms) : 7592, 7823
. : milestone, 7708,
section candidate
no_agent (4.389 ms) : 4337, 4441
. : milestone, 4389,
iast (9.084 ms) : 8939, 9230
. : milestone, 9084,
iast_FULL (13.85 ms) : 13572, 14128
. : milestone, 13850,
iast_GLOBAL (10.191 ms) : 10002, 10381
. : milestone, 10191,
profiling (9.068 ms) : 8927, 9209
. : milestone, 9068,
tracing (7.611 ms) : 7504, 7719
. : milestone, 7611,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
dateFormat X
axisFormat %s
section baseline
no_agent (1.481 ms) : 1469, 1493
. : milestone, 1481,
appsec (2.413 ms) : 2364, 2462
. : milestone, 2413,
iast (2.203 ms) : 2141, 2264
. : milestone, 2203,
iast_GLOBAL (2.253 ms) : 2190, 2315
. : milestone, 2253,
profiling (2.058 ms) : 2009, 2108
. : milestone, 2058,
tracing (2.014 ms) : 1966, 2061
. : milestone, 2014,
section candidate
no_agent (1.487 ms) : 1475, 1498
. : milestone, 1487,
appsec (2.406 ms) : 2357, 2454
. : milestone, 2406,
iast (2.191 ms) : 2130, 2252
. : milestone, 2191,
iast_GLOBAL (2.237 ms) : 2175, 2299
. : milestone, 2237,
profiling (2.027 ms) : 1978, 2075
. : milestone, 2027,
tracing (2.009 ms) : 1962, 2057
. : milestone, 2009,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
dateFormat X
axisFormat %s
section baseline
no_agent (14.728 s) : 14728000, 14728000
. : milestone, 14728000,
appsec (14.825 s) : 14825000, 14825000
. : milestone, 14825000,
iast (18.552 s) : 18552000, 18552000
. : milestone, 18552000,
iast_GLOBAL (17.869 s) : 17869000, 17869000
. : milestone, 17869000,
profiling (15.075 s) : 15075000, 15075000
. : milestone, 15075000,
tracing (14.906 s) : 14906000, 14906000
. : milestone, 14906000,
section candidate
no_agent (15.321 s) : 15321000, 15321000
. : milestone, 15321000,
appsec (15.047 s) : 15047000, 15047000
. : milestone, 15047000,
iast (18.99 s) : 18990000, 18990000
. : milestone, 18990000,
iast_GLOBAL (18.174 s) : 18174000, 18174000
. : milestone, 18174000,
profiling (15.423 s) : 15423000, 15423000
. : milestone, 15423000,
tracing (14.77 s) : 14770000, 14770000
. : milestone, 14770000,
|
* removing limits for version 5.0.0 on muzzle * add vertx-pg-client-4.4.2 * workaround * use java 11 for vertx mysql client muzzle * use java 11 for all the vertx 3.9 checking 5+ * fix muzzle for vertx-web * modifying vertx-web-4.0 * skip assertInverse for muzzle vertx web 4 --------- Co-authored-by: Andrea Marziali <[email protected]>
* skip should be for entire class * bump memory to combat oom issues * bump smoketests and forked tests memory and defaults
* first pass checks * add some debug * Revert "add some debug" This reverts commit 26a7adb. * Normalize node index and node total * fix normalization * actually use the normalized values * first pass: migrate tests to Gitlab * less scenarios for now * bump memory and workers * actually use task partitions * Add section start/end * Allow individual tests to control gradle memory * remove a little of the reportspam * use concrete docker image tags * configure test containers to use mirror * fix limit * fix missing container versions * Better gitlab naming with smarter split * update caches with test dep cache, save test results * Cleanup repositories {} blocks * update proxy url to use internal routing * no "s" * allow insecure everywhere * include apache with confluence * remove intentional fail * use gradlePluginCache, mavenCentral after pluginportal * proxy mule as well * change BUILD_CACHE_TYPE to CACHE_TYPE since its used for both * split cache like with circleci. Cache became too big * Some caches are too big to be saved * test fixes * more test fixes * disable some tests for now * more fixes * Use maven proxy with openliberty smoketests * missed one more mongodb case * enable some more of the matrix * Fix lettuce tests * Fix lettuce tests * cleanup imports * create asyncCommands * codenarc * Cleanup aggregator tests to try to reduce flakiness * debug maven logs for now * try non-ssh url * Revert "debug maven logs for now" This reverts commit c5c158f. * add some debug logging * codenarc * try fixed value of parallelism * enable debug logs for smoketest building only * limit number of reserved threads * some debug for why the smoketest is not running * more debug for gradle launcher failure * try fixing graal test * try to force the logging * codenarc * set available processors for jetty * codenarc * bluntly bump max threads * some more debug * remove verbose logging * try memcached test with new location * opt out of split by parallel * Try with 3 gradle workers * try decreasing reporting interval * Refractor CrashTrackingSmokeTest to use OutputThreads * Simple forked memory test * better ergonomics * try increasing timeout * extend agent test runner * don't override forkedMinHeapSize and forkedMaxHeapSize * print out requests * remove logging statement * Better crash tracking * remove intentionally failing test * use a test UDP server to record oom events * Make crash tracking smoke test more resiliant * loop through messages until the correct event * some debug * configurable dogstatsd port * enable everything * use sleep instead of park nanos * save body data ahead of time * retry once for system failures * spotless * cleanup * spotless/merge issues * Skip ExceptionHistogramTest on J9 * add a print statement around the body for debugging * comment out all but 8, 17, and 21 for now * fix spring 6 forwarding headers * revert logging statements * needs to be set in 3.1 for latestDepTest * faster OOM * property was set in incorrect location * ssi_smoke, flaky, and esoteric jvms on master only * use normal runner when TestContainers isn't needed * skip forwarding header filter here too * tweaks * limit executor size and fix IPV4 assumptions * try more workers with less memory * set max memory for all crashtracking tests * double retry * scale up memory, less jobs * revert memory bump, increase forked tests * tweaks * add flag for nondefault jvms and flaky tests * correct description * Externalize runtime processors override * ExceptionHistogramTest exclusions already handled by master * increase spring native build memory * bump gradle build timeout * bump graalvm compile memory --------- Co-authored-by: Santiago Mola <[email protected]>
Debugger benchmarksParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 5 unstable metrics. See unchanged results
Request duration reports for reportsgantt
title reports - request duration [CI 0.99] : candidate=None, baseline=None
dateFormat X
axisFormat %s
section baseline
noprobe (324.337 µs) : 276, 372
. : milestone, 324,
basic (314.767 µs) : 304, 325
. : milestone, 315,
loop (10.877 ms) : 10855, 10900
. : milestone, 10877,
section candidate
noprobe (305.17 µs) : 284, 327
. : milestone, 305,
basic (308.677 µs) : 301, 316
. : milestone, 309,
loop (10.84 ms) : 10807, 10873
. : milestone, 10840,
|
sarahchen6
force-pushed
the
sarahchen6/1.49.0-branch
branch
from
a616391 to
3023be2
Compare
Fix benchmark failures and improve results quality
Contributor
Author
|
1.50.0 has been released 🎉 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
This is a branch with 1.49.0 code and 1.50.0 build/CI.
Motivation
Compare benchmarks between this branch and
masterto see if code changes from 1.49.0 to 1.50.0 introduce any serious regressions.Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]