diff --git a/README.md b/README.md index 4269d207..a8e6b1a5 100644 --- a/README.md +++ b/README.md @@ -1,36 +1,36 @@ # GraphQL Gateways Benchmark -This repository is a collection is different scenarios and tests performed on different implementations of GraphQL gateways. +This repository is a collection is different scenarios and tests performed on different implementations of GraphQL gateways. This goals of this repo: 1. **Provide a transparent, accurate and descriptive benchmark testing for different tools:** - ➡️ All tests are running in Docker containers, with the same configuration. + ➡️ All tests are running in Docker containers, with the same configuration. - ➡️ During the test, tools like [cadvisor](https://github.com/google/cadvisor) are monitoring stats. + ➡️ During the test, tools like [cadvisor](https://github.com/google/cadvisor) are monitoring stats. - ➡️ Promethues and Grafana are stores results and stats for every run, and we export and embed it as PNG. + ➡️ Promethues and Grafana are stores results and stats for every run, and we export and embed it as PNG. - ➡️ Every scenario have a different report, based on what it measures. + ➡️ Every scenario have a different report, based on what it measures. - ➡️ We are using a dedicated GitHub Actions runners, with `concurrency=1` to make sure tests are running as standalone. + ➡️ We are using a dedicated GitHub Actions runners, with `concurrency=1` to make sure tests are running as standalone. 2. **Always be up to date:** - ➡️ This allows tools to improve over time. + ➡️ This allows tools to improve over time. - ➡️ All code in the repository is open and can be changed by the community. + ➡️ All code in the repository is open and can be changed by the community. - ➡️ Scenarios are executed for every PR, and reports are generated automatically per scenario. + ➡️ Scenarios are executed for every PR, and reports are generated automatically per scenario. - ➡️ [Renovate](https://github.com/renovatebot/renovate) keeps all dependencies up-to-date. + ➡️ [Renovate](https://github.com/renovatebot/renovate) keeps all dependencies up-to-date. 3. **Various scenarios** - ➡️ We are trying to create real-life scenarios based on [our experience and our customers](https://the-guild.dev). + ➡️ We are trying to create real-life scenarios based on [our experience and our customers](https://the-guild.dev). - ➡️ Each scenario has differnt setup and measure different stats. + ➡️ Each scenario has differnt setup and measure different stats. # Scenarios @@ -40,13 +40,13 @@ This goals of this repo: This scenario runs the following: -1. 4 GraphQL subgraphs in dedicated services +1. 4 GraphQL subgraphs in dedicated services 2. A GraphQL gateway compatible with the Apollo Federation spec 3. Constant rate of VUs over fixed time span This measures the following: -1. RPS (requests per second) rate +1. Successful RPS (requests that did not fail per second) rate 2. Request duration (average, p95) 3. Request failures (count) 4. CPU usage during the entire execution @@ -59,7 +59,7 @@ This measures the following: [Latest Results](./federation/scenarios/constant-vus-subgraphs-delay/README.md) -This scenario runs the same flow as `fed-constant-vus-over-time` but with an intentional delay on each upstream Subgraph. This creates more stress and increases memory in the server due to the more inflight requests. +This scenario runs the same flow as `fed-constant-vus-over-time` but with an intentional delay on each upstream Subgraph. This creates more stress and increases memory in the server due to the more inflight requests. ## `fed-constant-vus-subgraphs-delay-resources` @@ -73,16 +73,17 @@ This scenario runs the same flow as `fed-constant-vus-subgraphs-delay` but with This scenario runs the following: -1. 4 GraphQL subgraphs in dedicated services +1. 4 GraphQL subgraphs in dedicated services 2. A GraphQL gateway compatible with the Apollo Federation spec 3. Gradually ramping VUs to a high number, to demo a stress scenario This measures the following: -1. RPS (requests per second) rate -2. Request duration (average, med, max, p95) -3. Request failures (count) -4. CPU usage during the entire execution -5. RAM usage during the entire execution -6. HTTP layer timings + +1. Successful RPS (requests that did not fail per second) rate +1. Request duration (average, med, max, p95) +1. Request failures (count) +1. CPU usage during the entire execution +1. RAM usage during the entire execution +1. HTTP layer timings > This scenario uses Federation spec with all gateways that supports this kind of specification (not all gateways supports v2 spec). diff --git a/federation/gateways/hive-gateway-bun/docker-compose.yaml b/federation/gateways/hive-gateway-bun/docker-compose.yaml index 0c3e240a..32f2d98c 100644 --- a/federation/gateways/hive-gateway-bun/docker-compose.yaml +++ b/federation/gateways/hive-gateway-bun/docker-compose.yaml @@ -1,6 +1,6 @@ services: gateway: - image: ghcr.io/graphql-hive/gateway:1.13.6-bun + image: ghcr.io/graphql-hive/gateway:1.13.7-alpha-5c304a6911bf81bf710f865b4fbc9ad0d842b8ac-bun container_name: gateway networks: - test diff --git a/federation/gateways/hive-gateway/docker-compose.yaml b/federation/gateways/hive-gateway/docker-compose.yaml index f85d2c0c..f98d189a 100644 --- a/federation/gateways/hive-gateway/docker-compose.yaml +++ b/federation/gateways/hive-gateway/docker-compose.yaml @@ -1,6 +1,6 @@ services: gateway: - image: ghcr.io/graphql-hive/gateway:1.13.6 + image: ghcr.io/graphql-hive/gateway:1.13.7-alpha-5c304a6911bf81bf710f865b4fbc9ad0d842b8ac container_name: gateway networks: - test diff --git a/federation/scenarios/constant-vus-over-time/README.md b/federation/scenarios/constant-vus-over-time/README.md index e2c6a2c4..063722d0 100644 --- a/federation/scenarios/constant-vus-over-time/README.md +++ b/federation/scenarios/constant-vus-over-time/README.md @@ -10,18 +10,18 @@ This scenario was running 300 VUs over 60s ### Comparison -Comparison +Comparison -| Gateway | RPS ⬇️ | Requests | Duration | Notes | -| :--------------- | :----: | :-------------------: | :----------------------: | :----------------------------------------------------------------------- | -| cosmo | 181 | 11050 total, 0 failed | avg: 796ms, p95: 2225ms | ✅ | -| apollo-router | 165 | 10082 total, 0 failed | avg: 882ms, p95: 2311ms | ✅ | -| grafbase | 164 | 10009 total, 0 failed | avg: 888ms, p95: 2447ms | ✅ | -| hive-gateway-bun | 85 | 5455 total, 0 failed | avg: 3298ms, p95: 5420ms | ✅ | -| apollo-server | 82 | 5192 total, 21 failed | avg: 3520ms, p95: 3520ms | ❌ 21 failed requests, 21 non-200 responses, 21 unexpected GraphQL errors | -| hive-gateway | 76 | 4836 total, 20 failed | avg: 3795ms, p95: 5336ms | ❌ 20 failed requests, 20 non-200 responses, 20 unexpected GraphQL errors | -| mercurius | 74 | 4632 total, 0 failed | avg: 3918ms, p95: 5195ms | ✅ | +| Gateway | Successful RPS ⬇️ | Requests | Duration | Notes | +| :--------------- | :---------------: | :-------------------: | :----------------------: | :----------------------------------------------------------------------- | +| cosmo | 166 | 10179 total, 0 failed | avg: 880ms, p95: 2398ms | ✅ | +| grafbase | 165 | 10131 total, 0 failed | avg: 855ms, p95: 2525ms | ✅ | +| apollo-router | 157 | 9622 total, 0 failed | avg: 1008ms, p95: 2663ms | ✅ | +| apollo-server | 85 | 5418 total, 73 failed | avg: 3355ms, p95: 3398ms | ❌ 73 failed requests, 73 non-200 responses, 73 unexpected GraphQL errors | +| hive-gateway-bun | 83 | 5266 total, 0 failed | avg: 3421ms, p95: 5788ms | ✅ | +| hive-gateway | 75 | 4822 total, 25 failed | avg: 3794ms, p95: 5289ms | ❌ 25 failed requests, 25 non-200 responses, 25 unexpected GraphQL errors | +| mercurius | 73 | 4552 total, 0 failed | avg: 3985ms, p95: 5275ms | ✅ | @@ -40,22 +40,22 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 33090 ✗ 0 - data_received..................: 970 MB 16 MB/s - data_sent......................: 13 MB 215 kB/s - http_req_blocked...............: avg=1.31ms min=1.38µs med=3.46µs max=1.5s p(90)=5.52µs p(95)=11.53µs - http_req_connecting............: avg=1.03ms min=0s med=0s max=1.08s p(90)=0s p(95)=0s - http_req_duration..............: avg=795.75ms min=3.25ms med=619.55ms max=6.45s p(90)=1.63s p(95)=2.22s - { expected_response:true }...: avg=795.75ms min=3.25ms med=619.55ms max=6.45s p(90)=1.63s p(95)=2.22s - http_req_failed................: 0.00% ✓ 0 ✗ 11050 - http_req_receiving.............: avg=269.57ms min=29.88µs med=90.39µs max=5.89s p(90)=1.05s p(95)=1.68s - http_req_sending...............: avg=25.11ms min=8.1µs med=16.3µs max=3.24s p(90)=123.21µs p(95)=20.47ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=501.07ms min=3.14ms med=448.34ms max=2.38s p(90)=918.14ms p(95)=1.08s - http_reqs......................: 11050 181.48156/s - iteration_duration.............: avg=1.62s min=21.68ms med=1.31s max=9.28s p(90)=3.34s p(95)=4.08s - iterations.....................: 11030 181.153087/s - vus............................: 300 min=300 max=300 + checks.........................: 100.00% ✓ 30477 ✗ 0 + data_received..................: 893 MB 15 MB/s + data_sent......................: 12 MB 198 kB/s + http_req_blocked...............: avg=1.42ms min=1.96µs med=4.83µs max=3.33s p(90)=7.33µs p(95)=15.37µs + http_req_connecting............: avg=1.08ms min=0s med=0s max=3.33s p(90)=0s p(95)=0s + http_req_duration..............: avg=880.42ms min=4.01ms med=694.29ms max=6.3s p(90)=1.9s p(95)=2.39s + { expected_response:true }...: avg=880.42ms min=4.01ms med=694.29ms max=6.3s p(90)=1.9s p(95)=2.39s + http_req_failed................: 0.00% ✓ 0 ✗ 10179 + http_req_receiving.............: avg=349.97ms min=37.17µs med=130.02µs max=5.71s p(90)=1.22s p(95)=1.83s + http_req_sending...............: avg=24.31ms min=7.44µs med=26.25µs max=3.07s p(90)=194.34µs p(95)=29.21ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=506.13ms min=3.83ms med=486.27ms max=2.61s p(90)=895.97ms p(95)=1.03s + http_reqs......................: 10179 166.625928/s + iteration_duration.............: avg=1.77s min=25.8ms med=1.42s max=10.39s p(90)=3.71s p(95)=4.5s + iterations.....................: 10159 166.298536/s + vus............................: 77 min=77 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -63,25 +63,25 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview
- Summary for: `apollo-router` + Summary for: `grafbase` **K6 Output** @@ -95,22 +95,22 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 30186 ✗ 0 - data_received..................: 885 MB 15 MB/s - data_sent......................: 12 MB 196 kB/s - http_req_blocked...............: avg=1.44ms min=1.62µs med=3.32µs max=1.63s p(90)=5.42µs p(95)=11.23µs - http_req_connecting............: avg=1.09ms min=0s med=0s max=1.63s p(90)=0s p(95)=0s - http_req_duration..............: avg=881.81ms min=7.11ms med=697.07ms max=5.6s p(90)=1.85s p(95)=2.31s - { expected_response:true }...: avg=881.81ms min=7.11ms med=697.07ms max=5.6s p(90)=1.85s p(95)=2.31s - http_req_failed................: 0.00% ✓ 0 ✗ 10082 - http_req_receiving.............: avg=294.69ms min=31.87µs med=87.61µs max=5.18s p(90)=1.17s p(95)=1.75s - http_req_sending...............: avg=17.29ms min=7.86µs med=15.3µs max=3.34s p(90)=127.04µs p(95)=3.1ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=569.82ms min=7.03ms med=527.58ms max=2.63s p(90)=1.06s p(95)=1.21s - http_reqs......................: 10082 165.111479/s - iteration_duration.............: avg=1.79s min=30.92ms med=1.5s max=9.49s p(90)=3.68s p(95)=4.38s - iterations.....................: 10062 164.783942/s - vus............................: 70 min=70 max=300 + checks.........................: 100.00% ✓ 30333 ✗ 0 + data_received..................: 890 MB 15 MB/s + data_sent......................: 12 MB 197 kB/s + http_req_blocked...............: avg=1.07ms min=1.69µs med=3.82µs max=1.77s p(90)=5.67µs p(95)=11.5µs + http_req_connecting............: avg=669.73µs min=0s med=0s max=1.39s p(90)=0s p(95)=0s + http_req_duration..............: avg=855.01ms min=3.59ms med=633.81ms max=6.97s p(90)=1.99s p(95)=2.52s + { expected_response:true }...: avg=855.01ms min=3.59ms med=633.81ms max=6.97s p(90)=1.99s p(95)=2.52s + http_req_failed................: 0.00% ✓ 0 ✗ 10131 + http_req_receiving.............: avg=354.54ms min=33.5µs med=102.36µs max=6.94s p(90)=1.35s p(95)=1.99s + http_req_sending...............: avg=15ms min=8.06µs med=17.73µs max=2.73s p(90)=129.89µs p(95)=2.55ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=485.46ms min=3.51ms med=438.36ms max=2.45s p(90)=926.63ms p(95)=1.11s + http_reqs......................: 10131 165.997811/s + iteration_duration.............: avg=1.78s min=21.21ms med=1.46s max=10.27s p(90)=3.75s p(95)=4.55s + iterations.....................: 10111 165.670108/s + vus............................: 63 min=63 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -118,25 +118,25 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview
- Summary for: `grafbase` + Summary for: `apollo-router` **K6 Output** @@ -150,48 +150,48 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 29967 ✗ 0 - data_received..................: 880 MB 14 MB/s - data_sent......................: 12 MB 195 kB/s - http_req_blocked...............: avg=860.36µs min=1.64µs med=4.1µs max=1.61s p(90)=5.93µs p(95)=12.61µs - http_req_connecting............: avg=715.99µs min=0s med=0s max=1.3s p(90)=0s p(95)=0s - http_req_duration..............: avg=888.12ms min=3.15ms med=714.88ms max=7.53s p(90)=1.87s p(95)=2.44s - { expected_response:true }...: avg=888.12ms min=3.15ms med=714.88ms max=7.53s p(90)=1.87s p(95)=2.44s - http_req_failed................: 0.00% ✓ 0 ✗ 10009 - http_req_receiving.............: avg=337.6ms min=34.54µs med=100.16µs max=6.39s p(90)=1.25s p(95)=1.83s - http_req_sending...............: avg=28.73ms min=8.17µs med=21µs max=3.35s p(90)=207.11µs p(95)=19.64ms + checks.........................: 100.00% ✓ 28806 ✗ 0 + data_received..................: 844 MB 14 MB/s + data_sent......................: 11 MB 187 kB/s + http_req_blocked...............: avg=1.64ms min=1.74µs med=3.77µs max=1.72s p(90)=5.84µs p(95)=12.07µs + http_req_connecting............: avg=1.41ms min=0s med=0s max=1.51s p(90)=0s p(95)=0s + http_req_duration..............: avg=1s min=7.15ms med=818.99ms max=6s p(90)=2.13s p(95)=2.66s + { expected_response:true }...: avg=1s min=7.15ms med=818.99ms max=6s p(90)=2.13s p(95)=2.66s + http_req_failed................: 0.00% ✓ 0 ✗ 9622 + http_req_receiving.............: avg=352.18ms min=36.92µs med=94.9µs max=4.95s p(90)=1.38s p(95)=2.09s + http_req_sending...............: avg=19.03ms min=7.49µs med=17.88µs max=3.52s p(90)=147.12µs p(95)=12.58ms http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=521.77ms min=3.08ms med=461.62ms max=2.35s p(90)=1.04s p(95)=1.18s - http_reqs......................: 10009 164.3078/s - iteration_duration.............: avg=1.79s min=18.66ms med=1.49s max=10.3s p(90)=3.75s p(95)=4.54s - iterations.....................: 9989 163.97948/s - vus............................: 300 min=300 max=300 - vus_max........................: 300 min=300 max=300 + http_req_waiting...............: avg=636.67ms min=7.03ms med=600.1ms max=2.42s p(90)=1.1s p(95)=1.29s + http_reqs......................: 9622 157.728349/s + iteration_duration.............: avg=1.88s min=31.29ms med=1.61s max=8.69s p(90)=3.77s p(95)=4.53s + iterations.....................: 9602 157.400499/s + vus............................: 31 min=31 max=300 + vus_max........................: 300 min=300 max=300 ``` **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview
- Summary for: `hive-gateway-bun` + Summary for: `apollo-server` **K6 Output** @@ -199,54 +199,56 @@ This scenario was running 300 VUs over 60s ``` - ✓ response code was 200 - ✓ no graphql errors + ✗ response code was 200 + ↳ 98% — ✓ 5325 / ✗ 73 + ✗ no graphql errors + ↳ 98% — ✓ 5325 / ✗ 73 ✓ valid response structure █ setup - checks.........................: 100.00% ✓ 16305 ✗ 0 - data_received..................: 479 MB 7.5 MB/s - data_sent......................: 6.5 MB 102 kB/s - http_req_blocked...............: avg=743.12µs min=1.51µs med=3.71µs max=53.44ms p(90)=6.04µs p(95)=617.8µs - http_req_connecting............: avg=715.06µs min=0s med=0s max=53.39ms p(90)=0s p(95)=401.01µs - http_req_duration..............: avg=3.29s min=13.91ms med=3.03s max=7.34s p(90)=4.95s p(95)=5.41s - { expected_response:true }...: avg=3.29s min=13.91ms med=3.03s max=7.34s p(90)=4.95s p(95)=5.41s - http_req_failed................: 0.00% ✓ 0 ✗ 5455 - http_req_receiving.............: avg=45.24ms min=39.06µs med=114.63µs max=2.23s p(90)=5.17ms p(95)=278.28ms - http_req_sending...............: avg=1.2ms min=8.65µs med=18.92µs max=673.25ms p(90)=65.26µs p(95)=261.65µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.25s min=13.77ms med=3s max=7.34s p(90)=4.92s p(95)=5.4s - http_reqs......................: 5455 85.87497/s - iteration_duration.............: avg=3.4s min=284.72ms med=3.1s max=7.53s p(90)=5.1s p(95)=5.53s - iterations.....................: 5435 85.560121/s - vus............................: 104 min=104 max=300 - vus_max........................: 300 min=300 max=300 + checks.........................: 99.09% ✓ 15975 ✗ 146 + data_received..................: 470 MB 7.5 MB/s + data_sent......................: 6.4 MB 103 kB/s + http_req_blocked...............: avg=518.93µs min=1.53µs med=3.16µs max=22.3ms p(90)=5.25µs p(95)=877.84µs + http_req_connecting............: avg=499.87µs min=0s med=0s max=22.28ms p(90)=0s p(95)=826.89µs + http_req_duration..............: avg=3.35s min=11.34ms med=2s max=1m0s p(90)=2.53s p(95)=3.39s + { expected_response:true }...: avg=2.58s min=11.34ms med=1.99s max=59.11s p(90)=2.49s p(95)=3.01s + http_req_failed................: 1.34% ✓ 73 ✗ 5345 + http_req_receiving.............: avg=437.09µs min=0s med=105.36µs max=231.03ms p(90)=207.45µs p(95)=387.28µs + http_req_sending...............: avg=124.14µs min=8.88µs med=15.79µs max=47.65ms p(90)=37.52µs p(95)=261.23µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=3.35s min=11.27ms med=2s max=1m0s p(90)=2.53s p(95)=3.39s + http_reqs......................: 5418 86.852203/s + iteration_duration.............: avg=3.38s min=229.77ms med=2.02s max=1m0s p(90)=2.55s p(95)=3.42s + iterations.....................: 5398 86.531597/s + vus............................: 70 min=70 max=300 + vus_max........................: 300 min=300 max=300 ``` **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview
- Summary for: `apollo-server` + Summary for: `hive-gateway-bun` **K6 Output** @@ -254,50 +256,48 @@ This scenario was running 300 VUs over 60s ``` - ✗ response code was 200 - ↳ 99% — ✓ 5151 / ✗ 21 - ✗ no graphql errors - ↳ 99% — ✓ 5151 / ✗ 21 + ✓ response code was 200 + ✓ no graphql errors ✓ valid response structure █ setup - checks.........................: 99.72% ✓ 15453 ✗ 42 - data_received..................: 455 MB 7.2 MB/s - data_sent......................: 6.2 MB 98 kB/s - http_req_blocked...............: avg=2.1ms min=1.71µs med=3.85µs max=128.85ms p(90)=6.05µs p(95)=260.52µs - http_req_connecting............: avg=2.02ms min=0s med=0s max=128.81ms p(90)=0s p(95)=163.4µs - http_req_duration..............: avg=3.51s min=13.33ms med=2.79s max=1m0s p(90)=3.29s p(95)=3.51s - { expected_response:true }...: avg=3.29s min=13.33ms med=2.78s max=59.57s p(90)=3.28s p(95)=3.48s - http_req_failed................: 0.40% ✓ 21 ✗ 5171 - http_req_receiving.............: avg=389.67µs min=0s med=114.91µs max=88.9ms p(90)=215.54µs p(95)=368.99µs - http_req_sending...............: avg=223.84µs min=8.94µs med=20.9µs max=37.79ms p(90)=41.43µs p(95)=155.18µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.51s min=13.18ms med=2.79s max=1m0s p(90)=3.29s p(95)=3.51s - http_reqs......................: 5192 82.40553/s - iteration_duration.............: avg=3.55s min=442.54ms med=2.81s max=1m0s p(90)=3.31s p(95)=3.54s - iterations.....................: 5172 82.088098/s - vus............................: 6 min=6 max=300 - vus_max........................: 300 min=300 max=300 + checks.........................: 100.00% ✓ 15738 ✗ 0 + data_received..................: 462 MB 7.3 MB/s + data_sent......................: 6.3 MB 99 kB/s + http_req_blocked...............: avg=1.34ms min=1.46µs med=3.65µs max=77.81ms p(90)=6.29µs p(95)=3.77ms + http_req_connecting............: avg=1.28ms min=0s med=0s max=77.77ms p(90)=0s p(95)=2.94ms + http_req_duration..............: avg=3.42s min=16.81ms med=3.13s max=8.42s p(90)=5.13s p(95)=5.78s + { expected_response:true }...: avg=3.42s min=16.81ms med=3.13s max=8.42s p(90)=5.13s p(95)=5.78s + http_req_failed................: 0.00% ✓ 0 ✗ 5266 + http_req_receiving.............: avg=54.68ms min=38.41µs med=137.75µs max=2.31s p(90)=66.3ms p(95)=339.95ms + http_req_sending...............: avg=1.92ms min=9.4µs med=18.72µs max=733.43ms p(90)=138.3µs p(95)=7.35ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=3.36s min=16.67ms med=3.08s max=8.42s p(90)=5.07s p(95)=5.74s + http_reqs......................: 5266 83.576189/s + iteration_duration.............: avg=3.51s min=165.52ms med=3.18s max=8.43s p(90)=5.31s p(95)=5.94s + iterations.....................: 5246 83.258771/s + vus............................: 9 min=9 max=300 + vus_max........................: 300 min=300 max=300 ``` **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview
@@ -312,29 +312,29 @@ This scenario was running 300 VUs over 60s ``` ✗ response code was 200 - ↳ 99% — ✓ 4796 / ✗ 20 + ↳ 99% — ✓ 4777 / ✗ 25 ✗ no graphql errors - ↳ 99% — ✓ 4796 / ✗ 20 + ↳ 99% — ✓ 4777 / ✗ 25 ✓ valid response structure █ setup - checks.........................: 99.72% ✓ 14388 ✗ 40 - data_received..................: 423 MB 6.7 MB/s - data_sent......................: 5.7 MB 90 kB/s - http_req_blocked...............: avg=2.19ms min=1.85µs med=4.42µs max=94.86ms p(90)=6.49µs p(95)=18.15ms - http_req_connecting............: avg=2.12ms min=0s med=0s max=84.25ms p(90)=0s p(95)=16.75ms - http_req_duration..............: avg=3.79s min=15.79ms med=2.66s max=59.99s p(90)=3.66s p(95)=5.33s - { expected_response:true }...: avg=3.56s min=15.79ms med=2.65s max=59.55s p(90)=3.65s p(95)=5.16s - http_req_failed................: 0.41% ✓ 20 ✗ 4816 - http_req_receiving.............: avg=392.35µs min=0s med=116.2µs max=61.25ms p(90)=443.49µs p(95)=1.11ms - http_req_sending...............: avg=1.26ms min=9.96µs med=25.92µs max=72.25ms p(90)=71.42µs p(95)=4.4ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.79s min=15.68ms med=2.66s max=59.99s p(90)=3.66s p(95)=5.33s - http_reqs......................: 4836 76.057729/s - iteration_duration.............: avg=3.83s min=302.54ms med=2.68s max=1m0s p(90)=3.69s p(95)=5.4s - iterations.....................: 4816 75.743181/s - vus............................: 58 min=58 max=300 + checks.........................: 99.65% ✓ 14331 ✗ 50 + data_received..................: 421 MB 6.7 MB/s + data_sent......................: 5.7 MB 91 kB/s + http_req_blocked...............: avg=550.05µs min=1.56µs med=3.89µs max=25.27ms p(90)=6µs p(95)=2.05ms + http_req_connecting............: avg=528.12µs min=0s med=0s max=25.23ms p(90)=0s p(95)=1.35ms + http_req_duration..............: avg=3.79s min=15.98ms med=2.55s max=1m0s p(90)=3.51s p(95)=5.28s + { expected_response:true }...: avg=3.5s min=15.98ms med=2.54s max=59.4s p(90)=3.46s p(95)=4.99s + http_req_failed................: 0.51% ✓ 25 ✗ 4797 + http_req_receiving.............: avg=364.69µs min=0s med=112.37µs max=117.34ms p(90)=418.69µs p(95)=1.13ms + http_req_sending...............: avg=176.17µs min=8.41µs med=22µs max=24.85ms p(90)=55.29µs p(95)=464.02µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=3.79s min=15.85ms med=2.54s max=1m0s p(90)=3.51s p(95)=5.28s + http_reqs......................: 4822 76.307523/s + iteration_duration.............: avg=3.83s min=155.43ms med=2.57s max=1m0s p(90)=3.54s p(95)=5.31s + iterations.....................: 4802 75.991025/s + vus............................: 44 min=44 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -342,19 +342,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -374,22 +374,22 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 13836 ✗ 0 - data_received..................: 407 MB 6.6 MB/s - data_sent......................: 5.5 MB 89 kB/s - http_req_blocked...............: avg=1.38ms min=1.73µs med=4.16µs max=73.51ms p(90)=6.33µs p(95)=7.34ms - http_req_connecting............: avg=1.34ms min=0s med=0s max=73.48ms p(90)=0s p(95)=5.29ms - http_req_duration..............: avg=3.91s min=11.04ms med=3.94s max=8.5s p(90)=4.58s p(95)=5.19s - { expected_response:true }...: avg=3.91s min=11.04ms med=3.94s max=8.5s p(90)=4.58s p(95)=5.19s - http_req_failed................: 0.00% ✓ 0 ✗ 4632 - http_req_receiving.............: avg=6.51ms min=42.24µs med=108.72µs max=796.59ms p(90)=254.92µs p(95)=547.68µs - http_req_sending...............: avg=311.7µs min=9.22µs med=23.7µs max=22.59ms p(90)=45.69µs p(95)=564.77µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.91s min=10.94ms med=3.94s max=8.5s p(90)=4.58s p(95)=5.19s - http_reqs......................: 4632 74.757361/s - iteration_duration.............: avg=3.95s min=482.43ms med=3.95s max=8.51s p(90)=4.6s p(95)=5.2s - iterations.....................: 4612 74.434574/s - vus............................: 236 min=236 max=300 + checks.........................: 100.00% ✓ 13596 ✗ 0 + data_received..................: 400 MB 6.5 MB/s + data_sent......................: 5.4 MB 87 kB/s + http_req_blocked...............: avg=452.72µs min=1.85µs med=4.18µs max=44.66ms p(90)=6.52µs p(95)=1ms + http_req_connecting............: avg=423.3µs min=0s med=0s max=20.63ms p(90)=0s p(95)=802.16µs + http_req_duration..............: avg=3.98s min=12.01ms med=4s max=9.16s p(90)=4.65s p(95)=5.27s + { expected_response:true }...: avg=3.98s min=12.01ms med=4s max=9.16s p(90)=4.65s p(95)=5.27s + http_req_failed................: 0.00% ✓ 0 ✗ 4552 + http_req_receiving.............: avg=2.58ms min=41.63µs med=110.02µs max=887.16ms p(90)=279.72µs p(95)=678.34µs + http_req_sending...............: avg=162.01µs min=9.5µs med=23.68µs max=49.93ms p(90)=49.33µs p(95)=248.47µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=3.98s min=11.9ms med=4s max=9.15s p(90)=4.65s p(95)=5.27s + http_reqs......................: 4552 73.486529/s + iteration_duration.............: avg=4.02s min=171.46ms med=4.02s max=9.2s p(90)=4.67s p(95)=5.29s + iterations.....................: 4532 73.163653/s + vus............................: 239 min=239 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -397,19 +397,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview \ No newline at end of file diff --git a/federation/scenarios/constant-vus-over-time/generate-report.ts b/federation/scenarios/constant-vus-over-time/generate-report.ts index f1a6607d..0b820061 100644 --- a/federation/scenarios/constant-vus-over-time/generate-report.ts +++ b/federation/scenarios/constant-vus-over-time/generate-report.ts @@ -58,10 +58,7 @@ async function generateReport(artifactsRootPath: string) { withFileTypes: true, }) .filter((r) => r.isDirectory() && !IGNORED_DIRS.includes(r.name)) - .filter( - (r) => - r.name.startsWith(process.env.SCENARIO_ARTIFACTS_PREFIX!) - ) + .filter((r) => r.name.startsWith(process.env.SCENARIO_ARTIFACTS_PREFIX!)) .map((r) => r.name); console.info( @@ -121,13 +118,19 @@ async function generateReport(artifactsRootPath: string) { } const jsonSummary = JSON.parse(readFileSync(jsonSummaryFilePath, "utf8")); + const succReqs = + jsonSummary.metrics.http_reqs.values.count - + jsonSummary.metrics.http_req_failed.values.passes; return { name: dirName.replace(process.env.SCENARIO_ARTIFACTS_PREFIX!, ""), path: fullPath, jsonSummary, txtSummary: readFileSync(txtSummaryFilePath, "utf8"), - rps: Math.floor(jsonSummary.metrics.http_reqs.values.rate), + rps: Math.floor( + // calculate only successful requests + succReqs / (jsonSummary.state.testRunDurationMs / 1000) + ), overviewImageUrl, httpImageUrl, containersImageUrl, @@ -217,7 +220,6 @@ async function generateReport(artifactsRootPath: string) { (c) => c.name === "valid response structure" ); - if (http200Check.fails > 0) { notes.push(`${http200Check.fails} non-200 responses`); } @@ -247,7 +249,7 @@ async function generateReport(artifactsRootPath: string) { { columns: [ { name: "Gateway" }, - { name: "RPS ⬇️", align: "center" }, + { name: "Successful RPS ⬇️", align: "center" }, { name: "Requests", align: "center" }, { name: "Duration", align: "center" }, { name: "Notes", align: "left" }, diff --git a/federation/scenarios/constant-vus-subgraphs-delay-resources/README.md b/federation/scenarios/constant-vus-subgraphs-delay-resources/README.md index 89a5230c..93bf96ab 100644 --- a/federation/scenarios/constant-vus-subgraphs-delay-resources/README.md +++ b/federation/scenarios/constant-vus-subgraphs-delay-resources/README.md @@ -10,18 +10,18 @@ This scenario was running 500 VUs over 60s ### Comparison -Comparison +Comparison -| Gateway | RPS ⬇️ | Requests | Duration | Notes | -| :--------------- | :----: | :--------------------: | :-----------------------: | :-------------------------------------------------------------------------- | -| cosmo | 174 | 10745 total, 0 failed | avg: 1383ms, p95: 3295ms | ✅ | -| grafbase | 167 | 10316 total, 0 failed | avg: 1418ms, p95: 3235ms | ✅ | -| apollo-router | 158 | 9740 total, 15 failed | avg: 1487ms, p95: 3412ms | ❌ 15 failed requests, 15 non-200 responses, 15 unexpected GraphQL errors | -| hive-gateway-bun | 88 | 5727 total, 0 failed | avg: 5340ms, p95: 9045ms | ✅ | -| apollo-server | 87 | 5511 total, 249 failed | avg: 5488ms, p95: 45598ms | ❌ 249 failed requests, 249 non-200 responses, 249 unexpected GraphQL errors | -| hive-gateway | 82 | 5361 total, 138 failed | avg: 5691ms, p95: 22853ms | ❌ 138 failed requests, 138 non-200 responses, 138 unexpected GraphQL errors | -| mercurius | 73 | 4642 total, 0 failed | avg: 6574ms, p95: 8397ms | ✅ | +| Gateway | Successful RPS ⬇️ | Requests | Duration | Notes | +| :--------------- | :---------------: | :--------------------: | :-----------------------: | :-------------------------------------------------------------------------------------------------------------- | +| cosmo | 176 | 10838 total, 0 failed | avg: 1299ms, p95: 3063ms | ✅ | +| grafbase | 164 | 10122 total, 0 failed | avg: 1380ms, p95: 3840ms | ✅ | +| apollo-router | 158 | 9754 total, 10 failed | avg: 1343ms, p95: 3029ms | ❌ 10 failed requests, 10 non-200 responses, 11 unexpected GraphQL errors, non-compatible response structure (1) | +| hive-gateway-bun | 87 | 5634 total, 0 failed | avg: 5346ms, p95: 9143ms | ✅ | +| apollo-server | 83 | 5730 total, 262 failed | avg: 5348ms, p95: 46959ms | ❌ 262 failed requests, 262 non-200 responses, 262 unexpected GraphQL errors | +| hive-gateway | 78 | 5205 total, 145 failed | avg: 5854ms, p95: 26508ms | ❌ 145 failed requests, 145 non-200 responses, 145 unexpected GraphQL errors | +| mercurius | 77 | 4905 total, 0 failed | avg: 6247ms, p95: 8223ms | ✅ | @@ -40,22 +40,22 @@ This scenario was running 500 VUs over 60s █ setup - checks.........................: 100.00% ✓ 32175 ✗ 0 - data_received..................: 943 MB 15 MB/s - data_sent......................: 13 MB 208 kB/s - http_req_blocked...............: avg=2.82ms min=1.44µs med=3.38µs max=1.61s p(90)=5.62µs p(95)=110.5µs - http_req_connecting............: avg=2.13ms min=0s med=0s max=1.61s p(90)=0s p(95)=0s - http_req_duration..............: avg=1.38s min=3.26ms med=1.17s max=7.56s p(90)=2.76s p(95)=3.29s - { expected_response:true }...: avg=1.38s min=3.26ms med=1.17s max=7.56s p(90)=2.76s p(95)=3.29s - http_req_failed................: 0.00% ✓ 0 ✗ 10745 - http_req_receiving.............: avg=430.36ms min=32.43µs med=94.68µs max=6.59s p(90)=1.71s p(95)=2.36s - http_req_sending...............: avg=32.1ms min=7.41µs med=16.03µs max=5.42s p(90)=167.24µs p(95)=7.73ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=920.4ms min=3.14ms med=869.25ms max=4.43s p(90)=1.6s p(95)=1.92s - http_reqs......................: 10745 174.775045/s - iteration_duration.............: avg=2.81s min=24.75ms med=2.47s max=12.85s p(90)=5.42s p(95)=6.45s - iterations.....................: 10725 174.44973/s - vus............................: 251 min=251 max=500 + checks.........................: 100.00% ✓ 32454 ✗ 0 + data_received..................: 951 MB 16 MB/s + data_sent......................: 13 MB 209 kB/s + http_req_blocked...............: avg=3.18ms min=1.27µs med=3.19µs max=2.44s p(90)=5.12µs p(95)=55.39µs + http_req_connecting............: avg=2.39ms min=0s med=0s max=1.35s p(90)=0s p(95)=0s + http_req_duration..............: avg=1.29s min=3.33ms med=1.11s max=7.09s p(90)=2.53s p(95)=3.06s + { expected_response:true }...: avg=1.29s min=3.33ms med=1.11s max=7.09s p(90)=2.53s p(95)=3.06s + http_req_failed................: 0.00% ✓ 0 ✗ 10838 + http_req_receiving.............: avg=294.08ms min=33.51µs med=80.09µs max=5.41s p(90)=1.31s p(95)=2.02s + http_req_sending...............: avg=29.7ms min=7.77µs med=14.73µs max=3.15s p(90)=418.15µs p(95)=21.96ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=975.35ms min=3.21ms med=927.9ms max=4.17s p(90)=1.68s p(95)=2.06s + http_reqs......................: 10838 176.266643/s + iteration_duration.............: avg=2.76s min=20.46ms med=2.41s max=12.05s p(90)=5.27s p(95)=6.2s + iterations.....................: 10818 175.941367/s + vus............................: 261 min=261 max=500 vus_max........................: 500 min=500 max=500 ``` @@ -63,19 +63,19 @@ This scenario was running 500 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -95,22 +95,22 @@ This scenario was running 500 VUs over 60s █ setup - checks.........................: 100.00% ✓ 30888 ✗ 0 - data_received..................: 907 MB 15 MB/s - data_sent......................: 12 MB 199 kB/s - http_req_blocked...............: avg=2.85ms min=1.38µs med=3.53µs max=2.4s p(90)=6.02µs p(95)=167.94µs - http_req_connecting............: avg=2.35ms min=0s med=0s max=2.4s p(90)=0s p(95)=0s - http_req_duration..............: avg=1.41s min=3.07ms med=1.25s max=7.79s p(90)=2.6s p(95)=3.23s - { expected_response:true }...: avg=1.41s min=3.07ms med=1.25s max=7.79s p(90)=2.6s p(95)=3.23s - http_req_failed................: 0.00% ✓ 0 ✗ 10316 - http_req_receiving.............: avg=286.54ms min=32.71µs med=83.78µs max=4.87s p(90)=1.23s p(95)=1.7s - http_req_sending...............: avg=24.66ms min=7.75µs med=15.98µs max=4.35s p(90)=110.07µs p(95)=22.73ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=1.1s min=3.01ms med=1.01s max=5.61s p(90)=1.93s p(95)=2.59s - http_reqs......................: 10316 167.965228/s - iteration_duration.............: avg=2.9s min=23.84ms med=2.64s max=10.28s p(90)=5.32s p(95)=6.26s - iterations.....................: 10296 167.639588/s - vus............................: 207 min=207 max=500 + checks.........................: 100.00% ✓ 30306 ✗ 0 + data_received..................: 890 MB 14 MB/s + data_sent......................: 12 MB 195 kB/s + http_req_blocked...............: avg=1.71ms min=1.74µs med=3.82µs max=1.87s p(90)=6.15µs p(95)=185.66µs + http_req_connecting............: avg=847.17µs min=0s med=0s max=1.65s p(90)=0s p(95)=0s + http_req_duration..............: avg=1.38s min=3.25ms med=1.14s max=9.79s p(90)=2.72s p(95)=3.84s + { expected_response:true }...: avg=1.38s min=3.25ms med=1.14s max=9.79s p(90)=2.72s p(95)=3.84s + http_req_failed................: 0.00% ✓ 0 ✗ 10122 + http_req_receiving.............: avg=447.76ms min=32.47µs med=95.94µs max=7.77s p(90)=1.5s p(95)=2.53s + http_req_sending...............: avg=41.22ms min=7.21µs med=18.38µs max=6.3s p(90)=239.87µs p(95)=18.88ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=891.41ms min=3.17ms med=814.49ms max=3.7s p(90)=1.65s p(95)=1.98s + http_reqs......................: 10122 164.359589/s + iteration_duration.............: avg=2.95s min=25.2ms med=2.55s max=14.25s p(90)=5.73s p(95)=6.95s + iterations.....................: 10102 164.034832/s + vus............................: 197 min=197 max=500 vus_max........................: 500 min=500 max=500 ``` @@ -118,19 +118,19 @@ This scenario was running 500 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -145,29 +145,30 @@ This scenario was running 500 VUs over 60s ``` ✗ response code was 200 - ↳ 99% — ✓ 9705 / ✗ 15 + ↳ 99% — ✓ 9724 / ✗ 10 ✗ no graphql errors - ↳ 99% — ✓ 9705 / ✗ 15 - ✓ valid response structure + ↳ 99% — ✓ 9723 / ✗ 11 + ✗ valid response structure + ↳ 99% — ✓ 9723 / ✗ 1 █ setup - checks.........................: 99.89% ✓ 29115 ✗ 30 - data_received..................: 853 MB 14 MB/s + checks.........................: 99.92% ✓ 29170 ✗ 22 + data_received..................: 855 MB 14 MB/s data_sent......................: 12 MB 188 kB/s - http_req_blocked...............: avg=7.04ms min=1.36µs med=3.45µs max=5.17s p(90)=5.82µs p(95)=1.64ms - http_req_connecting............: avg=6.2ms min=0s med=0s max=5.17s p(90)=0s p(95)=1.05ms - http_req_duration..............: avg=1.48s min=6.82ms med=1.31s max=7.19s p(90)=2.83s p(95)=3.41s - { expected_response:true }...: avg=1.48s min=6.82ms med=1.31s max=7.19s p(90)=2.82s p(95)=3.4s - http_req_failed................: 0.15% ✓ 15 ✗ 9725 - http_req_receiving.............: avg=399.94ms min=0s med=92.9µs max=6.55s p(90)=1.65s p(95)=2.11s - http_req_sending...............: avg=41.78ms min=8.33µs med=15.93µs max=4.38s p(90)=522.73µs p(95)=24.53ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=1.04s min=6.75ms med=947.39ms max=6.08s p(90)=1.92s p(95)=2.36s - http_reqs......................: 9740 158.23854/s - iteration_duration.............: avg=3.07s min=35.65ms med=2.7s max=12.47s p(90)=5.96s p(95)=7.03s - iterations.....................: 9720 157.913615/s - vus............................: 282 min=282 max=500 + http_req_blocked...............: avg=5.25ms min=1.61µs med=3.57µs max=3.34s p(90)=6.29µs p(95)=6.33ms + http_req_connecting............: avg=5.18ms min=0s med=0s max=3.34s p(90)=0s p(95)=5.22ms + http_req_duration..............: avg=1.34s min=7.46ms med=1.22s max=7.17s p(90)=2.47s p(95)=3.02s + { expected_response:true }...: avg=1.34s min=7.46ms med=1.22s max=7.17s p(90)=2.47s p(95)=3.02s + http_req_failed................: 0.10% ✓ 10 ✗ 9744 + http_req_receiving.............: avg=318.15ms min=0s med=90.69µs max=5.74s p(90)=1.25s p(95)=1.91s + http_req_sending...............: avg=23.98ms min=7.22µs med=17.58µs max=4.78s p(90)=150.65µs p(95)=2.25ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=1s min=7.35ms med=936.02ms max=3.18s p(90)=1.83s p(95)=2.04s + http_reqs......................: 9754 158.687222/s + iteration_duration.............: avg=3.04s min=26.64ms med=2.78s max=12.05s p(90)=5.59s p(95)=6.53s + iterations.....................: 9734 158.361844/s + vus............................: 285 min=285 max=500 vus_max........................: 500 min=500 max=500 ``` @@ -175,19 +176,19 @@ This scenario was running 500 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -207,22 +208,22 @@ This scenario was running 500 VUs over 60s █ setup - checks.........................: 100.00% ✓ 17121 ✗ 0 - data_received..................: 503 MB 7.7 MB/s - data_sent......................: 6.8 MB 105 kB/s - http_req_blocked...............: avg=1.21ms min=1.92µs med=4.17µs max=113.51ms p(90)=13.57µs p(95)=10.84ms - http_req_connecting............: avg=1.09ms min=0s med=0s max=53.78ms p(90)=0s p(95)=9.98ms - http_req_duration..............: avg=5.33s min=17.47ms med=5.03s max=12.87s p(90)=7.78s p(95)=9.04s - { expected_response:true }...: avg=5.33s min=17.47ms med=5.03s max=12.87s p(90)=7.78s p(95)=9.04s - http_req_failed................: 0.00% ✓ 0 ✗ 5727 - http_req_receiving.............: avg=69.25ms min=41.26µs med=108.97µs max=3.37s p(90)=6.37ms p(95)=325.87ms - http_req_sending...............: avg=1.07ms min=9.61µs med=21.62µs max=595.14ms p(90)=217.63µs p(95)=758.85µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=5.26s min=17.23ms med=4.94s max=12.87s p(90)=7.63s p(95)=9.02s - http_reqs......................: 5727 88.242818/s - iteration_duration.............: avg=5.47s min=292.92ms med=5.13s max=13.77s p(90)=7.88s p(95)=9.11s - iterations.....................: 5707 87.934654/s - vus............................: 143 min=143 max=500 + checks.........................: 100.00% ✓ 16842 ✗ 0 + data_received..................: 495 MB 7.7 MB/s + data_sent......................: 6.7 MB 104 kB/s + http_req_blocked...............: avg=5.46ms min=1.83µs med=4.23µs max=170.54ms p(90)=13.91µs p(95)=53.41ms + http_req_connecting............: avg=5.31ms min=0s med=0s max=170.51ms p(90)=0s p(95)=51.77ms + http_req_duration..............: avg=5.34s min=16.76ms med=4.93s max=13.99s p(90)=7.92s p(95)=9.14s + { expected_response:true }...: avg=5.34s min=16.76ms med=4.93s max=13.99s p(90)=7.92s p(95)=9.14s + http_req_failed................: 0.00% ✓ 0 ✗ 5634 + http_req_receiving.............: avg=68.07ms min=33.55µs med=124.34µs max=2.89s p(90)=7.79ms p(95)=320.91ms + http_req_sending...............: avg=3.34ms min=8.4µs med=23.3µs max=580.53ms p(90)=1.1ms p(95)=26.06ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=5.27s min=16.57ms med=4.88s max=13.99s p(90)=7.82s p(95)=9.1s + http_reqs......................: 5634 87.744624/s + iteration_duration.............: avg=5.49s min=425.29ms med=5.05s max=14.16s p(90)=8.03s p(95)=9.24s + iterations.....................: 5614 87.433142/s + vus............................: 34 min=34 max=500 vus_max........................: 500 min=500 max=500 ``` @@ -230,19 +231,19 @@ This scenario was running 500 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -257,29 +258,29 @@ This scenario was running 500 VUs over 60s ``` ✗ response code was 200 - ↳ 95% — ✓ 5242 / ✗ 249 + ↳ 95% — ✓ 5448 / ✗ 262 ✗ no graphql errors - ↳ 95% — ✓ 5242 / ✗ 249 + ↳ 95% — ✓ 5448 / ✗ 262 ✓ valid response structure █ setup - checks.........................: 96.93% ✓ 15726 ✗ 498 - data_received..................: 463 MB 7.4 MB/s - data_sent......................: 6.5 MB 104 kB/s - http_req_blocked...............: avg=2.3ms min=1.42µs med=3.2µs max=53.31ms p(90)=11.88µs p(95)=24.66ms - http_req_connecting............: avg=2.27ms min=0s med=0s max=52.23ms p(90)=0s p(95)=24.58ms - http_req_duration..............: avg=5.48s min=11.45ms med=2.3s max=1m0s p(90)=3.09s p(95)=45.59s - { expected_response:true }...: avg=2.9s min=11.45ms med=2.25s max=59.13s p(90)=2.91s p(95)=3.16s - http_req_failed................: 4.51% ✓ 249 ✗ 5262 - http_req_receiving.............: avg=259.3µs min=0s med=105.52µs max=78.15ms p(90)=229.97µs p(95)=421.23µs - http_req_sending...............: avg=287.33µs min=8.63µs med=17.51µs max=60.55ms p(90)=150.06µs p(95)=1.31ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=5.48s min=11.33ms med=2.3s max=1m0s p(90)=3.09s p(95)=45.59s - http_reqs......................: 5511 87.829058/s - iteration_duration.............: avg=5.52s min=65.57ms med=2.32s max=1m0s p(90)=3.12s p(95)=45.94s - iterations.....................: 5491 87.510318/s - vus............................: 120 min=120 max=500 + checks.........................: 96.89% ✓ 16344 ✗ 524 + data_received..................: 481 MB 7.3 MB/s + data_sent......................: 6.8 MB 103 kB/s + http_req_blocked...............: avg=2.06ms min=1.56µs med=3.22µs max=177.82ms p(90)=168.7µs p(95)=17.67ms + http_req_connecting............: avg=2.01ms min=0s med=0s max=152ms p(90)=127.02µs p(95)=17.41ms + http_req_duration..............: avg=5.34s min=11.9ms med=2.12s max=1m0s p(90)=2.95s p(95)=46.95s + { expected_response:true }...: avg=2.72s min=11.9ms med=2.07s max=59.44s p(90)=2.69s p(95)=3.09s + http_req_failed................: 4.57% ✓ 262 ✗ 5468 + http_req_receiving.............: avg=218.98µs min=0s med=101.84µs max=48.96ms p(90)=207.08µs p(95)=336.81µs + http_req_sending...............: avg=407.59µs min=9.12µs med=16.79µs max=94.7ms p(90)=151.95µs p(95)=1.37ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=5.34s min=11.72ms med=2.12s max=1m0s p(90)=2.95s p(95)=46.95s + http_reqs......................: 5730 87.115614/s + iteration_duration.............: avg=5.38s min=389.07ms med=2.14s max=1m0s p(90)=2.97s p(95)=47.78s + iterations.....................: 5710 86.811546/s + vus............................: 79 min=79 max=500 vus_max........................: 500 min=500 max=500 ``` @@ -287,19 +288,19 @@ This scenario was running 500 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -314,49 +315,49 @@ This scenario was running 500 VUs over 60s ``` ✗ response code was 200 - ↳ 97% — ✓ 5203 / ✗ 138 + ↳ 97% — ✓ 5040 / ✗ 145 ✗ no graphql errors - ↳ 97% — ✓ 5203 / ✗ 138 + ↳ 97% — ✓ 5040 / ✗ 145 ✓ valid response structure █ setup - checks.........................: 98.26% ✓ 15609 ✗ 276 - data_received..................: 459 MB 7.1 MB/s - data_sent......................: 6.4 MB 98 kB/s - http_req_blocked...............: avg=6.84ms min=1.63µs med=4.38µs max=180.35ms p(90)=18.94µs p(95)=59.33ms - http_req_connecting............: avg=6.59ms min=0s med=0s max=172.48ms p(90)=0s p(95)=51.97ms - http_req_duration..............: avg=5.69s min=15.41ms med=3.22s max=1m0s p(90)=5.27s p(95)=22.85s - { expected_response:true }...: avg=4.25s min=15.41ms med=3.19s max=59.95s p(90)=4.64s p(95)=6.6s - http_req_failed................: 2.57% ✓ 138 ✗ 5223 - http_req_receiving.............: avg=1.46ms min=0s med=105.14µs max=373.2ms p(90)=737.99µs p(95)=2.77ms - http_req_sending...............: avg=2.25ms min=8.76µs med=25.15µs max=140.46ms p(90)=760.27µs p(95)=17.18ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=5.68s min=15.26ms med=3.22s max=1m0s p(90)=5.26s p(95)=22.82s - http_reqs......................: 5361 82.695239/s - iteration_duration.............: avg=5.77s min=535.09ms med=3.27s max=1m0s p(90)=5.34s p(95)=23.17s - iterations.....................: 5341 82.386732/s - vus............................: 81 min=81 max=500 - vus_max........................: 500 min=500 max=500 + checks.........................: 98.11% ✓ 15120 ✗ 290 + data_received..................: 444 MB 6.9 MB/s + data_sent......................: 6.2 MB 96 kB/s + http_req_blocked...............: avg=5.98ms min=2.14µs med=4.49µs max=135.7ms p(90)=33.01µs p(95)=62.26ms + http_req_connecting............: avg=5.81ms min=0s med=0s max=132.14ms p(90)=0s p(95)=61.6ms + http_req_duration..............: avg=5.85s min=17.15ms med=3.23s max=1m0s p(90)=5.49s p(95)=26.5s + { expected_response:true }...: avg=4.29s min=17.15ms med=3.19s max=59.63s p(90)=4.75s p(95)=6.74s + http_req_failed................: 2.78% ✓ 145 ✗ 5060 + http_req_receiving.............: avg=1.41ms min=0s med=110.27µs max=309.22ms p(90)=861.16µs p(95)=2.83ms + http_req_sending...............: avg=944.43µs min=9.57µs med=26.14µs max=105.01ms p(90)=607.59µs p(95)=3.88ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=5.85s min=17.03ms med=3.23s max=1m0s p(90)=5.47s p(95)=26.5s + http_reqs......................: 5205 80.76844/s + iteration_duration.............: avg=5.92s min=644.96ms med=3.27s max=1m0s p(90)=5.53s p(95)=26.66s + iterations.....................: 5185 80.45809/s + vus............................: 44 min=44 max=500 + vus_max........................: 500 min=500 max=500 ``` **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -376,22 +377,22 @@ This scenario was running 500 VUs over 60s █ setup - checks.........................: 100.00% ✓ 13866 ✗ 0 - data_received..................: 407 MB 6.5 MB/s - data_sent......................: 5.5 MB 87 kB/s - http_req_blocked...............: avg=2.66ms min=1.61µs med=4.09µs max=50.98ms p(90)=8.87ms p(95)=26.95ms - http_req_connecting............: avg=2.6ms min=0s med=0s max=44.66ms p(90)=8.38ms p(95)=25.62ms - http_req_duration..............: avg=6.57s min=11.13ms med=6.5s max=11.24s p(90)=8.16s p(95)=8.39s - { expected_response:true }...: avg=6.57s min=11.13ms med=6.5s max=11.24s p(90)=8.16s p(95)=8.39s - http_req_failed................: 0.00% ✓ 0 ✗ 4642 - http_req_receiving.............: avg=10.08ms min=40.62µs med=105.73µs max=853.3ms p(90)=303.54µs p(95)=802.49µs - http_req_sending...............: avg=197.35µs min=9.74µs med=22.43µs max=15.51ms p(90)=316.43µs p(95)=1.14ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=6.56s min=11.03ms med=6.5s max=11.24s p(90)=8.16s p(95)=8.39s - http_reqs......................: 4642 73.637425/s - iteration_duration.............: avg=6.64s min=447.36ms med=6.51s max=11.26s p(90)=8.19s p(95)=8.4s - iterations.....................: 4622 73.320159/s - vus............................: 24 min=24 max=500 + checks.........................: 100.00% ✓ 14655 ✗ 0 + data_received..................: 431 MB 6.8 MB/s + data_sent......................: 5.8 MB 92 kB/s + http_req_blocked...............: avg=1.64ms min=1.8µs med=3.71µs max=44.63ms p(90)=144.19µs p(95)=17.23ms + http_req_connecting............: avg=1.59ms min=0s med=0s max=34.62ms p(90)=102.95µs p(95)=16.86ms + http_req_duration..............: avg=6.24s min=11.99ms med=6.23s max=12.97s p(90)=7.33s p(95)=8.22s + { expected_response:true }...: avg=6.24s min=11.99ms med=6.23s max=12.97s p(90)=7.33s p(95)=8.22s + http_req_failed................: 0.00% ✓ 0 ✗ 4905 + http_req_receiving.............: avg=4.23ms min=39.07µs med=97.92µs max=753.07ms p(90)=246.62µs p(95)=507.99µs + http_req_sending...............: avg=142.09µs min=9.1µs med=20.53µs max=39.94ms p(90)=98.09µs p(95)=614.81µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=6.24s min=11.87ms med=6.23s max=12.97s p(90)=7.33s p(95)=8.22s + http_reqs......................: 4905 77.557933/s + iteration_duration.............: avg=6.3s min=465.57ms med=6.25s max=13.01s p(90)=7.35s p(95)=8.24s + iterations.....................: 4885 77.241693/s + vus............................: 82 min=82 max=500 vus_max........................: 500 min=500 max=500 ``` @@ -399,19 +400,19 @@ This scenario was running 500 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview \ No newline at end of file diff --git a/federation/scenarios/constant-vus-subgraphs-delay/README.md b/federation/scenarios/constant-vus-subgraphs-delay/README.md index c76274e6..d8d8c041 100644 --- a/federation/scenarios/constant-vus-subgraphs-delay/README.md +++ b/federation/scenarios/constant-vus-subgraphs-delay/README.md @@ -10,18 +10,18 @@ This scenario was running 300 VUs over 60s ### Comparison -Comparison +Comparison -| Gateway | RPS ⬇️ | Requests | Duration | Notes | -| :--------------- | :----: | :-------------------: | :----------------------: | :----------------------------------------------------------------------- | -| cosmo | 181 | 11036 total, 0 failed | avg: 871ms, p95: 2191ms | ✅ | -| grafbase | 170 | 10406 total, 0 failed | avg: 882ms, p95: 2440ms | ✅ | -| apollo-router | 163 | 9972 total, 0 failed | avg: 977ms, p95: 2467ms | ✅ | -| apollo-server | 85 | 5370 total, 73 failed | avg: 3392ms, p95: 3376ms | ❌ 73 failed requests, 73 non-200 responses, 73 unexpected GraphQL errors | -| hive-gateway-bun | 82 | 5185 total, 0 failed | avg: 3458ms, p95: 5924ms | ✅ | -| hive-gateway | 77 | 4907 total, 19 failed | avg: 3726ms, p95: 5489ms | ❌ 19 failed requests, 19 non-200 responses, 19 unexpected GraphQL errors | -| mercurius | 75 | 4698 total, 0 failed | avg: 3860ms, p95: 4942ms | ✅ | +| Gateway | Successful RPS ⬇️ | Requests | Duration | Notes | +| :--------------- | :---------------: | :-------------------: | :----------------------: | :----------------------------------------------------------------------- | +| cosmo | 176 | 10756 total, 0 failed | avg: 883ms, p95: 2323ms | ✅ | +| grafbase | 168 | 10304 total, 0 failed | avg: 895ms, p95: 2251ms | ✅ | +| apollo-router | 162 | 9898 total, 0 failed | avg: 943ms, p95: 2339ms | ✅ | +| apollo-server | 87 | 5523 total, 69 failed | avg: 3296ms, p95: 3032ms | ❌ 69 failed requests, 69 non-200 responses, 69 unexpected GraphQL errors | +| hive-gateway-bun | 85 | 5405 total, 0 failed | avg: 3310ms, p95: 5566ms | ✅ | +| hive-gateway | 80 | 5085 total, 13 failed | avg: 3599ms, p95: 5104ms | ❌ 13 failed requests, 13 non-200 responses, 13 unexpected GraphQL errors | +| mercurius | 77 | 4821 total, 0 failed | avg: 3770ms, p95: 4847ms | ✅ | @@ -40,21 +40,21 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 33048 ✗ 0 - data_received..................: 969 MB 16 MB/s - data_sent......................: 13 MB 215 kB/s - http_req_blocked...............: avg=1.22ms min=1.61µs med=3.35µs max=2.82s p(90)=5.36µs p(95)=10.64µs - http_req_connecting............: avg=712.27µs min=0s med=0s max=1.69s p(90)=0s p(95)=0s - http_req_duration..............: avg=870.98ms min=3.34ms med=732.98ms max=5.06s p(90)=1.77s p(95)=2.19s - { expected_response:true }...: avg=870.98ms min=3.34ms med=732.98ms max=5.06s p(90)=1.77s p(95)=2.19s - http_req_failed................: 0.00% ✓ 0 ✗ 11036 - http_req_receiving.............: avg=284.2ms min=33.54µs med=89.33µs max=4.47s p(90)=1.16s p(95)=1.64s - http_req_sending...............: avg=19.69ms min=8.06µs med=15.57µs max=3.22s p(90)=86.8µs p(95)=3.59ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=567.08ms min=3.26ms med=546.28ms max=2.85s p(90)=1.01s p(95)=1.12s - http_reqs......................: 11036 181.239537/s - iteration_duration.............: avg=1.63s min=20.18ms med=1.36s max=8.61s p(90)=3.31s p(95)=4s - iterations.....................: 11016 180.911085/s + checks.........................: 100.00% ✓ 32208 ✗ 0 + data_received..................: 944 MB 16 MB/s + data_sent......................: 13 MB 209 kB/s + http_req_blocked...............: avg=2.78ms min=1.69µs med=4.01µs max=3.67s p(90)=6.16µs p(95)=13.09µs + http_req_connecting............: avg=2.4ms min=0s med=0s max=3.67s p(90)=0s p(95)=0s + http_req_duration..............: avg=882.84ms min=3.46ms med=715.72ms max=5.35s p(90)=1.83s p(95)=2.32s + { expected_response:true }...: avg=882.84ms min=3.46ms med=715.72ms max=5.35s p(90)=1.83s p(95)=2.32s + http_req_failed................: 0.00% ✓ 0 ✗ 10756 + http_req_receiving.............: avg=279.54ms min=35.63µs med=99.18µs max=4.75s p(90)=1.08s p(95)=1.72s + http_req_sending...............: avg=27.6ms min=8.72µs med=21.29µs max=3.16s p(90)=140.32µs p(95)=20.21ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=575.69ms min=3.34ms med=550.79ms max=3.05s p(90)=1.02s p(95)=1.16s + http_reqs......................: 10756 176.403885/s + iteration_duration.............: avg=1.68s min=20.97ms med=1.37s max=9.63s p(90)=3.49s p(95)=4.24s + iterations.....................: 10736 176.075875/s vus............................: 300 min=300 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -63,19 +63,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -95,22 +95,22 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 31158 ✗ 0 - data_received..................: 914 MB 15 MB/s - data_sent......................: 12 MB 203 kB/s - http_req_blocked...............: avg=1.2ms min=1.54µs med=3.49µs max=2.08s p(90)=5.28µs p(95)=10.81µs - http_req_connecting............: avg=672.78µs min=0s med=0s max=1.18s p(90)=0s p(95)=0s - http_req_duration..............: avg=881.92ms min=3.22ms med=693.26ms max=6.68s p(90)=1.93s p(95)=2.43s - { expected_response:true }...: avg=881.92ms min=3.22ms med=693.26ms max=6.68s p(90)=1.93s p(95)=2.43s - http_req_failed................: 0.00% ✓ 0 ✗ 10406 - http_req_receiving.............: avg=318.45ms min=33.92µs med=88.96µs max=6.25s p(90)=1.3s p(95)=1.94s - http_req_sending...............: avg=19.43ms min=7.57µs med=16.29µs max=3.21s p(90)=129.46µs p(95)=11.89ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=544.03ms min=3.01ms med=516.23ms max=2.57s p(90)=998.73ms p(95)=1.18s - http_reqs......................: 10406 170.96353/s - iteration_duration.............: avg=1.72s min=19.56ms med=1.45s max=10.53s p(90)=3.57s p(95)=4.25s - iterations.....................: 10386 170.634943/s - vus............................: 300 min=300 max=300 + checks.........................: 100.00% ✓ 30852 ✗ 0 + data_received..................: 906 MB 15 MB/s + data_sent......................: 12 MB 201 kB/s + http_req_blocked...............: avg=1.59ms min=1.29µs med=3.53µs max=2.43s p(90)=5.3µs p(95)=11.47µs + http_req_connecting............: avg=856.19µs min=0s med=0s max=2.43s p(90)=0s p(95)=0s + http_req_duration..............: avg=894.74ms min=3.04ms med=743.12ms max=5.89s p(90)=1.84s p(95)=2.25s + { expected_response:true }...: avg=894.74ms min=3.04ms med=743.12ms max=5.89s p(90)=1.84s p(95)=2.25s + http_req_failed................: 0.00% ✓ 0 ✗ 10304 + http_req_receiving.............: avg=311.79ms min=35.78µs med=90.03µs max=5.67s p(90)=1.26s p(95)=1.65s + http_req_sending...............: avg=16.3ms min=7.56µs med=16.53µs max=2.36s p(90)=86.05µs p(95)=1.05ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=566.63ms min=2.99ms med=532.23ms max=2.44s p(90)=1.01s p(95)=1.24s + http_reqs......................: 10304 168.974767/s + iteration_duration.............: avg=1.75s min=23.65ms med=1.54s max=9.23s p(90)=3.48s p(95)=4.21s + iterations.....................: 10284 168.646788/s + vus............................: 17 min=17 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -118,19 +118,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -150,22 +150,22 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 29856 ✗ 0 - data_received..................: 875 MB 14 MB/s - data_sent......................: 12 MB 194 kB/s - http_req_blocked...............: avg=1.93ms min=1.49µs med=3.26µs max=2.12s p(90)=5.2µs p(95)=11.1µs - http_req_connecting............: avg=1.36ms min=0s med=0s max=2.12s p(90)=0s p(95)=0s - http_req_duration..............: avg=976.87ms min=7.24ms med=827.99ms max=5.56s p(90)=2s p(95)=2.46s - { expected_response:true }...: avg=976.87ms min=7.24ms med=827.99ms max=5.56s p(90)=2s p(95)=2.46s - http_req_failed................: 0.00% ✓ 0 ✗ 9972 - http_req_receiving.............: avg=284.29ms min=33.13µs med=83.71µs max=5.11s p(90)=1.24s p(95)=1.85s - http_req_sending...............: avg=28.76ms min=7.65µs med=15.2µs max=2.91s p(90)=142.74µs p(95)=20.16ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=663.81ms min=6.99ms med=636.26ms max=3.13s p(90)=1.11s p(95)=1.29s - http_reqs......................: 9972 163.382342/s - iteration_duration.............: avg=1.81s min=32.38ms med=1.55s max=8.38s p(90)=3.52s p(95)=4.23s - iterations.....................: 9952 163.05466/s - vus............................: 34 min=34 max=300 + checks.........................: 100.00% ✓ 29634 ✗ 0 + data_received..................: 869 MB 14 MB/s + data_sent......................: 12 MB 193 kB/s + http_req_blocked...............: avg=1.75ms min=1.41µs med=3.16µs max=1.66s p(90)=5.17µs p(95)=10.99µs + http_req_connecting............: avg=1.59ms min=0s med=0s max=1.66s p(90)=0s p(95)=0s + http_req_duration..............: avg=942.64ms min=6.63ms med=787.05ms max=5.52s p(90)=1.87s p(95)=2.33s + { expected_response:true }...: avg=942.64ms min=6.63ms med=787.05ms max=5.52s p(90)=1.87s p(95)=2.33s + http_req_failed................: 0.00% ✓ 0 ✗ 9898 + http_req_receiving.............: avg=277.46ms min=33.79µs med=84.59µs max=4.3s p(90)=1.16s p(95)=1.65s + http_req_sending...............: avg=17.57ms min=8.04µs med=14.92µs max=4.15s p(90)=112.12µs p(95)=15.71ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=647.61ms min=6.54ms med=615.65ms max=2.52s p(90)=1.1s p(95)=1.24s + http_reqs......................: 9898 162.297432/s + iteration_duration.............: avg=1.82s min=31.24ms med=1.56s max=10.88s p(90)=3.52s p(95)=4.29s + iterations.....................: 9878 161.969492/s + vus............................: 2 min=2 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -173,19 +173,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -200,29 +200,29 @@ This scenario was running 300 VUs over 60s ``` ✗ response code was 200 - ↳ 98% — ✓ 5277 / ✗ 73 + ↳ 98% — ✓ 5434 / ✗ 69 ✗ no graphql errors - ↳ 98% — ✓ 5277 / ✗ 73 + ↳ 98% — ✓ 5434 / ✗ 69 ✓ valid response structure █ setup - checks.........................: 99.08% ✓ 15831 ✗ 146 - data_received..................: 466 MB 7.4 MB/s - data_sent......................: 6.4 MB 102 kB/s - http_req_blocked...............: avg=183.15µs min=1.48µs med=3.35µs max=12.96ms p(90)=5.44µs p(95)=264.01µs - http_req_connecting............: avg=174.08µs min=0s med=0s max=12.15ms p(90)=0s p(95)=189.9µs - http_req_duration..............: avg=3.39s min=11.11ms med=1.98s max=1m0s p(90)=2.67s p(95)=3.37s - { expected_response:true }...: avg=2.61s min=11.11ms med=1.97s max=59.75s p(90)=2.62s p(95)=3s - http_req_failed................: 1.35% ✓ 73 ✗ 5297 - http_req_receiving.............: avg=288.86µs min=0s med=107.72µs max=302.69ms p(90)=208.39µs p(95)=342.87µs - http_req_sending...............: avg=150.84µs min=8.67µs med=18.59µs max=55.5ms p(90)=38.89µs p(95)=187.86µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.39s min=11.01ms med=1.98s max=1m0s p(90)=2.67s p(95)=3.37s - http_reqs......................: 5370 85.861949/s - iteration_duration.............: avg=3.42s min=231.91ms med=2.01s max=1m0s p(90)=2.69s p(95)=3.41s - iterations.....................: 5350 85.542165/s - vus............................: 98 min=98 max=300 + checks.........................: 99.16% ✓ 16302 ✗ 138 + data_received..................: 479 MB 7.7 MB/s + data_sent......................: 6.6 MB 105 kB/s + http_req_blocked...............: avg=304.17µs min=1.37µs med=2.95µs max=13.89ms p(90)=4.75µs p(95)=284.73µs + http_req_connecting............: avg=294.75µs min=0s med=0s max=13.37ms p(90)=0s p(95)=211.13µs + http_req_duration..............: avg=3.29s min=11.44ms med=1.86s max=1m0s p(90)=2.61s p(95)=3.03s + { expected_response:true }...: avg=2.57s min=11.44ms med=1.84s max=59.88s p(90)=2.59s p(95)=2.81s + http_req_failed................: 1.24% ✓ 69 ✗ 5454 + http_req_receiving.............: avg=649.96µs min=0s med=103.7µs max=243.72ms p(90)=215.01µs p(95)=390.66µs + http_req_sending...............: avg=142.41µs min=8.49µs med=14.67µs max=73.99ms p(90)=35.65µs p(95)=183.71µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=3.29s min=11.37ms med=1.85s max=1m0s p(90)=2.61s p(95)=3.03s + http_reqs......................: 5523 88.387569/s + iteration_duration.............: avg=3.32s min=352.6ms med=1.88s max=1m0s p(90)=2.63s p(95)=3.04s + iterations.....................: 5503 88.067498/s + vus............................: 90 min=90 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -230,19 +230,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -262,22 +262,22 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 15495 ✗ 0 - data_received..................: 455 MB 7.2 MB/s - data_sent......................: 6.2 MB 98 kB/s - http_req_blocked...............: avg=2.25ms min=1.52µs med=4.39µs max=76.27ms p(90)=6.91µs p(95)=20.38ms - http_req_connecting............: avg=2.2ms min=0s med=0s max=74.4ms p(90)=0s p(95)=20.17ms - http_req_duration..............: avg=3.45s min=17.51ms med=3.13s max=7.6s p(90)=5.13s p(95)=5.92s - { expected_response:true }...: avg=3.45s min=17.51ms med=3.13s max=7.6s p(90)=5.13s p(95)=5.92s - http_req_failed................: 0.00% ✓ 0 ✗ 5185 - http_req_receiving.............: avg=56.97ms min=41.5µs med=149.72µs max=1.98s p(90)=48.27ms p(95)=425.71ms - http_req_sending...............: avg=1.8ms min=8.04µs med=24.36µs max=745.92ms p(90)=154.47µs p(95)=7.01ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.39s min=17.36ms med=3.11s max=7.6s p(90)=5.11s p(95)=5.88s - http_reqs......................: 5185 82.472121/s - iteration_duration.............: avg=3.55s min=304.38ms med=3.19s max=7.66s p(90)=5.24s p(95)=6s - iterations.....................: 5165 82.154003/s - vus............................: 168 min=168 max=300 + checks.........................: 100.00% ✓ 16155 ✗ 0 + data_received..................: 474 MB 7.5 MB/s + data_sent......................: 6.4 MB 102 kB/s + http_req_blocked...............: avg=1.2ms min=1.47µs med=3.9µs max=82.87ms p(90)=6.44µs p(95)=950.4µs + http_req_connecting............: avg=1.16ms min=0s med=0s max=82.84ms p(90)=0s p(95)=456.01µs + http_req_duration..............: avg=3.3s min=15.93ms med=2.94s max=7.36s p(90)=5.15s p(95)=5.56s + { expected_response:true }...: avg=3.3s min=15.93ms med=2.94s max=7.36s p(90)=5.15s p(95)=5.56s + http_req_failed................: 0.00% ✓ 0 ✗ 5405 + http_req_receiving.............: avg=55.08ms min=39.17µs med=103µs max=1.95s p(90)=47.5ms p(95)=453.94ms + http_req_sending...............: avg=1.84ms min=9.11µs med=20.06µs max=798.33ms p(90)=128.87µs p(95)=3.94ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=3.25s min=15.69ms med=2.9s max=7.36s p(90)=5.1s p(95)=5.49s + http_reqs......................: 5405 85.551833/s + iteration_duration.............: avg=3.43s min=146.48ms med=3.03s max=7.37s p(90)=5.34s p(95)=5.78s + iterations.....................: 5385 85.235267/s + vus............................: 55 min=55 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -285,19 +285,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -312,29 +312,29 @@ This scenario was running 300 VUs over 60s ``` ✗ response code was 200 - ↳ 99% — ✓ 4868 / ✗ 19 + ↳ 99% — ✓ 5052 / ✗ 13 ✗ no graphql errors - ↳ 99% — ✓ 4868 / ✗ 19 + ↳ 99% — ✓ 5052 / ✗ 13 ✓ valid response structure █ setup - checks.........................: 99.74% ✓ 14604 ✗ 38 - data_received..................: 429 MB 6.8 MB/s - data_sent......................: 5.8 MB 92 kB/s - http_req_blocked...............: avg=849.92µs min=1.53µs med=4.03µs max=40.8ms p(90)=6.29µs p(95)=951.16µs - http_req_connecting............: avg=811.9µs min=0s med=0s max=38.29ms p(90)=0s p(95)=517.78µs - http_req_duration..............: avg=3.72s min=15.01ms med=2.55s max=1m0s p(90)=3.62s p(95)=5.48s - { expected_response:true }...: avg=3.5s min=15.01ms med=2.54s max=59.51s p(90)=3.57s p(95)=5.29s - http_req_failed................: 0.38% ✓ 19 ✗ 4888 - http_req_receiving.............: avg=856.56µs min=0s med=109.49µs max=176.99ms p(90)=469.02µs p(95)=1.39ms - http_req_sending...............: avg=457.02µs min=8.66µs med=22.74µs max=109.02ms p(90)=59.95µs p(95)=262.34µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.72s min=14.9ms med=2.54s max=1m0s p(90)=3.61s p(95)=5.48s - http_reqs......................: 4907 77.728414/s - iteration_duration.............: avg=3.76s min=208.29ms med=2.57s max=1m0s p(90)=3.65s p(95)=5.51s - iterations.....................: 4887 77.411608/s - vus............................: 40 min=40 max=300 + checks.........................: 99.82% ✓ 15156 ✗ 26 + data_received..................: 445 MB 7.0 MB/s + data_sent......................: 6.0 MB 95 kB/s + http_req_blocked...............: avg=1.07ms min=1.46µs med=3.57µs max=74.1ms p(90)=5.54µs p(95)=1.77ms + http_req_connecting............: avg=1.04ms min=0s med=0s max=67.62ms p(90)=0s p(95)=1.14ms + http_req_duration..............: avg=3.59s min=17.61ms med=2.52s max=1m0s p(90)=3.68s p(95)=5.1s + { expected_response:true }...: avg=3.45s min=17.61ms med=2.51s max=59.85s p(90)=3.66s p(95)=5.02s + http_req_failed................: 0.25% ✓ 13 ✗ 5072 + http_req_receiving.............: avg=369.41µs min=0s med=105.56µs max=47.35ms p(90)=423.84µs p(95)=1.22ms + http_req_sending...............: avg=216.23µs min=8.85µs med=20.48µs max=58.11ms p(90)=47.19µs p(95)=330.44µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=3.59s min=17.33ms med=2.52s max=1m0s p(90)=3.68s p(95)=5.1s + http_reqs......................: 5085 80.310172/s + iteration_duration.............: avg=3.63s min=585.53ms med=2.54s max=1m0s p(90)=3.73s p(95)=5.13s + iterations.....................: 5065 79.994301/s + vus............................: 43 min=43 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -342,19 +342,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -374,22 +374,22 @@ This scenario was running 300 VUs over 60s █ setup - checks.........................: 100.00% ✓ 14034 ✗ 0 - data_received..................: 412 MB 6.7 MB/s - data_sent......................: 5.6 MB 90 kB/s - http_req_blocked...............: avg=1.12ms min=1.75µs med=4.12µs max=68.63ms p(90)=6.34µs p(95)=5.07ms - http_req_connecting............: avg=1.07ms min=0s med=0s max=64.67ms p(90)=0s p(95)=4.64ms - http_req_duration..............: avg=3.85s min=11.66ms med=3.85s max=8.16s p(90)=4.67s p(95)=4.94s - { expected_response:true }...: avg=3.85s min=11.66ms med=3.85s max=8.16s p(90)=4.67s p(95)=4.94s - http_req_failed................: 0.00% ✓ 0 ✗ 4698 - http_req_receiving.............: avg=6.94ms min=41.4µs med=108.25µs max=829.46ms p(90)=297.74µs p(95)=708.37µs - http_req_sending...............: avg=795.2µs min=9.81µs med=23.08µs max=64.12ms p(90)=46.58µs p(95)=1.84ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.85s min=11.56ms med=3.85s max=8.16s p(90)=4.67s p(95)=4.94s - http_reqs......................: 4698 75.873186/s - iteration_duration.............: avg=3.9s min=471.52ms med=3.87s max=8.25s p(90)=4.68s p(95)=4.96s - iterations.....................: 4678 75.550184/s - vus............................: 229 min=229 max=300 + checks.........................: 100.00% ✓ 14403 ✗ 0 + data_received..................: 423 MB 6.8 MB/s + data_sent......................: 5.7 MB 92 kB/s + http_req_blocked...............: avg=564.75µs min=1.72µs med=3.67µs max=28.36ms p(90)=5.59µs p(95)=3.67ms + http_req_connecting............: avg=540.03µs min=0s med=0s max=17.49ms p(90)=0s p(95)=3.32ms + http_req_duration..............: avg=3.76s min=11.56ms med=3.76s max=7.44s p(90)=4.61s p(95)=4.84s + { expected_response:true }...: avg=3.76s min=11.56ms med=3.76s max=7.44s p(90)=4.61s p(95)=4.84s + http_req_failed................: 0.00% ✓ 0 ✗ 4821 + http_req_receiving.............: avg=1.09ms min=37.51µs med=104.59µs max=196.83ms p(90)=280.61µs p(95)=653.53µs + http_req_sending...............: avg=136.6µs min=8.54µs med=20.9µs max=109.98ms p(90)=42.29µs p(95)=198.92µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=3.76s min=11.45ms med=3.76s max=7.44s p(90)=4.61s p(95)=4.84s + http_reqs......................: 4821 77.681115/s + iteration_duration.............: avg=3.8s min=469.42ms med=3.77s max=7.44s p(90)=4.63s p(95)=4.86s + iterations.....................: 4801 77.358853/s + vus............................: 59 min=59 max=300 vus_max........................: 300 min=300 max=300 ``` @@ -397,19 +397,19 @@ This scenario was running 300 VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview \ No newline at end of file diff --git a/federation/scenarios/ramping-vus/README.md b/federation/scenarios/ramping-vus/README.md index 8f8fe797..5531ae3b 100644 --- a/federation/scenarios/ramping-vus/README.md +++ b/federation/scenarios/ramping-vus/README.md @@ -10,23 +10,23 @@ This scenario was trying to reach 2000 concurrent VUs over 60s ### Comparison -Comparison +Comparison -| Gateway | duration(p95)⬇️ | RPS | Requests | Durations | Notes | -| :--------------- | :-------------: | :---: | :---------------------: | :----------------------------------------------------: | :-------------------------------------------------------------------------- | -| cosmo | 6431ms | 171 | 12304 total, 0 failed | avg: 2742ms, p95: 6431ms, max: 14137ms, med: 2177ms | ✅ | -| apollo-router | 6823ms | 162 | 11460 total, 790 failed | avg: 2948ms, p95: 6823ms, max: 15587ms, med: 2767ms | ❌ 790 failed requests, 790 non-200 responses, 790 unexpected GraphQL errors | -| grafbase | 11738ms | 155 | 10907 total, 0 failed | avg: 3728ms, p95: 11739ms, max: 19528ms, med: 2329ms | ✅ | -| hive-gateway-bun | 29054ms | 88 | 7293 total, 0 failed | avg: 11810ms, p95: 29054ms, max: 39625ms, med: 10479ms | ✅ | -| mercurius | 42189ms | 52 | 4870 total, 0 failed | avg: 22296ms, p95: 42190ms, max: 43265ms, med: 21022ms | ✅ | -| hive-gateway | 53846ms | 79 | 7159 total, 0 failed | avg: 13421ms, p95: 53847ms, max: 58939ms, med: 3535ms | ✅ | -| apollo-server | 59999ms | 80 | 7423 total, 544 failed | avg: 12559ms, p95: 60000ms, max: 60194ms, med: 2296ms | ❌ 544 failed requests, 544 non-200 responses, 544 unexpected GraphQL errors | +| Gateway | duration(p95)⬇️ | Successful RPS | Requests | Durations | Notes | +| :--------------- | :-------------: | :------------: | :---------------------: | :----------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------- | +| apollo-router | 6463ms | 155 | 11448 total, 518 failed | avg: 2738ms, p95: 6463ms, max: 22126ms, med: 2552ms | ❌ 518 failed requests, 518 non-200 responses, 520 unexpected GraphQL errors, non-compatible response structure (3) | +| grafbase | 7506ms | 166 | 11690 total, 0 failed | avg: 3252ms, p95: 7506ms, max: 14963ms, med: 2937ms | ✅ | +| cosmo | 10942ms | 175 | 12323 total, 0 failed | avg: 2760ms, p95: 10942ms, max: 20059ms, med: 1930ms | ✅ | +| hive-gateway-bun | 29504ms | 85 | 7050 total, 0 failed | avg: 12182ms, p95: 29504ms, max: 36030ms, med: 10733ms | ✅ | +| mercurius | 40437ms | 55 | 5016 total, 0 failed | avg: 21529ms, p95: 40437ms, max: 41800ms, med: 20197ms | ✅ | +| hive-gateway | 54130ms | 77 | 7010 total, 0 failed | avg: 13754ms, p95: 54130ms, max: 59528ms, med: 3635ms | ✅ | +| apollo-server | 59999ms | 75 | 7517 total, 529 failed | avg: 12284ms, p95: 60000ms, max: 60028ms, med: 2240ms | ❌ 529 failed requests, 529 non-200 responses, 529 unexpected GraphQL errors |
- Summary for: `cosmo` + Summary for: `apollo-router` **K6 Output** @@ -34,54 +34,57 @@ This scenario was trying to reach 2000 concurrent VUs over 60s ``` - ✓ response code was 200 - ✓ no graphql errors - ✓ valid response structure + ✗ response code was 200 + ↳ 95% — ✓ 10910 / ✗ 518 + ✗ no graphql errors + ↳ 95% — ✓ 10908 / ✗ 520 + ✗ valid response structure + ↳ 99% — ✓ 10908 / ✗ 3 █ setup - checks.........................: 100.00% ✓ 36852 ✗ 0 - data_received..................: 1.1 GB 15 MB/s - data_sent......................: 15 MB 204 kB/s - http_req_blocked...............: avg=304.96ms min=1.69µs med=4.04µs max=10.53s p(90)=1.1s p(95)=2.61s - http_req_connecting............: avg=301.04ms min=0s med=0s max=10.53s p(90)=1.09s p(95)=2.59s - http_req_duration..............: avg=2.74s min=3.29ms med=2.17s max=14.13s p(90)=5.78s p(95)=6.43s - { expected_response:true }...: avg=2.74s min=3.29ms med=2.17s max=14.13s p(90)=5.78s p(95)=6.43s - http_req_failed................: 0.00% ✓ 0 ✗ 12304 - http_req_receiving.............: avg=375.4ms min=31.55µs med=90.65µs max=10.56s p(90)=1.13s p(95)=3.9s - http_req_sending...............: avg=154.52ms min=7.58µs med=20.96µs max=7.87s p(90)=317.32ms p(95)=763.91ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=2.21s min=3.12ms med=1.64s max=10.55s p(90)=5.21s p(95)=5.77s - http_reqs......................: 12304 171.935282/s - iteration_duration.............: avg=6.09s min=21.71ms med=5.14s max=28.37s p(90)=12.51s p(95)=15.19s - iterations.....................: 12284 171.655803/s - vus............................: 379 min=60 max=2000 - vus_max........................: 2000 min=2000 max=2000 + checks.........................: 96.91% ✓ 32726 ✗ 1041 + data_received..................: 959 MB 14 MB/s + data_sent......................: 14 MB 194 kB/s + http_req_blocked...............: avg=645.34ms min=1.7µs med=4.26µs max=12.01s p(90)=2.75s p(95)=5.16s + http_req_connecting............: avg=500.73ms min=0s med=0s max=8.96s p(90)=2.14s p(95)=3.35s + http_req_duration..............: avg=2.73s min=6.96ms med=2.55s max=22.12s p(90)=5.6s p(95)=6.46s + { expected_response:true }...: avg=2.73s min=6.96ms med=2.55s max=22.12s p(90)=5.66s p(95)=6.46s + http_req_failed................: 4.52% ✓ 518 ✗ 10930 + http_req_receiving.............: avg=392.79ms min=0s med=88.14µs max=8.62s p(90)=1.12s p(95)=2.33s + http_req_sending...............: avg=146ms min=7.37µs med=22.88µs max=16.27s p(90)=314.56ms p(95)=757.67ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=2.19s min=6.86ms med=2.05s max=10.5s p(90)=4.13s p(95)=4.86s + http_reqs......................: 11448 163.07624/s + iteration_duration.............: avg=6.24s min=8.98ms med=5.18s max=33.63s p(90)=12.92s p(95)=15.7s + iterations.....................: 11428 162.791341/s + vus............................: 6 min=6 max=1928 + vus_max........................: 2000 min=2000 max=2000 ``` **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview
- Summary for: `apollo-router` + Summary for: `grafbase` **K6 Output** @@ -89,56 +92,54 @@ This scenario was trying to reach 2000 concurrent VUs over 60s ``` - ✗ response code was 200 - ↳ 93% — ✓ 10650 / ✗ 790 - ✗ no graphql errors - ↳ 93% — ✓ 10650 / ✗ 790 + ✓ response code was 200 + ✓ no graphql errors ✓ valid response structure █ setup - checks.........................: 95.28% ✓ 31950 ✗ 1580 - data_received..................: 936 MB 13 MB/s - data_sent......................: 14 MB 193 kB/s - http_req_blocked...............: avg=644.39ms min=1.7µs med=4.39µs max=12.04s p(90)=3.14s p(95)=3.89s - http_req_connecting............: avg=639.29ms min=0s med=0s max=12.04s p(90)=3.13s p(95)=3.86s - http_req_duration..............: avg=2.94s min=6.87ms med=2.76s max=15.58s p(90)=5.44s p(95)=6.82s - { expected_response:true }...: avg=2.9s min=6.87ms med=2.72s max=15.58s p(90)=5.29s p(95)=6.84s - http_req_failed................: 6.89% ✓ 790 ✗ 10670 - http_req_receiving.............: avg=344.27ms min=0s med=93.17µs max=11.12s p(90)=1.27s p(95)=2.42s - http_req_sending...............: avg=192.54ms min=8.35µs med=23.74µs max=9.58s p(90)=656.31ms p(95)=1.04s - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=2.41s min=6.74ms med=2.14s max=12.91s p(90)=4.56s p(95)=5.19s - http_reqs......................: 11460 162.645424/s - iteration_duration.............: avg=6.34s min=26.78ms med=5.65s max=27.13s p(90)=12.61s p(95)=14.73s - iterations.....................: 11440 162.361575/s - vus............................: 387 min=74 max=1984 - vus_max........................: 2000 min=2000 max=2000 + checks.........................: 100.00% ✓ 35010 ✗ 0 + data_received..................: 1.0 GB 15 MB/s + data_sent......................: 14 MB 198 kB/s + http_req_blocked...............: avg=427.77ms min=1.6µs med=3.86µs max=13.97s p(90)=1.68s p(95)=3.91s + http_req_connecting............: avg=424.96ms min=0s med=0s max=10.76s p(90)=1.67s p(95)=3.91s + http_req_duration..............: avg=3.25s min=3.51ms med=2.93s max=14.96s p(90)=5.68s p(95)=7.5s + { expected_response:true }...: avg=3.25s min=3.51ms med=2.93s max=14.96s p(90)=5.68s p(95)=7.5s + http_req_failed................: 0.00% ✓ 0 ✗ 11690 + http_req_receiving.............: avg=136.44ms min=29.69µs med=78.33µs max=7.7s p(90)=235.6ms p(95)=926.48ms + http_req_sending...............: avg=164.69ms min=7.86µs med=19.15µs max=9.4s p(90)=392.66ms p(95)=777.7ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=2.95s min=3.43ms med=2.59s max=10.94s p(90)=5.34s p(95)=5.87s + http_reqs......................: 11690 166.764715/s + iteration_duration.............: avg=6.18s min=9.24ms med=5.17s max=28.53s p(90)=11.73s p(95)=14.52s + iterations.....................: 11670 166.479403/s + vus............................: 179 min=66 max=1946 + vus_max........................: 2000 min=2000 max=2000 ``` **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview
- Summary for: `grafbase` + Summary for: `cosmo` **K6 Output** @@ -152,22 +153,22 @@ This scenario was trying to reach 2000 concurrent VUs over 60s █ setup - checks.........................: 100.00% ✓ 32661 ✗ 0 - data_received..................: 959 MB 14 MB/s - data_sent......................: 13 MB 185 kB/s - http_req_blocked...............: avg=296.72ms min=1.76µs med=4.1µs max=10.84s p(90)=777.22ms p(95)=2.05s - http_req_connecting............: avg=293.04ms min=0s med=0s max=10.84s p(90)=759.78ms p(95)=2.05s - http_req_duration..............: avg=3.72s min=3.21ms med=2.32s max=19.52s p(90)=9.71s p(95)=11.73s - { expected_response:true }...: avg=3.72s min=3.21ms med=2.32s max=19.52s p(90)=9.71s p(95)=11.73s - http_req_failed................: 0.00% ✓ 0 ✗ 10907 - http_req_receiving.............: avg=270.67ms min=32.47µs med=86.3µs max=10.31s p(90)=683.15ms p(95)=1.94s - http_req_sending...............: avg=94.56ms min=7.6µs med=22.18µs max=7.99s p(90)=192.5ms p(95)=551.25ms + checks.........................: 100.00% ✓ 36909 ✗ 0 + data_received..................: 1.1 GB 15 MB/s + data_sent......................: 15 MB 209 kB/s + http_req_blocked...............: avg=327.71ms min=1.68µs med=3.73µs max=17.11s p(90)=844.84ms p(95)=1.89s + http_req_connecting............: avg=320.32ms min=0s med=0s max=17.05s p(90)=842.98ms p(95)=1.89s + http_req_duration..............: avg=2.76s min=3.38ms med=1.93s max=20.05s p(90)=5.77s p(95)=10.94s + { expected_response:true }...: avg=2.76s min=3.38ms med=1.93s max=20.05s p(90)=5.77s p(95)=10.94s + http_req_failed................: 0.00% ✓ 0 ✗ 12323 + http_req_receiving.............: avg=866.24ms min=33.39µs med=88.62µs max=17.42s p(90)=2.31s p(95)=8.22s + http_req_sending...............: avg=206.63ms min=8.05µs med=17.72µs max=13.48s p(90)=362.9ms p(95)=804.37ms http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=3.36s min=3.13ms med=1.95s max=19.24s p(90)=9.49s p(95)=11.44s - http_reqs......................: 10907 155.576551/s - iteration_duration.............: avg=6.52s min=13.13ms med=5.28s max=30.22s p(90)=14.22s p(95)=17.36s - iterations.....................: 10887 155.291273/s - vus............................: 45 min=45 max=1988 + http_req_waiting...............: avg=1.68s min=3.15ms med=1.5s max=12.73s p(90)=3.27s p(95)=3.93s + http_reqs......................: 12323 175.668787/s + iteration_duration.............: avg=5.78s min=9.12ms med=4.37s max=30.7s p(90)=13.62s p(95)=16.5s + iterations.....................: 12303 175.38368/s + vus............................: 90 min=71 max=1986 vus_max........................: 2000 min=2000 max=2000 ``` @@ -175,19 +176,19 @@ This scenario was trying to reach 2000 concurrent VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview
@@ -207,22 +208,22 @@ This scenario was trying to reach 2000 concurrent VUs over 60s █ setup - checks.........................: 100.00% ✓ 21819 ✗ 0 - data_received..................: 640 MB 7.8 MB/s - data_sent......................: 8.7 MB 105 kB/s - http_req_blocked...............: avg=12.06ms min=1.81µs med=4.74µs max=611.45ms p(90)=14.55ms p(95)=77.51ms - http_req_connecting............: avg=11.88ms min=0s med=0s max=611.34ms p(90)=14.15ms p(95)=76.08ms - http_req_duration..............: avg=11.81s min=16.83ms med=10.47s max=39.62s p(90)=25.43s p(95)=29.05s - { expected_response:true }...: avg=11.81s min=16.83ms med=10.47s max=39.62s p(90)=25.43s p(95)=29.05s - http_req_failed................: 0.00% ✓ 0 ✗ 7293 - http_req_receiving.............: avg=159.02ms min=41.74µs med=156.71µs max=7.5s p(90)=17.99ms p(95)=971.42ms - http_req_sending...............: avg=10.02ms min=9.38µs med=27.22µs max=2.24s p(90)=5.53ms p(95)=28.44ms + checks.........................: 100.00% ✓ 21090 ✗ 0 + data_received..................: 619 MB 7.5 MB/s + data_sent......................: 8.4 MB 102 kB/s + http_req_blocked...............: avg=14ms min=1.95µs med=5.01µs max=764.09ms p(90)=19.91ms p(95)=98.82ms + http_req_connecting............: avg=13.8ms min=0s med=0s max=557.93ms p(90)=19.57ms p(95)=98.43ms + http_req_duration..............: avg=12.18s min=16.46ms med=10.73s max=36.03s p(90)=26.14s p(95)=29.5s + { expected_response:true }...: avg=12.18s min=16.46ms med=10.73s max=36.03s p(90)=26.14s p(95)=29.5s + http_req_failed................: 0.00% ✓ 0 ✗ 7050 + http_req_receiving.............: avg=109.64ms min=41.06µs med=142.89µs max=7.69s p(90)=21.42ms p(95)=508.69ms + http_req_sending...............: avg=9.35ms min=9.62µs med=30.54µs max=1.23s p(90)=13.3ms p(95)=48.4ms http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=11.64s min=16.72ms med=10.41s max=39.62s p(90)=25.24s p(95)=29.03s - http_reqs......................: 7293 88.424772/s - iteration_duration.............: avg=12.07s min=165.82ms med=10.59s max=40.84s p(90)=26.07s p(95)=29.47s - iterations.....................: 7273 88.18228/s - vus............................: 93 min=0 max=1999 + http_req_waiting...............: avg=12.06s min=16.26ms med=10.65s max=36s p(90)=26.03s p(95)=29.46s + http_reqs......................: 7050 85.704/s + iteration_duration.............: avg=12.46s min=102.94ms med=10.92s max=38.06s p(90)=26.65s p(95)=30.02s + iterations.....................: 7030 85.460868/s + vus............................: 69 min=53 max=2000 vus_max........................: 2000 min=2000 max=2000 ``` @@ -230,19 +231,19 @@ This scenario was trying to reach 2000 concurrent VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -262,22 +263,22 @@ This scenario was trying to reach 2000 concurrent VUs over 60s █ setup - checks.........................: 100.00% ✓ 14394 ✗ 0 - data_received..................: 428 MB 4.6 MB/s - data_sent......................: 5.9 MB 64 kB/s - http_req_blocked...............: avg=316.03µs min=1.91µs med=5.26µs max=36.08ms p(90)=616.37µs p(95)=1ms - http_req_connecting............: avg=278.85µs min=0s med=0s max=35.92ms p(90)=533.25µs p(95)=866.61µs - http_req_duration..............: avg=22.29s min=12.2ms med=21.02s max=43.26s p(90)=41.43s p(95)=42.18s - { expected_response:true }...: avg=22.29s min=12.2ms med=21.02s max=43.26s p(90)=41.43s p(95)=42.18s - http_req_failed................: 0.00% ✓ 0 ✗ 4870 - http_req_receiving.............: avg=4.29ms min=39.32µs med=112.28µs max=1.53s p(90)=373.32µs p(95)=872.93µs - http_req_sending...............: avg=77.07µs min=9.23µs med=33.23µs max=18.52ms p(90)=81.89µs p(95)=139.76µs + checks.........................: 100.00% ✓ 14988 ✗ 0 + data_received..................: 440 MB 4.8 MB/s + data_sent......................: 6.0 MB 66 kB/s + http_req_blocked...............: avg=269.74µs min=1.66µs med=4.55µs max=12.52ms p(90)=597.41µs p(95)=982.66µs + http_req_connecting............: avg=231.8µs min=0s med=0s max=12.44ms p(90)=508.04µs p(95)=846µs + http_req_duration..............: avg=21.52s min=12.35ms med=20.19s max=41.79s p(90)=39.85s p(95)=40.43s + { expected_response:true }...: avg=21.52s min=12.35ms med=20.19s max=41.79s p(90)=39.85s p(95)=40.43s + http_req_failed................: 0.00% ✓ 0 ✗ 5016 + http_req_receiving.............: avg=140.99ms min=38.1µs med=120.12µs max=2.56s p(90)=416.64ms p(95)=1.1s + http_req_sending...............: avg=67.13µs min=8.68µs med=28.91µs max=18.09ms p(90)=76.12µs p(95)=101.78µs http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=22.29s min=12.08ms med=21.02s max=43.26s p(90)=41.38s p(95)=42.18s - http_reqs......................: 4870 52.892721/s - iteration_duration.............: avg=22.21s min=153.24ms med=20.02s max=43.37s p(90)=41.62s p(95)=42.39s - iterations.....................: 4745 51.535105/s - vus............................: 72 min=56 max=1999 + http_req_waiting...............: avg=21.38s min=12.25ms med=20.19s max=41.41s p(90)=39.76s p(95)=40.41s + http_reqs......................: 5016 55.167103/s + iteration_duration.............: avg=21.77s min=293.73ms med=20.25s max=43.37s p(90)=40.18s p(95)=40.83s + iterations.....................: 4996 54.947139/s + vus............................: 87 min=57 max=2000 vus_max........................: 2000 min=2000 max=2000 ``` @@ -285,19 +286,19 @@ This scenario was trying to reach 2000 concurrent VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -317,22 +318,22 @@ This scenario was trying to reach 2000 concurrent VUs over 60s █ setup - checks.........................: 100.00% ✓ 21417 ✗ 0 - data_received..................: 629 MB 7.0 MB/s - data_sent......................: 8.5 MB 94 kB/s - http_req_blocked...............: avg=2.6ms min=2.17µs med=4.67µs max=190.95ms p(90)=1.91ms p(95)=12.39ms - http_req_connecting............: avg=2.54ms min=0s med=0s max=190.75ms p(90)=1.65ms p(95)=11.98ms - http_req_duration..............: avg=13.42s min=15.4ms med=3.53s max=58.93s p(90)=47.47s p(95)=53.84s - { expected_response:true }...: avg=13.42s min=15.4ms med=3.53s max=58.93s p(90)=47.47s p(95)=53.84s - http_req_failed................: 0.00% ✓ 0 ✗ 7159 - http_req_receiving.............: avg=1.04ms min=41.47µs med=113.94µs max=1.01s p(90)=849.72µs p(95)=2.72ms - http_req_sending...............: avg=1.08ms min=9.26µs med=28.51µs max=120.12ms p(90)=168.62µs p(95)=2.87ms - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=13.41s min=15.28ms med=3.53s max=58.93s p(90)=47.44s p(95)=53.84s - http_reqs......................: 7159 79.350407/s - iteration_duration.............: avg=13.49s min=110.38ms med=3.58s max=58.99s p(90)=47.59s p(95)=53.93s - iterations.....................: 7139 79.128726/s - vus............................: 53 min=53 max=2000 + checks.........................: 100.00% ✓ 20970 ✗ 0 + data_received..................: 616 MB 6.8 MB/s + data_sent......................: 8.3 MB 92 kB/s + http_req_blocked...............: avg=2.23ms min=1.91µs med=4.47µs max=180.19ms p(90)=1.24ms p(95)=9.37ms + http_req_connecting............: avg=2.19ms min=0s med=0s max=180.11ms p(90)=1ms p(95)=9.19ms + http_req_duration..............: avg=13.75s min=15.68ms med=3.63s max=59.52s p(90)=48s p(95)=54.13s + { expected_response:true }...: avg=13.75s min=15.68ms med=3.63s max=59.52s p(90)=48s p(95)=54.13s + http_req_failed................: 0.00% ✓ 0 ✗ 7010 + http_req_receiving.............: avg=562.2µs min=43.63µs med=117.91µs max=84.29ms p(90)=891.49µs p(95)=2.58ms + http_req_sending...............: avg=864.45µs min=10.57µs med=28.31µs max=121.4ms p(90)=150.01µs p(95)=2.21ms + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=13.75s min=15.57ms med=3.63s max=59.52s p(90)=48s p(95)=54.12s + http_reqs......................: 7010 77.715875/s + iteration_duration.............: avg=13.82s min=96.87ms med=3.67s max=59.59s p(90)=48.03s p(95)=54.18s + iterations.....................: 6990 77.494146/s + vus............................: 43 min=43 max=2000 vus_max........................: 2000 min=2000 max=2000 ``` @@ -340,19 +341,19 @@ This scenario was trying to reach 2000 concurrent VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview @@ -367,29 +368,29 @@ This scenario was trying to reach 2000 concurrent VUs over 60s ``` ✗ response code was 200 - ↳ 92% — ✓ 6859 / ✗ 544 + ↳ 92% — ✓ 6968 / ✗ 529 ✗ no graphql errors - ↳ 92% — ✓ 6859 / ✗ 544 + ↳ 92% — ✓ 6968 / ✗ 529 ✓ valid response structure █ setup - checks.........................: 94.97% ✓ 20577 ✗ 1088 - data_received..................: 605 MB 6.5 MB/s - data_sent......................: 8.9 MB 96 kB/s - http_req_blocked...............: avg=543.77µs min=1.48µs med=4.24µs max=100.96ms p(90)=449.77µs p(95)=845.45µs - http_req_connecting............: avg=515.98µs min=0s med=0s max=100.9ms p(90)=365.13µs p(95)=695.19µs - http_req_duration..............: avg=12.55s min=11.81ms med=2.29s max=1m0s p(90)=56.99s p(95)=59.99s - { expected_response:true }...: avg=8.8s min=11.81ms med=2.2s max=59.99s p(90)=37.38s p(95)=48.84s - http_req_failed................: 7.32% ✓ 544 ✗ 6879 - http_req_receiving.............: avg=277.21µs min=0s med=113.9µs max=271.71ms p(90)=232.29µs p(95)=363.22µs - http_req_sending...............: avg=253.79µs min=9.35µs med=23.67µs max=96.9ms p(90)=74.33µs p(95)=152.95µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=12.55s min=11.67ms med=2.29s max=1m0s p(90)=56.99s p(95)=59.99s - http_reqs......................: 7423 80.049421/s - iteration_duration.............: avg=12.6s min=63.69ms med=2.31s max=1m0s p(90)=57.03s p(95)=1m0s - iterations.....................: 7403 79.833741/s - vus............................: 74 min=61 max=2000 + checks.........................: 95.18% ✓ 20904 ✗ 1058 + data_received..................: 614 MB 6.6 MB/s + data_sent......................: 9.0 MB 97 kB/s + http_req_blocked...............: avg=602.43µs min=1.44µs med=3.63µs max=98.6ms p(90)=426.53µs p(95)=825.46µs + http_req_connecting............: avg=561.23µs min=0s med=0s max=98.05ms p(90)=352.46µs p(95)=715.92µs + http_req_duration..............: avg=12.28s min=10.46ms med=2.24s max=1m0s p(90)=56.42s p(95)=59.99s + { expected_response:true }...: avg=8.67s min=10.46ms med=2.16s max=59.98s p(90)=37.57s p(95)=47.59s + http_req_failed................: 7.03% ✓ 529 ✗ 6988 + http_req_receiving.............: avg=306.24µs min=0s med=110.11µs max=283.5ms p(90)=238.36µs p(95)=413.99µs + http_req_sending...............: avg=297.71µs min=8.76µs med=19.22µs max=88.16ms p(90)=71.05µs p(95)=142.65µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=12.28s min=10.36ms med=2.23s max=1m0s p(90)=56.42s p(95)=59.99s + http_reqs......................: 7517 81.108159/s + iteration_duration.............: avg=12.33s min=57.59ms med=2.26s max=1m0s p(90)=56.47s p(95)=1m0s + iterations.....................: 7497 80.89236/s + vus............................: 84 min=62 max=2000 vus_max........................: 2000 min=2000 max=2000 ``` @@ -397,19 +398,19 @@ This scenario was trying to reach 2000 concurrent VUs over 60s **Performance Overview** -Performance Overview +Performance Overview **Subgraphs Overview** -Subgraphs Overview +Subgraphs Overview **HTTP Overview** -HTTP Overview +HTTP Overview \ No newline at end of file diff --git a/federation/scenarios/ramping-vus/generate-report.ts b/federation/scenarios/ramping-vus/generate-report.ts index fd391abe..aa874496 100644 --- a/federation/scenarios/ramping-vus/generate-report.ts +++ b/federation/scenarios/ramping-vus/generate-report.ts @@ -115,13 +115,19 @@ async function generateReport(artifactsRootPath: string) { } const jsonSummary = JSON.parse(readFileSync(jsonSummaryFilePath, "utf8")); + const succReqs = + jsonSummary.metrics.http_reqs.values.count - + jsonSummary.metrics.http_req_failed.values.passes; return { name: dirName.replace(process.env.SCENARIO_ARTIFACTS_PREFIX!, ""), path: fullPath, jsonSummary, txtSummary: readFileSync(txtSummaryFilePath, "utf8"), - rps: Math.floor(jsonSummary.metrics.http_reqs.values.rate), + rps: Math.floor( + // calculate only successful requests + succReqs / (jsonSummary.state.testRunDurationMs / 1000) + ), p95_duration: Math.floor( jsonSummary.metrics.http_req_duration.values["p(95)"] ), @@ -245,7 +251,7 @@ async function generateReport(artifactsRootPath: string) { columns: [ { name: "Gateway" }, { name: "duration(p95)⬇️", align: "center" }, - { name: "RPS", align: "center" }, + { name: "Successful RPS", align: "center" }, { name: "Requests", align: "center" }, { name: "Durations", align: "center" }, { name: "Notes", align: "left" },