Conversation
BenchmarksBenchmark execution time: 2026-02-18 15:21:03 Comparing candidate commit 01fa01b in PR branch Found 5 performance improvements and 12 performance regressions! Performance is the same for 164 metrics, 11 unstable metrics. scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1
scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0
scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1
scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8208) and master.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 193.89 ± (193.88 - 194.93) ms | 209.88 ± (210.96 - 213.27) ms | +8.2% | ❌⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 197.40 ± (197.15 - 197.74) ms | 212.03 ± (212.87 - 215.26) ms | +7.4% | ❌⬆️ |
Full Metrics Comparison
FakeDbCommand
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 74.82 ± (74.79 - 75.07) ms | 77.16 ± (77.17 - 77.59) ms | +3.1% | ✅⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 79.76 ± (79.59 - 79.98) ms | 81.63 ± (81.48 - 81.87) ms | +2.3% | ✅⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1078.54 ± (1078.43 - 1083.88) ms | 1093.38 ± (1093.28 - 1099.30) ms | +1.4% | ✅⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 22.73 ± (22.68 - 22.77) ms | 23.17 ± (23.11 - 23.23) ms | +2.0% | ✅⬆️ |
| process.time_to_main_ms | 86.01 ± (85.82 - 86.21) ms | 88.95 ± (88.75 - 89.14) ms | +3.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.92 ± (10.92 - 10.93) MB | 10.90 ± (10.90 - 10.91) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 12 ± (12 - 12) | 12 ± (12 - 12) | +0.0% | ✅ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 22.77 ± (22.71 - 22.83) ms | 23.09 ± (23.04 - 23.14) ms | +1.4% | ✅⬆️ |
| process.time_to_main_ms | 87.66 ± (87.47 - 87.86) ms | 89.78 ± (89.55 - 90.00) ms | +2.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.96 ± (10.96 - 10.97) MB | 10.94 ± (10.94 - 10.94) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 13 ± (13 - 13) | 13 ± (13 - 13) | +0.0% | ✅ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 240.65 ± (236.72 - 244.57) ms | 235.56 ± (231.80 - 239.32) ms | -2.1% | ✅ |
| process.time_to_main_ms | 492.60 ± (491.86 - 493.35) ms | 505.03 ± (504.30 - 505.76) ms | +2.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 47.62 ± (47.60 - 47.64) MB | 47.51 ± (47.49 - 47.54) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.0% | ✅ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 21.60 ± (21.57 - 21.64) ms | 22.14 ± (22.09 - 22.19) ms | +2.5% | ✅⬆️ |
| process.time_to_main_ms | 75.50 ± (75.36 - 75.64) ms | 77.94 ± (77.75 - 78.12) ms | +3.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.61 ± (10.61 - 10.62) MB | 10.63 ± (10.62 - 10.63) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 21.68 ± (21.63 - 21.73) ms | 22.28 ± (22.23 - 22.34) ms | +2.8% | ✅⬆️ |
| process.time_to_main_ms | 77.03 ± (76.84 - 77.21) ms | 80.41 ± (80.23 - 80.60) ms | +4.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.71 ± (10.70 - 10.72) MB | 10.74 ± (10.73 - 10.74) MB | +0.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 244.85 ± (240.65 - 249.05) ms | 243.17 ± (238.97 - 247.37) ms | -0.7% | ✅ |
| process.time_to_main_ms | 476.39 ± (475.76 - 477.01) ms | 484.03 ± (483.32 - 484.73) ms | +1.6% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 48.29 ± (48.27 - 48.32) MB | 48.24 ± (48.22 - 48.26) MB | -0.1% | ✅ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.0% | ✅ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 19.84 ± (19.80 - 19.89) ms | 20.37 ± (20.32 - 20.42) ms | +2.7% | ✅⬆️ |
| process.time_to_main_ms | 74.91 ± (74.75 - 75.06) ms | 77.74 ± (77.56 - 77.91) ms | +3.8% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.67 ± (7.66 - 7.68) MB | 7.68 ± (7.68 - 7.69) MB | +0.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 19.98 ± (19.93 - 20.03) ms | 20.31 ± (20.26 - 20.36) ms | +1.6% | ✅⬆️ |
| process.time_to_main_ms | 76.68 ± (76.52 - 76.84) ms | 78.48 ± (78.30 - 78.66) ms | +2.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.71 ± (7.70 - 7.71) MB | 7.73 ± (7.72 - 7.74) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 191.65 ± (190.81 - 192.49) ms | 195.10 ± (194.24 - 195.97) ms | +1.8% | ✅⬆️ |
| process.time_to_main_ms | 457.69 ± (456.92 - 458.47) ms | 469.13 ± (468.20 - 470.07) ms | +2.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 36.18 ± (36.13 - 36.23) MB | 36.22 ± (36.16 - 36.28) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 27 ± (27 - 27) | 27 ± (27 - 27) | -0.2% | ✅ |
HttpMessageHandler
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 193.89 ± (193.88 - 194.93) ms | 209.88 ± (210.96 - 213.27) ms | +8.2% | ❌⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 197.40 ± (197.15 - 197.74) ms | 212.03 ± (212.87 - 215.26) ms | +7.4% | ❌⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1154.48 ± (1159.64 - 1170.20) ms | 1208.76 ± (1208.04 - 1218.81) ms | +4.7% | ✅⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 190.94 ± (190.48 - 191.39) ms | 210.09 ± (208.46 - 211.71) ms | +10.0% | ✅⬆️ |
| process.time_to_main_ms | 82.45 ± (82.24 - 82.66) ms | 88.28 ± (87.74 - 88.81) ms | +7.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.07 ± (16.05 - 16.10) MB | 15.87 ± (15.85 - 15.88) MB | -1.3% | ✅ |
| runtime.dotnet.threads.count | 20 ± (19 - 20) | 20 ± (20 - 20) | +2.4% | ✅⬆️ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 189.01 ± (188.60 - 189.42) ms | 201.74 ± (200.47 - 203.02) ms | +6.7% | ✅⬆️ |
| process.time_to_main_ms | 83.44 ± (83.26 - 83.62) ms | 88.49 ± (88.00 - 88.99) ms | +6.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.15 ± (16.12 - 16.18) MB | 16.02 ± (16.00 - 16.05) MB | -0.8% | ✅ |
| runtime.dotnet.threads.count | 21 ± (21 - 21) | 21 ± (21 - 21) | +0.7% | ✅⬆️ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 436.21 ± (432.27 - 440.14) ms | 445.10 ± (440.92 - 449.28) ms | +2.0% | ✅⬆️ |
| process.time_to_main_ms | 489.18 ± (487.34 - 491.01) ms | 520.47 ± (518.55 - 522.39) ms | +6.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 57.81 ± (57.68 - 57.94) MB | 58.04 ± (57.91 - 58.18) MB | +0.4% | ✅⬆️ |
| runtime.dotnet.threads.count | 29 ± (29 - 29) | 30 ± (30 - 30) | +0.3% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 216.23 ± (214.71 - 217.75) ms | 212.91 ± (211.40 - 214.43) ms | -1.5% | ✅ |
| process.time_to_main_ms | 75.89 ± (75.50 - 76.27) ms | 75.69 ± (75.32 - 76.07) ms | -0.3% | ✅ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.15 ± (16.13 - 16.17) MB | 16.20 ± (16.18 - 16.22) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 19 ± (19 - 20) | 19 ± (19 - 19) | -0.6% | ✅ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 213.06 ± (211.67 - 214.45) ms | 213.79 ± (212.25 - 215.33) ms | +0.3% | ✅⬆️ |
| process.time_to_main_ms | 76.79 ± (76.46 - 77.11) ms | 77.57 ± (77.20 - 77.94) ms | +1.0% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.19 ± (16.17 - 16.21) MB | 16.24 ± (16.22 - 16.26) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 20 ± (20 - 20) | 21 ± (20 - 21) | +1.0% | ✅⬆️ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 459.58 ± (456.18 - 462.97) ms | 493.59 ± (485.73 - 501.45) ms | +7.4% | ✅⬆️ |
| process.time_to_main_ms | 475.27 ± (473.52 - 477.02) ms | 487.33 ± (484.73 - 489.93) ms | +2.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 57.66 ± (57.51 - 57.81) MB | 57.45 ± (57.24 - 57.66) MB | -0.4% | ✅ |
| runtime.dotnet.threads.count | 29 ± (29 - 29) | 30 ± (30 - 30) | +1.5% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 208.96 ± (207.76 - 210.15) ms | 211.68 ± (210.28 - 213.08) ms | +1.3% | ✅⬆️ |
| process.time_to_main_ms | 74.66 ± (74.26 - 75.06) ms | 75.20 ± (74.77 - 75.64) ms | +0.7% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.61 ± (11.59 - 11.63) MB | 11.59 ± (11.57 - 11.61) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 19 ± (19 - 19) | 19 ± (19 - 19) | +0.1% | ✅⬆️ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 209.68 ± (208.26 - 211.11) ms | 214.20 ± (212.47 - 215.93) ms | +2.2% | ✅⬆️ |
| process.time_to_main_ms | 76.08 ± (75.62 - 76.54) ms | 76.93 ± (76.55 - 77.32) ms | +1.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.62 ± (11.60 - 11.63) MB | 11.61 ± (11.59 - 11.63) MB | -0.1% | ✅ |
| runtime.dotnet.threads.count | 20 ± (20 - 20) | 20 ± (20 - 20) | +0.3% | ✅⬆️ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 436.17 ± (428.28 - 444.06) ms | 449.91 ± (441.69 - 458.13) ms | +3.1% | ✅⬆️ |
| process.time_to_main_ms | 459.97 ± (458.46 - 461.49) ms | 463.98 ± (462.27 - 465.69) ms | +0.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 49.44 ± (49.30 - 49.59) MB | 49.53 ± (49.39 - 49.66) MB | +0.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 29 ± (29 - 29) | 29 ± (29 - 29) | -0.3% | ✅ |
Comparison explanation
Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
- Welch test with statistical test for significance of 5%
- Only results indicating a difference greater than 5% and 5 ms are considered.
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8208) - mean (77ms) : 74, 81
master - mean (75ms) : 73, 77
section Bailout
This PR (8208) - mean (82ms) : 79, 84
master - mean (80ms) : 78, 82
section CallTarget+Inlining+NGEN
This PR (8208) - mean (1,096ms) : 1053, 1140
master - mean (1,081ms) : 1043, 1120
FakeDbCommand (.NET Core 3.1)
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8208) - mean (119ms) : 116, 122
master - mean (116ms) : 113, 119
section Bailout
This PR (8208) - mean (120ms) : 117, 123
master - mean (117ms) : 115, 120
section CallTarget+Inlining+NGEN
This PR (8208) - mean (778ms) : 718, 838
master - mean (763ms) : 699, 828
FakeDbCommand (.NET 6)
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8208) - mean (107ms) : 104, 110
master - mean (104ms) : 102, 106
section Bailout
This PR (8208) - mean (109ms) : 107, 112
master - mean (105ms) : 103, 108
section CallTarget+Inlining+NGEN
This PR (8208) - mean (758ms) : 683, 832
master - mean (750ms) : 669, 831
FakeDbCommand (.NET 8)
gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8208) - mean (106ms) : 103, 109
master - mean (103ms) : 100, 106
section Bailout
This PR (8208) - mean (107ms) : 105, 109
master - mean (104ms) : 102, 107
section CallTarget+Inlining+NGEN
This PR (8208) - mean (695ms) : 677, 712
master - mean (680ms) : 664, 695
HttpMessageHandler (.NET Framework 4.8)
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8208) - mean (212ms) : 195, 229
master - mean (194ms) : 189, 200
section Bailout
This PR (8208) - mean (214ms) : crit, 197, 231
master - mean (197ms) : 195, 200
section CallTarget+Inlining+NGEN
This PR (8208) - mean (1,213ms) : 1131, 1296
master - mean (1,165ms) : 1082, 1248
HttpMessageHandler (.NET Core 3.1)
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8208) - mean (311ms) : 281, 341
master - mean (282ms) : 273, 291
section Bailout
This PR (8208) - mean (302ms) : crit, 270, 334
master - mean (281ms) : 276, 286
section CallTarget+Inlining+NGEN
This PR (8208) - mean (1,006ms) : 924, 1088
master - mean (962ms) : 875, 1049
HttpMessageHandler (.NET 6)
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8208) - mean (300ms) : 269, 330
master - mean (302ms) : 273, 331
section Bailout
This PR (8208) - mean (301ms) : 276, 326
master - mean (301ms) : 278, 324
section CallTarget+Inlining+NGEN
This PR (8208) - mean (1,022ms) : 910, 1134
master - mean (969ms) : 909, 1030
HttpMessageHandler (.NET 8)
gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8208) - mean (301ms) : 271, 332
master - mean (297ms) : 275, 319
section Bailout
This PR (8208) - mean (305ms) : 269, 342
master - mean (299ms) : 272, 325
section CallTarget+Inlining+NGEN
This PR (8208) - mean (951ms) : 817, 1085
master - mean (933ms) : 818, 1049
andrewlock
left a comment
There was a problem hiding this comment.
Interesting 🤔 Makes me wonder if we should actually be pre-pulling and relying on that instead 😅 Maybe not though 🤷♂️
But regardless, I think we should probably make sure to apply this everywhere? 🤔 e.g. set it as a "top level" env var in ultimate-pipeline, and then pass it through to all of the docker-compose services? Otherwise I worry if/when we move to more test containers we'll end up running into the same issue again?
Thanks! I have added it to the other tests as well. Should we try with this setting before considering prepulling? Since we already take care of clean up, this should be ok? WDYT? |
andrewlock
left a comment
There was a problem hiding this comment.
Thanks! Yeah, I think this is def worth a try, no harm in it!
docker-compose.yml
Outdated
| - COSMOSDB_ENDPOINT=https://cosmosdb-emulator:8081 | ||
| - TEST_AGENT_HOST=test-agent | ||
| - CONTAINER_HOSTNAME=http://integrationtests | ||
| - TESTCONTAINERS_RYUK_DISABLED=true |
There was a problem hiding this comment.
If we add this to the ultimate-pipeline.yml variables, then we should change it in this file to just pass through:
| - TESTCONTAINERS_RYUK_DISABLED=true | |
| - TESTCONTAINERS_RYUK_DISABLED |
The advantage of that is we only have to toggle in one place 🙂
There was a problem hiding this comment.
Nice idea, thanks! Done!
Summary of changes
Disable Testcontainers Ryuk (Resource Reaper) in CI by setting
TESTCONTAINERS_RYUK_DISABLED=truein the tests of the docker-compose serviceReason for change
The Aerospike integration tests (and any other Testcontainers-based tests) intermittently fail in CI with:
Ryuk is a sidecar container that Testcontainers starts by default before every test container to handle resource cleanup on process exit. Since it needs to be pulled from Docker Hub, it is susceptible to transient failures (rate limiting, network issues), causing flaky test runs.
The CI pipeline already has its own Docker cleanup step (
.azure-pipelines/steps/clean-docker-containers.yml) that runsdocker rm -fon all containers after test execution, making Ryuk redundant.Skipping the Ryuk image pull saves CI time on every Docker-based integration test run.
Implementation details
Test coverage
Other details