Skip to content

Conversation

martincostello
Copy link
Contributor

Benchmark results: #2459 (comment)

Given that the hardware between my local setup and GitHub Actions is quite different, I suggest merging this PR once happy with the code changes with the failing benchmark comparison, then downloading the results from the run in main and opening a new PR to use those files to replace the contents of performance/benchmark/BenchmarkDotNet.Artifacts/results and update the baseline.

Changes

  • Avoid allocations from lambda closures.
  • Avoid allocations from context strings.
  • Avoid allocations from copying arrays.
  • Remove redundant null checks.
  • Ignore BenchmarkDotNet profiler files.
  • Ignore Visual Studio profiler session files.
  • Publish benchmark results to GitHub Actions workflow artifacts.

Publish benchmark results to GitHub Actions workflow artifacts.
- Ignore BenchmarkDotNet profiler files.
- Ignore Visual Studio profiler session files.
- Avoid allocations from lambda closures.
- Avoid allocations from context strings.
- Avoid allocations from copying arrays.
- Remove redundant null checks.
@Copilot Copilot AI review requested due to automatic review settings August 20, 2025 09:10
@martincostello martincostello requested a review from a team as a code owner August 20, 2025 09:10
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes the performance of the OpenApiWalker class by eliminating allocations from lambda closures, string concatenations, and array copying operations. The changes focus on replacing the existing Walk(context, action) pattern with more efficient WalkItem and WalkDictionary helper methods that avoid capturing variables in closures.

Key changes include:

  • Replace lambda closures with direct method calls to avoid allocation overhead
  • Add specialized WalkItem and WalkDictionary helper methods for efficient traversal
  • Implement optimized iteration patterns for collections to avoid array copying
  • Publish benchmark results as GitHub Actions artifacts for performance tracking

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 4 comments.

File Description
src/Microsoft.OpenApi/Services/OpenApiWalker.cs Core performance optimizations replacing lambda-based traversal with direct method calls
.github/workflows/ci-cd.yml Add artifact publishing for benchmark results

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@martincostello
Copy link
Contributor Author

FYI I ran each of the benchmarks using the Visual Studio profiler to get the allocations, and these are the top types:

image

Even more refactoring of the walker in a future PR to remove the delegates would probably significantly reduce the allocations:

image

@martincostello
Copy link
Contributor Author

Some of these might be new from the refactoring actually, so I'll see if I can shave it down more here...

@martincostello martincostello marked this pull request as draft August 20, 2025 11:40
Make all delegates static.
@martincostello
Copy link
Contributor Author

With the latest changes:

Method Mean Before Mean After Ratio Memory Before Memory After Ratio
PetStoreYaml 265.6 μs 293.3 μs 1.10 387.12 KB 386.91 KB 0.99
PetStoreJson 106.6 μs 134.4 μs 1.26 249.26 KB 249.06 KB 0.99
GHESYaml 774,833.9 μs 788,176.8 μs 1.01 400088.73 KB 384130.43 KB 0.96
GHESJson 364,114.2 μs 368,179.1 μs 1.01 261558.87 KB 262008.28 KB 1.00

BenchmarkDotNet v0.15.2, Windows 11 (10.0.26100.4946/24H2/2024Update/HudsonValley)
13th Gen Intel Core i7-13700H 2.90GHz, 1 CPU, 20 logical and 14 physical cores
.NET SDK 8.0.413
  [Host]   : .NET 8.0.19 (8.0.1925.36514), X64 RyuJIT AVX2
  ShortRun : .NET 8.0.19 (8.0.1925.36514), X64 RyuJIT AVX2

Job=ShortRun  IterationCount=3  LaunchCount=1  
WarmupCount=3  

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
PetStoreYaml 293.3 μs 23.48 μs 1.29 μs 31.2500 5.8594 - 386.91 KB
PetStoreJson 134.4 μs 49.91 μs 2.74 μs 19.5313 4.8828 - 249.06 KB
GHESYaml 788,176.8 μs 246,902.17 μs 13,533.54 μs 35000.0000 19000.0000 4000.0000 384130.43 KB
GHESJson 368,179.1 μs 23,852.48 μs 1,307.43 μs 20000.0000 11000.0000 2000.0000 262008.28 KB

@martincostello
Copy link
Contributor Author

Action<T> has dropped off top of the allocations now:

image

@martincostello martincostello marked this pull request as ready for review August 20, 2025 12:25
Resolve two CodeQL warnings.
Copy link
Member

@baywet baywet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is some great refactoring that was overdue here! Thanks for taking the time. I have left a few comments.

And yes... it's not the first time I see the action context capture using a lot of memory at scale...

@martincostello
Copy link
Contributor Author

martincostello commented Aug 20, 2025

Here's all the relevant benchmark data taken from the CI logs.

TL;DR:

10b46b9

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
PetStoreYaml 502.9 us 85.92 us 4.71 us 23.4375 3.9063 - 387.12 KB
PetStoreJson 215.0 us 10.97 us 0.60 us 14.6484 3.9063 - 248.97 KB
GHESYaml 1,063,967.0 us 51,235.65 us 2,808.40 us 27000.0000 20000.0000 3000.0000 400086.95 KB
GHESJson 509,285.0 us 133,784.60 us 7,333.19 us 17000.0000 10000.0000 2000.0000 261557.86 KB

90b3966

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
PetStoreYaml 553.4 us 19.28 us 1.06 us 26.3672 5.8594 - 445.71 KB
PetStoreJson 276.7 us 55.86 us 3.06 us 17.5781 3.9063 - 307.56 KB
GHESYaml 1,055,962.2 us 291,474.86 us 15,976.72 us 28000.0000 20000.0000 3000.0000 422172.41 KB
GHESJson 509,499.6 us 119,766.91 us 6,564.83 us 18000.0000 10000.0000 2000.0000 283643.26 KB

This PR

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
PetStoreYaml 511.4 us 68.15 us 3.74 us 23.4375 3.9063 - 386.9 KB
PetStoreJson 233.9 us 20.54 us 1.13 us 13.6719 3.9063 - 248.75 KB
GHESYaml 1,063,226.7 us 317,042.02 us 17,378.14 us 26000.0000 20000.0000 3000.0000 384128.47 KB
GHESJson 472,314.9 us 54,982.85 us 3,013.80 us 16000.0000 9000.0000 2000.0000 245598.41 KB

@martincostello
Copy link
Contributor Author

martincostello commented Aug 20, 2025

With the factoring suggestions the numbers get slightly worse than without, but it's probably within the realms of noise.

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
PetStoreYaml 533.9 us 467.5 us 25.62 us 23.4375 3.9063 - 387.26 KB
PetStoreJson 243.2 us 251.9 us 13.81 us 13.6719 1.9531 - 249.1 KB
GHESYaml 1,044,990.0 us 80,027.6 us 4,386.59 us 26000.0000 20000.0000 3000.0000 384490.94 KB
GHESJson 481,887.0 us 222,646.4 us 12,204.00 us 16000.0000 9000.0000 2000.0000 245957.72 KB

- Create helpers to reduce code duplication.
- Remove unused method.
@martincostello
Copy link
Contributor Author

@baywet Assuming you're otherwise happy with the PR and the changes to the perf numbers, do you want me to take the files from the latest CI run on this PR and check them in as the updated baseline (which should then make the PR green on the next run)?

Create common implementation to walk `OpenApiTag` and `OpenApiTagReference`.
Remove redundant comment.
baywet
baywet previously approved these changes Aug 20, 2025
Copy link
Member

@baywet baywet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for making the changes!

@martincostello
Copy link
Contributor Author

Latest numbers:

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
PetStoreYaml 529.5 us 62.50 us 3.43 us 23.4375 3.9063 - 387.26 KB
PetStoreJson 240.8 us 15.69 us 0.86 us 13.6719 1.9531 - 249.1 KB
GHESYaml 1,097,576.6 us 100,584.42 us 5,513.37 us 26000.0000 20000.0000 3000.0000 384492.38 KB
GHESJson 516,328.2 us 87,964.22 us 4,821.62 us 16000.0000 9000.0000 2000.0000 245957.5 KB

Would you like me to commit the results from that build into this PR before merge?

@baywet
Copy link
Member

baywet commented Aug 20, 2025

@martincostello yes please go ahead!

@baywet baywet enabled auto-merge August 20, 2025 15:11
@baywet baywet merged commit a007c03 into microsoft:main Aug 20, 2025
9 checks passed
@martincostello martincostello deleted the improve-OpenApiWalker-perf branch August 20, 2025 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants