Skip to content

Size template buffers from archive entries - #56193

Merged
JeremyKuhne merged 2 commits into
dotnet:mainfrom
JeremyKuhne:optimize-template-known-length-stream
Sep 10, 2026
Merged

Size template buffers from archive entries#56193
JeremyKuhne merged 2 commits into
dotnet:mainfrom
JeremyKuhne:optimize-template-known-length-stream

Conversation

@JeremyKuhne

Copy link
Copy Markdown
Member

Summary

Follow-up to #56185 and #56163.

  • expose archive-entry length through an internal IKnownLengthFile
  • size processing buffers before opening non-seekable ZIP entry streams
  • retain the existing 8 MiB fallback for unknown, invalid, oversized, or throwing lengths
  • keep IFile, IProcessorState, and ProcessorState unchanged
  • add boundary and real ZIP-entry tests

The source changes are independent from #56185. Performance was measured on top of its exact head.

Performance

Windows x64 Release ReadyToRun measurements used initialized dotnet new console --no-restore operations.

Delta from #56185

Metric #56185 This change Difference
Reported allocation 40,011,904 B 23,264,064 B -41.86%
Peak heap 25.06 MB 5.89 MB -19.17 MB
Promoted memory 16.40 MB 3.72 MB -12.68 MB
GC count 2 1 -1
Console latency 227.53 ms 225.15 ms 0.99% paired improvement
Sampled CPU 330.50 ms/op 324.41 ms/op -1.84%

The latency comparison used 15 alternating pairs; 11/15 candidate operations were faster with zero semantic mismatches.

Three-PR breakdown

PR Change Allocation result Time result
#56163 Remove template-host logging DI construction Host construction: 93,191 B to 9,801 B (-89.5%) Host CPU: 15.15 ms to 4.44 ms; console creation 5.32% faster
#56185 Store the common trie child inline Command allocation: 49,401,248 B to 40,047,344 B (-18.93%) Console latency effectively neutral; list 4.08% faster
This PR Size buffers from ZIP entry lengths Command allocation: 40,011,904 B to 23,264,064 B (-41.86%) Console creation 0.99% faster; sampled CPU 1.84% lower

Validation

  • Microsoft.TemplateEngine.Core.UnitTests
    • Debug and Release
    • net11.0 and net481
    • 370 passed, 0 failed per run
  • Microsoft.TemplateEngine.Edge.UnitTests
    • Debug and Release
    • net11.0 and net481
    • 269 passed, 0 failed per run
  • full Release ReadyToRun SDK build
  • exact measured SDK trees differed only in ReadyToRun Abstractions, Core, and Edge assemblies
  • all measured commands and semantic comparisons passed

Copilot AI lite review requested due to automatic review settings September 9, 2026 04:19
@JeremyKuhne
JeremyKuhne requested a review from a team as a code owner September 9, 2026 04:19
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new ZIP-entry unit test uses a single Stream.Read call assuming it fills the buffer, which is not guaranteed and can cause intermittent failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR improves template processing performance by sizing ProcessorState buffers based on known ZIP entry lengths (when available) before opening non-seekable entry streams, while preserving the existing 8 MiB default fallback for unknown/invalid lengths.

Changes:

  • Introduces an internal IKnownLengthFile contract in TemplateEngine Abstractions and exposes it to Core/Edge (and their unit tests) via InternalsVisibleTo.
  • Implements IKnownLengthFile for ZipFileFile so ZIP entry uncompressed length can be queried without opening the entry stream.
  • Updates Orchestrator to opportunistically select a smaller processing buffer size when IKnownLengthFile.Length is valid and below the default, and adds focused unit coverage (including ZIP-entry coverage).
File summaries
File Description
test/TemplateEngine/Microsoft.TemplateEngine.Edge.UnitTests/ZipFileFileTests.cs Adds ZIP-entry length validation coverage for ZipFileFile implementing known-length behavior.
test/TemplateEngine/Microsoft.TemplateEngine.Core.UnitTests/OrchestratorTests.cs Adds unit tests covering buffer-size selection behavior for known/invalid/throwing lengths.
src/TemplateEngine/Microsoft.TemplateEngine.Edge/Mount/Archive/ZipFileFile.cs Implements IKnownLengthFile by exposing ZIP entry uncompressed length.
src/TemplateEngine/Microsoft.TemplateEngine.Core/Util/Processor.cs Makes DefaultBufferSize accessible internally for orchestrator buffer-size decisions.
src/TemplateEngine/Microsoft.TemplateEngine.Core/Util/Orchestrator.cs Uses IKnownLengthFile.Length to choose a smaller buffer size before opening non-seekable streams.
src/TemplateEngine/Microsoft.TemplateEngine.Abstractions/Mount/IKnownLengthFile.cs Adds internal interface for files that can reliably report length.
src/TemplateEngine/Microsoft.TemplateEngine.Abstractions/Microsoft.TemplateEngine.Abstractions.csproj Grants friend access to internals for Core/Edge and their test assemblies.
Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@JeremyKuhne
JeremyKuhne merged commit 9e1ccc2 into dotnet:main Sep 10, 2026
23 checks passed
@JeremyKuhne
JeremyKuhne deleted the optimize-template-known-length-stream branch September 10, 2026 17:32
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants