Add Microsoft.Testing.Extensions.HtmlReport extension

[Evangelink]

Closes #5754.

Summary

Adds a new MTP extension Microsoft.Testing.Extensions.HtmlReport that produces a single, self-contained HTML test report at the end of a test session.

• CLI flags follow the TrxReport conventions:
  • --report-html — enable the HTML report.
  • --report-html-filename <name>.html — override the file name. Must be a pure file name (no path, no drive letter, no .., no leading/trailing whitespace, no invalid file-name char).
• The report file is published as a SessionFileArtifact, so other extensions and dotnet test pick it up like a TRX file.

Screenshots

Default view — failures first, dynamic toolbar

Grouped by Class

Grouped by Duration

Six buckets: < 10 ms, 10 ms – 100 ms, 100 ms – 1 s, 1 s – 10 s, 10 s – 1 min, ≥ 1 min.

Grouped by Trait — dark theme

The toolbar gets one Trait: <key> option per distinct trait/category found in the report.

Multiple results for the same TestNode.Uid

Some test frameworks emit several distinct terminal-state results sharing the same TestNode.Uid (parameterized rows, data-driven theories, in-process retries, framework bugs). The extension never deduplicates — every observation is preserved and the affected rows get a #N of M badge so users can tell them apart.

UX details

• Single HTML file, no CDN, all CSS/JS/data inlined → trivial to attach to PR comments, e-mail, archive as CI artifact.
• Native CSS variables, automatic light + dark theme via prefers-color-scheme.
• Header with summary cards (Total / Passed / Failed / Errored / Timed out / Skipped / Duration) and metadata block (framework, machine, user, started / finished, exit code).
• Free-text search, outcome filter pills (Failed selected by default when there are failures), sortable columns (outcome / name / duration).
• Group by: None / Class / Duration / Trait: <key> (one option per distinct trait key in the report). Group headers are collapsible and show per-outcome counts.
• Expandable per-row detail panel with error message, exception type, stack trace, standard output, standard error.
• Pagination (default 200 rows per page) keeps the report usable for very large runs.
• Filter / sort / search / grouping state is persisted in sessionStorage so refreshing the page keeps your view.

Security: handling test-controlled content safely

The data is embedded as a JSON island inside <script type="application/json">. A hand-rolled HTML-safe JSON encoder escapes <, >, &, ', \u2028, \u2029 and all control characters as \uXXXX so that test-controlled content (display names, stack traces, stdout/stderr) cannot break out of the script block. The renderer always uses textContent, never innerHTML.

A regression test (GenerateReportAsync_EscapesScriptInjection_InDisplayName) verifies that hostile payloads such as </script><img onerror=alert(1)> never appear literally in the produced HTML.

Duplicate UIDs

Some test frameworks misuse TestNode.Uid (or legitimately reuse it across parameterized rows / in-process retries). The consumer therefore:

1. Appends every terminal-state result — no observation is silently dropped.
2. The engine does a pre-pass to count results per UID and emits attemptIndex / attemptOf only on rows whose UID is non-unique.
3. The UI shows a #N of M badge on those rows and uses a synthetic per-row key for expand/collapse state so the rows stay independent.

Standard output / standard error

Already handled — captured from StandardOutputProperty / StandardErrorProperty and rendered in the expandable detail panel (truncated to 32 KiB with a UI-visible marker for very large streams).

Retries

For host-level retries (--retry-failed-tests), each attempt runs in its own results directory, so each gets its own HTML report — same behaviour as TRX. Within a single host run, in-process retries are surfaced via the #N of M mechanism above.

Performance

• No reflection-based JSON serialization → works on netstandard2.0 without a System.Text.Json dependency, AOT/trim safe.
• Per-test stdout / stderr (32 KiB) and stack traces (32 KiB) are truncated with a UI-visible suffix so 100k-test runs stay manageable.
• Browser-side pagination at 200 rows per page by default.

Future hook: live updates (out of scope for this PR)

The page exposes a global window.mtpRender(newReport) function. A future OpenTelemetry / SSE / WebSocket integration can stream deltas and call this function to refresh the page in place without reloading. None of the current code is structured in a way that closes the door on this.

Files

• src/Platform/Microsoft.Testing.Extensions.HtmlReport/
  • Microsoft.Testing.Extensions.HtmlReport.csproj (multi-targets netstandard2.0;$(SupportedNetFrameworks)).
  • HtmlReportExtensions.AddHtmlReportProvider() + TestingPlatformBuilderHook (auto-registered via MSBuild props).
  • HtmlReportGeneratorCommandLine, HtmlReportGenerator (IDataConsumer, ITestSessionLifetimeHandler, IDataProducer, IOutputDeviceDataProducer), HtmlReportEngine.
  • Templates/report-template.html (embedded resource).
  • Resources/ExtensionResources.resx + 12 .xlf files generated via dotnet msbuild ... /t:UpdateXlf.
  • PACKAGE.md, BannedSymbols.txt, PublicAPI/, build/, buildMultiTargeting/, buildTransitive/.
• TestFx.slnx: project registered.
• src/Platform/Microsoft.Testing.Platform/Microsoft.Testing.Platform.csproj: InternalsVisibleTo added.
• test/IntegrationTests/.../HelpInfoAllExtensionsTests.cs: expectations updated for --report-html / --report-html-filename lines, the new HtmlReportGeneratorCommandLine --info block, and the new PackageReference in the AllExtensions test asset.
• test/IntegrationTests/.../MSBuild.KnownExtensionRegistration.cs: assertion added for the new TestingPlatformBuilderHook and --report-html, plus a new PackageReference.
• test/UnitTests/.../HtmlReportGeneratorCommandLineTests.cs and HtmlReportEngineTests.cs: 21 new tests covering CLI validation, JSON shape, traits, duplicate-UID handling, output truncation and HTML/JS injection.
• docs/images/html-report/ — screenshots used in this PR description.

Public API

Only two public types / two members:

Microsoft.Testing.Extensions.HtmlReport.TestingPlatformBuilderHook
Microsoft.Testing.Extensions.HtmlReportExtensions

static TestingPlatformBuilderHook.AddExtensions(ITestApplicationBuilder, string[]) -> void
static HtmlReportExtensions.AddHtmlReportProvider(this ITestApplicationBuilder) -> void

Both declared in PublicAPI.Unshipped.txt. No init accessors.

Validation

• Microsoft.Testing.Extensions.HtmlReport builds clean across netstandard2.0;net8.0;net9.0;net462;net472.
• Microsoft.Testing.Platform and Microsoft.Testing.Platform.Acceptance.IntegrationTests build clean (0 warnings, 0 errors).
• All 105 unit tests in Microsoft.Testing.Extensions.UnitTests pass on net9.0 (84 pre-existing + 21 new).
• The acceptance HelpInfo / MSBuild known-extension-registration tests require ./build.cmd -pack to publish the new package before they can run end-to-end; the expectation files are updated so those tests will match on the next pack.

[Copilot]

Pull request overview

Adds a new Microsoft.Testing.Platform HTML report extension that can be auto-registered through MSBuild, exposes --report-html CLI options, generates a self-contained HTML session artifact, and adds unit/acceptance coverage for the new package.

Changes:

• Introduces Microsoft.Testing.Extensions.HtmlReport with CLI registration, report generation, embedded HTML template, resources, packaging, and public API declarations.
• Registers the extension in the solution, platform internals, MSBuild hook tests, and all-extension help/info acceptance expectations.
• Adds unit tests for filename validation, JSON/report generation, duplicate UIDs, traits, truncation, and injection escaping.

Show a summary per file

File	Description
TestFx.slnx	Registers the new HtmlReport project.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/Microsoft.Testing.Extensions.UnitTests.csproj	Adds test project reference to HtmlReport.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/HtmlReportGeneratorCommandLineTests.cs	Adds CLI validation tests.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/HtmlReportEngineTests.cs	Adds report engine/unit generation tests.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/MSBuild.KnownExtensionRegistration.cs	Adds MSBuild auto-registration assertions.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs	Updates help/info expectations and asset package references.
src/Platform/Microsoft.Testing.Platform/Microsoft.Testing.Platform.csproj	Grants internals access to HtmlReport.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/TestingPlatformBuilderHook.cs	Adds MSBuild builder hook.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Templates/report-template.html	Adds embedded HTML/CSS/JS report UI.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/ExtensionResources.resx	Adds localizable extension strings.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.cs.xlf	Adds Czech localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.de.xlf	Adds German localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.es.xlf	Adds Spanish localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.fr.xlf	Adds French localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.it.xlf	Adds Italian localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.ja.xlf	Adds Japanese localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.ko.xlf	Adds Korean localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.pl.xlf	Adds Polish localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.pt-BR.xlf	Adds Portuguese Brazil localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.ru.xlf	Adds Russian localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.tr.xlf	Adds Turkish localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.zh-Hans.xlf	Adds Simplified Chinese localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Resources/xlf/ExtensionResources.zh-Hant.xlf	Adds Traditional Chinese localization stub.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/PublicAPI/PublicAPI.Unshipped.txt	Declares new public API surface.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/PublicAPI/PublicAPI.Shipped.txt	Initializes shipped API file.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/PACKAGE.md	Adds NuGet package documentation.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/Microsoft.Testing.Extensions.HtmlReport.csproj	Defines project/package layout and resources.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/HtmlReportGeneratorCommandLine.cs	Adds CLI option provider and validation.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/HtmlReportGenerator.cs	Adds data consumer/lifetime handler that emits the artifact.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/HtmlReportExtensions.cs	Adds builder extension registration method.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/HtmlReportEngine.cs	Generates HTML-safe JSON and writes the report file.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/buildTransitive/Microsoft.Testing.Extensions.HtmlReport.props	Adds transitive package import.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/buildMultiTargeting/Microsoft.Testing.Extensions.HtmlReport.props	Adds MSBuild self-registration metadata.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/build/Microsoft.Testing.Extensions.HtmlReport.props	Adds build package import.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/BannedSymbols.txt	Adds analyzer banned-symbol rules.

Copilot's findings

• Files reviewed: 35/40 changed files
• Comments generated: 12

[Evangelink]

Review Summary — PR #8283: Microsoft.Testing.Extensions.HtmlReport

#	Dimension	Status
1	Algorithmic Correctness	✅ LGTM
2	Threading & Concurrency	✅ LGTM — ConsumeAsync is always invoked from a single reader task (SingleReader = true); _tests and _testStartTime are not subject to concurrent mutation
3	Security & IPC Contract Safety	✅ LGTM — AppendString correctly escapes <, >, &, ', \u2028, \u2029; IsValidPureFileName blocks path traversal
4	Public API & Binary Compatibility	✅ LGTM — all public types declared in PublicAPI.Unshipped.txt, XML docs present, no init accessors
5	Performance & Allocations	⚠️ MODERATE — OfType<TestMetadataProperty>() allocates per test node in BuildJson hot path
6	Cross-TFM Compatibility	✅ LGTM — #if NETCOREAPP guards, polyfills included, no missing API usage
7	Resource & IDisposable Management	✅ LGTM — IFileStream wrapped in using
8	Localization & Resources	✅ LGTM — all user-facing strings in .resx, no hardcoded UI strings
9	Test Completeness & Coverage	🚫 MAJOR — no unit tests ship with this new extension

Blocking Concern

The PR adds a non-trivial new extension — custom JSON serializer with HTML escaping, filename validation with six rejection branches, duplicate-UID attempt tracking, field truncation — but ships zero unit tests. The codebase's test infrastructure (TestFramework.ForTestingMSTest) is already available. Both IsValidPureFileName and BuildJson are well-suited to pure unit testing (deterministic inputs/outputs, no I/O).

Generated by Expert Code Review (on open) for issue #8283 · ● 19.4M

[Evangelink]

Thanks for the thorough review — addressed every finding. Force-pushed evangelink/html-reporter (f97c4ccf6).

build.cmd is clean: 0 Warning(s), 0 Error(s) across the entire repo (debug build, net462, net472, net8.0, net9.0, netstandard2.0).

Code changes

#	Comment	Fix
1	Default filter doesn't surface errored/timedOut-only runs	When summary has no failed but has errored or timedOut, open on that specific filter rather than failed. template.html state.filter init.
2	Path.GetInvalidFileNameChars() is OS-dependent (Linux accepts *?<>"|)	Replaced with a hard-coded InvalidFileNameChars array that is a strict superset of Windows + Unix invalid characters, so the same input is rejected on every host.
3	Grouped views bypass pagination, hangs browser on huge runs	Per-group rendering is now capped (default 200 rows/group) with Show more / Show all buttons. shownPerGroup state is tracked separately and persisted.
4	Synthetic key uid#N can collide with a real UID containing #N	Engine now emits a stable per-row rowKey (the result index). UI uses that key for expand-state — no derivation, no possible collision.
5	Default filename has second precision + FileMode.CreateNew ⇒ IOException on rapid reruns	Added a 5 s retry loop that appends _1, _2, … to the default name until a unique one is found. Explicit --report-html-filename still uses FileMode.Create (overwrite) with the existing warning. Mirrors TrxReportEngine.RetryWhenIOExceptionAsync.
6	Filter pills are <span>, not keyboard-accessible	Converted to <button> with aria-pressed state and :focus-visible styling.
7	Sortable column headers are <div> with click only	Converted to <button> with aria-sort (ascending/descending/none) and keyboard focus.
8	Test rows aren't focusable, no expanded/collapsed announcement	Rows with details now get role="button", tabindex="0", aria-expanded, and an Enter/Space keydown handler.
9	Group headers aren't keyboard accessible	Group headers got the same role="button", tabindex="0", aria-expanded and keyboard handler.
10	New provider not covered by ExtensionVersionTests	Added HtmlReportGeneratorCommandLine_UsesItsOwnAssemblyVersion.
11	Reserved Windows device names (CON.html, NUL.html, …) accepted	New WindowsReservedNames table; the bare name (without extension) is compared case-insensitively against CON, PRN, AUX, NUL, COM0..9, LPT0..9. New regression tests + a positive test that CONfig.html is still accepted.
12	Generator retains entire TestNodeUpdateMessage (with huge stdout/stderr) in memory	New CapturedTestResult DTO + TestResultCapture.TryCapture() projects each message at consume time. Truncation now caps at consume, so memory is bounded by the per-field caps × number of results — the engine no longer needs to truncate at all.
13	Missing engine tests	Added: outcome classification for every well-known state, truncation at +7 over the boundary, exact-boundary non-truncation, null-return for non-terminal states, escape tests for each of <, >, &, ', U+2028, U+2029, stable rowKey emission, default-filename retry, duplicate-UID attemptIndex/attemptOf annotation (with the unique-UID negative case).
14	OfType<TestMetadataProperty>() allocates per node	Replaced with foreach + is pattern in TestResultCapture so no enumerator object is allocated per test.
15	Missing CLI validator tests	Added cases for each rejection branch including the new cross-platform invalid chars and reserved Windows names; the positive case report.html is also covered.

Test counts

• Microsoft.Testing.Extensions.UnitTests filter ~HtmlReport: 50 tests, 0 failures on net9.0 (DataRow expansions included).

Screenshots refreshed

The screenshots in docs/images/html-report/ have been regenerated against the updated UI so the focus styles and ARIA-converted controls render correctly.

[Copilot]

Copilot's findings

• Files reviewed: 38/43 changed files
• Comments generated: 3

[Evangelink]

@copilot address review comments

[Copilot]

@copilot address review comments

Addressed all three remaining review comments in commit e4a8c0b:

1. sessionStorage key collision — storage key is now scoped to "mtp-state:" + location.href so each report file maintains independent UI state.

2. aria-sort semantic issue — added role="grid" to the .results container and role="columnheader" to each sortable header button. aria-sort is now on elements with a valid ARIA role (columnheader) within the required grid ownership chain.

3. Acceptance test — added HtmlReportTests.cs with four tests covering: (a) no HTML report generated when --report-html is absent, (b) HTML report generated and published as a session artifact when --report-html is specified (content verified), (c) invalid filename with path separator rejected, (d) --report-html-filename without --report-html rejected.

[Evangelink]

@copilot address review comments

[Copilot]

Copilot's findings

• Files reviewed: 39/44 changed files
• Comments generated: 3

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (2)

src/Platform/Microsoft.Testing.Extensions.HtmlReport/Templates/report-template.html:932

• Group count badges render the raw outcome keys (for example timedOut) instead of the user-facing labels used elsewhere (Timed out). This makes grouped headers inconsistent and exposes implementation casing in the report UI.
  src/Platform/Microsoft.Testing.Extensions.HtmlReport/Templates/report-template.html:994
• The per-row outcome pill displays the raw serialized outcome (timedOut) and relies on uppercase styling, so timeout results render as TIMEDOUT rather than the human-readable Timed out used by the filter and summary. Map outcomes to display labels before assigning the text.

• Files reviewed: 39/44 changed files
• Comments generated: 7

[Evangelink]

Update: extension marked as experimental

Pushed 29cbf13 which adds [Experimental("TPEXP")] on HtmlReportExtensions, updates PublicAPI.Unshipped.txt to use the [TPEXP] prefix, suppresses TPEXP in the acceptance test asset and adds an experimental notice to PACKAGE.md.

Review comments status

I re-checked each open review comment against the current code; all 21 inline comments are addressed:

Inline comments from Copilot reviewer (12)

• Default filter when no failed tests ΓÇö addressed in report-template.html (lines 542-550): falls back to errored ΓåÆ timedOut ΓåÆ all.
• OS-dependent invalid file name chars ΓÇö addressed in HtmlReportGeneratorCommandLine.cs: InvalidFileNameChars now hard-codes Windows-invalid characters (*, ?, ", <, >, |, control chars) so validation is consistent on Linux/macOS/Windows.
• Grouped views bypass pagination ΓÇö addressed: renderGrouped caps per-group rows via shownPerGroup/buildShowMore.
• Synthetic expansion key collision ΓÇö addressed: each row now uses the engine-supplied rowKey (unique row index), not a uid#attempt concatenation.
• FileMode.CreateNew without retry ΓÇö addressed by WriteWithRetryAsync: default file names retry with _<attempt> suffix on IOException for up to 5 s.
• Filter pills not keyboard-accessible ΓÇö addressed: replaced <span> with <button type="button" aria-pressed>.
• Sortable headers not keyboard-accessible ΓÇö addressed: now <button role="columnheader" aria-sort> inside the role="row`" header.
• Expandable rows not keyboard-accessible ΓÇö addressed: rows with detail get role="button" tabindex="0" aria-expanded plus an Enter/Space keydown handler.
• Group headers not keyboard-accessible ΓÇö addressed identically (role="button" tabindex="0" aria-expanded + keydown handler).
• ExtensionVersionTests coverage ΓÇö addressed: HtmlReportGeneratorCommandLine_UsesItsOwnAssemblyVersion added.
• Reserved Windows device names ΓÇö addressed: WindowsReservedNames list (CON, PRN, AUX, NUL, COM0-COM9, LPT0-LPT9) is checked against the bare file name.
• Retain full TestNodeUpdateMessage ΓÇö addressed: TestResultCapture.TryCapture projects each message into a capped CapturedTestResult DTO on the consumer thread, the engine never sees raw TestNode instances.

Inline comments from expert reviewer (3)

• BuildJson not unit-tested ΓÇö addressed: HtmlReportEngineTests.cs covers every outcome branch, duplicate-UID attemptIndex/attemptOf, truncation at the exact MaxMessageLength/MaxStackTraceLength/MaxStandardStreamLength boundaries and HTML-unsafe escaping (<, >, &, ', \u2028, \u2029).
• LINQ allocation in hot path ΓÇö addressed in TestResultCapture.cs: trait collection now uses a plain foreach + is check, no enumerator allocation.
• IsValidPureFileName not unit-tested ΓÇö addressed in HtmlReportGeneratorCommandLineTests.cs: all six rejection branches plus reserved device names are covered.

Inline comments from follow-up Copilot review (6)

• sessionStorage key collision across reports ΓÇö addressed: storageKey is now scoped per report via location.href (line 526).
• aria-sort placement ΓÇö addressed: sortable buttons now carry role="columnheader" aria-sort inside a role="row" header inside a role="grid" results container, which is a valid ARIA grid pattern.
• Missing acceptance test ΓÇö addressed: HtmlReportTests.cs registers --report-html against a real test host and asserts the artifact is published as an in-process file artifact.
• Custom file name path not covered E2E ΓÇö addressed: Html_WhenReportHtmlFilenameIsSpecified_HtmlReportIsGeneratedWithThatName runs the host with --report-html-filename my-custom-report.html and verifies the named file exists.
• Negative assertion looking for Out of process ΓÇö addressed: the negative assertion in Html_WhenReportHtmlIsNotSpecified_HtmlReportIsNotGenerated now checks the In process file artifacts produced block (the one used for the HTML artifact).
• CancelledTestNodeStateProperty.Exception not captured ΓÇö addressed in TestResultCapture.cs (line 44): the obsolete cancellation state is now included in the exception-extraction switch (under CS0618/MTP0001 suppression).

Let me know if anything still looks off ΓÇö happy to iterate.

[Evangelink]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Merge conflicts resolved in commit a8c743a91. The only conflict was in HelpInfoAllExtensionsTests.cs where the --report-azdo* options (added in main via #8298) and --report-html* options (from this branch) overlapped. Both sets of options are now included in alphabetical order.

[Copilot]

Copilot's findings

• Files reviewed: 40/45 changed files
• Comments generated: 1

[Copilot]

Copilot's findings

• Files reviewed: 39/44 changed files
• Comments generated: 0 new