Navigation Refactor: Generic Type System, O(1) Re-homing, and improved assembly routines (#1995)

* Add CLI command documentation drafts

- Introduced draft documentation for all CLI commands under `docs/cli`.
- Updated `_docset.yml` to include new CLI documentation.
- Adjusted navigation order and reinstated missing migration files.

* organize cli command documentation

* stage changes

* Finish documenting cli commands

* Cleanup migration

* touchups

* stage

* stage

* First pass of claude over building blocks

* Expanded building blocks

* update building blocks

* update building blocks

* Include navigation building blocks

* Small touchups

* update redirects

* Apply suggestions from code review

Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]>

* Sentence case

* fix links

* List items full stop

* colons in list not dashes

* fix test

* Start on isolated docset config reader

* Stage before invoking AI

* Start on new isolated navigation building

* toc nesting restrictions

* Continue with documentation set navigation, generated a lot of tests, url is dynamic based on parent toc

* Reorg tests

* chop of /index

* encapsulate more rules

* explicitly return IndexFileRef when deserialzing the configuration for typechecks > filepath checks

* Read through nested toc.yml file definitions

* Brand new navigation.yml deserialization

* Initial version of SiteNavigation

* Post claude test fixes

* Start on SiteDocumentationNavigation tests

* stage broken state

* stage

* stage

* fix tests manually

* path_prefix is required now and added more tests

* Stage work on path prefix providers

* Stage work tests passing again

* Add failing test

* Refactor TOC navigation handling to simplify nested validation logic and update related tests.

* fix tests

* dotnet format namespaces

* dotnet format namespaces

* Add explicit YamlMember

* inject creating IDocumentationFile

* stage

* Preserve covariance of interface

* Start on moving the new navigation into position

* Started on removing GlobalNavigation

* TocItem just reads path (dumb) its up to nav to preserve relative path

* commit failing test

* fix deeply linked children, especially those with a parent deep/linked/index.md

* increased complexity of test

* Fix FileInfo property

* Get rid od Markdown.Url, navigation builds urls

* Remove TableOfContentsTree

* remove CrossLinkNavigationItem

* remove FileNavigationItem

* Remove DocumentationGroup

* remove namespace usages

* Move new navigation into place

* fix compilation of ApiExplorer needs revisiting

* stage commit

* stage

* Simplify folder and TOC children validation logic

* Update tests

* Refactor TOC path construction to support TemporaryNavigationPlaceholder

* add nested folder back

* fix test

* fix test

* fix test

* fix test

* Fixing markdown tests

* Fixing markdown tests

* Esnure path prefix on build context is reflected in the urls

* Fix test

* fix typo

* fix tests

* fix test

* Fix tests

* Fix navigation rendering

* Include node navigations in crosslink lookup

* Fix markdownlookup for nodes

* load docs-content correctly

* Remove table of contents old

* stage

* stage

* stage

* Add `FileInfoValidationTests` for verifying file navigation items in documentation navigation

* stage

* Focus on DocumentationSetFile Toc paths

* Validate file references in PhysicalDocsetTests and clean up redundant diagnostics in FileInfoValidationTests

* Validate no diagnostics are emitted in FileInfoValidationTests

* Refactor TOC and folder navigation logic in `DocumentationSetNavigation`:

- Remove redundant `AutoDiscoverFolderFiles` logic, integrating auto-discovery directly into `ResolveFolderRef`.
- Ensure children validation is streamlined across TOC and folder navigation.
- Add validation for duplicate file references in `FileInfoValidationTests`.

* Add test to verify context resolution for TOC and file references in DocumentationSetFile

* Refactor TOC navigation logic and improve error handling:

- Replace `parentPath` with `parentContextPath` for clarity.
- Remove redundant `ResolveTocPath` method.
- Simplify TOC file path resolution logic.
- Update error messages to include context for better diagnostics.

* Refactor TOC navigation to simplify path handling and improve consistency:

- Remove `parentContextPath` parameter from navigation methods.
- Consolidate logic for TOC path resolution.
- Adjust children validation to rely on pre-resolved paths.
- Update relevant tests to cover new path handling behavior.

* Update TOC path expectations and add comments for TOC children limitations:

- Adjust `advancedToc.Path` test to match new path structure (`guides/advanced`).
- Add TODO comments to clarify known TOC children limitations in parent YAML.

* stage

* stage

* stage

* stage

* stage

* stage

* Add support for combined folder and file path resolution in DocumentationSetFile:

- Handle `folder` + `file` combinations as a single `FileRef` or `IndexFileRef`.
- Maintain existing behavior for standalone `file`, `folder`, and `hidden` keys.
- Update variable naming to enhance readability and clarity.

* Add FolderIndexFileRef to handle folder+file combinations:

- Introduce `FolderIndexFileRef` to represent YAML `folder` + `file` combinations.
- Update TOC resolution to ensure children resolve relative to folder paths.
- Adjust file reference creation logic to preserve specific types.
- Add comments for clarity and improve edge case handling.

* Add README that can be used as context.

* Emit hints for deep-linking virtual files during validation:

- Add checks to identify files with deep paths and children, suggesting the use of `folder` for better structure.
- Extend test coverage for virtual file scenarios to validate hint emission and suppression.
- Refactor diagnostics in physical docset navigation to include hints while ensuring no errors or warnings occur.

* Allow TOC references from narrative repository without requiring `path_prefix`.

* Added explicit support for folder with an index field

* Hints and suppresshints

* Normalize `sitePrefix` in `SiteNavigation`, update URL handling, and extend test coverage:

- Add `NormalizeSitePrefix` method to standardize `sitePrefix` with leading slash and no trailing slash.
- Apply normalized `sitePrefix` to navigation URLs.
- Improve `GetDescription` error messaging in `LlmsNavigationEnhancer`.
- Update tests to validate `sitePrefix` normalization and navigation item URL adjustments.

* Ensure documentationset passes path prefix to documentationsetnavigation

* stage

* stage

* stage

* stage

* Simplify building nodes without intermediary nav items

* stage

* update url logic to be relative conditionally

* cache url

* Index no longer part of NavigationItems

* fix some more tests

* Ensure nested toc nodes receive index and homeprovider

* fix some more tests

* stage

* fix navigationbuilding integration test

* clean up

* ensure urls end with / again

* TestDiagnosticsCollector improvements

* fix test diagnostics compiler issue

* fix authoring tests

* update synthetics tests

* tackle windows test failures

* fix unit test to allow for windows paths

* fix integration tests

* fix navigation validation for clashes

* skip test on CI

* fix windows test

* Fix DocumentationSet.Url for reporting first interesting url

* Allow skipping tests

* fix windows test

* fix EnsureIsSubpathOf

* Fix remaining covariance issues

* Additional cleanups and test fixtures

* reorg

* touchups

* start on navigation documentation

* moar docs

* moar docs

* fix test

* fix local build docs errors

* fix cross links

* Fix tests expecting crosslink navigation item

* fix bad test assertion

* post merge fixes

* post merge fixes

* Ensure url has no trailing slash

* Give SiteNavigation a fixed navigation title

* Fix smoke test check for trailing slash

* fix tests expecting trailing slash

* Re-enable the detection rule extension.

* assembler build fixes for detection rule files

* Clean up configuration folder

* Fix tests

* Implementsome of copilots feedback

* remove left over bad imports

---------

Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]>

Author: Mpdreamz

.github/workflows/smoke-test.yml

@@ -12,11 +12,11 @@ jobs:
       matrix: 
         include:
           - repository: elastic/docs-content
-            landing-page-path-output: /docs/
+            landing-page-path-output: /docs
           - repository: elastic/elasticsearch
-            landing-page-path-output: /docs/reference/
+            landing-page-path-output: /docs/reference
           - repository: elastic/opentelemetry
-            landing-page-path-output: /docs/reference/
+            landing-page-path-output: /docs/reference
 
             # This is a random repository that should not have a docset.yml
           - repository: elastic/oblt-actions

Directory.Packages.props

@@ -19,13 +19,15 @@
     <PackageVersion Include="Amazon.Lambda.S3Events" Version="3.1.0" />
     <PackageVersion Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.4.4" />
     <PackageVersion Include="Amazon.Lambda.SQSEvents" Version="2.2.0" />
+    <PackageVersion Include="AngleSharp" Version="1.1.2" />
     <PackageVersion Include="Aspire.Hosting" Version="9.5.2" />
     <PackageVersion Include="Aspire.Hosting.Testing" Version="9.4.2" />
     <PackageVersion Include="AWSSDK.Core" Version="4.0.3" />
     <PackageVersion Include="AWSSDK.SQS" Version="4.0.2.2" />
     <PackageVersion Include="AWSSDK.S3" Version="4.0.7.14" />
     <PackageVersion Include="Elastic.OpenTelemetry" Version="1.1.0" />
     <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.13.0" />
+    <PackageVersion Include="Generator.Equals" Version="3.2.1" PrivateAssets="all" IncludeAssets="runtime; build; native; contentfiles; analyzers; buildtransitive" />
     <PackageVersion Include="KubernetesClient" Version="17.0.14" />
     <PackageVersion Include="Elastic.Aspire.Hosting.Elasticsearch" Version="9.3.0" />
     <PackageVersion Include="Elastic.Clients.Elasticsearch" Version="9.1.4" />