Add basic content on what's needed to build docs with docs-builder (#338)

* add basic conversion doc

* add more updates

* add migration tooling docs

Author: bmorelli25

docs/configure/content-set/attributes.md

@@ -1,3 +1,25 @@
 # Attributes
 
-tbd
+To promote consistency across documentation, AsciiDoc uses shared attributes for common terms, URLs, and versions.
+
+In the AsciiDoctor-based system, shared attributes are defined at the site-level and live in the [`shared/` directory](https://github.com/elastic/docs/blob/master/shared) in the elastic/docs repo. The most used files in this directory are:
+
+* The [`attributes.asciidoc` file](https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc), which contains URLs, common words and phrases, and more.
+* The files in the [`versions/stack` directory](https://github.com/elastic/docs/tree/master/shared/versions/stack), which contain the latest versions for various products for a given Stack version.
+
+## In `docs-builder`
+
+Attributes are defined at the content set-level. Use the `subs` key to define attributes as key-value pairs in the content sets `docset.yml` or `toc.yml` file.
+
+Example:
+
+```yml
+subs:
+  attr-name:   "attr value"
+  ea:   "Elastic Agent"
+  es:   "Elasticsearch"
+```
+
+## Use attributes
+
+Attributes can be referenced in documentation file with the following syntax: `{{attr-name-here}}`.

docs/configure/content-set/file-structure.md

@@ -1,3 +1,9 @@
 # File structure
 
-tbd
+The file structure of each content set directly impacts the url path structure of each page. In other words, the directory structure you pull into docs-builder is the same as the directory structure it spits out.
+
+## Navigation
+
+Navigation is set independent of directory and file structure in docset and toc files.
+
+See [Navigation](./navigation.md) to learn more.

docs/configure/content-set/index.md

@@ -1,12 +1,20 @@
 ---
-navigation_title: Content set
+navigation_title: Content set-level
 ---
 
-# Content set configuration
+# Content set-level configuration
 
-Now we'll zoom into the AsciiDoc book level, and explore the V3 equivalent: content sets. At the book level, the system consists of:
+Elastic documentation is spread across multiple repositories. Each repository can contain one or more content sets. A content set is a group of documentation pages whose navigation is defined by a single `docset.yml` file. `docs-builder` builds each content set in isolation, ensuring that changes in one repository don’t affect another.
+
+A content set in `docs-builder` is equivalent to an AsciiDoc book. At this level, the system consists of:
 
 | System property | Asciidoc | V3 |
 | -------------------- | -------------------- | -------------------- |
 | **Content source files** --> A whole bunch of markup files as well as any other assets used in the docs (for example, images, videos, and diagrams). | **Markup**: AsciiDoc files **Assets**: Images, videos, and diagrams | **Markup**: MD files **Assets**: Images, videos, and diagrams |
-| **Information architecture** --> A way to specify the order in which these text-based files should appear in the information architecture of the book. | `index.asciidoc` file (this can be spread across several AsciiDoc files, but generally starts with the index file specified in the `conf.yaml` file)) | TBD |
+| **Information architecture** --> A way to specify the order in which these text-based files should appear in the information architecture of the book. | `index.asciidoc` file (this can be spread across several AsciiDoc files, but generally starts with the index file specified in the `conf.yaml` file)) | `docset.yml` and/or `toc.yml` file(s) |
+
+## Learn more
+
+* [File structure](./file-structure.md)
+* [Navigation](./navigation.md)
+* [Attributes](./attributes.md)

docs/configure/content-set/navigation.md

@@ -1,3 +1,125 @@
 # Navigation
 
-tbd
+Two types of nav files are supported: `docset.yml` and `toc.yml`.
+
+## `docset.yml`
+
+Example:
+
+```yaml
+project: 'PROJECT_NAME'
+
+external_hosts:
+  - EXTERNAL_LINKS_HERE
+
+exclude:
+  - 'EXCLUDED_FILES'
+
+toc:
+  - file: index.md
+  - toc: elastic-basics
+  - folder: top-level-bucket-a
+    children:
+      - file: index.md
+      - file: file-a.md
+      - file: file-b.md
+  - folder: top-level-bucket-b
+    children:
+      - file: index.md
+      - folder: second-level-bucket-c
+        children:
+          - file: index.md
+```
+
+### `project`
+
+The name of the project.
+
+Example:
+
+```yaml
+project: 'APM Java agent reference'
+```
+
+### `external_hosts`
+
+All links to external hosts must be declared in this section of `docset.yml`.
+
+Example:
+
+```yaml
+external_hosts:
+  - google.com
+  - github.com
+```
+
+### `exclude`
+
+Files to exclude from the TOC. Supports glob patterns.
+
+The following example excludes all markdown files beginning with `_`:
+
+```yaml
+exclude:
+  - '_*.md'
+```
+
+### `toc`
+
+Defines the table of contents (navigation) for the content set. A minimal toc is:
+
+```yaml
+toc:
+  - file: index.md
+```
+
+The table of contents can be created independent of the directory structure of the files it defines. You can use directories to define nesting in the TOC, but you don't have to. For example, both of the following create the same nav structure:
+
+```yaml
+  ...
+  - file: subsection/index.md
+    children:
+      - file: subsection/page-one.md
+      - file: subsection/page-two.md
+```
+
+```yaml
+  ...
+  - folder: subsection
+    children:
+      - file: index.md
+      - file: page-one.md
+      - file: page-two.md
+```
+
+#### Nest `toc`
+
+The `toc` key can include nested `toc.yml` files.
+
+The following example includes two sub-`toc.yml` files located in directories named `elastic-basics` and `solutions`:
+
+```yml
+toc:
+  - file: index.md
+  - toc: elastic-basics
+  - toc: solutions
+```
+
+### Attributes
+
+Example:
+
+```yml
+subs:
+  attr-name:   "attr value"
+  ea:   "Elastic Agent"
+  es:   "Elasticsearch"
+```
+
+See [Attributes](./attributes.md) to learn more.
+
+## `toc.yml`
+
+As a rule, each `docset.yml` file can only be included once in the assembler. This prevents us from accidentally duplicating pages in the docs. However, there are times when you want to split content sets and include them partially in different areas of the TOC. That's what `toc.yml` files are for. These files split one documentation set into multiple “sub-TOCs,” each mapped to a different navigation node.
+
+All configuration options that `docset.yml` supports are supported by `toc.yml`.

docs/configure/index.md

@@ -4,6 +4,40 @@ navigation_title: Configuration reference
 
 # Configure Elastic Docs
 
-* [Page configuration](./page.md)
-* [Content-set configuration](./content-set/index.md)
-* [Site configuration](./site/index.md)
+It's first important to understand how we build Elastic Docs:
+
+| System property | Asciidoc | V3 |
+| -------------------- | -------------------- | -------------------- |
+| **Docs build tool** --> An engine used to take the markup in the content sources and transform it into web pages. | Customized version of AsciiDoctor (lives in [**elastic/docs**](https://github.com/elastic/docs)) | Customized doc builder using open source tools (lives in [**elastic/docs-builder**](https://github.com/elastic/docs-builder)) |
+
+When working with `docs-builder`, there are three levels at which you can configure Elastic documentation:
+
+1. Site-level
+2. Content set-level
+3. Page-level
+
+## Site-level configuration
+
+At the site level, you can configure:
+
+* Content sources: where files live
+* Global navigation: how navigations are compiled and presented to users
+
+[Site configuration](./site/index.md)
+
+## Content-level configuration
+
+At the content set level, you can configure:
+
+* Content-set-level and sub-content-set-level navigation: how smaller groups of files are organized and presented to users
+* Attributes: variables that will be substituted at build-time for pre-defined values
+
+[Content-set configuration](./content-set/index.md)
+
+## Page-level configuration
+
+At the page level, you can configure:
+
+* Frontmatter that influences on-page UX to benefit the user in some way
+
+[Page configuration](./page.md)

docs/configure/page.md

@@ -1,11 +1,7 @@
 ---
-navigation_title: Page
+navigation_title: Page-level
 ---
 
-# Page configuration
+# Page-level configuration
 
-tbd
-
-go into frontmatter here?
-
-for now, see [frontmatter](../syntax/frontmatter.md).
+For now, page configuration is minimal. See [frontmatter](../syntax/frontmatter.md) for supported options.

docs/configure/site/content.md

@@ -1,8 +1,26 @@
-# Content config
+# `assembler.yml`
 
-In both the AsciiDoctor- and V3-based system, there is site-wide configuration where you list all content sources, where to find those sources, and in what order they should be added to the site.
+The global navigation is defined in the `assembler.yml` file. This file can roughly be thought of as the V3 equivalent of conf.yaml in the asciidoc build system. This file, which writers own, allows for arbitrary nesting of `docset.yml` and `toc.yml` references. This file will live in the `elastic/docs-content` repository, but will not build or have any influence over the `docs-builder` builds in that repo.
 
-In the AsciiDoctor system, this all happens in one YAML file in the /docs repo. In the V3 system, this ...
+The global navigation that is defined in `assembler.yml` can be composed of three main resources:
+1. Local TOC files: toc.yml files that live in the docs-content repository.
+2. External TOC files: A subset of a content set (represented by a toc.yml file) that is external to the docs-content repository.
+3. External content set declarations: An entire docset.yml file that is external to the docs-content repository.
+
+The `assembler.yml` file might look something like this:
+
+```yaml
+
+```
+
+## Assembler constraints
+
+To maintain each docset’s immutability and self-containment, resources in the global navigation (assembler.yml) must follow specific rules.
+1. A link resource can appear only once in the global navigation, and it must nest directly under another resource (not under individual pages from a different content set). In other words:
+    1. Each docset.yml or toc.yml file can appear only once in the overall navigation.
+    1. Any TOC declarations—whether in docset.yml or toc.yml—must be placed either at the top level or directly under another TOC entry, with no individual files in between.
+1. Nested resources must appear either before or after (default) the parent’s TOC, and they cannot be placed arbitrarily among the parent’s pages (e.g., between two files of the parent).
+1. If an external TOC is linked, all of its child TOCs must also be included in the global navigation.
 
 ## AsciiDoctor conf.yml
 
@@ -35,7 +53,3 @@ In the AsciiDoctor-powered site, content configuration at the site level is done
       repo:   docs
       path:   shared/settings.asciidoc
 ```
-
-## V3 configuration
-
-TO DO

docs/configure/site/index.md

@@ -1,25 +1,42 @@
 ---
-navigation_title: Site
+navigation_title: Site-level
 ---
 
-# Site configuration
-
-Start by understanding how the new V3 system works at the site level compared to how our custom AsciiDoctor system works. The system consists of:
+# Site-level configuration
 
+Start by understanding how the new V3 system works at the site level compared to how our custom AsciiDoctor system works. At the site-level, we have:
 
 | System property | Asciidoc | V3 |
 | -------------------- | -------------------- | -------------------- |
 | **Content sources** --> Collections of markup files containing doc content. These are split up across many docs repos. | _Books_ | _Content sets_ |
-| **Content configuration** --> A way to specify where to find those content sources, and in what order they should be added to the site. | Configuration file ([`conf.yml`](https://github.com/elastic/docs/blob/master/conf.yaml) in elastic/docs) | Config file location TBD |
-| **Cross-site values** --> Key-value pairs that should be substituted across all sources as web pages are built. | Shared attributes ([`shared/`](https://github.com/elastic/docs/tree/master/shared) in elastic/docs) | Shared attrs file TBD |
-| **Docs build tool** --> An engine used to take the markup in the content sources and transform it into web pages. | Customized version of AsciiDoctor (lives in [**elastic/docs**](https://github.com/elastic/docs)) | Customized doc builder using open source tools (lives in [**elastic/docs-builder**](https://github.com/elastic/docs-builder)) |
+| **Content configuration** --> A way to specify where to find those content sources, and in what order they should be added to the site. | Configuration file ([`conf.yml`](https://github.com/elastic/docs/blob/master/conf.yaml) in elastic/docs) | Config file `assembler.yml` |
 
 Where these pieces live and what format they are in varies between the two systems, but they generally achieve the same goal.
 
-## Asciidoc
+## Content sources
 
-![site-level config in the asciidoc system](./img/site-level-asciidoctor.png)
+TBD
+
+## Content configuration
+
+In both the AsciiDoctor- and V3-based system, there is site-wide configuration where you list all content sources, where to find those sources, and in what order they should be added to the site.
+
+In the AsciiDoctor system, this all happens in one YAML file in the `/docs` repo. In the V3 system, this happens in the `assembler.yml` file in `docs-content`.
+
+[assembler.yml](./content.md)
+
+## Landing page
 
-## V3
+See [landing page](./landing-page.md)
 
-DIAGRAM NEEDED
+## Redirects
+
+Plan still needed
+
+## V3 site-level diagram
+
+DIAGRAM NEEDED
+
+## Asciidoc site-level diagram
+
+![site-level config in the asciidoc system](./img/site-level-asciidoctor.png)

docs/configure/site/landing-page.md

@@ -1,3 +1,3 @@
 # Landing page
 
-tbd
+The landing page will initially be an HTML template. To update the landing page, you will directly edit the HTML template. Eventually, we will templetize the landing page for easier updates.

docs/configure/site/redirects.md

@@ -18,7 +18,7 @@ STRATEGY TBD
 
 ## Migration redirects
 
-Docs inventory script: https://github.com/elastic/docs-helpers/tree/main/docs-inventory
-Docs inventory sheet: https://docs.google.com/spreadsheets/d/1LfPI3TZqdpONGxOmL8B8V-Feo1flLwObz9_ibCEMkIQ/edit?gid=605983744#gid=605983744
+* Docs inventory script: https://github.com/elastic/docs-helpers/tree/main/docs-inventory
+* Docs inventory sheet: https://docs.google.com/spreadsheets/d/1LfPI3TZqdpONGxOmL8B8V-Feo1flLwObz9_ibCEMkIQ/edit?gid=605983744#gid=605983744
 
 This sheet will be used to request redirects after migration from 8.current to 9.current.