feat(api-markdown-documenter): Add documentation domain to mdast transformation (microsoft#24787)

Updates the library with a new transformation layer that converts the
`DocumentationNode` representation to
[mdast](https://github.com/syntax-tree/mdast), and replaces the
library's custom Markdown rendering with
[mdast-util-to-markdown](https://github.com/syntax-tree/mdast-util-to-markdown).

For an example of how the website build will be updated in response to
these changes, see here:
microsoft#25072.

Author: Josmithr

tools/api-markdown-documenter/CHANGELOG.md

@@ -2,6 +2,31 @@
 
 ## 0.21.0
 
+### Add DocumentationNode -> mdast transformation layer
+
+Adds transformation library for generating [mdast](https://github.com/syntax-tree/mdast) from `DocumentationNode`s.
+
+#### Example
+
+```typescript
+const modelDirectoryPath = "<PATH-TO-YOUR-DIRECTORY-CONTAINING-API-REPORTS>";
+
+// Create the API Model from our API reports
+const apiModel = await loadModel({
+	modelDirectoryPath,
+});
+
+// Transform the API Model to documents
+const documents = transformApiModel({
+	apiModel,
+});
+
+// Convert the documents to Markdown via mdast
+const markdownDocuments = documents.map((document) => documentToMarkdown(document, {}));
+
+// Use the resulting Markdown documents with your favorite mdast-compatible library!
+```
+
 ### List parsing
 
 Markdown-like list syntax is now supported in TSDoc comments.

tools/api-markdown-documenter/README.md

@@ -176,12 +176,12 @@ title: Markdown
 graph LR
     A[ApiModel]
     B[Documentation AST]
-    C[Markdown AST~]
+    C[Markdown AST]
     D[raw Markdown]
 
     A-->|transformApiModel|B
-    B-->|documentToMarkdown~|C
-    C-->|MarkdownRenderer.renderMarkdown~|D
+    B-->|documentToMarkdown|C
+    C-->|MarkdownRenderer.renderMarkdown|D
 
     A-.->|MarkdownRenderer.renderApiModel|D
     B-.->|MarkdownRenderer.renderDocument|D
@@ -207,10 +207,6 @@ graph LR
     B-.->|HtmlRenderer.renderDocument|D
 ```
 
-**Note:** APIs above marked with an `*` are in preview, and may change without notice.
-
-**Note:** APIs above marked with an `~` are planned, but do not yet exist.
-
 For more details on the interior `Documentation AST` ([Abstract Syntax Tree][]) domain, see [Documentation Domain](#documentation-domain) below.
 
 ### API-Extractor
@@ -241,7 +237,7 @@ Then, if you need the new node to be usable in specific contexts (e.g., `Phrasin
 
 ##### Example
 
-<!-- AUTO-GENERATED-CONTENT:START (INCLUDE_CODE:path=./examples/ExtendDocumentationDomain.ts&language=typescript&start=13&end=35) -->
+<!-- AUTO-GENERATED-CONTENT:START (INCLUDE_CODE:path=./examples/ExtendDocumentationDomain.ts&language=typescript&start=17&end=39) -->
 
 <!-- prettier-ignore-start -->
 <!-- NOTE: This section is automatically generated by embedding the referenced file contents. Do not update these generated contents directly. -->
@@ -289,6 +285,38 @@ If you would like to add rendering support for a custom `Documentation Domain` n
 
 If you would like to change any or all of this library's default rendering policies, you may simply override the default policies for the desired `type`s.
 
+### ToMarkdown Transformation
+
+This library includes APIs for transforming `Documentation Domain` trees to Markdown syntax trees using [mdast](https://github.com/syntax-tree/mdast).
+
+<!-- AUTO-GENERATED-CONTENT:START (INCLUDE_CODE:path=./examples/ToMarkdown.ts&language=typescript&start=11&end=-5) -->
+
+<!-- prettier-ignore-start -->
+<!-- NOTE: This section is automatically generated by embedding the referenced file contents. Do not update these generated contents directly. -->
+
+```typescript
+const modelDirectoryPath = "<PATH-TO-YOUR-DIRECTORY-CONTAINING-API-REPORTS>";
+
+// Create the API Model from our API reports
+const apiModel = await loadModel({
+	modelDirectoryPath,
+});
+
+// Transform the API Model to documents
+const documents = transformApiModel({
+	apiModel,
+});
+
+// Convert the documents to Markdown via mdast
+const markdownDocuments = documents.map((document) => documentToMarkdown(document, {}));
+
+// Use the resulting Markdown documents with your favorite mdast-compatible library!
+```
+
+<!-- prettier-ignore-end -->
+
+<!-- AUTO-GENERATED-CONTENT:END -->
+
 ### HtmlRenderer
 
 This library includes APIs for HTML rendering.
@@ -367,10 +395,12 @@ Other validation may be added in the future as needed.
 
 -   Rather than repeatedly walking up a given `ApiItem`'s hierarchy when evaluating paths, links, etc., we could pass down transformation context object containing a running record of the item's hierarchy as we walk the tree.
 
-## Longer-term work
+### Longer-term work
 
--   Add `documentToMarkdown` API that generates `mdast` output.
-    -   Update Markdown rendering APIs to leverage the `mdast` and `hast` domain outputs.
+- Replace [Documentation Domain](#documentation-domain) with `mdast`.
+  - The `mdast` ecosystem already supports conversion from `mdast` to HTML and other formats.
+    There really isn't a reason for this library to have its own proprietary format.
+    For the couple of special concepts this library does have (e.g. hierarchical sections with contextual heading levels), we can leverage `mdast`'s extensibility model.
 
 <!-- AUTO-GENERATED-CONTENT:START (README_FOOTER) -->
 

tools/api-markdown-documenter/api-report/api-markdown-documenter.beta.api.md

@@ -25,16 +25,25 @@ import { ApiProperty } from '@microsoft/api-extractor-model';
 import { ApiPropertySignature } from '@microsoft/api-extractor-model';
 import { ApiTypeAlias } from '@microsoft/api-extractor-model';
 import { ApiVariable } from '@microsoft/api-extractor-model';
+import type { BlockContent as BlockContent_2 } from 'mdast';
 import type { Data } from 'unist';
 import { DocSection } from '@microsoft/tsdoc';
 import { Excerpt } from '@microsoft/api-extractor-model';
+import type { ListItem } from 'mdast';
 import type { Literal } from 'unist';
 import { NewlineKind } from '@rushstack/node-core-library';
 import type { Node as Node_2 } from 'unist';
 import type { Nodes } from 'hast';
+import type { Nodes as Nodes_2 } from 'mdast';
+import { Options } from 'mdast-util-to-markdown';
 import type { Parent } from 'unist';
+import type { PhrasingContent as PhrasingContent_2 } from 'mdast';
 import { ReleaseTag } from '@microsoft/api-extractor-model';
 import type { Root } from 'hast';
+import type { Root as Root_2 } from 'mdast';
+import type { RootContent } from 'mdast';
+import type { TableCell } from 'mdast';
+import type { TableRow } from 'mdast';
 import { TypeParameter } from '@microsoft/api-extractor-model';
 
 // @public
@@ -173,6 +182,14 @@ export interface BlockContentMap {
     table: TableNode;
 }
 
+// @public
+export function blockContentToMarkdown(node: BlockContent, context: ToMarkdownContext): [BlockContent_2];
+
+// @public
+export type BlockContentToMarkdownTransformations = {
+    readonly [K in keyof BlockContentMap]: ToMarkdownTransformation<BlockContentMap[K], BlockContent_2[]>;
+};
+
 // @public @sealed
 export class CodeSpanNode extends DocumentationLiteralNodeBase<string> {
     constructor(value: string);
@@ -338,6 +355,9 @@ export interface DocumentNodeProps {
 // @public
 export function documentToHtml(document: DocumentNode, config: ToHtmlConfiguration): Root;
 
+// @public
+export function documentToMarkdown(document: DocumentNode, config: ToMarkdownConfiguration): Root_2;
+
 // @public
 export interface DocumentWriter {
     decreaseIndent(): void;
@@ -360,6 +380,7 @@ export namespace DocumentWriter {
 export class FencedCodeBlockNode extends DocumentationLiteralNodeBase<string> {
     constructor(value: string, language?: string);
     static createFromPlainText(text: string, language?: string): FencedCodeBlockNode;
+    static readonly Empty: FencedCodeBlockNode;
     get isEmpty(): boolean;
     readonly language?: string;
     readonly type = "fencedCode";
@@ -640,38 +661,19 @@ export interface LoggingConfiguration {
 // @public
 export type LoggingFunction = (message: string | Error, ...parameters: unknown[]) => void;
 
-// @public
-export interface MarkdownRenderConfiguration extends LoggingConfiguration {
-    readonly customRenderers?: MarkdownRenderers;
-    readonly startingHeadingLevel?: number;
-}
-
-// @public
-export interface MarkdownRenderContext extends TextFormatting {
-    readonly customRenderers?: MarkdownRenderers;
-    readonly headingLevel: number;
-    readonly insideCodeBlock?: boolean;
-    readonly insideTable?: boolean;
-}
-
 declare namespace MarkdownRenderer {
     export {
         RenderApiModelAsMarkdownOptions as RenderApiModelOptions,
         renderApiModelAsMarkdown as renderApiModel,
         RenderDocumentsAsMarkdownOptions as RenderDocumentsOptions,
         renderDocumentsAsMarkdown as renderDocuments,
         renderDocument_2 as renderDocument,
-        renderNode,
-        renderNodes
+        RenderDocumentAsMarkdownConfiguration,
+        renderMarkdown
     }
 }
 export { MarkdownRenderer }
 
-// @public
-export interface MarkdownRenderers {
-    readonly [documentationNodeKind: string]: (node: DocumentationNode, writer: DocumentWriter, context: MarkdownRenderContext) => void;
-}
-
 export { NewlineKind }
 
 // @public @sealed
@@ -699,6 +701,14 @@ export interface PhrasingContentMap {
     text: PlainTextNode;
 }
 
+// @public
+export function phrasingContentToMarkdown(node: PhrasingContent, context: ToMarkdownContext): [PhrasingContent_2];
+
+// @public
+export type PhrasingContentToMarkdownTransformations = {
+    readonly [K in keyof PhrasingContentMap]: ToMarkdownTransformation<PhrasingContentMap[K], PhrasingContent_2[]>;
+};
+
 // @public @sealed
 export class PlainTextNode extends DocumentationLiteralNodeBase<string> {
     constructor(text: string);
@@ -716,24 +726,28 @@ export { ReleaseTag }
 function renderApiModelAsMarkdown(options: RenderApiModelAsMarkdownOptions): Promise<void>;
 
 // @public
-interface RenderApiModelAsMarkdownOptions extends ApiItemTransformationOptions, MarkdownRenderConfiguration, FileSystemConfiguration {
+interface RenderApiModelAsMarkdownOptions extends ApiItemTransformationOptions, RenderDocumentAsMarkdownConfiguration, FileSystemConfiguration {
 }
 
 // @public
 function renderDocument(document: DocumentNode, config: RenderDocumentAsHtmlConfiguration): string;
 
 // @public
-function renderDocument_2(document: DocumentNode, config: MarkdownRenderConfiguration): string;
+function renderDocument_2(document: DocumentNode, config: RenderDocumentAsMarkdownConfiguration): string;
 
 // @public @sealed
 export interface RenderDocumentAsHtmlConfiguration extends ToHtmlConfiguration, RenderHtmlConfiguration {
 }
 
+// @public @sealed
+export interface RenderDocumentAsMarkdownConfiguration extends ToMarkdownConfiguration, RenderMarkdownConfiguration {
+}
+
 // @public
 function renderDocumentsAsMarkdown(documents: readonly DocumentNode[], options: RenderDocumentsAsMarkdownOptions): Promise<void>;
 
 // @public
-interface RenderDocumentsAsMarkdownOptions extends MarkdownRenderConfiguration, FileSystemConfiguration {
+interface RenderDocumentsAsMarkdownOptions extends RenderDocumentAsMarkdownConfiguration, FileSystemConfiguration {
 }
 
 // @public
@@ -745,14 +759,19 @@ export interface RenderHtmlConfiguration {
 }
 
 // @public
-function renderNode(node: DocumentationNode, writer: DocumentWriter, context: MarkdownRenderContext): void;
+function renderMarkdown(tree: Nodes_2, config: RenderMarkdownConfiguration): string;
 
-// @public
-function renderNodes(children: DocumentationNode[], writer: DocumentWriter, childContext: MarkdownRenderContext): void;
+// @public @sealed
+export interface RenderMarkdownConfiguration {
+    readonly mdastToMarkdownOptions?: Partial<Options>;
+}
 
 // @public
 export type SectionContent = BlockContent | SectionNode;
 
+// @public
+export function sectionContentToMarkdown(node: SectionContent, context: ToMarkdownContext): RootContent[];
+
 // @public @sealed
 export interface SectionHierarchyConfiguration extends DocumentationHierarchyConfigurationBase {
     readonly kind: HierarchyKind.Section;
@@ -870,6 +889,31 @@ export interface ToHtmlTransformations {
     readonly [documentationNodeKind: string]: ToHtmlTransformation;
 }
 
+// @public
+export interface ToMarkdownConfiguration extends LoggingConfiguration {
+    readonly customTransformations?: Partial<ToMarkdownTransformations>;
+    readonly startingHeadingLevel?: number;
+}
+
+// @public
+export interface ToMarkdownContext {
+    readonly headingLevel: number;
+    readonly logger: Logger;
+    readonly transformations: ToMarkdownTransformations;
+}
+
+// @public
+export type ToMarkdownTransformation<TIn extends DocumentationNode = DocumentationNode, TOut extends Nodes_2[] = [Nodes_2]> = (node: TIn, context: ToMarkdownContext) => TOut;
+
+// @public
+export type ToMarkdownTransformations = BlockContentToMarkdownTransformations & PhrasingContentToMarkdownTransformations & {
+    readonly ["heading"]: ToMarkdownTransformation<HeadingNode, BlockContent_2[]>;
+    readonly ["listItem"]: ToMarkdownTransformation<ListItemNode, [ListItem]>;
+    readonly ["section"]: ToMarkdownTransformation<SectionNode, RootContent[]>;
+    readonly ["tableCell"]: ToMarkdownTransformation<TableCellNode, [TableCell]>;
+    readonly ["tableRow"]: ToMarkdownTransformation<TableRowNode, [TableRow]>;
+};
+
 // @public
 export type TransformApiItemWithChildren<TApiItem extends ApiItem> = (apiItem: TApiItem, config: ApiItemTransformationConfiguration, generateChildSection: (apiItem: ApiItem) => SectionNode[]) => SectionNode[];