Update contributor documentation to reflect changes in input discovery (#1060)

* Update contributor documentation about documentation input discovery

rdar://139494777

* Update copyright year for updated contributor documentation article

* Avoid deprecated API in code example in contributor documentation

* Mention catalog content file extensions in top-level contributor documentation

* Add more documentation for `DocumentationContext/InputsProvider`

* Correct "catalog" terminology in some contributor documentation

* Add more documentation for `DocumentationBundle`

* Apply suggestions from code review

Co-authored-by: Maya Epps <[email protected]>

---------

Co-authored-by: Maya Epps <[email protected]>

Author: d-ronnqvist

Sources/SwiftDocC/Infrastructure/DocumentationBundle.swift

@@ -10,20 +10,30 @@
 
 import Foundation
 
-/**
- A documentation bundle.
-
- A documentation bundle stores all of the authored content and metadata for a collection of topics and/or frameworks.
-
- No content data is immediately loaded when creating a `DocumentationBundle` except for its `Info.plist`. Its purpose is to provide paths on disk for documentation resources.
-
- ## Topics
- ### Bundle Metadata
-
- - ``displayName``
- - ``identifier``
- - ``version``
- */
+/// A collection of the build inputs for a unit of documentation.
+///
+/// A unit of documentation may for example cover a framework, library, or tool.
+/// Projects or packages may have multiple units of documentation to represent the different consumable products in that project or package.
+///
+/// ## Topics
+///
+/// ### Input files
+///
+/// - ``markupURLs``
+/// - ``symbolGraphURLs``
+/// - ``miscResourceURLs``
+///
+/// ### Render customization
+///
+/// - ``customHeader``
+/// - ``customFooter``
+/// - ``themeSettings``
+///
+/// ### Metadata
+///
+/// - ``info``
+/// - ``displayName``
+/// - ``identifier``
 public struct DocumentationBundle {
     public enum PropertyListError: DescribedError {
         case invalidVersionString(String)
@@ -39,21 +49,17 @@ public struct DocumentationBundle {
         }
     }
 
-    /// Information about this documentation bundle that's unrelated to its documentation content.
+    /// Non-content information or metadata about this unit of documentation.
     public let info: Info
 
-    /**
-     The bundle's human-readable display name.
-     */
+    /// A human-readable display name for this unit of documentation.
     public var displayName: String {
         info.displayName
     }
 
-    /**
-     The documentation bundle identifier.
-
-     An identifier string that specifies the app type of the bundle. The string should be in reverse DNS format using only the Roman alphabet in upper and lower case (A–Z, a–z), the dot (“.”), and the hyphen (“-”).
-     */
+    /// A identifier for this unit of documentation
+    ///
+    /// The string is typically in reverse DNS format using only the Roman alphabet in upper and lower case (A–Z, a–z), the dot (“.”), and the hyphen (“-”).
     public var identifier: String {
         info.identifier
     }
@@ -72,13 +78,25 @@ public struct DocumentationBundle {
     @available(*, deprecated, message: "This deprecated API will be removed after 6.1 is released")
     public var attributedCodeListings: [String: AttributedCodeListing] = [:]
 
-    /// Symbol Graph JSON files for the modules documented by this bundle.
+    /// Symbol graph JSON input files for the module that's represented by this unit of documentation.
+    ///
+    /// Tutorial or article-only documentation won't have any symbol graph JSON files.
+    ///
+    /// ## See Also
+    ///
+    /// - ``DocumentationBundleFileTypes/isSymbolGraphFile(_:)``
     public let symbolGraphURLs: [URL]
 
-    /// DocC Markup files of the bundle.
+    /// Documentation markup input files for this unit of documentation.
+    ///
+    /// Documentation markup files include both articles, documentation extension files, and tutorial files.
+    ///
+    /// ## See Also
+    ///
+    /// - ``DocumentationBundleFileTypes/isMarkupFile(_:)``
     public let markupURLs: [URL]
 
-    /// Miscellaneous resources of the bundle.
+    /// Miscellaneous resources (for example images, videos, or downloadable assets) for this unit of documentation.
     public let miscResourceURLs: [URL]
 
     /// A custom HTML file to use as the header for rendered output.
@@ -90,21 +108,19 @@ public struct DocumentationBundle {
     /// A custom JSON settings file used to theme renderer output.
     public let themeSettings: URL?
 
-    /**
-    A URL prefix to be appended to the relative presentation URL.
-    
-    This is used when a bundle's documentation is hosted in a known location.
-    */
+    /// A URL prefix to be appended to the relative presentation URL.
+    ///
+    /// This is used when a built documentation is hosted in a known location.
     public let baseURL: URL
 
-    /// Creates a documentation bundle.
+    /// Creates a new collection of build inputs for a unit of documentation.
     ///
     /// - Parameters:
-    ///   - info: Information about the bundle.
+    ///   - info: Non-content information or metadata about this unit of documentation.
     ///   - baseURL: A URL prefix to be appended to the relative presentation URL.
-    ///   - symbolGraphURLs: Symbol Graph JSON files for the modules documented by the bundle.
-    ///   - markupURLs: DocC Markup files of the bundle.
-    ///   - miscResourceURLs: Miscellaneous resources of the bundle.
+    ///   - symbolGraphURLs: Symbol graph JSON input files for the module that's represented by this unit of documentation.
+    ///   - markupURLs: Documentation markup input files for this unit of documentation.
+    ///   - miscResourceURLs: Miscellaneous resources (for example images, videos, or downloadable assets) for this unit of documentation.
     ///   - customHeader: A custom HTML file to use as the header for rendered output.
     ///   - customFooter: A custom HTML file to use as the footer for rendered output.
     ///   - themeSettings: A custom JSON settings file used to theme renderer output.

Sources/SwiftDocC/Infrastructure/Input Discovery/DocumentationInputsProvider.swift

@@ -14,6 +14,43 @@ import SymbolKit
 extension DocumentationContext {
 
     /// A type that provides inputs for a unit of documentation.
+    ///
+    /// The inputs provider discovers documentation catalogs on the file system and creates a ``DocumentationBundle`` from the discovered catalog content.
+    ///
+    /// The input provider categorizes the catalog content based on corresponding ``DocumentationBundleFileTypes`` conditions:
+    ///
+    ///  Category                                 | Condition
+    ///  ---------------------------------------- | -------------------------------------------------
+    ///  ``DocumentationBundle/markupURLs``       | ``DocumentationBundleFileTypes/isMarkupFile(_:)``
+    ///  ``DocumentationBundle/symbolGraphURLs``  | ``DocumentationBundleFileTypes/isSymbolGraphFile(_:)``
+    ///  ``DocumentationBundle/info``             | ``DocumentationBundleFileTypes/isInfoPlistFile(_:)``
+    ///  ``DocumentationBundle/themeSettings``    | ``DocumentationBundleFileTypes/isThemeSettingsFile(_:)``
+    ///  ``DocumentationBundle/customHeader``     | ``DocumentationBundleFileTypes/isCustomHeader(_:)``
+    ///  ``DocumentationBundle/customFooter``     | ``DocumentationBundleFileTypes/isCustomFooter(_:)``
+    ///  ``DocumentationBundle/miscResourceURLs`` | Any file not already matched above.
+    ///
+    /// ## Topics
+    ///
+    /// ### Catalog discovery
+    ///
+    /// Discover documentation catalogs and create documentation build inputs from the discovered catalog's content.
+    ///
+    /// - ``findCatalog(startingPoint:allowArbitraryCatalogDirectories:)``
+    /// - ``makeInputs(contentOf:options:)``
+    ///
+    /// ### Input discovery
+    ///
+    /// Discover documentation build inputs from a mix of discovered documentation catalogs and other command line options.
+    ///
+    /// - ``inputsAndDataProvider(startingPoint:allowArbitraryCatalogDirectories:options:)``
+    ///
+    /// ### Errors
+    ///
+    /// Errors that the inputs provider can raise while validating the discovered inputs.
+    ///
+    /// - ``MultipleCatalogsError``
+    /// - ``NotEnoughInformationError``
+    /// - ``InputsFromSymbolGraphError``
     package struct InputsProvider {
         /// The file manager that the provider uses to read file and directory contents from the file system.
         private var fileManager: FileManagerProtocol

Sources/SwiftDocC/SwiftDocC.docc/SwiftDocC.md

@@ -4,15 +4,17 @@ Combine code comments with markup prose to produce structured, semantic document
 
 ## Overview
 
-DocC comes with built-in support for several types of input files. You organize these files by placing them in a folder with a `.docc` extension. This folder is called a documentation bundle, and can include these file types:
+DocC comes with built-in support for several types of input files. You group these files by placing them in a folder with a `.docc` extension. This folder is called a documentation catalog, and can include these file types:
 
- - Symbol-graph files, in JSON format, that describe available symbols in a framework.
- - Lightweight markdown files that contain free-form articles and more.
- - Tutorial files that include dynamic, learning content.
- - Asset files like images, videos, and archived projects for download.
- - An `Info.plist` file that contains metadata about the bundle.
+ - Lightweight markdown files that contain free-form articles or additional symbol documentation, with an `.md` extension.
+ - Tutorial files that include dynamic learning content, with a `.tutorial` extension.
+ - Asset files like images, videos, and archived projects for download, with known extensions like `.png`, `.jpg`, `.mov`, and `.zip`.
+ - Symbol-graph files that describe the symbols of your API, with an `.symbols.json` extension.
+ - An `Info.plist` file with optional metadata about the documentation.
+ - A `theme-settings.json` with theming customizations for the rendered output.
 
-SwiftDocC provides the APIs you use to load a bundle, parse the symbol-graph meta-information, extract symbol documentation, and optionally pair that symbol documentation with external file content. DocC represents the compiled documentation in an in-memory model that you can further convert in a persistable representation for writing to disk.
+SwiftDocC provides the APIs you use to discover documentation inputs, load catalog content, parse the symbol-graph meta-information, extract symbol documentation, and pair that symbol documentation with external file content. 
+DocC represents the compiled documentation in an in-memory model that you can further convert in a persistable representation for writing to disk.
 
 ## Topics 
 
@@ -22,7 +24,7 @@ SwiftDocC provides the APIs you use to load a bundle, parse the symbol-graph met
 
 ### Content Discovery
 
-- <doc:DocumentationWorkspaceGroup>
+- <doc:InputDiscovery>
 - <doc:DocumentationContextGroup>
 
 ### Resolving documentation links

Sources/SwiftDocC/SwiftDocC.docc/SwiftDocC/Benchmarking.md

@@ -8,7 +8,7 @@ When you are working on a PR to add a feature or fix a bug you should evaluate t
 
 ## Running a benchmark
 
-To benchmark the `convert` command with a given documentation bundle `MyFramework.docc` run:
+To benchmark the `convert` command with a given documentation catalog `MyFramework.docc` run:
 
 ```
 swift run --package-path bin/benchmark benchmark --docc-arguments convert MyFramework.docc
@@ -80,4 +80,4 @@ benchmark(add: BundlesCount(context: context))
 - ``benchmark(end:benchmarkLog:)``
 - ``benchmark(wrap:benchmarkLog:body:)``
 
-<!-- Copyright (c) 2021-2022 Apple Inc and the Swift Project authors. All Rights Reserved. -->
+<!-- Copyright (c) 2021-2024 Apple Inc and the Swift Project authors. All Rights Reserved. -->

Sources/SwiftDocC/SwiftDocC.docc/SwiftDocC/CompilerPipeline.md

@@ -6,23 +6,24 @@ Get to know the steps involved in documentation compilation.
 
 This article describes the discrete and sequential steps of compiling documentation with DocC.
 
-DocC starts with content discovery by parsing the documentation sources in your documentation bundle. Next, it validates and semantically analyzes them and then builds an in-memory model of the compiled documentation. Once the in-memory model is finalized, DocC converts each topic into a persistable representation it can store on disk.
-
-To use the compiled documentation, either query the in-memory model directly or convert its nodes to their render-friendly representation. For example, the `SwiftDocCUtilities` framework enumerates all the nodes in DocC's in-memory model, converts each node for rendering, and finally writes the complete documentation to the disk.
+DocC starts with input discovery by categorizing the documentation sources in your documentation catalog. Next, it loads and parses the those inputs to create in-memory models of the documentation pages. Once the in-memory model is finalized, DocC converts each topic into a persistable, render-friendly representation it can store on disk.
 
 ### Discovery
 
-DocC starts discovery by creating a ``DocumentationWorkspace`` to interact with the file system and a ``DocumentationContext`` that manages the in-memory model for the built documentation.
+DocC starts by creating a ``DocumentationContext/InputsProvider`` to discover the inputs from the user-provided command line arguments. These inputs are:
 
-When a documentation bundle is found in the workspace by a ``DocumentationWorkspaceDataProvider``, the following files are recognized and processed (others are ignored):
+ - Markup files, tutorial files, and assets (for example images)
+ - Symbol graph files, describing the symbols in a given module (types, functions, variables, etc.) and their relationships (inheritance, conformance, etc.)
+ - Meta information about this "unit" of documentation (for example a custom display name)
+ - Customizations to the render template.
 
-- An `Info.plist` file containing meta information like the bundle display name.
-- Symbol-graph files with the `.symbols.json` extension.
-- Authored markup files with an `.md` extension
-- Authored tutorial files with a `.tutorial` extension
-- Additional documentation assets with known extensions like `.png`, `.jpg`, `.mov`, and `.zip`.
+Markup, tutorials, assets, and render-template-customization can only be discovered as files inside of a documentation catalog (`.docc` directory).
+Symbol graph files can either be discovered as files inside of a documentation catalog or as additional files provided via user-provided command line arguments.
+Meta information can either be discovered from an optional top-level `Info.plist` file inside of a documentation catalog or as provided values via user-provided command line arguments. All meta information is optional.
 
-You can organize the files in any way, as long as `Info.plist` is in the root of the directory tree. Here is an example of a bundle, that groups topic files in logical groups with an additional directory for shared asset files:
+You can organize the files inside the documentation catalog according to your preference, 
+as long as the optional `Info.plist`--containing optional meta information--and the optional render customization files are top-level.
+For example, this catalog groups files based on their topic with an additional directory for shared asset files:
 
 ```none
 SwiftDocC.docc
@@ -45,9 +46,10 @@ SwiftDocC.docc
 
 ### Analysis and Registration
 
-This phase starts with registering all symbols from the available symbol graphs into a documentation *topic graph* in memory. 
+This phase starts with creating a ``DocumentationContext`` using the discovered inputs from the previous phase. 
+This begins loading and registering the inputs with the context.
 
-The symbol graph files are machine generated and describe all available symbols in a framework (types, functions, variables, etc.) and their relationships, for example, inheritance and conformance.
+The first input that the context registers is the symbol information. The symbol information comes from "symbol graph files" which are machine generated and describe all available symbols in a framework (types, functions, variables, etc.) and their relationships (inheritance, conformance, etc.).
 
 Each symbol becomes a documentation node in the topic graph and thus a documentation *topic* (an entity in the documentation model). The symbol's topic could optionally be extended with authored documentation from a markup file.
 
@@ -57,7 +59,7 @@ Next, all the remaining markup files are analyzed and converted to documents (fo
 
 Finally, if you reference any symbols from another framework, and DocC knows how to resolve those, the symbols are fetched and added to the graph too.
 
-### Curation
+#### Curation
 
 At this point the in-memory topic graph accurately represents the machine generated description of the documented framework. However, documentation is often better consumed when it's curated into logical groups and into an incremental learning experience.
 
@@ -102,4 +104,4 @@ The file hierarchy under the output path represents the complete, compiled docum
 ╰ videos
 ```
 
-<!-- Copyright (c) 2021 Apple Inc and the Swift Project authors. All Rights Reserved. -->
+<!-- Copyright (c) 2021-2024 Apple Inc and the Swift Project authors. All Rights Reserved. -->

Sources/SwiftDocC/SwiftDocC.docc/SwiftDocC/DocumentationContextGroup.md

@@ -4,8 +4,9 @@ Build and query the in-memory documentation model.
 
 ## Discussion
 
-The documentation context generally manages the in-memory documentation including:
- 
+A documentation context is the the in-memory representation of a "unit" of documentation (for example a module, package, or technology). 
+The context is generally responsible for:
+
  - Analyzing bundle file contents and converting to semantic models.
  - Managing a graph of documentation nodes (a single node representing one documentation topic).
  - Processing assets like media files or download archives.
@@ -14,12 +15,17 @@ The documentation context generally manages the in-memory documentation includin
 
 ### Creating a Context
 
+Use ``DocumentationContext/init(bundle:dataProvider:diagnosticEngine:configuration:)`` to create a context for a given bundle:
+
 ```swift
-let workspace = DocumentationWorkspace()
-let context = try DocumentationContext(dataProvider: workspace)
-```
+let inputsProvider = DocumentationContext.InputsProvider()
+let (bundle, dataProvider) = try inputsProvider.inputsAndDataProvider(
+    startingPoint: catalogURL, 
+    options: bundleDiscoveryOptions
+)
 
-During initialization the context will inspect the available bundles in the workspace and load any symbol graph files and markup files.
+let context = try DocumentationContext(bundle: bundle, dataProvider: dataProvider)
+```
 
 ### Accessing Documentation
 
@@ -40,19 +46,11 @@ To find out the location of the source file for a given documentation node use:
 let sourceFileURL = try context.documentURL(for: reference)
 ```
 
-And finally to print all known paths in the context:
-
-```swift
-context.knownIdentifiers.forEach({ print($0) })
-```
-
 ## Topics
 
 ### Documentation Context
 
 - ``DocumentationContext``
-- ``DocumentationContextDataProvider``
-- ``DocumentationContextDataProviderDelegate``
 - ``AutomaticCuration``
 
 ### Documentation Nodes