✨ NEW: Create ToC from file structure (#14)

This is a re-working of `jupyter-book toc` (by @choldgraf) but with improvements such as:

- configurable index name and suffixes
- using "natural sort order" for files/folders
- using `fnmatch` for matching files/folders to skip
- better testing

Author: chrisjsewell

.pre-commit-config.yaml

@@ -5,6 +5,7 @@ exclude: >
     (?x)^(
       \.vscode/settings\.json|
       tests/.*xml|
+      tests/.*txt|
     )$
 
 repos:

README.md

@@ -205,6 +205,75 @@ meta:
   - doc3
 ```
 
+To build a ToC file from an existing site:
+
+```console
+$ sphinx-etoc create-toc path/to/folder
+```
+
+Some rules used:
+
+- Files/folders will be skipped if they match a pattern added by `-s` (based on [fnmatch](https://docs.python.org/3/library/fnmatch.html) Unix shell-style wildcards)
+- Sub-folders with no content files inside will be skipped
+- File and folder names will be sorted by [natural order](https://en.wikipedia.org/wiki/Natural_sort_order)
+- If there is a file called `index` (or the name set by `-i`) in any folder, it will be treated as the index file, otherwise the first file by ordering will be used.
+
+The command can also guess a `title` for each file, based on its path:
+
+- The folder name is used for index files, otherwise the file name
+- Words are split by `_`
+- The first "word" is removed if it is an integer
+
+For example, for a site with files:
+
+```
+index.rst
+1_a_title.rst
+11_another_title.rst
+.hidden_file.rst
+.hidden_folder/index.rst
+1_a_subfolder/index.rst
+2_another_subfolder/index.rst
+2_another_subfolder/other.rst
+3_subfolder/1_no_index.rst
+3_subfolder/2_no_index.rst
+14_subfolder/index.rst
+14_subfolder/subsubfolder/index.rst
+14_subfolder/subsubfolder/other.rst
+```
+
+will create the ToC:
+
+```console
+$ sphinx-etoc create-toc path/to/folder -i index -s ".*" -e ".rst" -t
+root: index
+sections:
+- file: 1_a_title
+  title: A title
+- file: 11_another_title
+  title: Another title
+- file: 1_a_subfolder/index
+  title: A subfolder
+- file: 2_another_subfolder/index
+  title: Another subfolder
+  sections:
+  - file: 2_another_subfolder/other
+    title: Other
+- file: 3_subfolder/1_no_index
+  title: No index
+  sections:
+  - file: 3_subfolder/2_no_index
+    title: No index
+- file: 14_subfolder/index
+  title: Subfolder
+  sections:
+  - file: 14_subfolder/subsubfolder/index
+    title: Subsubfolder
+    sections:
+    - file: 14_subfolder/subsubfolder/other
+      title: Other
+```
+
 ## API
 
 The ToC file is parsed to a `SiteMap`, which is a `MutableMapping` subclass, with keys representing docnames mapping to a `DocItem` that stores information on the toctrees it should contain:
@@ -239,20 +308,6 @@ intro:
 
 ## Development Notes
 
-Want to have a built-in CLI including commands:
-
-- generate toc from existing documentation toctrees (and remove toctree directives)
-- generate toc from existing documentation, but just from its structure (i.e. `jupyter-book toc mybookpath/`)
-- generate documentation skeleton from toc
-- check toc (without running sphinx)
-
-Process:
-
-- Read toc ("builder-inited" event), error if toc not found
-  - Note, in jupyter-book: if index page does not exist, works out first page from toc and creates an index page that just redirects to it)
-- adds toctree node to page doctree after it is parsed ("doctree-read" event)
-  - Note, in jupyter-book this was done by physically adding to the text before parsing ("source-read" event), but this is not as robust.
-
 Questions / TODOs:
 
 - Should `titlesonly` default to `True` (as in jupyter-book)?
@@ -264,7 +319,11 @@ Questions / TODOs:
   it will add `doc.rst` to the excluded patterns but then, when looking for `doc.md`,
   will still select `doc.rst` (since it is first in `source_suffix`).
   Maybe open an issue on sphinx, that `doc2path` should respect exclude patterns.
-- Intergrate https://github.com/executablebooks/sphinx-multitoc-numbering into this extension? (or upstream PR)
+- Integrate https://github.com/executablebooks/sphinx-multitoc-numbering into this extension? (or upstream PR)
+- document suppressing warnings
+- test against orphan file
+- https://github.com/executablebooks/sphinx-book-theme/pull/304
+- CLI command to generate toc from existing documentation `toctrees` (and then remove toctree directives)
 
 [github-ci]: https://github.com/executablebooks/sphinx-external-toc/workflows/continuous-integration/badge.svg?branch=main
 [github-link]: https://github.com/executablebooks/sphinx-external-toc

sphinx_external_toc/api.py

@@ -306,12 +306,14 @@ def _docitem_to_dict(
 
     def _parse_section(item):
         if isinstance(item, FileItem):
-            return _docitem_to_dict(
-                site_map[item],
-                site_map,
-                skip_defaults=skip_defaults,
-                parsed_docnames=parsed_docnames,
-            )
+            if item in site_map:
+                return _docitem_to_dict(
+                    site_map[item],
+                    site_map,
+                    skip_defaults=skip_defaults,
+                    parsed_docnames=parsed_docnames,
+                )
+            return {FILE_KEY: str(item)}
         if isinstance(item, GlobItem):
             return {GLOB_KEY: str(item)}
         if isinstance(item, UrlItem):

sphinx_external_toc/cli.py

@@ -1,9 +1,11 @@
+from pathlib import PurePosixPath
+
 import click
 import yaml
 
 from sphinx_external_toc import __version__
-from sphinx_external_toc.api import parse_toc_yaml
-from sphinx_external_toc.tools import create_site_from_toc
+from sphinx_external_toc.api import parse_toc_yaml, create_toc_dict
+from sphinx_external_toc.tools import create_site_from_toc, create_site_map_from_path
 
 
 @click.group(context_settings={"help_option_names": ["-h", "--help"]})
@@ -44,3 +46,62 @@ def create_site(toc_file, path, extension, overwrite):
         toc_file, root_path=path, default_ext="." + extension, overwrite=overwrite
     )
     # TODO option to add basic conf.py?
+    click.secho("SUCCESS!", fg="green")
+
+
+@main.command("create-toc")
+@click.argument(
+    "site_folder", type=click.Path(exists=True, file_okay=False, dir_okay=True)
+)
+@click.option(
+    "-e",
+    "--extension",
+    multiple=True,
+    default=[".rst", ".md"],
+    show_default=True,
+    help="File extensions to consider as documents (use multiple times)",
+)
+@click.option(
+    "-i",
+    "--index",
+    default="index",
+    show_default=True,
+    help="File name (without suffix) considered as the index file in a folder",
+)
+@click.option(
+    "-s",
+    "--skip-match",
+    multiple=True,
+    default=[".*"],
+    show_default=True,
+    help="File/Folder names which match will be ignored (use multiple times)",
+)
+@click.option(
+    "-t",
+    "--guess-titles",
+    is_flag=True,
+    help="Guess titles of documents from path names",
+)
+def create_toc(site_folder, extension, index, skip_match, guess_titles):
+    """Create a ToC file from a file structure."""
+    site_map = create_site_map_from_path(
+        site_folder,
+        suffixes=extension,
+        default_index=index,
+        ignore_matches=skip_match,
+    )
+    if guess_titles:
+        for docname in site_map:
+            # don't give a title to the root document
+            if docname == site_map.root.docname:
+                continue
+            filepath = PurePosixPath(docname)
+            # use the folder name for index files
+            name = filepath.parent.name if filepath.name == index else filepath.name
+            # split into words
+            words = name.split("_")
+            # remove first word if is an integer
+            words = words[1:] if words and all(c.isdigit() for c in words[0]) else words
+            site_map[docname].title = " ".join(words).capitalize()
+    data = create_toc_dict(site_map)
+    click.echo(yaml.dump(data, sort_keys=False, default_flow_style=False))