diff --git a/docs/source/cmd/index_internal.rst b/docs/source/cmd/index_internal.rst index ab9c13aba42..d485be705c4 100644 --- a/docs/source/cmd/index_internal.rst +++ b/docs/source/cmd/index_internal.rst @@ -3,150 +3,3 @@ Internal commands for developers .. autocmdgroup:: internal :members: - -Writing command help --------------------- - -- use `chformal` as an example -- generated help content below - -.. _chformal autocmd: - -.. autocmd:: chformal - :noindex: - -The ``formatted_help()`` method -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- ``PrettyHelp::get_current()`` -- ``PrettyHelp::set_group()`` - - + used with ``.. autocmdgroup:: `` - + can assign group and return false - + if no group is set, will try to use ``source_location`` and assign group - from path to source file - -- return value - - + true means help content added to current ``PrettyHelp`` - + false to use ``Pass::help()`` - -- adding content - - + help content is a list of ``ContentListing`` nodes, each one having a type, - body, and its own list of children ``ContentListing``\ s - + ``PrettyHelp::get_root()`` returns the root ``ContentListing`` (``type="root"``) - + ``ContentListing::{usage, option, codeblock, paragraph}`` each add a - ``ContentListing`` to the current node, with type the same as the method - - * the first argument is the body of the new node - * ``usage`` shows how to call the command (i.e. its "signature") - * ``paragraph`` content is formatted as a paragraph of text with line breaks - added automatically - * ``codeblock`` content is displayed verbatim, use line breaks as desired; - takes an optional ``language`` argument for assigning the language in RST - output for code syntax highlighting (use ``yoscrypt`` for yosys script - syntax highlighting) - * ``option`` lists a single option for the command, usually starting with a - dash (``-``); takes an optional second argument which adds a paragraph - node as a means of description - - + ``ContentListing::open_usage`` creates and returns a new usage node, can be - used to e.g. add text/options specific to a given usage of the command - + ``ContentListing::open_option`` creates and returns a new option node, can - be used to e.g. add multiple paragraphs to an option's description - + paragraphs are treated as raw RST, allowing for inline formatting and - references as if it were written in the RST file itself - -.. literalinclude:: /generated/chformal.cc - :language: c++ - :start-at: bool formatted_help() - :end-before: void execute - :caption: ``ChformalPass::formatted_help()`` from :file:`passes/cmds/chformal.cc` - -Dumping command help to json -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- `help -dump-cmds-json cmds.json` - - + generates a ``ContentListing`` for each command registered in Yosys - + tries to parse unformatted ``Pass::help()`` output if - ``Pass::formatted_help()`` is unimplemented or returns false - - * if a line starts with four spaces followed by the name of the command then - a space, it is parsed as a signature (usage node) - * if a line is indented and starts with a dash (``-``), it is parsed as an - option - * anything else is parsed as a codeblock and added to either the root node - or the current option depending on the indentation - - + dictionary of command name to ``ContentListing`` - - * uses ``ContentListing::to_json()`` recursively for each node in root - * root node used for source location of class definition - * includes flags set during pass constructor (e.g. ``experimental_flag`` set - by ``Pass::experimental()``) - * also title (``short_help`` argument in ``Pass::Pass``), group, and class - name - - + dictionary of group name to list of commands in that group - -- used by sphinx autodoc to generate help content - -.. literalinclude:: /generated/cmds.json - :language: json - :start-at: "chformal": { - :end-before: "chparam": { - :caption: `chformal` in generated :file:`cmds.json` - -.. note:: Synthesis command scripts are special cased - - If the final block of help output starts with the string `"The following - commands are executed by this synthesis command:\n"`, then the rest of the - code block is formatted as ``yoscrypt`` (e.g. `synth_ice40`). The caveat - here is that if the ``script()`` calls ``run()`` on any commands *prior* to - the first ``check_label`` then the auto detection will break and revert to - unformatted code (e.g. `synth_fabulous`). - -Command line rendering -~~~~~~~~~~~~~~~~~~~~~~ - -- if ``Pass::formatted_help()`` returns true, will call - ``PrettyHelp::log_help()`` - - + traverse over the children of the root node and render as plain text - + effectively the reverse of converting unformatted ``Pass::help()`` text - + lines are broken at 80 characters while maintaining indentation (controlled - by ``MAX_LINE_LEN`` in :file:`kernel/log_help.cc`) - + each line is broken into words separated by spaces, if a given word starts - and ends with backticks they will be stripped - -- if it returns false it will call ``Pass::help()`` which should call ``log()`` - directly to print and format help text - - + if ``Pass::help()`` is not overridden then a default message about missing - help will be displayed - -.. literalinclude:: /generated/chformal.log - :lines: 2- - -RST generated from autocmd -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- below is the raw RST output from ``autocmd`` (``YosysCmdDocumenter`` class in - :file:`docs/util/cmd_documenter.py`) for `chformal` command -- heading will be rendered as a subheading of the most recent heading (see - `chformal autocmd`_ above rendered under `Writing command help`_) -- ``.. cmd:def:: `` line is indexed for cross references with ``:cmd:ref:`` - directive (`chformal autocmd`_ above uses ``:noindex:`` option so that - `chformal` still links to the correct location) - - + ``:title:`` option controls text that appears when hovering over the - `chformal` link - -- commands with warning flags (experimental or internal) add a ``.. warning`` - block before any of the help content -- if a command has no ``source_location`` the ``.. note`` at the bottom will - instead link to :doc:`/cmd/index_other` - -.. autocmd_rst:: chformal diff --git a/docs/source/yosys_internals/extending_yosys/documenting.rst b/docs/source/yosys_internals/extending_yosys/documenting.rst new file mode 100644 index 00000000000..c0f51f63797 --- /dev/null +++ b/docs/source/yosys_internals/extending_yosys/documenting.rst @@ -0,0 +1,504 @@ +Generated help messages and documentation +========================================= + +All Yosys commands and built-in cell types should include help text, documenting +their functionality for users. This help text is made available through the +`help` command, and online via `ReadtheDocs`_ as part of the :doc:`/cmd_ref` and +:doc:`/cell_index` documentation. When running locally, any commands provided +by loaded plugins (either from the command line when calling ``yosys``, or +dynamically with the `plugin` command) will also be available to the `help` +command. + +.. _ReadtheDocs: https://about.readthedocs.com/ + +.. note:: + + Since help text for commands is generated from compiled code, the online help + may differ from that produced by `help`. Some commands, like `abc`, may be + completely unavailable depending on compile flags; while others may limit + specific features, such as whether the `synth` script pass uses ABC. + +Command help +------------ + +The first stop for command help text is the ``Pass::short_help``. This is a +short sentence describing the pass, and is set in the ``Pass`` constructor with +the name of the pass, as demonstrated here with `chformal`. + +.. literalinclude:: /generated/chformal.cc + :language: c++ + :start-at: public Pass { + :end-at: ChformalPass() + :caption: ``ChformalPass()`` from :file:`passes/cmds/chformal.cc` + :append: + // ... + } ChformalPass; + +All currently available commands are listed with their ``short_help`` string +when calling `help` without arguments, and is more or less the same as the +unlisted :ref:`command index `. The string is also used when +hovering over links to commands in the documentation, and in section headings +like :ref:`chformal autocmd`. + +The next section shows the complete help text for the `chformal` command. This +can be displayed locally by using ``help `` (or ``yosys -h `` +from the command line). The general format is to show each usage signature (how +the command is called), followed by a paragraph describing what the pass does, +and a list of options or flags available. Additional arguments in the signature +or option may use square brackets (``[]``) to indicate optional parts, and angle +brackets (``<>``) for required parts. The pipe character ``|`` may be used to +indicate mutually exclusive arguments. + +.. note:: + + Remember that when using ``Frontend`` and ``Backend`` the pass name will be + be prefixed with ``read_`` or ``write_`` respectively. Usage signatures must + match the pass name available in commands/scripts, which is available as + ``Pass::pass_name``. + +.. todo:: decide on a formatting style for pass options + +.. _chformal autocmd: + +.. autocmd:: chformal + :noindex: + +The ``Pass::help()`` method +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is the original way to provide help text, and as of this writing is still +the most common. The ``log()`` function should be called directly to print and +format the help text, and each line should be limited to 80 (printed) +characters. While it is possible to provide arbitrary formatting, it is +preferred to follow the guidelines here to maintain consistency with other +passes and to assist in correct parsing and formatting during RST generation +(i.e. these docs). + +The first line should always be a blank line, followed by the primary usage +signature for the command. Each usage signature should be indented with 4 +spaces, and followed by a blank line. Each option or flag should start on a new +line indented with 4 spaces, followed by a description of the option which is +indented by a further 4 spaces, and then a blank line. Option descriptions +typically start with lower case, and may forgo a trailing period (``.``). Where +multiple options share a description the blank line between options should be +omitted. + +.. note:: + + `Dumping to json`_ has more on how formatting in ``help()`` gets parsed. + + +The ``Pass::formatted_help()`` method +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``formatted_help`` method serves two purposes in help generation, both of +which are optional. In both cases, any pass which uses the method should +``#include "kernel/log_help.h"``, and begin the method by calling ``auto *help = +PrettyHelp::get_current();``. The method finishes by returning a boolean value. +``true`` means help content has been added to the current ``PrettyHelp``, while +``false`` indicates that ``Pass::help()`` should be called instead. + +Setting a command group +^^^^^^^^^^^^^^^^^^^^^^^ + +Command groups are used when `Dumping to json`_, so that related +commands can be presented together in documentation. For example, all of the +formal commands (which `chformal` is one of) are listed under +:doc:`/cmd/index_formal`, by using the ``autocmdgroup`` directive in +:file:`docs/source/cmd/index_formal.rst`. By default, commands are grouped by +their source location, such that the group is the same as the path to the source +file. + +.. note:: + + Source location tracking requires :makevar:`ENABLE_HELP_SOURCE` to be set in + the makefile. Some passes, like the ``opt_*`` family, are able to be grouped + by the name of the pass; but most will be assigned the ``unknown`` group. + + For frontends and backends, source code is structured such that different + formats are located in different folders. Default behavior is to instead + group all of these passes as :doc:`/cmd/index_frontends` and + :doc:`/cmd/index_backends` respectively. Without location tracking, the + fallback is to look for passes that start with ``read_`` or ``write_``. + +It is possible to set the group of a command explicitly with the +``PrettyHelp::set_group()`` method. This allows grouping of commands which may +not share a common source location, as well as ensuring that commands are still +grouped when location tracking is disabled. Because ``Pass::formatted_help()`` +returns if it produced help content, it is completely valid to override the +method, get the current instance of ``PrettyHelp``, set the command group, and +then return ``false``. + +.. warning:: + + There is currently no warning available for groups that do not have a + corresponding ``autocmdgroup``. If you add a new command group, make sure + that it has a corresponding index page. + + +Rich help text +^^^^^^^^^^^^^^ + +The second purpose of ``Pass::formatted_help`` is to provide richer help +content which is able to take advantage of the reStructuredText formatting used +here in the web docs. It also provides a more fluid way of writing help text, +without getting caught up in the terminal-first spacing requirements of writing +for ``Pass::help()``. + +Help content is a list of ``ContentListing`` nodes on a root node, which can be +found by calling ``PrettyHelp::get_root()``. Each node has a type, a body, and +its own list of children ``ContentListing``\ s. Adding content is done with the +``ContentListing::{usage, option, codeblock, paragraph}`` methods, which each +add a new child node with a type set to the calling method. Let's take a look +at the source code for `chformal`. + +.. literalinclude:: /generated/chformal.cc + :language: c++ + :start-at: bool formatted_help() + :end-before: void execute + :caption: ``ChformalPass::formatted_help()`` from :file:`passes/cmds/chformal.cc` + +We can see that each of the ``ContentListing`` methods have the body of the new +node as the first argument. For a ``usage`` node, this is how to call the +command (i.e. its usage signature). ``paragraph`` nodes contain a paragraph of +text with line breaks added automatically; the argument itself should contain +any line breaks, but the string can be broken across multiple lines as shown. +The body of a ``paragraph`` node is treated as raw RST, allowing for inline +formatting and references as if it were written in the RST file itself. As +shown in the example (and the :ref:`formatted output above `), +this includes using single backticks for linking to cells or commands, and +double backticks for raw code. + +The ``option`` method lists a single option for the command, usually starting +with a dash (``-``). An optional second argument can be provided with adds a +paragraph node as a child of the option, and is used for describing the option. +Where multiple options share a description, it should be added to the last +option. + +.. note:: + + To add multiple paragraphs to an option's description, + ``ContentListing::open_option()`` should be used instead. This method + returns the option node, which can then be used to call + ``ContentList::paragraph()`` multiple times. + +``codeblock`` content is displayed verbatim, and content should include line +breaks as desired. No extra formatting will be applied to the text, and it will +be rendered with a monospace font; making it perfect for code sections or ASCII +art diagrams which render the same on the web as they do in the terminal. An +optional second argument is available for specifying the language in RST output +for code syntax highlighting (use ``yoscrypt`` for yosys script syntax +highlighting). + +.. + not recommended since it (currently) doesn't render in the terminal + + The final method available is ``ContentListing::open_usage``. As with + ``open_option`` creates and returns a new node which can have additional content + added to it directly. For the usage node, this can be used for example to add + text/options specific to a given usage of the command. In the web documentation + any content added in this way will be indented under the usage signature. + + +Command line rendering +~~~~~~~~~~~~~~~~~~~~~~ + +- if ``Pass::formatted_help()`` returns true, will call + ``PrettyHelp::log_help()`` + + + traverse over the children of the root node and render as plain text + + effectively the reverse of converting unformatted ``Pass::help()`` text + + lines are broken at 80 characters while maintaining indentation (controlled + by ``MAX_LINE_LEN`` in :file:`kernel/log_help.cc`) + + each line is broken into words separated by spaces, if a given word starts + and ends with backticks they will be stripped + +- if it returns false it will call ``Pass::help()`` which should call ``log()`` + directly to print and format help text + + + if ``Pass::help()`` is not overridden then a default message about missing + help will be displayed + +.. literalinclude:: /generated/chformal.log + :lines: 2- + + +Cell help +--------- + +- :file:`techlibs/common/simcells.v` and :file:`techlibs/common/simlib.v` +- parsed by :file:`techlibs/common/cellhelp.py` + + + unlike commands, cell help text is generated at compile time + + only formatting occurs at run time + + no support for custom cells in plugins + +- comments preceding cell type converted to a ``SimHelper`` struct, defined in + :file:`kernel/register.cc` +- ``#include``\ d in :file:`kernel/register.cc`, creating a map of cell types to + their ``SimHelper`` struct. + +- ``help -cells`` + + - lists all cells with their input/output ports + - again an unlisted :ref:`cell index ` + +- ``help `` + + - prints help message for ```` + - constructed from ``SimHelper`` content depending on version + +- ``help +`` + + - prints verilog code for ```` + +v1 (default) +~~~~~~~~~~~~ + +- Legacy format +- Expects formatting as follows: + +.. code-block:: verilog + + //- + //- $ () + //* group + //- + //- + //- + module \$ (); + // ... + endmodule + +- ``//* group`` line is generally after the cell signature, but can appear + anywhere in the comment block + + - determines where the cell is included in sphinx + - check :file:`docs/source/cell` for current groups + - a missing group will raise an error + - assigning an unused group will result in the cell not being included in the + sphinx docs + +- the port signature line (``//- $ ()``) must start with (at + least) 4 spaces (not tabs) + + - the empty lines (``//-``) before/after the signature are required + +- cell description can be multiple lines, but each line must start with ``//-`` + and a space + + - description should have a trailing empty line + - line breaks are preserved in `help` calls but will be rendered as text in + sphinx docs, with empty lines being required between paragraphs + +- cells in :file:`techlibs/common/simcells.v` can have optional truth table at + the end of the cell description which is rendered in sphinx docs as a literal + code block +- e.g. `$_NOT_`: + +.. code-block:: verilog + + //- + //- $_NOT_ (A, Y) + //* group comb_simple + //- + //- An inverter gate. + //- + //- Truth table: A | Y + //- ---+--- + //- 0 | 1 + //- 1 | 0 + //- + +v2 (more expressive) +~~~~~~~~~~~~~~~~~~~~ + +- each field of the ``SimHelper`` struct can be assigned with: + +.. code-block:: verilog + + //* + +- ``ver`` must be explicitly set as ``2`` +- optional fields ``title`` and ``tags`` + + - title used for better displaying of cell + - tags is a space-separated list of :doc:`/cell/properties` + +- the port signature is automatically generated from the ``module`` definition +- cell description is the same as v1 +- can link to commands or passes using backticks (`````) +- e.g. `$nex`: + +.. code-block:: verilog + + //* ver 2 + //* title Case inequality + //* group binary + //* tags x-aware + //- This corresponds to the Verilog '!==' operator. + //- + //- Refer to `$eqx` for more details. + //- + +- code blocks can be included as following: + +.. code-block:: verilog + + //- text + //- :: + //- + //- monospaced text + //- + //- indentation and line length will be preserved, giving a scroll bar if necessary for the browser window + //- + //- more text + +- the empty line after the ``::`` and before the text continues are required +- the ``::`` line will be ignored in the `help` call while sphinx docs will + treat everything that follows as a literal block until the next unindented + line: + + text + :: + + monospaced text + + indentation and line length will be preserved, giving a scroll bar if necessary for the browser window + + more text + + +Dumping to json +--------------- + +- `help -dump-cmds-json cmds.json` + + + generates a ``ContentListing`` for each command registered in Yosys + + tries to parse unformatted ``Pass::help()`` output if + ``Pass::formatted_help()`` is unimplemented or returns false + + * if a line starts with four spaces followed by the name of the command then + a space, it is parsed as a signature (usage node) + * if a line is indented and starts with a dash (``-``), it is parsed as an + option + * anything else is parsed as a codeblock and added to either the root node + or the current option depending on the indentation + + + dictionary of command name to ``ContentListing`` + + * uses ``ContentListing::to_json()`` recursively for each node in root + * root node used for source location of class definition + * includes flags set during pass constructor (e.g. ``experimental_flag`` set + by ``Pass::experimental()``) + * also title (``short_help`` argument in ``Pass::Pass``), group, and class + name + + + dictionary of group name to list of commands in that group + +- used by sphinx autodoc to generate help content + +.. literalinclude:: /generated/cmds.json + :language: json + :start-at: "chformal": { + :end-before: "chparam": { + :caption: `chformal` in generated :file:`cmds.json` + +.. note:: Synthesis command scripts are special cased + + If the final block of help output starts with the string ``"The following + commands are executed by this synthesis command:\n"``, then the rest of the + code block is formatted as ``yoscrypt`` (e.g. `synth_ice40`). The caveat + here is that if the ``script()`` calls ``run()`` on any commands *prior* to + the first ``check_label`` then the auto detection will break and revert to + unformatted code (e.g. `synth_fabulous`). + + +Cells and commands in Sphinx +---------------------------- + +To support the rich documentation of commands and cells in Yosys, we use two +custom `Sphinx Domains`_. + +.. _Sphinx Domains: https://www.sphinx-doc.org/en/master/usage/domains/index.html + +- ``yoscrypt`` role allows inline code to have yosys script syntax highlighting +- default role of ``autoref`` + + + any text in single backticks without an explicit role will be assigned this one + + will convert to ``cell:ref`` if it begins with ``$``, otherwise ``cmd:ref`` + + to attempt linking there must be no spaces, and it must not begin with a + dash (``-``) + + ```chformal``` (or ``:autoref:`chformal```) -> ``:cmd:ref:`chformal``` -> `chformal` + + also works with two words, if the first one is ``help`` + + ```help $add``` -> ``:cell:ref:`help $add <$add>``` -> `help $add` + + fallback to formatting as inline yoscrypt + + ```-remove``` -> `-remove` + + ```chformal -remove``` -> ``:yoscrypt:`chformal -remove``` -> `chformal -remove` + +Using autodoc +~~~~~~~~~~~~~ + +- below is the raw RST output from ``autocmd`` (``YosysCmdDocumenter`` class in + :file:`docs/util/cmd_documenter.py`) for `chformal` command +- heading will be rendered as a subheading of the most recent heading (see + `chformal autocmd`_ above rendered under `Command help`_) +- ``.. cmd:def:: `` line is indexed for cross references with ``:cmd:ref:`` + directive (`chformal autocmd`_ above uses ``:noindex:`` option so that + `chformal` still links to the correct location) + + + ``:title:`` option controls text that appears when hovering over the + `chformal` link + +- commands with warning flags (experimental or internal) add a ``.. warning`` + block before any of the help content +- if a command has no ``source_location`` the ``.. note`` at the bottom will + instead link to :doc:`/cmd/index_other` + +.. _showing autocmd generated rst: + +.. autocmd_rst:: chformal + +- command groups documented with ``autocmdgroup `` + + + with ``:members:`` option this is the same as calling ``autocmd`` for each + member of the group + +- ``autocell`` and ``autocellgroup`` + + + very similar to ``autocmd`` and ``autocmdgroup`` but for cells instead of + commands (``YosysCellDocumenter`` in :file:`docs/util/cell_documenter.py`) + + optionally includes verilog source for cell(s) with ``:source:`` option + (plus ``:linenos:``) + + cell definitions do not include titles + + cells can have properties (:ref:`propindex`) + +- bonus ``autocmd_rst``, used exclusively on this page for `showing autocmd + generated rst`_ + +Our custom Sphinx domains +~~~~~~~~~~~~~~~~~~~~~~~~~ + +- ``cmd`` and ``cell`` +- Directives + + + ``cmd:def`` provide command definition + + ``cmd:usage`` used by ``autocmd`` for command usage signatures + + ``cell:def`` provide cell definition + + ``cell:defprop`` provide cell property definition (used in + :doc:`/cell/properties`) + + ``cell:source`` used by ``autocell`` for simulation models + +- Roles + + + ``cmd:ref`` link to a ``cmd:def`` with the same name + + ``cmd:title`` same as ``cmd:ref``, but includes the short help in the text + + - ``:cmd:title:`chformal``` -> :cmd:title:`chformal` + + + ``cell:ref`` link to a ``cell:def`` with the same name + + ``cell:title`` + + - ``:cell:title:`$nex``` -> :cell:title:`$nex` + + + ``cell:prop`` link to a ``cell:defprop`` of the same name diff --git a/docs/source/yosys_internals/extending_yosys/index.rst b/docs/source/yosys_internals/extending_yosys/index.rst index 72843ecd61e..3b8aad54179 100644 --- a/docs/source/yosys_internals/extending_yosys/index.rst +++ b/docs/source/yosys_internals/extending_yosys/index.rst @@ -13,5 +13,6 @@ of interest for developers looking to customise Yosys builds. functional_ir advanced_bugpoint contributing + documenting test_suites diff --git a/kernel/log_help.cc b/kernel/log_help.cc index 93b91b08b83..cb22d236e8d 100644 --- a/kernel/log_help.cc +++ b/kernel/log_help.cc @@ -44,7 +44,7 @@ void ContentListing::usage(const string &text, void ContentListing::option(const string &text, const string &description, const source_location location) { - auto option = open_option(text); + auto option = open_option(text, location); if (description.length()) option->add_content("text", description, location); } diff --git a/kernel/rtlil.cc b/kernel/rtlil.cc index adfda7d5f30..ead667cbde6 100644 --- a/kernel/rtlil.cc +++ b/kernel/rtlil.cc @@ -2475,7 +2475,8 @@ namespace { * * Things to do after finalizing the cell interface: * - Add support to kernel/satgen.h for the new cell type - * - Add to docs/source/CHAPTER_CellLib.rst (or just add a fixme to the bottom) + * - Maybe add v2 cell help fields (title, tags) + * - Add extra details to relevant docs/source/cell/word_*.rst (or just add a todo to the top) * - Maybe add support to the Verilog backend for dumping such cells as expression * */