doc: properly handle preformatted blocks

Lowdown requires a blank line before all preformatted blocks, or it doesn't
recognize them. `tools/md2man.sh` contained some ad-hoc efforts at fixing up
some locations where these required blank lines are absent from the output of
`tools/fromschema.py`, but it missed some. Instead of playing Whack-a-Mole, use
a blanket sed expression to ensure that a blank line precedes _every_ opening
```.

`esc_underscores(…)` in `tools/fromschema.py` did not work correctly on strings
containing an odd number of backticks, notably the ``` delimiters surrounding
preformatted text blocks. Specifically, it was dropping the last backtick since
none of the alternatives in the regex matched it. Add a new alternative that
matches a whole preformatted block as a single unit.

`output_member(…)` in `tools/fromschema.py` was passing each line of a member's
description through `esc_underscores(…)` individually, but that breaks
preformatted text blocks that are naturally multi-line and leads to mistakenly
escaping underscores inside such blocks. Rewrite the code to make use of the
`outputs(…)` utility function that joins all the provided lines together before
passing the whole text through `esc_underscores(…)`.

Drive-by fix a couple of flubbed preformatted blocks in schemas.

[ Added shellcheck suppression for md2man.sh --RR ]
Changelog-None

Author: whitslack

contrib/msggen/msggen/schema.json

@@ -28990,9 +28990,9 @@
             "description": [
               "Determines what action is taken:",
               "  - *subcommand* **start** takes a *path* to an executable as argument and starts it as plugin. *path* may be an absolute path or a path relative to the plugins directory (default *~/.lightning/plugins*). If the plugin is already running and the executable (checksum) has changed, the plugin is killed and restarted except if its an important (or builtin) plugin. If the plugin doesn't complete the 'getmanifest' and 'init' handshakes within 60 seconds, the command will timeout and kill the plugin. Additional *options* may be passed to the plugin, but requires all parameters to be passed as keyword=value pairs using the `-k|--keyword` option which is recommended. For example the following command starts the plugin helloworld.py (present in the plugin directory) with the option greeting set to 'A crazy':",
-              "  ```shell.",
-              "  lightning-cli -k plugin subcommand=start plugin=helloworld.py greeting='A crazy'.",
-              "  ```.",
+              "    ```shell",
+              "    lightning-cli -k plugin subcommand=start plugin=helloworld.py greeting='A crazy'",
+              "    ```",
               "  - *subcommand* **stop** takes a plugin executable *path* or *name* as argument and stops the plugin. If the plugin subscribed to 'shutdown', it may take up to 30 seconds before this command returns. If the plugin is important and dynamic, this will shutdown `lightningd`.",
               "  - *subcommand* **startdir** starts all executables it can find in *directory* (excl. subdirectories) as plugins. Checksum and timeout behavior as in **start** applies.",
               "  - *subcommand* **rescan** starts all plugins in the default plugins directory (default *~/.lightning/plugins*) that are not already running. Checksum and timeout behavior as in **start** applies.",
@@ -33060,7 +33060,10 @@
           "relative_amount": {
             "type": "integer",
             "description": [
-              "A positive or negative amount of satoshis to add or subtract from the channel. Note you may need to add a double dash (--) after splice_init if using a negative *relative_amount* so it is not interpretted as a command modifier. For example: ```shell lightning-cli splice_init -- $CHANNEL_ID -100000 ```."
+              "A positive or negative amount of satoshis to add or subtract from the channel. Note you may need to add a double dash (--) after splice_init if using a negative *relative_amount* so it is not interpretted as a command modifier. For example:",
+              "```shell",
+              "lightning-cli splice_init -- $CHANNEL_ID -100000",
+              "```"
             ]
           },
           "initialpsbt": {

doc/schemas/plugin.json

@@ -31,9 +31,9 @@
         "description": [
           "Determines what action is taken:",
           "  - *subcommand* **start** takes a *path* to an executable as argument and starts it as plugin. *path* may be an absolute path or a path relative to the plugins directory (default *~/.lightning/plugins*). If the plugin is already running and the executable (checksum) has changed, the plugin is killed and restarted except if its an important (or builtin) plugin. If the plugin doesn't complete the 'getmanifest' and 'init' handshakes within 60 seconds, the command will timeout and kill the plugin. Additional *options* may be passed to the plugin, but requires all parameters to be passed as keyword=value pairs using the `-k|--keyword` option which is recommended. For example the following command starts the plugin helloworld.py (present in the plugin directory) with the option greeting set to 'A crazy':",
-          "  ```shell.",
-          "  lightning-cli -k plugin subcommand=start plugin=helloworld.py greeting='A crazy'.",
-          "  ```.",
+          "    ```shell",
+          "    lightning-cli -k plugin subcommand=start plugin=helloworld.py greeting='A crazy'",
+          "    ```",
           "  - *subcommand* **stop** takes a plugin executable *path* or *name* as argument and stops the plugin. If the plugin subscribed to 'shutdown', it may take up to 30 seconds before this command returns. If the plugin is important and dynamic, this will shutdown `lightningd`.",
           "  - *subcommand* **startdir** starts all executables it can find in *directory* (excl. subdirectories) as plugins. Checksum and timeout behavior as in **start** applies.",
           "  - *subcommand* **rescan** starts all plugins in the default plugins directory (default *~/.lightning/plugins*) that are not already running. Checksum and timeout behavior as in **start** applies.",

doc/schemas/splice_init.json

@@ -24,7 +24,10 @@
       "relative_amount": {
         "type": "integer",
         "description": [
-          "A positive or negative amount of satoshis to add or subtract from the channel. Note you may need to add a double dash (--) after splice_init if using a negative *relative_amount* so it is not interpretted as a command modifier. For example: ```shell lightning-cli splice_init -- $CHANNEL_ID -100000 ```."
+          "A positive or negative amount of satoshis to add or subtract from the channel. Note you may need to add a double dash (--) after splice_init if using a negative *relative_amount* so it is not interpretted as a command modifier. For example:",
+          "```shell",
+          "lightning-cli splice_init -- $CHANNEL_ID -100000",
+          "```"
         ]
       },
       "initialpsbt": {

tools/fromschema.py

@@ -21,7 +21,7 @@ def output_title(title, underline='-', num_leading_newlines=1, num_trailing_newl
 
 def esc_underscores(s):
     """Backslash-escape underscores outside of backtick-enclosed spans"""
-    return ''.join(['\\_' if x == '_' else x for x in re.findall(r'[^`_\\]+|`(?:[^`\\]|\\.)*`|\\.|_', s)])
+    return ''.join(['\\_' if x == '_' else x for x in re.findall(r'(?ms:^[ \t]*```.*?^[ \t]*```)|[^`_\\\n]+|`(?:[^`\\]|\\.)*`|\\.|[_\n]', s)])
 
 
 def json_value(obj):
@@ -165,8 +165,8 @@ def output_member(propname, properties, is_optional, indent, print_type=True, pr
     output_range(properties)
 
     if 'description' in properties:
-        for i in range(0, len(properties['description'])):
-            output('{} {}{}'.format(':' if i == 0 else '', esc_underscores(properties['description'][i]), '' if i + 1 == len(properties['description']) else '\n'))
+        output(': ')
+        outputs(properties['description'], '\n ')
 
     if 'default' in properties:
         output(' The default is {}.'.format(esc_underscores(properties['default']) if isinstance(properties['default'], str) else properties['default']))

tools/md2man.sh

@@ -28,15 +28,17 @@ TITLELINE="$(head -n1 "$SOURCE")"
 
 # Replace lightning-cli with $ lightning-cli but do not replace it if it is preceded with (
 # because it is used in the examples to run it in the shell, eg. $(lightning-cli listpeerchannels)
+# shellcheck disable=SC2016 # These are not variables, shellcheck!
 SOURCE=$(tail -n +3 "$SOURCE" | sed -E '
     :a;N;$!ba;
-    s#EXAMPLES\n------------#\nEXAMPLES\n------------\n#g;
-    s#Request:#Request:\n#g;
-    s#Response:#Response:\n#g;
     s#(\(lightning-cli)#\x1#ig;
     s#lightning-cli#$ lightning-cli#g;
     s#\x1#(lightning-cli#g;
-    s#\*\*Notification (1|2|3)\*\*:#**Notification \1**:\n#g;
+' |
+# Lowdown requires a blank line before every preformatted text block
+sed '
+	/^$/{:0;N;/\n$/b0};s/^[[:blank:]]*```/\n\0/;
+	/\n[[:blank:]]*```/{:1;n;/^[[:blank:]]*```/!b1}
 ')
 
 # Output to the target file