rename forms, introduce (h2), move (column_name) into (line)

fixes for url, column_name, etc.

Author: justinmk

README.md

@@ -13,14 +13,29 @@ Notes
 - vimdoc format "spec":
     - [:help help-writing](https://neovim.io/doc/user/helphelp.html#help-writing)
     - https://github.com/nanotee/vimdoc-notes
-- `(code_block)` is contained by `(line)` because `>` can start a code block at the end of a line.
+- whitespace is intentionally captured in `(word)`, because it is often necessary to be
+  able to correctly layout vim help files (especially old/legacy).
+- `(codeblock)` is contained by `(line)` because `>` can start a code block at the end of a line.
+- `(column_heading)` is contained by `(line)` because `>` (to close
+  a `(codeblock)` can appear at the start of `(column_heading)`.
+- `h1` ("Heading 1"): `======` followed by text and optional `*tags*`.
+- `h2` ("Heading 2"): `------` followed by text and optional `*tags*`.
+- `h3` ("Heading 3"): only UPPERCASE WORDS, followed by optional `*tags*`.
 
 Known issues
 ------------
 
-- `line` in a `code_block` does not contain `word` atoms, it's just the full
+- `line_li` ("list item") is _experimental_. It doesn't support nesting yet and
+  it may not work well; you can treat it as a normal `line` for layout purposes.
+- `codeblock` ">" must not be preceded only by tabs, a space char is required (" >").
+  See `:help lcs-tab` for example. Currently the grammar doesn't enforce this.
+- `codeblock` terminated by an "implicit stop" (i.e. no terminating `<`)
+  consumes the first char of the terminating line, and continues the parent
+  `block`, preventing top-level forms like `h1`, `h2` from being recognized
+  until a blank line is encountered.
+- `line` in a `codeblock` does not contain `word` atoms, it's just the full
   raw text line including whitespace. This is somewhat dictated by its
-  "preformatted" nature; parsing the contents implies loading a "child"
+  "preformatted" nature; parsing the contents would require loading a "child"
   language (injection). See [#2](https://github.com/vigoux/tree-sitter-vimdoc/issues/2).
 - `url` doesn't handle _surrounding_ parens. E.g. `(https://example.com/#yay)` yields `word`
 - `url` doesn't handle _nested_ parens. E.g. `(https://example.com/(foo)#yay)`
@@ -37,9 +52,3 @@ TODO
 - `line_noeol` is a special-case to support documents that don't end in EOL.
   Grammar could be a bit simpler if we just require EOL at end of document.
 - `line_modeline` (only at EOF)
-- `column_heading` should not allow hotlinks. This is sometimes used in old help files to show results of a code example, e.g. in `usr_41.txt`:
-  ```
-  List concatenation is done with +: >
-          :echo alist + ['foo', 'bar']
-  <	['foo', 'bar', 'foo', 'bar'] ~
-  ```

corpus/arguments.txt

@@ -34,7 +34,7 @@ multiple arguments on the same line
 ================================================================================
 NOT an argument
 ================================================================================
-{foo "{bar}" `{baz}` |{baz| }
+{foo "{bar}" `{baz}` |{baz| } {}
 
 --------------------------------------------------------------------------------
 
@@ -45,8 +45,9 @@ NOT an argument
         (word)
         (ERROR))
       (word)
-      (backtick
+      (codespan
         (word))
-      (hotlink
+      (taglink
         (word))
+      (word)
       (word))))

corpus/code_block.txt renamed to corpus/codeblock.txt

@@ -21,21 +21,21 @@ block3:
 (help_file
   (block
     (line
-      (code_block
+      (codeblock
         (line))))
   (block
     (line
       (word))
     (line
-      (code_block
+      (codeblock
         (line)))
     (line
       (word)))
   (block
     (line
       (word))
     (line
-      (code_block
+      (codeblock
         (line))))
   (block
     (line
@@ -68,7 +68,7 @@ text
     (line
       (word)
       (word)
-      (code_block
+      (codeblock
         (line)
         (line)
         (line))))
@@ -79,11 +79,11 @@ text
   (block
     (line
       (word)
-      (hotlink
+      (taglink
         (word))
       (word))
     (line
-      (code_block
+      (codeblock
         (line)
         (line)
         (line))))
@@ -92,23 +92,36 @@ text
       (word))))
 
 ================================================================================
-codeblock with implicit stop
+codeblock with implicit stop (FIXME)
 ================================================================================
 >
   line1
   line2
 
-Blehke
+-------------------------------
+h1-headline *foo*
+
+-------------------------------
+h1-headline *foo*
+
 --------------------------------------------------------------------------------
 
 (help_file
   (block
     (line
-      (code_block
+      (codeblock
         (line)
         (line)
         (line)))
     (line
+      (word))
+    (line
+      (word)
+      (tag
+        (word))))
+  (h2
+    (word)
+    (tag
       (word))))
 
 ================================================================================
@@ -133,7 +146,7 @@ x
   (block
     (line
       (word)
-      (code_block
+      (codeblock
         (line)
         (line)
         (line)
@@ -159,7 +172,7 @@ tricky
 (help_file
   (block
     (line
-      (code_block
+      (codeblock
         (line)
         (line)
         (line))))
@@ -192,7 +205,7 @@ To test for a non-empty string, use empty(): >
       (word)
       (word)
       (word)
-      (code_block
+      (codeblock
         (line)
         (line)))
     (line
@@ -216,7 +229,7 @@ To test for a non-empty string, use empty(): >
       (word)
       (word)
       (word)
-      (code_block
+      (codeblock
         (line)
         (line)))
     (line
@@ -228,5 +241,5 @@ To test for a non-empty string, use empty(): >
       (word)
       (word)
       (word)
-      (code_block
+      (codeblock
         (line)))))

corpus/backtick.txt renamed to corpus/codespan.txt

@@ -1,5 +1,5 @@
 ================================================================================
-simple backtick
+simple codespan
 ================================================================================
 
 a `foobar` b `:echo`
@@ -10,14 +10,14 @@ a `foobar` b `:echo`
   (block
     (line
       (word)
-      (backtick
+      (codespan
         (word))
       (word)
-      (backtick
+      (codespan
         (word)))))
 
 ================================================================================
-backtick in text
+codespan in text
 ================================================================================
 
 Hello `world`, I am `markup language`. But `this is
@@ -29,24 +29,24 @@ an error`.
   (block
     (line
       (word)
-      (backtick
+      (codespan
         (word))
       (word)
       (word)
       (word)
-      (backtick
+      (codespan
         (word))
       (word)
       (word)
-      (backtick
+      (codespan
         (word)
         (MISSING "`")))
     (line
       (word)
       (word))))
 
 ================================================================================
-NOT a codespan / backtick
+NOT a codespan
 ================================================================================
 						*'* *'a* *`* *`a*
 '{a-z}  `{a-z}		Jump to the mark.

corpus/headline.txt renamed to corpus/heading1_2.txt

@@ -1,36 +1,64 @@
 ================================================================================
-simple headline
+h1 h2 heading
 ================================================================================
 
 ==============================================================================
-Headline                                                             *headline*
+Heading1                                                            *heading1*
 
 Text
 
+-----------------------------------------------------------------------
+Heading two                                                *tag2.1*   *tag2.2*
+
+Text2
+
 --------------------------------------------------------------------------------
 
 (help_file
-  (headline
+  (h1
     (word)
     (tag
       (word)))
+  (block
+    (line
+      (word)))
+  (h2
+    (word)
+    (word)
+    (tag
+      (word))
+    (tag
+      (word)))
   (block
     (line
       (word))))
 
 ================================================================================
-headline with tag at the start
+h1 h2 heading with tag at the start
 ================================================================================
 
 ==============================================================================
 *03.1*	Word movement
 
 Text
 
+------------------------------------------------------------------------------
+*05.6*	heading two
+
+Text
+
 --------------------------------------------------------------------------------
 
 (help_file
-  (headline
+  (h1
+    (tag
+      (word))
+    (word)
+    (word))
+  (block
+    (line
+      (word)))
+  (h2
     (tag
       (word))
     (word)
@@ -40,26 +68,26 @@ Text
       (word))))
 
 ================================================================================
-NOT a headline
+NOT h1 h2 heading
 ================================================================================
 
 ====
-Not-a-headline-1                                                     *headline1*
+Not-a-heading-1                                                     *heading1*
 
 Text
 
 ---
-Not-a-headline-2 more words                                          *headline2*
+Not-a-heading-2 more words                                          *heading2*
 
 Text
 
 ===========a======b========3089uaafaewf_-----awefpoij===========================
-Not-a-headline-3                                                     *headline3*
+Not-a-heading-3                                                     *heading3*
 
 Text
 
 ------------===============--------------------=================================
-Not-a-headline-4                                                     *headline4*
+Not-a-heading-4                                                     *heading4*
 Text
 
 --------------------------------------------------------------------------------