feat: listitem (line_li)

- BREAKING CHANGE: give up on block_end, line_noeol. Too much trouble
  and doesn't matter for consumers anyway.
- line_li now consumes all adjacent lines until blank line, "<", or
  next listitem. This makes it much easier for applications to present
  lists--even nested lists--because now the application only needs to
  compare the indent of the previous sibling listitem.

Author: justinmk

README.md

@@ -1,32 +1,33 @@
 tree-sitter-vimdoc
 ==================
 
-This grammar intentionally support a subset of the vimdoc "spec"; predictable
-results are the primary goal, so that _output_ formats (e.g. HTML) are
-well-formed; the _input_ (vimdoc) is secondary. The first step should always be
-to try to fix the input (within reason) rather than insist on a grammar that
-handles vimdoc's endless quirks.
+This grammar intentionally support a subset of the vimdoc "spec"
+([ref1](https://neovim.io/doc/user/helphelp.html#help-writing),
+[ref2](https://github.com/nanotee/vimdoc-notes));
+predictable results are the primary goal, so that _output_ formats (e.g. HTML)
+are well-formed; the _input_ (vimdoc) is secondary. The first step should
+always be to try to fix the input rather than insist on a grammar that handles
+vimdoc's endless quirks.
 
 Overview
 --------
 
-- vimdoc format "spec":
-    - [:help help-writing](https://neovim.io/doc/user/helphelp.html#help-writing)
-    - https://github.com/nanotee/vimdoc-notes
-- whitespace is intentionally captured in all atoms, because it is often used
-  for "layout" and ascii art in legacy help files.
-- `block` is the main top-level node which contains `line` nodes.
-  - ends at blank line(s) or a line starting with `<`.
+- `block` is the main top-level node which contains `line` and `line_li` nodes.
+  - delimited by blank line(s) or any line starting with `<` (codeblock terminator).
 - `line`:
   - contains atoms (words, tags, taglinks, …)
-  - contains `codeblock` because `>` can start a codeblock at the end of a line.
   - contains headings (`h1`, `h2`, `h3`) because `codeblock` terminated by
     "implicit stop" (no terminating `<`) consumes blank lines, so `block` has
     no way to end.
   - contains `column_heading` because `<` (the `codeblock` terminating char)
     can appear at the start of `column_heading`.
+- `line_li` ("list item")
+  - consumes lines until blank line, codeblock, or next listitem.
+  - nesting is ignored: indented listitems are parsed as siblings.
 - `codeblock`:
-  - contains `line` nodes which do not contain `word` nodes, it's just the full
+  - contained by `line` or `line_li`. Because ">" can start
+    a codeblock at the end of any line.
+  - contains `line` nodes without `word` nodes, it's just the full
     raw text line including whitespace. This is somewhat dictated by its
     "preformatted" nature; parsing the contents would require loading a "child"
     language (injection). See [#2](https://github.com/neovim/tree-sitter-vimdoc/issues/2).
@@ -38,16 +39,20 @@ Overview
 Known issues
 ------------
 
-- `line_li` ("list item") is experimental. It doesn't support nesting yet.
+- Input must end with newline/EOL (`\n`). Grammar does not support files without EOL.
+- Input must end with a blank line. Though this doesn't seem to matter in practice.
 - Spec requires that `codeblock` delimiter ">" must be preceded by a space
   (" >"), not a tab. But currently the grammar doesn't enforce this. Example:
   `:help lcs-tab`.
 - `url` doesn't handle _surrounding_ parens. E.g. `(https://example.com/#yay)` yields `word`
 - `url` doesn't handle _nested_ parens. E.g. `(https://example.com/(foo)#yay)`
+- `column_heading` currently only recognizes tilde "~" preceded by space (i.e.
+  "foo ~" not "foo~"). This covers 99% of :help files, but the grammar should
+  probably support "foo~" also.
 
 TODO
 ----
 
-- `line_noeol` is a special-case to support documents that don't end in EOL.
-  Grammar could be simpler if we require EOL at end of document.
 - `line_modeline` ?
+- `tag_heading` : line(s) containing only tags, typically implies a "heading"
+  before a block.

corpus/arguments.txt

@@ -2,6 +2,8 @@
 simple argument
 ================================================================================
 This in an argument: {arg}
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -19,6 +21,7 @@ multiple arguments on the same line
 ================================================================================
 {foo} {bar} {baz}
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -36,6 +39,7 @@ NOT an argument
 ================================================================================
 {foo "{bar}" `{baz}` |{baz| } {}
 
+
 --------------------------------------------------------------------------------
 
 (help_file

corpus/codeblock.txt

@@ -16,6 +16,8 @@ block3:
 <
 
   text
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -62,6 +64,7 @@ From luaref.txt:
 
 text
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -116,6 +119,7 @@ h2-headline *foo*
 
 H3 HEADLINE *foo*
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -162,8 +166,8 @@ codeblock with empty lines
 
 Example: >
 
-    let buf = nvim_create_buf(v:false, v:true)
-    call nvim_buf_set_lines(buf, 0, -1, v:true, ["test", "text"])
+    - let buf = nvim_create_buf(v:false, v:true)
+    - call nvim_buf_set_lines(buf, 0, -1, v:true, ["test", "text"])
 
     let opts = {'relative': 'cursor', 'width': 10, 'height': 2, 'col': 0,
         \ 'row': 1, 'anchor': 'NW', 'style': 'minimal'}
@@ -172,6 +176,7 @@ Example: >
     call nvim_win_set_option(win, 'winhl', 'Normal:MyHighlight')
 x
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -211,6 +216,7 @@ tricky codeblock
 
 tricky
 
+
 --------------------------------------------------------------------------------
 
 (help_file

corpus/codespan.txt

@@ -4,6 +4,7 @@ simple codespan
 
 a `foobar` b `:echo`
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -23,6 +24,7 @@ codespan in text
 Hello `world`, I am `markup language`. But `this is
 an error`.
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -52,6 +54,7 @@ NOT codespan
 				*g'* *g'a* *g`* *g`a*
 g'{mark}  g`{mark}
 
+
 --------------------------------------------------------------------------------
 
 (help_file

corpus/heading1_2.txt

@@ -12,6 +12,7 @@ Heading two                                                *tag2.1*   *tag2.2*
 
 Text2
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -51,6 +52,7 @@ Text
 
 Text
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -104,6 +106,7 @@ Text
 Not-a-heading-4                                                     *heading4*
 Text
 
+
 --------------------------------------------------------------------------------
 
 (help_file

corpus/heading3-column_heading.txt

@@ -5,8 +5,10 @@ HELLO
 
 Test
 
-GUI ~
+GUI COOL ~
 Text
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -20,7 +22,8 @@ Text
   (block
     (line
       (column_heading
-        (uppercase_name)))
+        (word)
+        (word)))
     (line
       (word))))
 
@@ -30,6 +33,8 @@ h3 (uppercase) heading, multiple words
 HELLO WORLD
 
   LEADING-SPACE-NOT-A-HEADING
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -47,6 +52,8 @@ h3 heading with tag
 HELLO WORLD        *abc*
 
 Test
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -63,9 +70,11 @@ Test
 ================================================================================
 column_heading lowercase
 ================================================================================
-Test~
+test-Test ~
 
 Below
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -78,17 +87,35 @@ Below
       (word))))
 
 ================================================================================
-column_heading with multiple words
+column_heading no space before tilde (FIXME)
 ================================================================================
-Multiple words~
+test~
+
+Below
+
+
+--------------------------------------------------------------------------------
+
+(help_file
+  (block
+    (line
+      (word)))
+  (block
+    (line
+      (word))))
 
-text
+================================================================================
+column_heading with multiple words
+================================================================================
+Multiple words ~
 
 nvim_ui_try_resize({width}, {height})                   *nvim_ui_try_resize()*
     TODO: Documentation
 
     Attributes: ~
         |RPC| only
+
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -97,9 +124,6 @@ nvim_ui_try_resize({width}, {height})                   *nvim_ui_try_resize()*
       (column_heading
         (word)
         (word))))
-  (block
-    (line
-      (word)))
   (block
     (line
       (word)
@@ -132,6 +156,7 @@ column_heading should NOT parse atoms (links, tags, etc.) (FIXME)
 
     Text.
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -165,6 +190,7 @@ ABC not-h3
 5. PRODUCT of 3.
    ...`/lua`-path,-leaving:
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -208,6 +234,7 @@ NOT column_heading
 ================================================================================
 tilde ~/foo/bar foo~ bar
 
+
 --------------------------------------------------------------------------------
 
 (help_file
@@ -216,5 +243,4 @@ tilde ~/foo/bar foo~ bar
       (word)
       (word)
       (word)
-      (word)
       (word))))