Skip to content

Commit c418f88

Browse files
authored
Merge #25 fix: codeblock without "<" consumes extra char
2 parents eee1c58 + 24498ae commit c418f88

File tree

11 files changed

+2883
-2455
lines changed

11 files changed

+2883
-2455
lines changed

README.md

Lines changed: 29 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -7,48 +7,47 @@ well-formed; the _input_ (vimdoc) is secondary. The first step should always be
77
to try to fix the input (within reason) rather than insist on a grammar that
88
handles vimdoc's endless quirks.
99

10-
Notes
11-
-----
10+
Overview
11+
--------
1212

1313
- vimdoc format "spec":
1414
- [:help help-writing](https://neovim.io/doc/user/helphelp.html#help-writing)
1515
- https://github.com/nanotee/vimdoc-notes
16-
- whitespace is intentionally captured in `(word)`, because it is often necessary to be
17-
able to correctly layout vim help files (especially old/legacy).
18-
- `(codeblock)` is contained by `(line)` because `>` can start a code block at the end of a line.
19-
- `(column_heading)` is contained by `(line)` because `>` (to close
20-
a `(codeblock)` can appear at the start of `(column_heading)`.
21-
- `h1` ("Heading 1"): `======` followed by text and optional `*tags*`.
22-
- `h2` ("Heading 2"): `------` followed by text and optional `*tags*`.
23-
- `h3` ("Heading 3"): only UPPERCASE WORDS, followed by optional `*tags*`.
16+
- whitespace is intentionally captured in all atoms, because it is often used
17+
for "layout" and ascii art in legacy help files.
18+
- `block` is the main top-level node which contains `line` nodes.
19+
- ends at blank line(s) or a line starting with `<`.
20+
- `line`:
21+
- contains atoms (words, tags, taglinks, …)
22+
- contains `codeblock` because `>` can start a codeblock at the end of a line.
23+
- contains headings (`h1`, `h2`, `h3`) because `codeblock` terminated by
24+
"implicit stop" (no terminating `<`) consumes blank lines, so `block` has
25+
no way to end.
26+
- contains `column_heading` because `<` (the `codeblock` terminating char)
27+
can appear at the start of `column_heading`.
28+
- `codeblock`:
29+
- contains `line` nodes which do not contain `word` nodes, it's just the full
30+
raw text line including whitespace. This is somewhat dictated by its
31+
"preformatted" nature; parsing the contents would require loading a "child"
32+
language (injection). See [#2](https://github.com/neovim/tree-sitter-vimdoc/issues/2).
33+
- the terminating `<` (and any following whitespace) is discarded (anonymous).
34+
- `h1` = "Heading 1": `======` followed by text and optional `*tags*`.
35+
- `h2` = "Heading 2": `------` followed by text and optional `*tags*`.
36+
- `h3` = "Heading 3": only UPPERCASE WORDS, followed by optional `*tags*`.
2437

2538
Known issues
2639
------------
2740

28-
- `line_li` ("list item") is _experimental_. It doesn't support nesting yet and
29-
it may not work well; you can treat it as a normal `line` for layout purposes.
30-
- `codeblock` ">" must not be preceded only by tabs, a space char is required (" >").
31-
See `:help lcs-tab` for example. Currently the grammar doesn't enforce this.
32-
- `codeblock` terminated by an "implicit stop" (i.e. no terminating `<`)
33-
consumes the first char of the terminating line, and continues the parent
34-
`block`, preventing top-level forms like `h1`, `h2` from being recognized
35-
until a blank line is encountered.
36-
- `line` in a `codeblock` does not contain `word` atoms, it's just the full
37-
raw text line including whitespace. This is somewhat dictated by its
38-
"preformatted" nature; parsing the contents would require loading a "child"
39-
language (injection). See [#2](https://github.com/vigoux/tree-sitter-vimdoc/issues/2).
41+
- `line_li` ("list item") is experimental. It doesn't support nesting yet.
42+
- Spec requires that `codeblock` delimiter ">" must be preceded by a space
43+
(" >"), not a tab. But currently the grammar doesn't enforce this. Example:
44+
`:help lcs-tab`.
4045
- `url` doesn't handle _surrounding_ parens. E.g. `(https://example.com/#yay)` yields `word`
4146
- `url` doesn't handle _nested_ parens. E.g. `(https://example.com/(foo)#yay)`
42-
- Ideally `block_end` should consume the last block of the document _only_ if that
43-
block is missing a trailing blank line or EOL ("\n").
44-
- TODO: consider simply _not supporting_ docs without EOL?
45-
- Ideally `line_noeol` should consume the last line of the document _only_ if
46-
that line is missing EOL ("\n").
47-
- TODO: consider simply _not supporting_ docs without EOL?
4847

4948
TODO
5049
----
5150

5251
- `line_noeol` is a special-case to support documents that don't end in EOL.
53-
Grammar could be a bit simpler if we just require EOL at end of document.
54-
- `line_modeline` (only at EOF)
52+
Grammar could be simpler if we require EOL at end of document.
53+
- `line_modeline` ?

corpus/arguments.txt

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,10 @@ NOT an argument
4343
(line
4444
(argument
4545
(word)
46-
(ERROR))
46+
(MISSING "}"))
47+
(word)
48+
(argument
49+
(word))
4750
(word)
4851
(codespan
4952
(word))

corpus/codeblock.txt

Lines changed: 115 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ block3:
2828
(word))
2929
(line
3030
(codeblock
31-
(line)))
31+
(line))))
32+
(block
3233
(line
3334
(word)))
3435
(block
@@ -92,17 +93,28 @@ text
9293
(word))))
9394

9495
================================================================================
95-
codeblock with implicit stop (FIXME)
96+
codeblock with implicit stop
9697
================================================================================
9798
>
9899
line1
99100
line2
100101

101-
-------------------------------
102+
===============================
102103
h1-headline *foo*
104+
line1
105+
106+
line2
107+
108+
>
109+
line1
103110

104111
-------------------------------
105-
h1-headline *foo*
112+
h2-headline *foo*
113+
114+
>
115+
line1
116+
117+
H3 HEADLINE *foo*
106118

107119
--------------------------------------------------------------------------------
108120

@@ -114,15 +126,35 @@ h1-headline *foo*
114126
(line)
115127
(line)))
116128
(line
117-
(word))
129+
(h1
130+
(word)
131+
(tag
132+
(word))))
118133
(line
119-
(word)
120-
(tag
121-
(word))))
122-
(h2
123-
(word)
124-
(tag
125-
(word))))
134+
(word)))
135+
(block
136+
(line
137+
(word)))
138+
(block
139+
(line
140+
(codeblock
141+
(line)
142+
(line)))
143+
(line
144+
(h2
145+
(word)
146+
(tag
147+
(word)))))
148+
(block
149+
(line
150+
(codeblock
151+
(line)
152+
(line)))
153+
(line
154+
(h3
155+
(uppercase_name)
156+
(tag
157+
(word))))))
126158

127159
================================================================================
128160
codeblock with empty lines
@@ -155,7 +187,9 @@ x
155187
(line)
156188
(line)
157189
(line)
158-
(line)))))
190+
(line)))
191+
(line
192+
(word))))
159193

160194
================================================================================
161195
tricky codeblock
@@ -166,7 +200,17 @@ tricky codeblock
166200
< line3
167201
<
168202

203+
Example: >
204+
205+
vim.spell.check()
206+
-->
207+
{
208+
{'quik', 'bad', 4}
209+
}
210+
<
211+
169212
tricky
213+
170214
--------------------------------------------------------------------------------
171215

172216
(help_file
@@ -176,6 +220,16 @@ tricky
176220
(line)
177221
(line)
178222
(line))))
223+
(block
224+
(line
225+
(word)
226+
(codeblock
227+
(line)
228+
(line)
229+
(line)
230+
(line)
231+
(line)
232+
(line))))
179233
(block
180234
(line
181235
(word))))
@@ -243,3 +297,51 @@ To test for a non-empty string, use empty(): >
243297
(word)
244298
(codeblock
245299
(line)))))
300+
301+
================================================================================
302+
codeblock stop and start on same line
303+
================================================================================
304+
Examples: >
305+
:lua vim.api.nvim_command('echo "Hello, Nvim!"')
306+
< LuaJIT: >
307+
:lua =jit.version
308+
<
309+
*:lua-heredoc*
310+
:lua << [endmarker]
311+
{script}
312+
313+
Example: >
314+
lua << EOF
315+
EOF
316+
<
317+
318+
--------------------------------------------------------------------------------
319+
320+
(help_file
321+
(block
322+
(line
323+
(word)
324+
(codeblock
325+
(line))))
326+
(block
327+
(line
328+
(word)
329+
(codeblock
330+
(line))))
331+
(block
332+
(line
333+
(tag
334+
(word)))
335+
(line
336+
(word)
337+
(word)
338+
(word))
339+
(line
340+
(argument
341+
(word))))
342+
(block
343+
(line
344+
(word)
345+
(codeblock
346+
(line)
347+
(line)))))

corpus/codespan.txt

Lines changed: 3 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -46,11 +46,10 @@ an error`.
4646
(word))))
4747

4848
================================================================================
49-
NOT a codespan
49+
NOT codespan
5050
================================================================================
51-
*'* *'a* *`* *`a*
52-
'{a-z} `{a-z} Jump to the mark.
53-
*g'* *g'a* *g`* *g`a*
51+
*'* *'a* *`* *`a*
52+
*g'* *g'a* *g`* *g`a*
5453
g'{mark} g`{mark}
5554

5655
--------------------------------------------------------------------------------
@@ -66,14 +65,6 @@ g'{mark} g`{mark}
6665
(word))
6766
(tag
6867
(word)))
69-
(ERROR)
70-
(line
71-
(argument
72-
(word))
73-
(word)
74-
(word)
75-
(word)
76-
(word))
7768
(line
7869
(tag
7970
(word))

corpus/heading1_2.txt

Lines changed: 29 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -15,20 +15,24 @@ Text2
1515
--------------------------------------------------------------------------------
1616

1717
(help_file
18-
(h1
19-
(word)
20-
(tag
21-
(word)))
18+
(block
19+
(line
20+
(h1
21+
(word)
22+
(tag
23+
(word)))))
2224
(block
2325
(line
2426
(word)))
25-
(h2
26-
(word)
27-
(word)
28-
(tag
29-
(word))
30-
(tag
31-
(word)))
27+
(block
28+
(line
29+
(h2
30+
(word)
31+
(word)
32+
(tag
33+
(word))
34+
(tag
35+
(word)))))
3236
(block
3337
(line
3438
(word))))
@@ -50,19 +54,23 @@ Text
5054
--------------------------------------------------------------------------------
5155

5256
(help_file
53-
(h1
54-
(tag
55-
(word))
56-
(word)
57-
(word))
57+
(block
58+
(line
59+
(h1
60+
(tag
61+
(word))
62+
(word)
63+
(word))))
5864
(block
5965
(line
6066
(word)))
61-
(h2
62-
(tag
63-
(word))
64-
(word)
65-
(word))
67+
(block
68+
(line
69+
(h2
70+
(tag
71+
(word))
72+
(word)
73+
(word))))
6674
(block
6775
(line
6876
(word))))

0 commit comments

Comments
 (0)