-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathextensions.html
373 lines (359 loc) · 16 KB
/
extensions.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<meta name="author" content="by Darius J Chuck" />
<title>The Jevko Syntax: Extensions Specification</title>
<style>
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
/* The extra [class] is a hack that increases specificity enough to
override a similar rule in reveal.js */
ul.task-list[class]{list-style: none;}
ul.task-list li input[type="checkbox"] {
font-size: inherit;
width: 0.8em;
margin: 0 0.8em 0.2em -1.6em;
vertical-align: middle;
}
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
</style>
<link rel="stylesheet" href="/style.css?stamp=1704054960818" />
<link rel="stylesheet" href="/highlight/style.css?stamp=1704054960818" />
<link rel="icon" type="image/png" href="/jevko.png" />
<meta name="og-image" property="og:image" content="/jevko.png" />
<meta name="twitter:image" content="/jevko.png" />
<!--[if lt IE 9]>
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
<![endif]-->
</head>
<body>
<header id="top" class="top black-bg">
<div class="header">
<a href="/index.html">
<img class="logo" src="/jevko.png" alt="logo" />
</a>
<div class="right links" style="flex-grow: 1">
<div>
<a href="https://jevko-org.translate.goog/?_x_tr_sl=en&_x_tr_tl=zh-TW&_x_tr_hl=en" title="Click here to Google Translate jevko.org"><!--
--><img src="/translate_icon.svg" style="width: 2.5rem;" alt="[^translate]" /><!--
--></a>
<a href="https://xtao.org" title="Click here to go to xtao.org, the parent page of jevko.org"><!--
--><img src="/tao-logo.svg" style="width: 2.5rem;" alt="[^TAO]" /><!--
--></a>
</div>
<div>
<a href="https://archive.org/details/@jevko" title="Click here to go to the official Jevko archive.org profile"><!--
--><img src="/archive_icon.svg" style="width: 2.5rem;" alt="[^archive]" /><!--
--></a>
<a href="https://www.youtube.com/@jevko-org" title="Click here to go to the official Jevko YouTube channel"><!--
--><img src="/youtube_icon.svg" style="width: 2.5rem;" alt="[^youtube]" /><!--
--></a>
<a href="https://codeberg.org/jevko-org" title="Click here to go to the official Jevko Codeberg organization"><!--
--><img src="/codeberg_icon.svg" style="width: 2.5rem;" alt="[^codeberg]" /><!--
--></a>
<a href="https://github.com/jevko" title="Click here to go to the official Jevko GitHub organization"><!--
--><img src="/github_icon.svg" style="width: 2.5rem;" alt="[^github]" /><!--
--></a>
<a rel="me" href="https://layer8.space/@jevko" title="Click here to go to the official Jevko Mastodon channel"><!--
--><img src="/mastodon_icon.svg" style="width: 2.5rem;" alt="[^mastodon]" /><!--
--></a>
<a href="https://reddit.com/r/jevko" title="Click here to go to the official Jevko subreddit"><!--
--><img src="/reddit_icon.svg" style="width: 2.5rem;" alt="[^reddit]" /><!--
--></a>
<a href="https://layer8.space/@jevko.rss" title="Click here to download the official Jevko RSS channel"><!--
--><img src="/rss_icon.svg" style="width: 2.5rem;" alt="[rss]" /><!--
--></a>
</div>
<div>
<a href="/spec.html" title="Click here to go to see the official Jevko speficication">[specification]</a>
<a href="/diagram.xhtml" title="Click here to see the official Jevko syntax railroad diagrams">[diagrams]</a>
<a href="/extensions.html" title="Click here to go to see the official Jevko extensions specification">[extensions]</a>
<!-- <a href="/license.html" title="Click here to see the license Jevko is available under">[license]</a> -->
</div>
</div>
</div>
</header>
<header id="title-block-header">
<h1 class="title">The Jevko Syntax: Extensions Specification</h1>
<p class="author">by Darius J Chuck</p>
<p class="date">First Draft: January 2024.</p>
</header>
<p>These extensions complement the <a
href="spec.html">specification</a>.</p>
<h1 id="fencedtext">FencedText</h1>
<p>A Jevko extension which enables arbitrary text without escaping,
delimited similarly to fenced code blocks in Markdown.</p>
<h2 id="status">Status</h2>
<p>This extension is strongly recommended for implementation in all
Jevko processors.</p>
<h2 id="examples">Examples</h2>
<p>Some examples of valid Jevkos using this extension:</p>
<pre><code>`'hello'`</code></pre>
<pre><code>```'
some
lines
with [arbitrary] `characters`
'```</code></pre>
<pre><code>[`'hello'`]
[```'
some
lines
with [arbitrary] `characters`
'```]</code></pre>
<p>Note that there is no whitespace or any other characters before and
after the enclosing quotes in any of these examples.</p>
<p>These are also valid, but not recommended:</p>
<pre><code>`'hello'`[]
[]```'
some
lines
with [arbitrary] `characters`
'```</code></pre>
<h2 id="formal-definition">Formal definition</h2>
<p>We will now formally define the <code>FencedText</code> extension in
ABNF.</p>
<p>We change the definition of the Jevko Text rule as follows:</p>
<pre class="abnf"><code>Text = FencedText / OpenText</code></pre>
<p>where <code>OpenText</code> is defined the same as the original
<code>Text</code> rule:</p>
<pre class="abnf"><code>OpenText = *Symbol</code></pre>
<p>and the new <code>FencedText</code> rule is defined as:</p>
<pre class="abnf"><code>FencedText = FencedText_1
/ FencedText_3
/ FencedText_5
/ FencedText_7
/ FencedText_9
/ FencedText_11
/ FencedText_13
/ FencedText_15
FencedText_1 = "`'" FencedContent_1 "'`"
FencedText_3 = "```'" FencedContent_3 "'```"
FencedText_5 = "`````'" FencedContent_5 "'`````"
FencedText_7 = "```````'" FencedContent_7 "'```````"
FencedText_9 = "`````````'" FencedContent_9 "'`````````"
FencedText_11 = "```````````'"
FencedContent_11
"'```````````"
FencedText_13 = "`````````````'"
FencedContent_13
"'`````````````"
FencedText_15 = "```````````````'"
FencedContent_15
"'```````````````"
FencedContent_1 = *AnyCodepoint
; can't contain "'`" ("[" / "]" / EOF)
FencedContent_3 = *AnyCodepoint
; can't contain "'```" ("[" / "]" / EOF)
FencedContent_5 = *AnyCodepoint
; can't contain "'`````" ("[" / "]" / EOF)
FencedContent_7 = *AnyCodepoint
; can't contain "'```````" ("[" / "]" / EOF)
FencedContent_9 = *AnyCodepoint
; can't contain "'`````````" ("[" / "]" / EOF)
FencedContent_11 = *AnyCodepoint
; can't contain "'```````````" ("[" / "]" / EOF)
FencedContent_13 = *AnyCodepoint
; can't contain "'`````````````" ("[" / "]" / EOF)
FencedContent_15 = *AnyCodepoint
; can't contain "'```````````````" ("[" / "]" / EOF)
; any Unicode codepoint
AnyCodepoint = %x0-10ffff</code></pre>
<h2 id="detailed-formal-definition">Detailed formal definition</h2>
<p><code>FencedText</code> is abstractly defined as follows.</p>
<p>We’ll use ABNF with some necessary additions which will be explained
as we go.</p>
<pre class="abnf"><code>FencedText(n) = {n}Escaper Fencer
FencedContent(n)
Fencer {n}Escaper</code></pre>
<p>where:</p>
<ul>
<li><code>n > 0 && n <= 15 && n % 2 == 1</code>
<ul>
<li>in other words <code>n ∈ {1, 3, 5, 7, 9, 11, 13, 15}</code></li>
<li><em>the reason even numbers are excluded is backwards
compatibility</em></li>
</ul></li>
<li><code>{n}Rule</code> means <code>Rule</code> repeated exactly
<code>n</code> times</li>
</ul>
<p>The upper limit of 15 for n is chosen conservatively for now. It may
be extended in the future.</p>
<p>Note: <code>FencedText(n)</code> is technically not a valid ABNF
rule. Rather, from a formal point of view it is a hyperrule with one
parameter (<code>n</code>).</p>
<p>Similarly, <code>FencedContent(n)</code> is a hyperrule which matches
an almost arbitrary sequence of codepoints:</p>
<pre class="abnf"><code>FencedContent(n) = *AnyCodepoint</code></pre>
<p>where:</p>
<ul>
<li><code>AnyCodepoint</code> means any Unicode codepoint
(<code>%x0-10ffff</code>)</li>
<li><code>FencedContent(n)</code> can’t contain the sequence
<code>Fencer {n}Escaper (Opener / Closer / EOF)</code>
<ul>
<li>which is our closing delimiter/fence</li>
<li><code>EOF</code> matches the End Of Input</li>
</ul></li>
</ul>
<p>Put in practical terms: we have to parse the input lazily, matching
the rule as soon as we find the closing fence.</p>
<p>Now we define:</p>
<pre class="abnf"><code>Fencer = "'"</code></pre>
<p>Equivalently:</p>
<pre class="abnf"><code>Fencer = %x27 ; apostrophe</code></pre>
<p>So we can restate the rule more concretely:</p>
<pre class="abnf"><code>FencedText(n) = {n}"`" "'" FencedContent(n) "'" {n}"`"</code></pre>
<ul>
<li>so <code>FencedContent(n)</code> can’t contain
<code>"'" {n}"`" ("[" / "]" / EOF)</code></li>
</ul>
<p>Finally, we can expand the hyperrules into multiple concrete
pure-ABNF rules:</p>
<pre class="abnf"><code>FencedText = FencedText_1
/ FencedText_3
/ FencedText_5
/ ...
/ FencedText_15
FencedText_1 = "`'" FencedContent_1 "'`"
FencedText_3 = "```'" FencedContent_3 "'```"
FencedText_5 = "`````'" FencedContent_5 "'`````"
...
FencedText_15 = "```````````````'"
FencedContent_15
"'```````````````"
FencedContent_1 = *AnyCodepoint
; can't contain "'`" ("[" / "]" / EOF)
FencedContent_3 = *AnyCodepoint
; can't contain "'```" ("[" / "]" / EOF)
FencedContent_5 = *AnyCodepoint
; can't contain "'`````" ("[" / "]" / EOF)
...
FencedContent_15 = *AnyCodepoint
; can't contain "'```````````````" ("[" / "]" / EOF)</code></pre>
<h1 id="taggedtext">TaggedText</h1>
<p>A Jevko extension which enables arbitrary text without escaping,
delimited similarly to heredocs or dollar quoted literals in
PostgreSQL.</p>
<p>Even though this extension is technically independent, this
description assumes <code>FencedText</code> has been implemented as
well.</p>
<h2 id="status-1">Status</h2>
<p>This extension is experimental and its specification may change in
the future.</p>
<h2 id="examples-1">Examples</h2>
<p>Some examples of valid Jevkos using this extension:</p>
<pre><code>`//hello//</code></pre>
<pre><code>`/tag/
some
lines
with [arbitrary] `characters`
/tag/</code></pre>
<pre><code>[`//hello//]
[`/end/
some
lines
with [arbitrary] `characters`
/end/]</code></pre>
<p>These are also valid, but not recommended:</p>
<pre><code>`//hello//[]
[]`/end/
some
lines
with [arbitrary] `characters`
/end/</code></pre>
<h2 id="formal-definition-1">Formal definition</h2>
<p>The <code>TaggedText</code> extension is formally defined similarly
to <code>FencedText</code> (therefore see the definition of
<code>FencedText</code> for more details).</p>
<p>We further extend the <code>Text</code> rule, assuming we’ve already
added <code>FencedText</code>:</p>
<pre class="abnf"><code>Text = TaggedText / FencedText / OpenText</code></pre>
<p>Exhaustive definition of <code>TaggedText</code> in terms of pure
ABNF rules is not viable, as the rule admits an enormous number of
unique user-defined tags to use as delimiters.</p>
<p>Because of that, we only show ABNF definitions only for a few example
tags, namely the empty tag, <code>END</code>, and <code>done</code>.</p>
<pre class="abnf"><code>TaggedText = TaggedText_
/ TaggedText_END
/ TaggedText_done
/ ...
TaggedText_ = "`//" TaggedContent_ "//"
TaggedText_END = "`/END/" TaggedContent_END "/END/"
TaggedText_done = "`/done/" TaggedContent_done "/done/"
...
TaggedContent_ = *AnyCodepoint
; can't contain "//" ("[" / "]" / EOF)
TaggedContent_END = *AnyCodepoint
; can't contain "/END/" ("[" / "]" / EOF)
TaggedContent_done = *AnyCodepoint
; can't contain "/done/" ("[" / "]" / EOF)
...</code></pre>
<h2 id="detailed-formal-defintion">Detailed formal defintion</h2>
<p>Abstractly we define two hyperrules:</p>
<pre class="abnf"><code>TaggedText(Tag) = Escaper Tagger Tag Tagger
TaggedContent(Tag)
Tagger Tag Tagger
TaggedContent(n) = *AnyCodepoint</code></pre>
<p>where <code>TaggedContent(Tag)</code> can’t contain
<code>Tagger Tag Tagger (Opener / Closer / EOF)</code>.</p>
<p>Now we define:</p>
<pre class="abnf"><code>Tagger = "/"</code></pre>
<p>Equivalently:</p>
<pre class="abnf"><code>Tagger = %x2f ; slash</code></pre>
<p>and:</p>
<pre class="abnf"><code>Tag = *255(ALPHA / DIGIT / "_")</code></pre>
<p>Equivalently:</p>
<pre class="abnf"><code>Tag = 0*255(%x41–5A / %x61–7A / %x30–39 / %x5f)</code></pre>
<p>or, using a regular expression:</p>
<pre class="regexp"><code>Tag = /[a-zA-Z0-9_]{0,255}/</code></pre>
<p>In other words <code>Tag</code> is an identifier of length 0-255
characters which can contain only alphanumeric characters and
underscores.</p>
<p>Note: the tag length is currently limited to 255. If need be, this
maximum length as well as the allowed characters may be extended in the
future.</p>
<p>Now we can restate the hyperrule more concretely:</p>
<pre class="abnf"><code>TaggedText(Tag) = "`/" Tag "/"
TaggedContent(Tag)
"/" Tag "/"</code></pre>
<p>where <code>TaggedContent(Tag)</code> can’t contain
<code>"/" Tag "/" ("[" / "]" / EOF)</code>.</p>
<p>Finally, we can expand this into a very large number of pure ABNF
rules:</p>
<pre class="abnf"><code>TaggedText = TaggedText_
/ TaggedText_END
/ TaggedText_done
/ ...
TaggedText_ = "`//" TaggedContent_ "//"
TaggedText_END = "`/END/" TaggedContent_END "/END/"
TaggedText_done = "`/done/" TaggedContent_done "/done/"
...
TaggedContent_ = *AnyCodepoint
; can't contain "//" ("[" / "]" / EOF)
TaggedContent_END = *AnyCodepoint
; can't contain "/END/" ("[" / "]" / EOF)
TaggedContent_done = *AnyCodepoint
; can't contain "/done/" ("[" / "]" / EOF)
...</code></pre>
<footer id="bottom" class="black-bg">
<div class="footer">
<!-- <a href="/index.html" class="footer-link"> -->
<div class="left" style="margin-bottom: 1.25rem;">© 2024 Jevko.org</div>
<!-- </a> -->
<div class="right links">
<a href="/attribution.html" title="Click here to go to the [attribution] subpage"><!--
-->[attribution]<!--
--></a>
</div>
</div>
</footer>
</body>
</html>