-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathF-Editor-Tooling-Part1.html
350 lines (225 loc) · 21.2 KB
/
F-Editor-Tooling-Part1.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Behind the F# editor tootling. Part 1: introduction to compilers.</title>
<meta name="description" content="Behind the F# editor tootling. Part 1: introduction to compilers.">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="shortcut icon" href="/favicon/favicon.ico">
<link rel="apple-touch-icon-precomposed" href="/favicon/lattice-optic-illusion-152-204321.png" >
<meta name="msapplication-TileColor" content="#ffffff" />
<meta name="msapplication-TileImage" content="/favicon/lattice-optic-illusion-144-204321.png">
<link rel="stylesheet" type="text/css" href="//fonts.googleapis.com/css?family=Lato:400,400italic,700,700italic" />
<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.4.0/styles/default.min.css">
<link rel="stylesheet" href="/css/main.css" />
<link rel="stylesheet" href="/css/font-awesome.min.css" />
<link rel="canonical" href="http://localhost:4000/F-Editor-Tooling-Part1">
<link rel="alternate" type="application/rss+xml" title="F# Reflections Feed" href="http://localhost:4000/feed.xml">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.2/jquery.min.js"></script>
</head>
<body>
<nav id="top" class="site-navigation">
<div class="inner">
<button id="menu-toggle" aria-expanded="false">Menu</button>
<div class="nav-menu">
<ul class="menu">
<li class="">
<a class="" href="/">Home</a>
</li>
<li class="">
<a class="" href="/about">About</a>
</li>
<li class="">
<a class="" href="/tags/">Archives</a>
</li>
<li class="">
<a class="" href="/2016">2016</a>
</li>
</ul>
</div><!-- .nav-menu -->
<div class="subscribe-button">
<a class="fa-feed square fill-horizontal" href="/feed.xml"><span class="screen-reader-text">RSS</span></a>
</div><!-- .social-links -->
</div><!-- .inner -->
</nav><!-- .site-navigation -->
<header class="site-header">
<div class="inner">
<h1 class="site-title"><a class="logo-text" href="/">F# Reflections</a></h1>
<p class="site-description">Thoughts, stories and ideas.</p>
</div><!-- .inner -->
</header>
<div class="site-content">
<div class="inner">
<main class="site-main">
<article class="post">
<header class="entry-header">
<div class="entry-header-wrap">
<div class="post-thumbnail">
<img src="/images/posts/landscape_4.jpg" alt="Behind the F# editor tootling. Part 1: introduction to compilers.">
</div><!-- .post-thumbnail -->
<h1 class="entry-title">Behind the F# editor tootling. Part 1: introduction to compilers.</h1>
</div><!-- .entry-header-wrap -->
<div class="entry-meta">
by <span class="post-author">Krzysztof Cieslak</span> on <time class="published" datetime="2019-08-27">August 27, 2019</time>
</div><!-- .entry-meta -->
</header><!-- .entry-header -->
<div class="entry-content">
<h1>Behind the F# editor tootling. Part 1: introduction to compilers.</h1>
<p>F# editor tooling, while it’s not on the level of mainstream languages like C# or Java, has been in a unique spot among Functional Programming languages. F# tooling is available cross-platform, it’s feature-rich, stable, innovative and, what’s also essential, it’s commercially supported and actively developed. Moreover, all this is available while communities of some other FP languages are discussing whether displaying simple tooltip is not too expensive.</p>
<p>In the series, I’d like to dive behind what we see in the editor and show some of the magic that lets us have this excellent developer tooling experience. In this post I’d like to do a gentle introduction to compiler design, introduce some concepts and vocabulary that will be useful later, and give a general description of the potential approaches to the editor tooling in context of compilers.</p>
<!--more-->
<blockquote>
<p>Please keep in mind that this post is a high-level introduction to the concepts with some simplifications and without too many details. If you’re one of a few people that work on compilers or editor tooling this post is probably not for you ;-)</p>
</blockquote>
<h2>Compilers 101</h2>
<p>If you ever took university course about compilers or read any introductory book to the topic you probably know that compilers are often treated as a black box - compiler takes a list of files, some options, do magic, and then it outputs some executable or library.</p>
<p>Typically inside the compiler, several things are happening in order:</p>
<ul>
<li>Lexing - it’s the process that transforms text (source code) into a list of tokens that can be understood by the compiler</li>
<li>Parsing - it takes the list of tokens produced by lexing and transforms it into Abstract Syntax Tree (AST). AST is a tree that represents the structure of your code. Each node of the tree denotes a construct occurring in the source code. The syntax is “abstract” in the sense that it does not represent every detail appearing in the real syntax, but rather just the structural or content-related details. For example, construct like an if-condition-then expression may be denoted as a single node with three branches. Parsing can produce errors if your code contains structural errors (for example, missing parentheses, or wrong indentation)</li>
<li>Typechecking - In a statically typed language, it’s a process of verifying and enforcing the constraints of the types. Usually it takes as an input AST and any additional context (for example external references) and ensures that everything is fine with your code regarding type system - for example, it checks what types are variables you use, if they contain the members you use if your function returned declared type. A result of type checking is a different type of the syntax tree - in F# compiler we call it Typed Abstract Syntax Tree (TAST). TAST is built on a higher level of abstraction - it contains fewer details about the structure of the code, but more information about entities (variables, objects, functions, etc.) including full type information. Type checking can produce a lot of different types of errors and is one of the primary sources of them.</li>
<li>Optimisations - is a process where compiler perform a different type of optimisations on your code such as removing dead code or inlining. Every optimisation is taking TAST as an input and returns new TAST as an output, and the process is repeated as long as the output TAST is different from input one.</li>
<li>Code generation - it’s the final step of the process, in which compiler takes final TAST and generates output code from it (either in the form of the byte code of some form - IL in case of F# - or just native code)</li>
</ul>
<h2>Compiler 101 vs editor tooling</h2>
<p>Unfortunately, the naive, black-box approach to the building compilers is not enough. With such design, you can only start getting information about code only after code is built, inspecting the final result. It is problematic for multiple reasons:</p>
<ul>
<li>You don’t have any information connected to the structure of code - you only have final results. It may be suitable enough to get some tooltip information or maybe a list of usages of the given Symbol, but it’s not enough in case of any editor features depending on code structure (refactorings, autocomplete lists, formatters etc.)</li>
<li>You only have up-to-date information about your code after you’ve saved the file and built project - for example, you don’t get updated errors in your code while you’re typing and making changes in the file</li>
<li>Saving the file and building a project usually takes a while - if our compiler process is entirely stateless it needs to perform all phases. Also, everything has its costs. Especially optimisations and code generation is not interesting for us from editor tooling point of view - we want to provide tooling in a context of the source code we edit, not in the context of optimised, final artefact we get from the compiler.</li>
</ul>
<p>Due to those problems editor tooling for languages using a black-box design of the compiler usually have huge problems - either they’re limited in what’s possible in such editor tooling, or attempt to work around those problems by re-inventing what compiler does by using technologies such as tree-sitter or just by writing “presentation compiler”.</p>
<p>However, such workarounds are never as accurate and fully featured as systems based on the compiler directly. While building performant, accurate and productive editor tooling, we need to have direct access to the internal state of a compiler (such as AST, TAST, and Symbol information) and that’s why we need a different approach to building compilers.</p>
<h2>Modern compiler design</h2>
<p>In recent years many mainstream programming languages have moved towards compiler service approach to the editor tooling. In such approach, editor tooling is powered by internal capabilities of the compiler which ensures that editor functionality. Examples of such approach are [C# Roslyn API] or [F# Compiler Service] (on which we will focus more in the next parts of this series).</p>
<p>The modern compiler needs to be:</p>
<ul>
<li>API - compiler needs to be accessible as a normal library or some other kind of API. We need to be able to run it inside another process, and we need to be able to communicate with different layers of a compiler (usually lexer, parser and type-checker)</li>
<li>Server - it needs to be ready to run as a long-running, stateful application that gives us the ability to do partial operations (for example single file parsing) in the context of a whole compilation unit. The fact it’s the stateful, long-running and asynchronous process has vast implications on the design of compiler and the performance optimisations.</li>
<li>Database - as mentioned above compiler is a long-running stateful process. Additionally, we need to have a way of getting what is usually internal information of the compiler - Tokens, AST, TAST, list of known by the compiler symbols and more.</li>
</ul>
<p>With such compiler capabilities, we can host it directly in editor process (for example F# or C# tools in VS) or in another application providing higher-level API that is also communication layer which can be used from editors (for example language server using Language Server Protocol)</p>
<h2>Summary</h2>
<p>I hope this blog post is good enough, the high-level introduction of compiler design, concepts and vocabulary that you’ll need to know in the future posts from this series.</p>
</div><!-- .entry-content -->
<footer class="entry-footer">
<div class="share-post">
<span>Share</span>
<a class="fa-twitter" target="_blank" href="https://twitter.com/intent/tweet?text=Behind%20the%20F#%20editor%20tootling.%20Part%201:%20introduction%20to%20compilers.&url=http://localhost:4000/F-Editor-Tooling-Part1"><span class="screen-reader-text">Twitter</span></a>
<a class="fa-facebook" target="_blank" href="https://www.facebook.com/sharer/sharer.php?u=http://localhost:4000/F-Editor-Tooling-Part1&t=Behind%20the%20F#%20editor%20tootling.%20Part%201:%20introduction%20to%20compilers."><span class="screen-reader-text">Facebook</span></a>
<a class="fa-google-plus" target="_blank" href="http://plus.google.com/share?url=http://localhost:4000/F-Editor-Tooling-Part1"><span class="screen-reader-text">Google+</span></a>
</div><!-- .share-post -->
<div class="tag-links">
Tags: <a href="/tags/index.html#F%23" title="Pages tagged F#" rel="tag">F#</a><a href="/tags/index.html#Tooling" title="Pages tagged Tooling" rel="tag">Tooling</a><a href="/tags/index.html#Compiler" title="Pages tagged Compiler" rel="tag">Compiler</a><a href="/tags/index.html#FSharp.Compiler.Services" title="Pages tagged FSharp.Compiler.Services" rel="tag">FSharp.Compiler.Services</a>
</div>
</footer><!-- .entry-footer -->
</article><!-- .post -->
<div class="author-box">
<div class="author-info">
<a href="http://kcieslak.io/about" target="_blank"> <div class="author-avatar" style="background-image: url(/images/authorimage.jpg)"><span class="screen-reader-text">Krzysztof Cieslak's Picture</span></div> </a>
<div class="author-details">
<h2 class="author-title">About Krzysztof Cieslak</h2>
<p class="author-bio">Krzysztof is a F# developer and consultant, open source contributor, conference speaker</p>
<span class="author-location"><i class="fa-map-marker" aria-hidden="true"></i> Lodz, Poland</span>
<span class="author-website"><i class="fa-chain" aria-hidden="true"></i> <a href="http://kcieslak.io" target="_blank"> http://kcieslak.io</a></span>
</div><!-- .author-details -->
</div><!-- .author-info -->
</div><!-- .author-box -->
<nav class="post-navigation">
<h2 class="screen-reader-text">Post navigation</h2>
<div class="nav-previous">
<div class="nav-inside">
<a href="/Ionide-Introducing-Info-Panel" class="nav-thumb" style="background-image: url(/images/posts/landscape_8.jpg)"></a>
<div class="nav-before">Previous</div>
<div class="nav-title"><a href="/Ionide-Introducing-Info-Panel">Ionide — Introducing Info Panel</a></div>
<div class="nav-date">May 6, 2019</div>
</div><!-- .nav-inside -->
</div><!-- .nav-previous -->
</nav><!-- .post-navigation -->
</main><!-- .site-main -->
<aside class="sidebar">
<section class="widget widget-text">
<h2 class="widget-title">About Krzysztof Cieslak</h2>
<p>Krzysztof is a F# developer and consultant, open source contributor, conference speaker</p>
</section><!-- .widget-text -->
<section class="widget widget-text">
<p><a href="http://lambdafactory.io" target="_blank"><img src="/images/banner.png" alt="Optional Banner Ad" /></a></p>
</section><!-- .widget-text-->
<section class="widget widget-recent-posts">
<h2 class="widget-title">Latest Posts</h2>
<ul class="recent-posts">
<li class="recent-item"><a href="/F-Editor-Tooling-Part1">Behind the F# editor tootling. Part 1: introduction to compilers.</a> <span>August 27, 2019</span></li>
<li class="recent-item"><a href="/Ionide-Introducing-Info-Panel">Ionide — Introducing Info Panel</a> <span>May 6, 2019</span></li>
<li class="recent-item"><a href="/Ionide-New-Hope">Ionide — A New Hope</a> <span>March 24, 2019</span></li>
<li class="recent-item"><a href="/PureScript-on-Azure-Functions">PureScript on Azure Functions</a> <span>January 12, 2019</span></li>
<li class="recent-item"><a href="/Future-of-F-Cross-platform-editor-tooling">Future of F# cross-platform editor tooling</a> <span>December 31, 2018</span></li>
</ul><!-- .recent-posts -->
</section><!-- .widget-recent-posts -->
<!--Create a sorted array of tags-->
<section class="widget widget-tags">
<h2 class="widget-title">Tags</h2>
<div class="tagcloud">
<a href="/tags/#Azure">Azure</a>
<a href="/tags/#Azure+Functions">Azure Functions</a>
<a href="/tags/#Community">Community</a>
<a href="/tags/#Compiler">Compiler</a>
<a href="/tags/#F%23">F#</a>
<a href="/tags/#FAKE">FAKE</a>
<a href="/tags/#FSharp.Compiler.Services">FSharp.Compiler.Services</a>
<a href="/tags/#Fable">Fable</a>
<a href="/tags/#JS">JS</a>
<a href="/tags/#MVP">MVP</a>
<a href="/tags/#OAuth">OAuth</a>
<a href="/tags/#Open+Source">Open Source</a>
<a href="/tags/#Paket">Paket</a>
<a href="/tags/#PureScript">PureScript</a>
<a href="/tags/#Saturn">Saturn</a>
<a href="/tags/#Serverless">Serverless</a>
<a href="/tags/#Tooling">Tooling</a>
<a href="/tags/#VS+Code">VS Code</a>
<a href="/tags/#Web">Web</a>
<a href="/tags/#Webpack">Webpack</a>
</div><!-- .tagcloud -->
</section><!-- .widget -->
<section class="widget widget-text">
<h2 class="widget-title">Contact me</h2>
<a href="https://twitter.com/k_cieslak" class="fa-twitter square fill-horizontal"><span class="screen-reader-text">Twitter</span></a>
<a href="https://github.com/Krzysztof-Cieslak" class="fa-github-alt square fill-horizontal"><span class="screen-reader-text">GitHub</span></a>
<a href="https://gitlab.com/Krzysztof-Cieslak" class="fa-gitlab square fill-horizontal"><span class="screen-reader-text">GitHub</span></a>
<a href="mailto:[email protected]" class="fa-envelope square fill-horizontal"><span class="screen-reader-text">E-Mail</span></a>
</section><!-- .widget-text -->
</aside><!-- .sidebar -->
</div><!-- .inner -->
</div><!-- .site-content -->
<footer class="site-footer">
<div class="inner">
<div class="social-links">
<a href="https://twitter.com/k_cieslak" class="fa-twitter square fill-horizontal"><span class="screen-reader-text">Twitter</span></a>
<a href="https://github.com/Krzysztof-Cieslak" class="fa-github-alt square fill-horizontal"><span class="screen-reader-text">GitHub</span></a>
<a href="https://gitlab.com/Krzysztof-Cieslak" class="fa-gitlab square fill-horizontal"><span class="screen-reader-text">GitHub</span></a>
<a href="mailto:[email protected]" class="fa-envelope square fill-horizontal"><span class="screen-reader-text">E-Mail</span></a>
<a class="fa-feed square fill-horizontal" href="/feed.xml"><span class="screen-reader-text">RSS</span></a>
</div><!-- .social-links -->
<div class="site-info">
© <a href="/">Krzysztof Cieslak</a> all rights reserved. <br />Powered by <a target="_blank" href="https://jekyllrb.com/">Jekyll</a>. BlogInn theme by <a target="_blank" href="http://justgoodthemes.com/">JustGoodThemes</a>
<a href="#top" id="top-link" class="fa-chevron-up top-link square fill-horizontal"><span class="screen-reader-text">Back to the top</span></a>
</div><!-- .site-info -->
</div><!-- .inner -->
</footer><!-- .site-footer -->
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.4.0/highlight.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.4.0/languages/fsharp.min.js"></script>
<script type="text/javascript" src="/js/plugins.js"></script>
<script type="text/javascript" src="/js/custom.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<!-- Asynchronous Google Analytics snippet -->
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-74514389-1', 'auto');
ga('require', 'linkid', 'linkid.js');
ga('send', 'pageview');
</script>
</body>
</html>