turbo-editors/turbo-go · colouring-and-completion.md

⑂ main

colouring-and-completion.md · 122 lines · 11.4 KBmarkdown    Blame HistoryRaw
   1# Colouring and completion — explanation
3## What is this about?
5The two features that make Turbo Go an editor *for Go* rather than a text editor that happens to be written in it: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.
7## Colouring: the compiler's own tokeniser
9Turbo Go does not have a syntax definition. It calls `go/scanner` — the lexer the Go toolchain itself uses — and turns the tokens it gets back into coloured spans.
11This means keywords, literals and operators are recognised **exactly** as the compiler recognises them. Raw string literals, automatic semicolon insertion, `0x_FF` separators, the lot. There is no regular expression to get subtly wrong, and no table to update when the language changes.
13### Tolerance is the whole point
15Source under a cursor is syntactically broken most of the time it is being typed. Half a string, an unclosed brace, an identifier that stops mid-word. A highlighter that gives up on invalid input is a highlighter that flickers off exactly when you are looking at it.
17So the scanner runs in its most forgiving mode and **every syntax error is discarded**. It still returns usable tokens: an unterminated string comes back as a string running to the end of the line, an unclosed `/*` as a comment running to the end of the file. Which is precisely the behaviour you want — the colours stay steady, and they tell you what is wrong.
19### What the editor adds
21Three distinctions the scanner does not make, because they are about reading rather than parsing:
23- an identifier before `(`, or after `func`, is a **function**
24- an identifier after `type`, `struct` or `interface` is a **type**
25- brackets, commas, dots and semicolons are **punctuation**, split out from the operators that compute, so a theme can quiet them down
27Predeclared names — `int`, `error`, `nil`, `len`, `min` — are recognised by name, not by keyword, because they are not keywords: a file may shadow them, and colouring the shadowed one anyway is what every other Go editor does too.
29### Why it is fast enough to do naively
31Scanning the whole file on every keystroke sounds wasteful, and would be. It does not happen: the buffer keeps a revision counter, bumped on every change, and the highlighter re-scans only when the number has moved. The editor redraws far more often than the text changes — every cursor move, every scroll — and all of those redraws are free.
33### The cost of this choice
35**A language is coloured only if someone wrote a scanner for it.** There is no definition language to write a definition in, so each one is Go code.
37Eight of them — TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell — live in turbo-core, because every editor built on it meets them whatever language it is for. The Go scanner lives here, in `internal/golang`, and is registered at start-up with `syntax.Register`. That is why a `.rs` file opens as plain text in Turbo Go: this editor registers Go and nothing else.
39That is a real limitation, and it was accepted deliberately: this is an editor for Go. A generic highlighter would have brought a dependency, a definition format, and a permanent gap between "what the highlighter thinks Go is" and what Go is.
41### The other eight languages
43TOML came first, and it earned its scanner by being unavoidable: the editor reads two TOML files — theme files and a project's `.turbo-go/settings.toml` — and both are meant to be edited in the editor itself. Shipping a settings file full of explanatory comments and then showing it in flat grey would have been an odd thing to do.
45Markdown, JavaScript, HTML and shell followed for a plainer reason: they are what sits beside Go in a Go project. A repository has a `README.md`, some scripts, and often a page or a bit of JavaScript, and an editor that colours only the `.go` files makes you leave it for the rest. YAML, XML and Dockerfiles joined them on the same reasoning: the files a Go project keeps beside its code are now just as likely to be a compose file, a CI workflow or an image build, and a compose file in flat grey is exactly the file you most want the shape of. They are now shared: written once here, and inherited by every editor built on turbo-core.
47Each is a few hundred lines and they share one small piece of machinery — a line, a position in it, and the spans found so far. What they do **not** share is any attempt at a general engine. There is no pattern language, no grammar format, no table of regular expressions: each scanner is ordinary Go that a reader can follow, and adding one means writing one rather than learning a notation.
49They stop short in ways the [reference](../reference/languages.md) states outright, and the stopping points were chosen rather than run out of:
51- **No JavaScript regular expressions.** Telling `/x/g` from a division needs to know whether the previous token could end an expression. A wrong guess colours the rest of the line as a string — a much louder failure than leaving a regex the colour of an operator.
52- **No shell heredocs.** Following `<<EOF` to its delimiter means carrying an arbitrary word across lines, with the `<<-` and quoted spellings on top, for a construct that is usually a few lines of plain text.
53- **No JavaScript inside `<script>`.** It means following an element across lines and mapping another scanner's columns back out, and the same argument would then demand CSS.
54- **No language inside a Markdown fence.** ```` ```go ```` is one colour. Colouring it properly means every scanner has to be reachable from every other, which is the beginning of the general engine this package does not have.
56The unifying rule is that a scanner **guesses nothing**. Where a construct cannot be recognised without knowing more than one line holds, it is left alone rather than approximated, because a highlighter that is wrong is worse than one that is quiet.
58### Five classes that Go has nothing to say about
60The twelve classes the Go tokeniser produces cover the eight other languages almost entirely — a string is a string in all of them. Five things had no home: a Markdown heading, its emphasis and its links, and an HTML tag and attribute.
62Reusing existing classes was the cheaper option and was taken for TOML, where a table header genuinely reads as a type and a key as an identifier. It does not survive the markup languages: a heading is not a keyword and a tag is not one either, and a theme that wanted headings quiet and keywords loud could not say so. So `syntax.heading`, `syntax.tag`, `syntax.attribute`, `syntax.emphasis` and `syntax.link` exist.
64The cost is real and falls on themes written elsewhere: one that sets none of them falls back along the dots to `syntax` and then to `default`, so Markdown stays readable but its headings are not distinct. Every shipped theme sets all five, and a test fails if one stops doing so.
66## Completion: someone else's program
68Completion works the other way round. Turbo Go knows nothing about Go's type system and does not try to: it asks `gopls`, over the Language Server Protocol, and draws the answer.
70### Optional, and not by accident
72The editor is fully usable with no language server. Not degraded — the buffer, the colouring, the themes, the windows, the search, all work exactly the same. Only completion, hover and go-to-definition are missing, and the status bar says so with the single command that fixes it.
74This is enforced by construction rather than by discipline. `Language` wraps the entire conversation, and with no server every method does nothing at all. There is no `if server != nil` anywhere else in the editor, because there is nothing to check.
76### The trap in the protocol
78The protocol counts columns in **UTF-16 code units**. The editor counts them in runes. On ASCII the two are the same number, which is exactly why getting this wrong survives testing — until someone opens a file with an accent in a comment, and every completion after it lands one column off.
80So every position crossing that boundary is converted, and the conversion is tested with a musical clef, which needs a surrogate pair and therefore counts as two.
82### The other trap
84`gopls` asks its client questions. During start-up it requests `workspace/configuration` — and **waits for the reply**. A client that only sends requests and only reads responses never finishes initialising, and hangs with no error at all.
86So the connection routes server-to-client requests to a handler, and the client answers with empty settings, which means "use your defaults".
88### Bounded, always
90Every request has a deadline: three seconds for a completion, thirty for the handshake, because a cold `gopls` has a module graph to load before it can say hello. A server that stops answering slows the editor down and never stops it.
92A completion that arrives after you have typed three more characters is not a completion, it is an interruption — which is why the request is made synchronously and given up on quickly, rather than being delivered late.
94## Nine questions, one connection
96Completion is the loudest thing the language server does and the least revealing. The same connection answers eight more, and they divide into three kinds by what comes back.
98**Something to read.** `hover` — what is this? — drawn in a box.
100**Places in the code.** `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — a Go interface has as many definitions as it has implementations, and for a long time this editor took the first and threw the rest away.
102**Names.** `documentSymbol` for a file's own outline, `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.
104And one thing nobody asks for at all: **`publishDiagnostics` arrives unbidden**, whenever the server has an opinion, for whatever files it has loaded — which are usually more than the one in front of you. That is why Problems lists every file rather than the current one, and why the mark in the gutter appears without anything being pressed.
106The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here, and it would have been inherited by all eight for free.
108## Two features, two shapes
110It is worth noticing why these ended up so different.
112Colouring must be **instant and always right enough**, on text that is usually invalid. That calls for a local, tolerant, cheap answer — and the tokeniser is already in the standard library.
114Completion must be **occasionally right about the whole program**, including its dependencies. That is a compiler's job, it is expensive, and it is already solved by a program that does nothing else.
116The first was worth writing. The second was worth asking for.
118## How it relates to the rest
120- Where these two live in the codebase: [Architecture](architecture.md)
121- The dependency policy that shaped both: [Design decisions](design-decisions.md)
122- Getting completion running: [How to enable Go completion](../how-to/enable-completion.md)

📦 Turbo Go 3d7798b k33g 8h ago	1	# Colouring and completion — explanation
	2
	3	## What is this about?
	4
	5	The two features that make Turbo Go an editor for Go rather than a text editor that happens to be written in it: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.
	6
	7	## Colouring: the compiler's own tokeniser
	8
	9	Turbo Go does not have a syntax definition. It calls `go/scanner` — the lexer the Go toolchain itself uses — and turns the tokens it gets back into coloured spans.
	10
	11	This means keywords, literals and operators are recognised exactly as the compiler recognises them. Raw string literals, automatic semicolon insertion, `0x_FF` separators, the lot. There is no regular expression to get subtly wrong, and no table to update when the language changes.
	12
	13	### Tolerance is the whole point
	14
	15	Source under a cursor is syntactically broken most of the time it is being typed. Half a string, an unclosed brace, an identifier that stops mid-word. A highlighter that gives up on invalid input is a highlighter that flickers off exactly when you are looking at it.
	16
	17	So the scanner runs in its most forgiving mode and every syntax error is discarded. It still returns usable tokens: an unterminated string comes back as a string running to the end of the line, an unclosed `/*` as a comment running to the end of the file. Which is precisely the behaviour you want — the colours stay steady, and they tell you what is wrong.
	18
	19	### What the editor adds
	20
	21	Three distinctions the scanner does not make, because they are about reading rather than parsing:
	22
	23	- an identifier before `(`, or after `func`, is a function
	24	- an identifier after `type`, `struct` or `interface` is a type
	25	- brackets, commas, dots and semicolons are punctuation, split out from the operators that compute, so a theme can quiet them down
	26
	27	Predeclared names — `int`, `error`, `nil`, `len`, `min` — are recognised by name, not by keyword, because they are not keywords: a file may shadow them, and colouring the shadowed one anyway is what every other Go editor does too.
	28
	29	### Why it is fast enough to do naively
	30
	31	Scanning the whole file on every keystroke sounds wasteful, and would be. It does not happen: the buffer keeps a revision counter, bumped on every change, and the highlighter re-scans only when the number has moved. The editor redraws far more often than the text changes — every cursor move, every scroll — and all of those redraws are free.
	32
	33	### The cost of this choice
	34
	35	A language is coloured only if someone wrote a scanner for it. There is no definition language to write a definition in, so each one is Go code.
	36
	37	Eight of them — TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell — live in turbo-core, because every editor built on it meets them whatever language it is for. The Go scanner lives here, in `internal/golang`, and is registered at start-up with `syntax.Register`. That is why a `.rs` file opens as plain text in Turbo Go: this editor registers Go and nothing else.
	38
	39	That is a real limitation, and it was accepted deliberately: this is an editor for Go. A generic highlighter would have brought a dependency, a definition format, and a permanent gap between "what the highlighter thinks Go is" and what Go is.
	40
	41	### The other eight languages
	42
	43	TOML came first, and it earned its scanner by being unavoidable: the editor reads two TOML files — theme files and a project's `.turbo-go/settings.toml` — and both are meant to be edited in the editor itself. Shipping a settings file full of explanatory comments and then showing it in flat grey would have been an odd thing to do.
	44
	45	Markdown, JavaScript, HTML and shell followed for a plainer reason: they are what sits beside Go in a Go project. A repository has a `README.md`, some scripts, and often a page or a bit of JavaScript, and an editor that colours only the `.go` files makes you leave it for the rest. YAML, XML and Dockerfiles joined them on the same reasoning: the files a Go project keeps beside its code are now just as likely to be a compose file, a CI workflow or an image build, and a compose file in flat grey is exactly the file you most want the shape of. They are now shared: written once here, and inherited by every editor built on turbo-core.
	46
	47	Each is a few hundred lines and they share one small piece of machinery — a line, a position in it, and the spans found so far. What they do not share is any attempt at a general engine. There is no pattern language, no grammar format, no table of regular expressions: each scanner is ordinary Go that a reader can follow, and adding one means writing one rather than learning a notation.
	48
	49	They stop short in ways the [reference](../reference/languages.md) states outright, and the stopping points were chosen rather than run out of:
	50
	51	- No JavaScript regular expressions. Telling `/x/g` from a division needs to know whether the previous token could end an expression. A wrong guess colours the rest of the line as a string — a much louder failure than leaving a regex the colour of an operator.
	52	- No shell heredocs. Following `<<EOF` to its delimiter means carrying an arbitrary word across lines, with the `<<-` and quoted spellings on top, for a construct that is usually a few lines of plain text.
	53	- No JavaScript inside `<script>`. It means following an element across lines and mapping another scanner's columns back out, and the same argument would then demand CSS.
	54	- No language inside a Markdown fence. ```` ```go ```` is one colour. Colouring it properly means every scanner has to be reachable from every other, which is the beginning of the general engine this package does not have.
	55
	56	The unifying rule is that a scanner guesses nothing. Where a construct cannot be recognised without knowing more than one line holds, it is left alone rather than approximated, because a highlighter that is wrong is worse than one that is quiet.
	57
	58	### Five classes that Go has nothing to say about
	59
	60	The twelve classes the Go tokeniser produces cover the eight other languages almost entirely — a string is a string in all of them. Five things had no home: a Markdown heading, its emphasis and its links, and an HTML tag and attribute.
	61
	62	Reusing existing classes was the cheaper option and was taken for TOML, where a table header genuinely reads as a type and a key as an identifier. It does not survive the markup languages: a heading is not a keyword and a tag is not one either, and a theme that wanted headings quiet and keywords loud could not say so. So `syntax.heading`, `syntax.tag`, `syntax.attribute`, `syntax.emphasis` and `syntax.link` exist.
	63
	64	The cost is real and falls on themes written elsewhere: one that sets none of them falls back along the dots to `syntax` and then to `default`, so Markdown stays readable but its headings are not distinct. Every shipped theme sets all five, and a test fails if one stops doing so.
65
66	## Completion: someone else's program
67
68	Completion works the other way round. Turbo Go knows nothing about Go's type system and does not try to: it asks `gopls`, over the Language Server Protocol, and draws the answer.
69
70	### Optional, and not by accident
71
72	The editor is fully usable with no language server. Not degraded — the buffer, the colouring, the themes, the windows, the search, all work exactly the same. Only completion, hover and go-to-definition are missing, and the status bar says so with the single command that fixes it.
73
74	This is enforced by construction rather than by discipline. `Language` wraps the entire conversation, and with no server every method does nothing at all. There is no `if server != nil` anywhere else in the editor, because there is nothing to check.
75
76	### The trap in the protocol
77
78	The protocol counts columns in UTF-16 code units. The editor counts them in runes. On ASCII the two are the same number, which is exactly why getting this wrong survives testing — until someone opens a file with an accent in a comment, and every completion after it lands one column off.
79
80	So every position crossing that boundary is converted, and the conversion is tested with a musical clef, which needs a surrogate pair and therefore counts as two.
81
82	### The other trap
83
84	`gopls` asks its client questions. During start-up it requests `workspace/configuration` — and waits for the reply. A client that only sends requests and only reads responses never finishes initialising, and hangs with no error at all.
85
86	So the connection routes server-to-client requests to a handler, and the client answers with empty settings, which means "use your defaults".
87
88	### Bounded, always
89
90	Every request has a deadline: three seconds for a completion, thirty for the handshake, because a cold `gopls` has a module graph to load before it can say hello. A server that stops answering slows the editor down and never stops it.
91
92	A completion that arrives after you have typed three more characters is not a completion, it is an interruption — which is why the request is made synchronously and given up on quickly, rather than being delivered late.
93
94	## Nine questions, one connection
95
96	Completion is the loudest thing the language server does and the least revealing. The same connection answers eight more, and they divide into three kinds by what comes back.
97
98	Something to read. `hover` — what is this? — drawn in a box.
99
100	Places in the code. `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — a Go interface has as many definitions as it has implementations, and for a long time this editor took the first and threw the rest away.
101
102	Names. `documentSymbol` for a file's own outline, `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.
103
104	And one thing nobody asks for at all: `publishDiagnostics` arrives unbidden, whenever the server has an opinion, for whatever files it has loaded — which are usually more than the one in front of you. That is why Problems lists every file rather than the current one, and why the mark in the gutter appears without anything being pressed.
105
106	The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here, and it would have been inherited by all eight for free.
107
108	## Two features, two shapes
109
110	It is worth noticing why these ended up so different.
111
112	Colouring must be instant and always right enough, on text that is usually invalid. That calls for a local, tolerant, cheap answer — and the tokeniser is already in the standard library.
113
114	Completion must be occasionally right about the whole program, including its dependencies. That is a compiler's job, it is expensive, and it is already solved by a program that does nothing else.
115
116	The first was worth writing. The second was worth asking for.
117
118	## How it relates to the rest
119
120	- Where these two live in the codebase: [Architecture](architecture.md)
121	- The dependency policy that shaped both: [Design decisions](design-decisions.md)
122	- Getting completion running: [How to enable Go completion](../how-to/enable-completion.md)