Colouring and completion — explanation

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122

# Colouring and completion — explanation

## What is this about?

The two features that make Turbo Go an editor *for Go* rather than a text editor that happens to be written in it: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.

## Colouring: the compiler's own tokeniser

Turbo Go does not have a syntax definition. It calls `go/scanner` — the lexer the Go toolchain itself uses — and turns the tokens it gets back into coloured spans.

This means keywords, literals and operators are recognised **exactly** as the compiler recognises them. Raw string literals, automatic semicolon insertion, `0x_FF` separators, the lot. There is no regular expression to get subtly wrong, and no table to update when the language changes.

### Tolerance is the whole point

Source under a cursor is syntactically broken most of the time it is being typed. Half a string, an unclosed brace, an identifier that stops mid-word. A highlighter that gives up on invalid input is a highlighter that flickers off exactly when you are looking at it.

So the scanner runs in its most forgiving mode and **every syntax error is discarded**. It still returns usable tokens: an unterminated string comes back as a string running to the end of the line, an unclosed `/*` as a comment running to the end of the file. Which is precisely the behaviour you want — the colours stay steady, and they tell you what is wrong.

### What the editor adds

Three distinctions the scanner does not make, because they are about reading rather than parsing:

- an identifier before `(`, or after `func`, is a **function**
- an identifier after `type`, `struct` or `interface` is a **type**
- brackets, commas, dots and semicolons are **punctuation**, split out from the operators that compute, so a theme can quiet them down

Predeclared names — `int`, `error`, `nil`, `len`, `min` — are recognised by name, not by keyword, because they are not keywords: a file may shadow them, and colouring the shadowed one anyway is what every other Go editor does too.

### Why it is fast enough to do naively

Scanning the whole file on every keystroke sounds wasteful, and would be. It does not happen: the buffer keeps a revision counter, bumped on every change, and the highlighter re-scans only when the number has moved. The editor redraws far more often than the text changes — every cursor move, every scroll — and all of those redraws are free.

### The cost of this choice

**A language is coloured only if someone wrote a scanner for it.** There is no definition language to write a definition in, so each one is Go code.

Eight of them — TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell — live in turbo-core, because every editor built on it meets them whatever language it is for. The Go scanner lives here, in `internal/golang`, and is registered at start-up with `syntax.Register`. That is why a `.rs` file opens as plain text in Turbo Go: this editor registers Go and nothing else.

That is a real limitation, and it was accepted deliberately: this is an editor for Go. A generic highlighter would have brought a dependency, a definition format, and a permanent gap between "what the highlighter thinks Go is" and what Go is.

### The other eight languages

TOML came first, and it earned its scanner by being unavoidable: the editor reads two TOML files — theme files and a project's `.turbo-go/settings.toml` — and both are meant to be edited in the editor itself. Shipping a settings file full of explanatory comments and then showing it in flat grey would have been an odd thing to do.

Markdown, JavaScript, HTML and shell followed for a plainer reason: they are what sits beside Go in a Go project. A repository has a `README.md`, some scripts, and often a page or a bit of JavaScript, and an editor that colours only the `.go` files makes you leave it for the rest. YAML, XML and Dockerfiles joined them on the same reasoning: the files a Go project keeps beside its code are now just as likely to be a compose file, a CI workflow or an image build, and a compose file in flat grey is exactly the file you most want the shape of. They are now shared: written once here, and inherited by every editor built on turbo-core.

Each is a few hundred lines and they share one small piece of machinery — a line, a position in it, and the spans found so far. What they do **not** share is any attempt at a general engine. There is no pattern language, no grammar format, no table of regular expressions: each scanner is ordinary Go that a reader can follow, and adding one means writing one rather than learning a notation.

They stop short in ways the [reference](../reference/languages.md) states outright, and the stopping points were chosen rather than run out of:

- **No JavaScript regular expressions.** Telling `/x/g` from a division needs to know whether the previous token could end an expression. A wrong guess colours the rest of the line as a string — a much louder failure than leaving a regex the colour of an operator.
- **No shell heredocs.** Following `<<EOF` to its delimiter means carrying an arbitrary word across lines, with the `<<-` and quoted spellings on top, for a construct that is usually a few lines of plain text.
- **No JavaScript inside `<script>`.** It means following an element across lines and mapping another scanner's columns back out, and the same argument would then demand CSS.
- **No language inside a Markdown fence.** ```` ```go ```` is one colour. Colouring it properly means every scanner has to be reachable from every other, which is the beginning of the general engine this package does not have.

The unifying rule is that a scanner **guesses nothing**. Where a construct cannot be recognised without knowing more than one line holds, it is left alone rather than approximated, because a highlighter that is wrong is worse than one that is quiet.

### Five classes that Go has nothing to say about

The twelve classes the Go tokeniser produces cover the eight other languages almost entirely — a string is a string in all of them. Five things had no home: a Markdown heading, its emphasis and its links, and an HTML tag and attribute.

Reusing existing classes was the cheaper option and was taken for TOML, where a table header genuinely reads as a type and a key as an identifier. It does not survive the markup languages: a heading is not a keyword and a tag is not one either, and a theme that wanted headings quiet and keywords loud could not say so. So `syntax.heading`, `syntax.tag`, `syntax.attribute`, `syntax.emphasis` and `syntax.link` exist.

The cost is real and falls on themes written elsewhere: one that sets none of them falls back along the dots to `syntax` and then to `default`, so Markdown stays readable but its headings are not distinct. Every shipped theme sets all five, and a test fails if one stops doing so.

## Completion: someone else's program

Completion works the other way round. Turbo Go knows nothing about Go's type system and does not try to: it asks `gopls`, over the Language Server Protocol, and draws the answer.

### Optional, and not by accident

The editor is fully usable with no language server. Not degraded — the buffer, the colouring, the themes, the windows, the search, all work exactly the same. Only completion, hover and go-to-definition are missing, and the status bar says so with the single command that fixes it.

This is enforced by construction rather than by discipline. `Language` wraps the entire conversation, and with no server every method does nothing at all. There is no `if server != nil` anywhere else in the editor, because there is nothing to check.

### The trap in the protocol

The protocol counts columns in **UTF-16 code units**. The editor counts them in runes. On ASCII the two are the same number, which is exactly why getting this wrong survives testing — until someone opens a file with an accent in a comment, and every completion after it lands one column off.

So every position crossing that boundary is converted, and the conversion is tested with a musical clef, which needs a surrogate pair and therefore counts as two.

### The other trap

`gopls` asks its client questions. During start-up it requests `workspace/configuration` — and **waits for the reply**. A client that only sends requests and only reads responses never finishes initialising, and hangs with no error at all.

So the connection routes server-to-client requests to a handler, and the client answers with empty settings, which means "use your defaults".

### Bounded, always

Every request has a deadline: three seconds for a completion, thirty for the handshake, because a cold `gopls` has a module graph to load before it can say hello. A server that stops answering slows the editor down and never stops it.

A completion that arrives after you have typed three more characters is not a completion, it is an interruption — which is why the request is made synchronously and given up on quickly, rather than being delivered late.

## Nine questions, one connection

Completion is the loudest thing the language server does and the least revealing. The same connection answers eight more, and they divide into three kinds by what comes back.

**Something to read.** `hover` — what is this? — drawn in a box.

**Places in the code.** `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — a Go interface has as many definitions as it has implementations, and for a long time this editor took the first and threw the rest away.

**Names.** `documentSymbol` for a file's own outline, `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.

And one thing nobody asks for at all: **`publishDiagnostics` arrives unbidden**, whenever the server has an opinion, for whatever files it has loaded — which are usually more than the one in front of you. That is why Problems lists every file rather than the current one, and why the mark in the gutter appears without anything being pressed.

The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here, and it would have been inherited by all eight for free.

## Two features, two shapes

It is worth noticing why these ended up so different.

Colouring must be **instant and always right enough**, on text that is usually invalid. That calls for a local, tolerant, cheap answer — and the tokeniser is already in the standard library.

Completion must be **occasionally right about the whole program**, including its dependencies. That is a compiler's job, it is expensive, and it is already solved by a program that does nothing else.

The first was worth writing. The second was worth asking for.

## How it relates to the rest

- Where these two live in the codebase: [Architecture](architecture.md)
- The dependency policy that shaped both: [Design decisions](design-decisions.md)
- Getting completion running: [How to enable Go completion](../how-to/enable-completion.md)