turbo-editors/turbo-rust · colouring-and-completion.md

⑂ v1.0.2

colouring-and-completion.md · 95 lines · 8.3 KBmarkdown    Blame HistoryRaw
📦 Turbo Rust   1# Colouring and completion — explanation
3## What is this about?
5The two features that make Turbo Rust an editor *for Rust* rather than a text editor that happens to open `.rs` files: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.
7## Colouring is ours; completion is not
9Colouring is done here, in about six hundred lines of hand-written Go. Completion is done by rust-analyzer, and Turbo Rust only asks and draws.
11That split is not an accident of effort. Colouring has to be **instant and tolerant**: it runs on every keystroke, on text that is invalid most of the time it is being typed, and a highlighter that stops to think or gives up on broken input is worse than no highlighter. Completion has to be **correct**, which for Rust means knowing the trait system, the crate graph and every dependency's public API — and nothing that has to be instant can also be that.
13So the editor draws colours it computed itself, and shows completions somebody else computed.
15## Why Rust is scanned by hand
17Go has a lexer in its standard library, and Turbo Go uses it: `go/scanner` is the same code the compiler uses, so the editor and the compiler agree about what a token is, with nothing to keep in step.
19Rust has no such thing available here. `rustc` is not a Go library, and rust-analyzer's own parser is a Rust crate. The choices were a hand-written scanner, or shelling out to something for every keystroke.
21The scanner it is. Six hundred lines, one file each for the dispatcher, the literals and the words — and no attempt at a general engine. There is no pattern language, no grammar format and no table of regular expressions: it is ordinary Go that a reader can follow, which is the same rule the eight scanners in turbo-core follow.
23## The three things that cross a line break
25Almost everything in Rust can be decided from the line in front of you. Three things cannot, and each is carried explicitly rather than approximated:
27**Block comments, with their depth.** Rust nests them: `/* a /* b */ c */` is one comment. A boolean "in a comment" flag closes it at the first `*/` and colours `c */` as code — which is not a subtle failure, it is half a screen of the wrong colour. So the state is an integer.
29**Raw strings, with their hash count.** `r#"a "quoted" thing"#` ends at a quote followed by *exactly* the number of hashes it opened with, and has no escapes at all. Carrying a boolean would end it at the inner quote.
31**Ordinary strings.** Rust allows a real newline inside `"…"`, so a string running past the end of a line is not the error state it would be in most languages.
33Everything else — attributes included — is decided within one line. An attribute that does not close is coloured to the end of its line and not carried, because an unclosed `#[` is nearly always a half-typed one, and carrying it would paint the rest of the file.
35## The one genuine ambiguity
37`'` opens a character literal and a lifetime, and Rust settles it by what follows: `'a'` is a character, `'a` is a lifetime.
39The rule here is to look for the closing quote where a character literal would have to put it — one rune along, or further for an escape — and to read a lifetime when it is not there. That gets `'static`, `'\n'`, `'a'`, `'\u{1F600}'` and `'a` all right, from the line alone.
41Getting it wrong is expensive: read `'a` as an unterminated character and the rest of the line becomes a string. `fn longest<'a>(x: &'a str) -> &'a str` has three of them, and it has a test of its own.
43## Where the scanner leans on convention
45**A leading capital means a type.** Rust's naming convention is strong enough to use: a type, a trait and an enum variant are all `UpperCamelCase`, and nothing else is. That is a heuristic, not a rule, and it is visibly one in a single place — a `SCREAMING_SNAKE_CASE` constant is coloured as a type.
47That could be fixed with a second rule ("all capitals and underscores means a constant"), and it was not: the rule would then mis-colour a type whose name is an acronym, and trading one wrong answer for another is not progress. The reference [says so plainly](../reference/languages.md) rather than leaving somebody to discover it.
49## What the scanner refuses to guess
51Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet:
53| Not recognised | Because |
54| --- | --- |
55| Which macro is being invoked | `println!` and a macro you wrote are both builtins; telling them apart needs the crate's expansion |
56| The inside of a `macro_rules!` body | Coloured as ordinary Rust, which is usually right and sometimes not |
57| The Markdown inside a `///` comment | A doc comment is one comment; colouring two languages at once is the general engine this does not have |
59## The other eight languages come free
61TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell are coloured by turbo-core, not here. A Rust project has a `Cargo.toml`, a `README.md`, some scripts, a CI workflow in YAML and often a Dockerfile, and an editor that coloured only the `.rs` files would make you leave it for the rest.
63That they are shared rather than copied is the point of the library: they were written once, for Turbo Go, and Turbo Rust got them by importing a package.
65## Completion, and why it can fail silently
67Turbo Rust knows nothing about Rust's type system and does not try to. It asks rust-analyzer over the Language Server Protocol and draws the answer.
69Two things about that are worth knowing, because both look like "completion is broken":
71**rust-analyzer answers nothing until it has loaded the workspace.** It reads `Cargo.toml`, resolves the dependency graph and indexes it, which takes seconds on a small crate and much longer on a large one. It says so with a `$/progress` notification this client does not read, so what you see meanwhile is an empty list.
73**A server given the wrong root loads the wrong code, and then answers nothing at all — with no error.** That is why the editor walks up from the file to the nearest `Cargo.toml` rather than using the working directory, and it is the single most confusing way completion can fail.
75The editor's answer to both is [Run ▸ Language server status](../reference/menus.md), which says what it found, where it started it and whether it is ready — because "nothing happened" is not something a user can act on.
77## Nine questions, one connection
79Completion is the loudest thing the language server does and the least revealing. The same connection answers eight more, and they divide into three kinds by what comes back.
81**Something to read.** `hover` — what is this? — drawn in a box.
83**Places in the code.** `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — a trait has as many implementations as somebody cared to write, and for a long time this editor took the first and threw the rest away.
85**Names.** `documentSymbol` for a file's own outline, `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.
87And one thing nobody asks for at all: **`publishDiagnostics` arrives unbidden**, whenever the server has an opinion, for whatever files it has loaded — which are usually more than the one in front of you. That is why Problems lists every file rather than the current one, and why the mark in the gutter appears without anything being pressed.
89The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here, and it would have been inherited by all eight for free.
91## How it relates to the rest
93- Exactly what is recognised: [Languages coloured](../reference/languages.md)
94- Getting completion working: [How to enable Rust completion](../how-to/enable-completion.md)
95- Where the scanner lives and why: [Architecture](architecture.md)

📦 Turbo Rust 713ea5c k33g 10h ago	1	# Colouring and completion — explanation
	2
	3	## What is this about?
	4
	5	The two features that make Turbo Rust an editor for Rust rather than a text editor that happens to open `.rs` files: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.
	6
	7	## Colouring is ours; completion is not
	8
	9	Colouring is done here, in about six hundred lines of hand-written Go. Completion is done by rust-analyzer, and Turbo Rust only asks and draws.
	10
	11	That split is not an accident of effort. Colouring has to be instant and tolerant: it runs on every keystroke, on text that is invalid most of the time it is being typed, and a highlighter that stops to think or gives up on broken input is worse than no highlighter. Completion has to be correct, which for Rust means knowing the trait system, the crate graph and every dependency's public API — and nothing that has to be instant can also be that.
	12
	13	So the editor draws colours it computed itself, and shows completions somebody else computed.
	14
	15	## Why Rust is scanned by hand
	16
	17	Go has a lexer in its standard library, and Turbo Go uses it: `go/scanner` is the same code the compiler uses, so the editor and the compiler agree about what a token is, with nothing to keep in step.
	18
	19	Rust has no such thing available here. `rustc` is not a Go library, and rust-analyzer's own parser is a Rust crate. The choices were a hand-written scanner, or shelling out to something for every keystroke.
	20
	21	The scanner it is. Six hundred lines, one file each for the dispatcher, the literals and the words — and no attempt at a general engine. There is no pattern language, no grammar format and no table of regular expressions: it is ordinary Go that a reader can follow, which is the same rule the eight scanners in turbo-core follow.
	22
	23	## The three things that cross a line break
	24
	25	Almost everything in Rust can be decided from the line in front of you. Three things cannot, and each is carried explicitly rather than approximated:
	26
	27	Block comments, with their depth. Rust nests them: `/* a /* b / c /` is one comment. A boolean "in a comment" flag closes it at the first `/` and colours `c /` as code — which is not a subtle failure, it is half a screen of the wrong colour. So the state is an integer.
	28
	29	Raw strings, with their hash count. `r#"a "quoted" thing"#` ends at a quote followed by exactly the number of hashes it opened with, and has no escapes at all. Carrying a boolean would end it at the inner quote.
	30
	31	Ordinary strings. Rust allows a real newline inside `"…"`, so a string running past the end of a line is not the error state it would be in most languages.
	32
	33	Everything else — attributes included — is decided within one line. An attribute that does not close is coloured to the end of its line and not carried, because an unclosed `#[` is nearly always a half-typed one, and carrying it would paint the rest of the file.
	34
	35	## The one genuine ambiguity
	36
	37	`'` opens a character literal and a lifetime, and Rust settles it by what follows: `'a'` is a character, `'a` is a lifetime.
	38
	39	The rule here is to look for the closing quote where a character literal would have to put it — one rune along, or further for an escape — and to read a lifetime when it is not there. That gets `'static`, `'\n'`, `'a'`, `'\u{1F600}'` and `'a` all right, from the line alone.
	40
	41	Getting it wrong is expensive: read `'a` as an unterminated character and the rest of the line becomes a string. `fn longest<'a>(x: &'a str) -> &'a str` has three of them, and it has a test of its own.
	42
	43	## Where the scanner leans on convention
	44
	45	A leading capital means a type. Rust's naming convention is strong enough to use: a type, a trait and an enum variant are all `UpperCamelCase`, and nothing else is. That is a heuristic, not a rule, and it is visibly one in a single place — a `SCREAMING_SNAKE_CASE` constant is coloured as a type.
	46
	47	That could be fixed with a second rule ("all capitals and underscores means a constant"), and it was not: the rule would then mis-colour a type whose name is an acronym, and trading one wrong answer for another is not progress. The reference [says so plainly](../reference/languages.md) rather than leaving somebody to discover it.
	48
	49	## What the scanner refuses to guess
	50
	51	Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet:
	52
	53	\| Not recognised \| Because \|
	54	\| --- \| --- \|
	55	\| Which macro is being invoked \| `println!` and a macro you wrote are both builtins; telling them apart needs the crate's expansion \|
	56	\| The inside of a `macro_rules!` body \| Coloured as ordinary Rust, which is usually right and sometimes not \|
	57	\| The Markdown inside a `///` comment \| A doc comment is one comment; colouring two languages at once is the general engine this does not have \|
	58
	59	## The other eight languages come free
	60
	61	TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell are coloured by turbo-core, not here. A Rust project has a `Cargo.toml`, a `README.md`, some scripts, a CI workflow in YAML and often a Dockerfile, and an editor that coloured only the `.rs` files would make you leave it for the rest.
	62
	63	That they are shared rather than copied is the point of the library: they were written once, for Turbo Go, and Turbo Rust got them by importing a package.
	64
65	## Completion, and why it can fail silently
66
67	Turbo Rust knows nothing about Rust's type system and does not try to. It asks rust-analyzer over the Language Server Protocol and draws the answer.
68
69	Two things about that are worth knowing, because both look like "completion is broken":
70
71	rust-analyzer answers nothing until it has loaded the workspace. It reads `Cargo.toml`, resolves the dependency graph and indexes it, which takes seconds on a small crate and much longer on a large one. It says so with a `$/progress` notification this client does not read, so what you see meanwhile is an empty list.
72
73	A server given the wrong root loads the wrong code, and then answers nothing at all — with no error. That is why the editor walks up from the file to the nearest `Cargo.toml` rather than using the working directory, and it is the single most confusing way completion can fail.
74
75	The editor's answer to both is [Run ▸ Language server status](../reference/menus.md), which says what it found, where it started it and whether it is ready — because "nothing happened" is not something a user can act on.
76
77	## Nine questions, one connection
78
79	Completion is the loudest thing the language server does and the least revealing. The same connection answers eight more, and they divide into three kinds by what comes back.
80
81	Something to read. `hover` — what is this? — drawn in a box.
82
83	Places in the code. `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — a trait has as many implementations as somebody cared to write, and for a long time this editor took the first and threw the rest away.
84
85	Names. `documentSymbol` for a file's own outline, `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.
86
87	And one thing nobody asks for at all: `publishDiagnostics` arrives unbidden, whenever the server has an opinion, for whatever files it has loaded — which are usually more than the one in front of you. That is why Problems lists every file rather than the current one, and why the mark in the gutter appears without anything being pressed.
88
89	The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here, and it would have been inherited by all eight for free.
90
91	## How it relates to the rest
92
93	- Exactly what is recognised: [Languages coloured](../reference/languages.md)
94	- Getting completion working: [How to enable Rust completion](../how-to/enable-completion.md)
95	- Where the scanner lives and why: [Architecture](architecture.md)