Colouring and completion — explanation
What is this about?
The two features that make Turbo JS an editor for JavaScript rather than a text editor that happens to open .js files: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.
Colouring is ours; completion is not
Colouring is done here, in about six hundred lines of hand-written Go. Completion is done by typescript-language-server — the same engine behind VS Code's JavaScript support — and Turbo JS only asks and draws.
That split is not an accident of effort. Colouring has to be instant and tolerant: it runs on every keystroke, on text that is invalid most of the time it is being typed, and a highlighter that stops to think or gives up on broken input is worse than no highlighter. Completion has to be correct, which for JavaScript means parsing the file, following its import and require lines into node_modules, inferring types through a language that declares none, and knowing what every method of every global takes — and nothing that has to be instant can also be that.
So the editor draws colours it computed itself, and shows completions somebody else computed.
Why JavaScript is scanned here when the library already scans it
turbo-core colours JavaScript for every editor in the family, because every editor meets it — in a README's code fence, in a web page's <script>, in the front end of a project written in something else. Its scanner is written for that use: keywords, strings, template literals, comments, numbers, the common globals, and a deliberate refusal to recognise regular expressions.
The refusal is reasoned. Telling /x/g from a / b / c needs to know whether the token before the slash could end an expression, and a wrong guess colours the rest of a line as a string — a worse failure than leaving a regular expression the colour of an operator, for a language you only pass through.
An editor for JavaScript makes the opposite trade, and can afford to. Regular expressions are on every other line of a Node program — routing, parsing, validation — and a reader wants them told from strings. And the guess can be bounded: a regular-expression literal may not contain a newline, so a slash with no closing slash on its line cannot open one, whatever came before it. The rule here is the one every editor uses — a slash after a value divides, a slash after an operator, a bracket, a keyword or at the start of a line opens a literal — with that bound added, so a wrong guess costs at most one line and never the rest of the file.
So the editor registers a scanner of its own under the library's name, javascript, and the library's is replaced. That is a door the library built on purpose: syntax.Register says the later registration wins because it is the more specific statement. Nothing in turbo-core changed, and a snippets file saying languages = ["javascript"] or a ```js fence in an agent window reaches this scanner without knowing which one it is.
What the replacement adds, besides regular expressions: the hashbang line, Node's globals, the name after function and class, private #names, @decorators, ... and ?. as single spans, Unicode identifiers, and a leading capital read as a class.
What crosses a line break, and why so little does
Two constructs may run from one line to the next, and each is carried as a flag rather than a depth because neither nests:
- A block comment runs from
/*to the first*/, wherever that is. JavaScript does not nest them —/* a /* b */ c */ends afterb, andc */is code — so a depth would be a claim about a different language. - A template literal runs from a backtick to the next unescaped backtick, interpolations included. It is the one string in the language that may hold a real newline.
An ordinary string does not cross a line. '…' and "…" end at their quote or at the end of their line, because the language says a newline inside one is an error, so an unterminated string colours the rest of its line and no more — and the next line is code again. A regular expression is the same. That is the opposite of what Turbo Golo's scanner does with Golo's strings, and for the opposite reason: Golo's lexer reads to the closing quote wherever it is, JavaScript's refuses to.
Where the scanner leans on the language, and where on convention
Keywords, constants and globals are tables. The keywords are the language's reserved words plus the contextual ones a reader meets as keywords — async, await, of, get, set, static, from, as. The globals are the standard library's constructors and namespaces, and — because this is a Node editor — Node's own: process, Buffer, require, module, exports, __dirname, __filename, setImmediate, and the web platform pieces Node ships, fetch, URL, TextEncoder, AbortController, performance, crypto. They are recognised by name, as Go's predeclared identifiers are: a file that shadows Map still has it coloured as a global, which is what every editor does.
A word after a dot is a property, whatever it is spelt like. map.get(k) is a call, options.default is a field, promise.catch(…) is a method, and none of them is the keyword it would be on its own. The scanner remembers the dot — and ?. — and reads the next word as a name. This is the rule that separates a JavaScript scanner from a keyword list: a Map has a method called get and set, a Promise one called catch and finally, an object a property called default, and colouring them as keywords would make every line of ordinary code look reserved.
A contextual keyword being called is a function. get name() {} in a class body has get as a keyword; get(key) with nothing in front of it is a function called get. The word alone cannot say; the parenthesis after it can. async is left out of that rule on purpose — async (x) => x is an arrow function, and async there is not a call.
A capitalised name is a class, and here that is a convention rather than a rule. JavaScript has no case rule: const Count = 1 is legal. But classes and constructors are capitalised by everybody — Greeter, EventEmitter, MyError — and nothing else customarily is, so the scanner colours by the convention, as Turbo Rust and Turbo Golo do for their languages. It holds even in front of a parenthesis, because new Greeter() is a constructor and not a call. What it costs is that a SCREAMING_SNAKE_CASE constant is coloured as a class too: nothing in the spelling separates the two conventions, and the reference says so rather than leaving somebody to find out.
The name after function is a function, and the name after class is a class, by position. Everywhere else a name is a function because a parenthesis follows it, and a declaration is where that is already true — function parse(input) has its parenthesis — but a generator's * and a class's name do not, so both are read by what came before them rather than after.
A regular expression is coloured as a character literal. The class set is closed — seventeen classes, so that one theme colours every language an editor will ever learn — and JavaScript has no character literal, so syntax.char is free. A regular expression is the other kind of quoted thing in the language, and worth telling from a string by colour; in turbo-classic the two happen to share a green, and other themes separate them.
Names may be almost anything. JavaScript identifiers are Unicode: café and 名前 are names, and so are $ and _, which jQuery and lodash made ordinary. The scanner uses Go's unicode.IsLetter rather than turbo-core's ASCII predicate, so they are coloured.
What the scanner refuses to guess
Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet:
| Not recognised | Because |
|---|---|
The code inside ${…} |
Colouring it means the scanner reaching back into itself with a nesting depth to carry — an interpolation may hold a template that holds an interpolation — for a construct that is usually one short expression. The whole literal is a string, and a backtick inside an interpolation ends it early |
| JSX | <div className="x"> is markup inside an expression, and a JSX scanner is an HTML scanner that hands back to a JavaScript scanner at every brace. It is a different language with its own extension, and it is not this one |
| TypeScript | The language server serves .ts files without being asked, and the scanner does not colour them. TypeScript's keywords, its type annotations and its generics are a scanner of their own, and an editor that coloured half of it would be wrong in exactly the lines that make a file TypeScript rather than JavaScript |
Whether MAX_SIZE is a constant |
The leading capital says class, and there is no second rule that would not mis-colour a class whose name is an acronym |
| Whether a name is bound in this scope | Nothing here reads more than one line at a time; that is the language server's question, and F1 answers it |
JSON, and why it is here
package.json is the first file in every Node project and the one this editor uses to find the project's root, and turbo-core does not colour JSON. Thirty lines in json.go do: a string followed by a colon is a key and is coloured as an attribute, any other string is a value, and the two sides of the colon read apart — which is the whole of what anyone wants from a coloured manifest. Comments are tolerated, because tsconfig.json and every editor's own settings file have them, and a scanner that painted them as broken would be wrong in exactly the files most likely to hold one.
It is registered under its own name, json, so a ```json fence in an agent window is coloured too.
The other seven languages come free
TOML, YAML, Markdown, HTML, XML, Dockerfiles and shell are coloured by turbo-core, not here. A Node project has a README.md, a compose.yaml for the database it talks to, a Dockerfile to ship as, a .github/workflows/ci.yml, and an editor that coloured only the .js files would make you leave it for the rest.
That they are shared rather than copied is the point of the library: they were written once, for Turbo Go, and Turbo JS got them by importing a package. JavaScript is the eighth, and this editor took it over rather than inheriting it.
Completion, and why it can fail silently
Turbo JS knows nothing about JavaScript's semantics and does not try to. It asks typescript-language-server over the Language Server Protocol and draws the answer.
Four things about that are worth knowing, because all four look like "completion is broken":
The server needs TypeScript 6 beside it, not 7. typescript-language-server is a thin layer over the typescript package's tsserver.js, and it does not depend on that package: you install both. TypeScript 7 is the native port, and it ships a compiler with a language server of its own inside it and no tsserver.js at all — so a server installed beside a current typescript starts and then refuses, with Could not find a valid TypeScript installation, before the editor has asked it anything. The install hint pins typescript@6 for exactly this reason, and the status bar shows the hint when the server is missing.
Why not TypeScript 7's own server, then. tsc --lsp --stdio was measured against the same nine end-to-end tests this repository runs against typescript-language-server. It answers all eight requests, four times faster, and it publishes no diagnostics — it offers them pull-style, through textDocument/diagnostic, a request turbo-core does not make. An editor whose gutter is blank because the server is waiting to be asked looks exactly like one with nothing to report, so the mature pair is the one named here. The day the library learns to pull diagnostics, one npm install -g typescript is the better answer.
Diagnostics for plain JavaScript are syntax errors. tsserver checks the types of a .js file only when asked — with // @ts-check at the top of the file, or checkJs in a jsconfig.json — so a .js file with a type mistake gets no mark, and a file with a missing bracket does. That is the server's design, and it is written down rather than worked around.
The project is where package.json is. The server is started in the nearest directory at or above the file you opened that holds a package.json, because that is where node_modules is and where import lines resolve from. Open a file from another project in the same session and the server answers about it from the first project's point of view.
The editor's answer to the first is Run ▸ Language server status, which says what it found, where it started it and whether it is ready — because "nothing happened" is not something a user can act on.
Nine questions, one connection
Completion is the loudest thing the language server does and the least revealing. The same connection asks eight more questions, and they divide into three kinds by what comes back.
Something to read. hover — what is this? — drawn in a box. For a function you declared, that is its signature and the JSDoc comment written immediately above it; for a method of a global, its signature and the standard library's documentation of it.
Places in the code. definition, typeDefinition, implementation, references. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — the references to a function declared once and called twice are three places, and the implementations of a class with two subclasses are two.
Names. documentSymbol for a file's own outline — its functions and classes, methods nested under their class — and workspace/symbol for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.
And one thing nobody asks for at all: publishDiagnostics arrives unbidden, whenever the server has an opinion, on open and on every edit. That is why the mark in the gutter appears without anything being pressed.
typescript-language-server answers all nine for plain JavaScript, and a test in this repository drives each of them against the real server, so this paragraph cannot go stale quietly. The editor asks for none of them until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here.
How it relates to the rest
- Exactly what is recognised: Languages coloured
- Getting completion working: How to enable completion
- Where the scanner lives and why: Architecture
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
|