| 📦 Turbo JS 91999d1 k33g 12h ago | 1 | # Colouring and completion — explanation |
| 2 | |
| 3 | ## What is this about? |
| 4 | |
| 5 | The two features that make Turbo JS an editor *for JavaScript* rather than a text editor that happens to open `.js` files: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive. |
| 6 | |
| 7 | ## Colouring is ours; completion is not |
| 8 | |
| 9 | Colouring is done here, in about six hundred lines of hand-written Go. Completion is done by `typescript-language-server` — the same engine behind VS Code's JavaScript support — and Turbo JS only asks and draws. |
| 10 | |
| 11 | That split is not an accident of effort. Colouring has to be **instant and tolerant**: it runs on every keystroke, on text that is invalid most of the time it is being typed, and a highlighter that stops to think or gives up on broken input is worse than no highlighter. Completion has to be **correct**, which for JavaScript means parsing the file, following its `import` and `require` lines into `node_modules`, inferring types through a language that declares none, and knowing what every method of every global takes — and nothing that has to be instant can also be that. |
| 12 | |
| 13 | So the editor draws colours it computed itself, and shows completions somebody else computed. |
| 14 | |
| 15 | ## Why JavaScript is scanned here when the library already scans it |
| 16 | |
| 17 | turbo-core colours JavaScript for every editor in the family, because every editor meets it — in a README's code fence, in a web page's `<script>`, in the front end of a project written in something else. Its scanner is written for that use: keywords, strings, template literals, comments, numbers, the common globals, and a deliberate refusal to recognise regular expressions. |
| 18 | |
| 19 | The refusal is reasoned. Telling `/x/g` from `a / b / c` needs to know whether the token before the slash could end an expression, and a wrong guess colours the rest of a line as a string — a worse failure than leaving a regular expression the colour of an operator, for a language you only pass through. |
| 20 | |
| 21 | An editor *for* JavaScript makes the opposite trade, and can afford to. Regular expressions are on every other line of a Node program — routing, parsing, validation — and a reader wants them told from strings. And the guess can be **bounded**: a regular-expression literal may not contain a newline, so a slash with no closing slash on its line cannot open one, whatever came before it. The rule here is the one every editor uses — a slash after a value divides, a slash after an operator, a bracket, a keyword or at the start of a line opens a literal — with that bound added, so a wrong guess costs at most one line and never the rest of the file. |
| 22 | |
| 23 | So the editor registers a scanner of its own under the library's name, `javascript`, and the library's is replaced. That is a door the library built on purpose: `syntax.Register` says the later registration wins because it is the more specific statement. Nothing in turbo-core changed, and a snippets file saying `languages = ["javascript"]` or a ```js fence in an agent window reaches this scanner without knowing which one it is. |
| 24 | |
| 25 | What the replacement adds, besides regular expressions: the hashbang line, Node's globals, the name after `function` and `class`, private `#names`, `@decorators`, `...` and `?.` as single spans, Unicode identifiers, and a leading capital read as a class. |
| 26 | |
| 27 | ## What crosses a line break, and why so little does |
| 28 | |
| 29 | Two constructs may run from one line to the next, and each is carried as a flag rather than a depth because neither nests: |
| 30 | |
| 31 | - **A block comment** runs from `/*` to the first `*/`, wherever that is. JavaScript does not nest them — `/* a /* b */ c */` ends after `b`, and `c */` is code — so a depth would be a claim about a different language. |
| 32 | - **A template literal** runs from a backtick to the next unescaped backtick, interpolations included. It is the one string in the language that may hold a real newline. |
| 33 | |
| 34 | An ordinary string does not cross a line. `'…'` and `"…"` end at their quote or at the end of their line, because the language says a newline inside one is an error, so an unterminated string colours the rest of its line and no more — and the next line is code again. A regular expression is the same. That is the opposite of what Turbo Golo's scanner does with Golo's strings, and for the opposite reason: Golo's lexer reads to the closing quote wherever it is, JavaScript's refuses to. |
| 35 | |
| 36 | ## Where the scanner leans on the language, and where on convention |
| 37 | |
| 38 | **Keywords, constants and globals are tables.** The keywords are the language's reserved words plus the contextual ones a reader meets as keywords — `async`, `await`, `of`, `get`, `set`, `static`, `from`, `as`. The globals are the standard library's constructors and namespaces, and — because this is a Node editor — Node's own: `process`, `Buffer`, `require`, `module`, `exports`, `__dirname`, `__filename`, `setImmediate`, and the web platform pieces Node ships, `fetch`, `URL`, `TextEncoder`, `AbortController`, `performance`, `crypto`. They are recognised by name, as Go's predeclared identifiers are: a file that shadows `Map` still has it coloured as a global, which is what every editor does. |
| 39 | |
| 40 | **A word after a dot is a property, whatever it is spelt like.** `map.get(k)` is a call, `options.default` is a field, `promise.catch(…)` is a method, and none of them is the keyword it would be on its own. The scanner remembers the dot — and `?.` — and reads the next word as a name. This is the rule that separates a JavaScript scanner from a keyword list: a Map has a method called `get` and `set`, a Promise one called `catch` and `finally`, an object a property called `default`, and colouring them as keywords would make every line of ordinary code look reserved. |
| 41 | |
| 42 | **A contextual keyword being called is a function.** `get name() {}` in a class body has `get` as a keyword; `get(key)` with nothing in front of it is a function called `get`. The word alone cannot say; the parenthesis after it can. `async` is left out of that rule on purpose — `async (x) => x` is an arrow function, and `async` there is not a call. |
| 43 | |
| 44 | **A capitalised name is a class, and here that is a convention rather than a rule.** JavaScript has no case rule: `const Count = 1` is legal. But classes and constructors are capitalised by everybody — `Greeter`, `EventEmitter`, `MyError` — and nothing else customarily is, so the scanner colours by the convention, as Turbo Rust and Turbo Golo do for their languages. It holds even in front of a parenthesis, because `new Greeter()` is a constructor and not a call. What it costs is that a `SCREAMING_SNAKE_CASE` constant is coloured as a class too: nothing in the spelling separates the two conventions, and the reference says so rather than leaving somebody to find out. |
| 45 | |
| 46 | **The name after `function` is a function, and the name after `class` is a class, by position.** Everywhere else a name is a function because a parenthesis follows it, and a declaration is where that is already true — `function parse(input)` has its parenthesis — but a generator's `*` and a class's name do not, so both are read by what came before them rather than after. |
| 47 | |
| 48 | **A regular expression is coloured as a character literal.** The class set is closed — seventeen classes, so that one theme colours every language an editor will ever learn — and JavaScript has no character literal, so `syntax.char` is free. A regular expression is the other kind of quoted thing in the language, and worth telling from a string by colour; in `turbo-classic` the two happen to share a green, and other themes separate them. |
| 49 | |
| 50 | **Names may be almost anything.** JavaScript identifiers are Unicode: `café` and `名前` are names, and so are `$` and `_`, which jQuery and lodash made ordinary. The scanner uses Go's `unicode.IsLetter` rather than turbo-core's ASCII predicate, so they are coloured. |
| 51 | |
| 52 | ## What the scanner refuses to guess |
| 53 | |
| 54 | Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet: |
| 55 | |
| 56 | | Not recognised | Because | |
| 57 | | --- | --- | |
| 58 | | The code inside `${…}` | Colouring it means the scanner reaching back into itself with a nesting depth to carry — an interpolation may hold a template that holds an interpolation — for a construct that is usually one short expression. The whole literal is a string, and a backtick inside an interpolation ends it early | |
| 59 | | JSX | `<div className="x">` is markup inside an expression, and a JSX scanner is an HTML scanner that hands back to a JavaScript scanner at every brace. It is a different language with its own extension, and it is not this one | |
| 60 | | TypeScript | The language server serves `.ts` files without being asked, and the scanner does not colour them. TypeScript's keywords, its type annotations and its generics are a scanner of their own, and an editor that coloured half of it would be wrong in exactly the lines that make a file TypeScript rather than JavaScript | |
| 61 | | Whether `MAX_SIZE` is a constant | The leading capital says class, and there is no second rule that would not mis-colour a class whose name is an acronym | |
| 62 | | Whether a name is bound in this scope | Nothing here reads more than one line at a time; that is the language server's question, and [F1 answers it](../how-to/ask-about-code.md) | |
| 63 | |
| 64 | ## JSON, and why it is here |
| 65 | |
| 66 | `package.json` is the first file in every Node project and the one this editor uses to find the project's root, and turbo-core does not colour JSON. Thirty lines in `json.go` do: a string followed by a colon is a **key** and is coloured as an attribute, any other string is a **value**, and the two sides of the colon read apart — which is the whole of what anyone wants from a coloured manifest. Comments are tolerated, because `tsconfig.json` and every editor's own settings file have them, and a scanner that painted them as broken would be wrong in exactly the files most likely to hold one. |
| 67 | |
| 68 | It is registered under its own name, `json`, so a ```json fence in an agent window is coloured too. |
| 69 | |
| 70 | ## The other seven languages come free |
| 71 | |
| 72 | TOML, YAML, Markdown, HTML, XML, Dockerfiles and shell are coloured by turbo-core, not here. A Node project has a `README.md`, a `compose.yaml` for the database it talks to, a `Dockerfile` to ship as, a `.github/workflows/ci.yml`, and an editor that coloured only the `.js` files would make you leave it for the rest. |
| 73 | |
| 74 | That they are shared rather than copied is the point of the library: they were written once, for Turbo Go, and Turbo JS got them by importing a package. JavaScript is the eighth, and this editor took it over rather than inheriting it. |
| 75 | |
| 76 | ## Completion, and why it can fail silently |
| 77 | |
| 78 | Turbo JS knows nothing about JavaScript's semantics and does not try to. It asks `typescript-language-server` over the Language Server Protocol and draws the answer. |
| 79 | |
| 80 | Four things about that are worth knowing, because all four look like "completion is broken": |
| 81 | |
| 82 | **The server needs TypeScript 6 beside it, not 7.** `typescript-language-server` is a thin layer over the `typescript` package's `tsserver.js`, and it does not depend on that package: you install both. TypeScript 7 is the native port, and it ships a compiler with a language server of its own inside it and no `tsserver.js` at all — so a server installed beside a current `typescript` starts and then refuses, with *Could not find a valid TypeScript installation*, before the editor has asked it anything. The install hint pins `typescript@6` for exactly this reason, and the status bar shows the hint when the server is missing. |
| 83 | |
| 84 | **Why not TypeScript 7's own server, then.** `tsc --lsp --stdio` was measured against the same nine end-to-end tests this repository runs against `typescript-language-server`. It answers all eight requests, four times faster, and it publishes **no** diagnostics — it offers them pull-style, through `textDocument/diagnostic`, a request turbo-core does not make. An editor whose gutter is blank because the server is waiting to be asked looks exactly like one with nothing to report, so the mature pair is the one named here. The day the library learns to pull diagnostics, one `npm install -g typescript` is the better answer. |
| 85 | |
| 86 | **Diagnostics for plain JavaScript are syntax errors.** `tsserver` checks the types of a `.js` file only when asked — with `// @ts-check` at the top of the file, or `checkJs` in a `jsconfig.json` — so a `.js` file with a type mistake gets no mark, and a file with a missing bracket does. That is the server's design, and it is written down rather than worked around. |
| 87 | |
| 88 | **The project is where `package.json` is.** The server is started in the nearest directory at or above the file you opened that holds a `package.json`, because that is where `node_modules` is and where `import` lines resolve from. Open a file from another project in the same session and the server answers about it from the first project's point of view. |
| 89 | |
| 90 | The editor's answer to the first is [Run ▸ Language server status](../reference/menus.md), which says what it found, where it started it and whether it is ready — because "nothing happened" is not something a user can act on. |
| 91 | |
| 92 | ## Nine questions, one connection |
| 93 | |
| 94 | Completion is the loudest thing the language server does and the least revealing. The same connection asks eight more questions, and they divide into three kinds by what comes back. |
| 95 | |
| 96 | **Something to read.** `hover` — what is this? — drawn in a box. For a function you declared, that is its signature and the JSDoc comment written immediately above it; for a method of a global, its signature and the standard library's documentation of it. |
| 97 | |
| 98 | **Places in the code.** `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — the references to a function declared once and called twice are three places, and the implementations of a class with two subclasses are two. |
| 99 | |
| 100 | **Names.** `documentSymbol` for a file's own outline — its functions and classes, methods nested under their class — and `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn. |
| 101 | |
| 102 | And one thing nobody asks for at all: **`publishDiagnostics` arrives unbidden**, whenever the server has an opinion, on open and on every edit. That is why the mark in the gutter appears without anything being pressed. |
| 103 | |
| 104 | `typescript-language-server` answers all nine for plain JavaScript, and a test in this repository drives each of them against the real server, so this paragraph cannot go stale quietly. The editor asks for none of them until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here. |
| 105 | |
| 106 | ## How it relates to the rest |
| 107 | |
| 108 | - Exactly what is recognised: [Languages coloured](../reference/languages.md) |
| 109 | - Getting completion working: [How to enable completion](../how-to/enable-completion.md) |
| 110 | - Where the scanner lives and why: [Architecture](architecture.md) |