| 📦 Turbo Golo d710c1b k33g 19h ago | 1 | # Colouring and completion — explanation |
| 2 | |
| 3 | ## What is this about? |
| 4 | |
| 5 | The two features that make Turbo Golo an editor *for Golo* rather than a text editor that happens to open `.golo` files: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive. |
| 6 | |
| 7 | ## Colouring is ours; completion is not |
| 8 | |
| 9 | Colouring is done here, in about six hundred lines of hand-written Go. Completion is done by `golo lsp` — the interpreter itself, in language-server mode — and Turbo Golo only asks and draws. |
| 10 | |
| 11 | That split is not an accident of effort. Colouring has to be **instant and tolerant**: it runs on every keystroke, on text that is invalid most of the time it is being typed, and a highlighter that stops to think or gives up on broken input is worse than no highlighter. Completion has to be **correct**, which for Golo means parsing the file, following its `import` lines into the modules embedded in the binary, and knowing what every one of 157 builtins takes — and nothing that has to be instant can also be that. |
| 12 | |
| 13 | So the editor draws colours it computed itself, and shows completions somebody else computed. |
| 14 | |
| 15 | ## Why Golo is scanned by hand, with a lexer sitting right there |
| 16 | |
| 17 | GoloScript is written in Go, and its `lexer` package is a tokeniser for exactly this language. Turbo Go goes through `go/scanner` in the same situation — the standard library analysing its own language, so the editor and the compiler agree about what a token is with nothing to keep in step. The obvious move was to import `golo/lexer` and convert its offsets with the library's `LineIndex`. |
| 18 | |
| 19 | It cannot be imported. GoloScript's `go.mod` says `module golo`: a bare name, no host, and Go's module system has no way to fetch a module by such a path. Importing it needs a `replace` directive pointing at a checkout beside this one, and a committed `replace` breaks every clone that has no such checkout — which is why the release script refuses to tag one. Vendoring the two packages was the other route, and it would mean a copy of somebody else's lexer that stops being theirs the day they change it. |
| 20 | |
| 21 | So the lexer is the **specification** rather than a dependency. `lexer/lexer.go` and `token/token.go` say what a token is — which runes open a comment, how a number ends, which words are reserved — and this scanner says the same thing in the library's `LineScanner` style. Where the two could disagree, the lexer's line is quoted in the code beside the decision. |
| 22 | |
| 23 | The scanner it is. Some six hundred lines, one file each for the dispatcher, the literals and the words — and no attempt at a general engine. There is no pattern language, no grammar format and no table of regular expressions: it is ordinary Go that a reader can follow, which is the same rule the eight scanners in turbo-core follow. |
| 24 | |
| 25 | ## What crosses a line break, and why it is carried rather than cut |
| 26 | |
| 27 | Four constructs may run from one line to the next, and the interpreter's lexer is the authority on each: |
| 28 | |
| 29 | - **A block comment** runs from one `----` to the next, wherever that is. Three dashes are two minus signs and a third; a fifth dash is part of the text. |
| 30 | - **A string** runs to its closing quote. The lexer reads it with `for l.ch != '"' && l.ch != 0`, which stops at the quote or the end of the file and at nothing in between — a newline inside a string is part of the string. |
| 31 | - **A `"""` multi-line string** runs to the next three quotes, with no escapes considered on the way. |
| 32 | - **A `'…'` character literal** is read by the same loop as a string, and so behaves the same way. |
| 33 | |
| 34 | Every other editor in this family stops a literal at the end of its line when the closing quote is missing, and Turbo MoonBit makes a point of it: MoonBit's grammar says a newline before the closing quote is an *error*, so there is nothing to carry. Golo's lexer says the opposite, and the scanner follows the lexer: **an unterminated string paints the rest of the file until a quote turns up**, because that is exactly what the interpreter will read as string. The colour is not a warning, it is a statement about what the program means — and a screenful of green after a stray quote is that statement made visible. |
| 35 | |
| 36 | What is carried is a value saying *which* of the four is open. None of them nests, so a depth would be a claim the language does not make. |
| 37 | |
| 38 | ## Where the scanner leans on the language, and where on convention |
| 39 | |
| 40 | **Keywords, constants and builtins are tables read out of GoloScript, not remembered.** The 38 keywords are `token/token.go`'s table less the three literals; the 157 builtins are what `evaluator.BuiltinNames()` answers less the five test counters that begin with a double underscore — the same five the language server keeps out of its completions. A test holds that table to a running `golo lsp` in both directions, which is how it stays a table read from the toolchain rather than a table somebody once typed. |
| 41 | |
| 42 | **A capitalised name is a type, and here that is a convention rather than a rule.** Golo has no case rule in its lexer: `let Count = 1` is legal. But structs, unions and their variants are capitalised by everybody — `Point`, `Shape`, `Circle`, `Some`, `None` — and nothing else customarily is, so the scanner colours by the convention, as Turbo Python does with PEP 8. What that costs is that a variant's constructor is coloured as a type (`Circle(1.0)` and `Result_Failure("no")` look like types applied to arguments) and a capitalised variable is coloured as one too. Nothing in the syntax separates them. |
| 43 | |
| 44 | **`Some`, `None`, `Ok` and `Err` are types here, not constants.** Turbo Rust and Turbo MoonBit colour them as constants because a reader meets them everywhere and reads them as built in. In Golo they are not: they are the variants of ordinary unions declared in `gololang.Errors`, available only after `import gololang.Errors`, and colouring them as the language's own would tell a reader they need no import when they do. |
| 45 | |
| 46 | **The name after `function` is a function, and the path after `module` or `import` is one name.** Everywhere else a name is a function because a parenthesis follows it, and a declaration — `function main = |args|` — is the one place that is not true; without a special case every function a file declares would be coloured as an ordinary variable at the one place a reader looks for it. A module path — `hello.World`, `gololang.Errors` — is one span and one colour because it is one name, and the reading it has to be saved from is the one where `gololang.Errors` looks like a variable with something done to it. |
| 47 | |
| 48 | **Names may be almost anything.** The lexer's `isLetter` admits any Unicode letter or mark, an underscore, and four blocks of emoji, so `let 😀 = 1` and `function 🚀launch = …` are legal Golo. The scanner uses the same predicate rather than turbo-core's ASCII one, so they are coloured — and so are `été` and `名前`, which Turbo MoonBit leaves uncoloured for its own language. |
| 49 | |
| 50 | ## What the lexer reads and the parser refuses |
| 51 | |
| 52 | This is the boundary worth stating plainly, because it is not one the scanner can see. GoloScript's lexer and its parser were written at different times, and the lexer is ahead: it reads several tokens that the parser, as of v0.1.1, then rejects. |
| 53 | |
| 54 | | The lexer reads | The parser says | |
| 55 | | --- | --- | |
| 56 | | `42L`, a long | `could not parse "42L" as integer` | |
| 57 | | `3.14F`, `2.0f`, a float | `could not parse "3.14F" as float` | |
| 58 | | `'x'`, a character | `no prefix parse function for CHAR found` | |
| 59 | | `1..3`, a range | `expected next token to be ), got .. instead` | |
| 60 | | `orIfNull`, `oftype` | `expected next token to be ), got orIfNull instead` | |
| 61 | | `local function …` | `no prefix parse function for LOCAL found` | |
| 62 | |
| 63 | The scanner colours what the lexer reads, because the lexer is the specification of what a token *is* and the parser's opinion of what to do with it may change tomorrow. So `42L` is one number and `orIfNull` is a keyword, and **a coloured token is not a promise that the interpreter accepts it**. The language server tells you when it does not: open `demos/syntax-tour/lexer-only.golo` and every one of those lines gets a mark in the gutter. |
| 64 | |
| 65 | ## What the scanner refuses to guess |
| 66 | |
| 67 | Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet: |
| 68 | |
| 69 | | Not recognised | Because | |
| 70 | | --- | --- | |
| 71 | | Digit separators and other bases | The lexer has no `1_000`, no `0xFF`, no `0b1010`. `1_000` is the number `1` followed by the name `_000`, and `0xFF` is `0` followed by `xFF` — which is what the interpreter sees, and colouring either as one number would be inventing a literal it rejects | |
| 72 | | A leading dot as a number | The lexer requires a digit before the point, so `.5` is a dot and then `5` | |
| 73 | | A lower-case `l` as a long suffix | The lexer accepts only `L`; `42l` is `42` and the name `l` | |
| 74 | | A keyword used after a colon as a method name | `obj: match()` keeps `match` a keyword. The scanner does not track what a colon introduces, and the lexer would refuse the word anyway | |
| 75 | | Escapes inside `"""…"""` | The lexer appends every rune until the three quotes, so `"""a\"""` ends at the first `"""` whatever the backslash meant | |
| 76 | | Whether a name is bound in this scope | Nothing here reads more than one line at a time; that is the language server's question, and [F1 answers it](../how-to/ask-about-code.md) | |
| 77 | |
| 78 | ## The other eight languages come free |
| 79 | |
| 80 | TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell are coloured by turbo-core, not here. A Golo project has a `README.md`, a `compose.yaml` for the service it talks to, a Dockerfile to ship as, and an editor that coloured only the `.golo` files would make you leave it for the rest. |
| 81 | |
| 82 | That they are shared rather than copied is the point of the library: they were written once, for Turbo Go, and Turbo Golo got them by importing a package. |
| 83 | |
| 84 | ## Completion, and why it can fail silently |
| 85 | |
| 86 | Turbo Golo knows nothing about Golo's semantics and does not try to. It asks `golo lsp` over the Language Server Protocol and draws the answer. |
| 87 | |
| 88 | Three things about that are worth knowing, because all three look like "completion is broken": |
| 89 | |
| 90 | **The server is the interpreter.** There is no separate `golo-lsp` binary to install and no toolchain it depends on: `golo lsp` reuses the interpreter's lexer, parser and AST. So "no completion" on a machine that runs Golo scripts has exactly one cause — the editor cannot find `golo` — and the status bar says so, with the address of the release page. |
| 91 | |
| 92 | **Only top-level declarations are offered.** The server's completion lists keywords, builtins, the functions and unions declared at the top level of the file, and the symbols pulled in by `import` from the modules embedded in the binary. A function you declared inside another function's body is not in the list, and nor is anything from a `.golo` file of your own on disk: imports of user modules are not resolved. That is the server's design, and it is written down rather than worked around. |
| 93 | |
| 94 | **Diagnostics are about parsing, not running.** `golo lsp` publishes syntax errors and two lints — a `:`/`.` confusion, and C-style `//` or `/* */` comments where Golo wants `#` and `----`. A program that parses and then fails at run time gets no mark, because the server never runs it. And a syntax error highlights a whole line: the parser's messages carry a line number and no column, so the mark lands on the line. |
| 95 | |
| 96 | The editor's answer to the first is [Run ▸ Language server status](../reference/menus.md), which says what it found, where it started it and whether it is ready — because "nothing happened" is not something a user can act on. |
| 97 | |
| 98 | ## Nine questions, one connection — and the four `golo lsp` does not answer |
| 99 | |
| 100 | Completion is the loudest thing the language server does and the least revealing. The same connection asks eight more questions, and they divide into three kinds by what comes back. |
| 101 | |
| 102 | **Something to read.** `hover` — what is this? — drawn in a box. For a function you declared, that is the `#` comments written immediately above it; for a builtin, its signature and a worked example; for a keyword, a sentence. |
| 103 | |
| 104 | **Places in the code.** `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — and for a long time this family's editors took the first and threw the rest away. |
| 105 | |
| 106 | **Names.** `documentSymbol` for a file's own outline, `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn. |
| 107 | |
| 108 | And one thing nobody asks for at all: **`publishDiagnostics` arrives unbidden**, whenever the server has an opinion, on open and on every edit. That is why the mark in the gutter appears without anything being pressed. |
| 109 | |
| 110 | **One of the nine comes back empty with `golo lsp`, and that is the server's boundary rather than the editor's.** It advertises `completion`, `hover`, `definition`, `documentSymbol`, `references`, `implementation` and `workspaceSymbol` — and not `typeDefinition`, so **Code ▸ Type definition** reports nothing found. Until GoloScript v0.2.0 it advertised only the first four, and **Shift-F12** (references), **Code ▸ Find implementations** and **Ctrl-T** (a symbol anywhere in the project) came back empty too; the test that pins this boundary failed the day the server started answering them, and this paragraph was revised — which is what the test is for. The gap is written down rather than hidden because the alternative — greying out a menu item depending on what a server said at start-up — makes the menu a different shape on different machines, and a user who has read this page knows more than one who found a greyed item. |
| 111 | |
| 112 | The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here. |
| 113 | |
| 114 | ## How it relates to the rest |
| 115 | |
| 116 | - Exactly what is recognised: [Languages coloured](../reference/languages.md) |
| 117 | - Getting completion working: [How to enable Golo completion](../how-to/enable-completion.md) |
| 118 | - Where the scanner lives and why: [Architecture](architecture.md) |