Colouring and completion — explanation
What is this about?
The two features that make Turbo MoonBit an editor for MoonBit rather than a text editor that happens to open .mbt files: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.
Colouring is ours; completion is not
Colouring is done here, in about six hundred lines of hand-written Go. Completion is done by moon-lsp, and Turbo MoonBit only asks and draws.
That split is not an accident of effort. Colouring has to be instant and tolerant: it runs on every keystroke, on text that is invalid most of the time it is being typed, and a highlighter that stops to think or gives up on broken input is worse than no highlighter. Completion has to be correct, which for MoonBit means following imports, resolving a name through the class hierarchy it was assigned in, and reading every installed package's public surface — and nothing that has to be instant can also be that.
So the editor draws colours it computed itself, and shows completions somebody else computed.
Why MoonBit is scanned by hand
MoonBit has no lexer available as a Go package. Turbo Go can go through go/scanner — the standard library analysing its own language, so the editor and the compiler agree about what a token is with nothing to keep in step. Turbo MoonBit has no such thing, and the three ways round it were weighed.
Running a real MoonBit lexer would mean starting moonc and asking it for tokens: a process per keystroke, and a dependency on a toolchain the editor should not need in order to colour a file.
Embedding a grammar — tree-sitter or the like — would mean a native library, a build step, and a binary that no longer compiles everywhere. turbo-core holds itself to two and a half direct dependencies; this is not where the third gets added.
Writing a scanner by hand costs one more file and gives something that runs on every keystroke without allocating anything surprising, never breaks on invalid text, and reads like ordinary Go.
The scanner it is. Some six hundred lines, one file each for the dispatcher, the literals and the words — and no attempt at a general engine. There is no pattern language, no grammar format and no table of regular expressions: it is ordinary Go that a reader can follow, which is the same rule the eight scanners in turbo-core follow.
The published grammar made that affordable. MoonBit's documentation carries a complete lexical specification — the productions for every literal, the keyword list, the rule that an integer ends before .. — so this scanner was written against a specification rather than against a pile of examples, and the places where it deliberately departs from that specification are named below.
Nothing crosses a line break
Every other editor in this family threads real state through its scanner. Turbo Go and Turbo Rust carry a block-comment depth; Turbo Rust also carries a raw string's delimiter; Turbo Python carries which of the two quotes opened a triple-quoted literal. Turbo MoonBit carries nothing at all, and that is a fact about the language rather than a shortcut:
- There is no block comment. The grammar says so in as many words: "MoonBit has no block-comment form."
//runs to the end of the line,///is a doc comment that does the same. - No literal may reach the next line. For strings, bytes, regexes, characters and byte characters alike, "a newline before the closing quote reports an unterminated string literal". A line that ends inside a literal is broken source, not a construct.
- A multi-line string is not one literal spanning lines. It is a run of lines each prefixed
#|or$|, each a complete token, which the compiler joins with a newline afterwards. - An attribute is explicitly one line: "everything through the next newline is the raw payload".
So the carry type is empty, and it is a named type rather than struct{} written inline, so that the reasoning has somewhere to live. If MoonBit ever grows a construct that crosses a line break, that type is what gains a field.
What this buys is worth stating plainly: a stray quote cannot paint the rest of the file. In every other editor here, an unterminated string is a case the scanner has to decide to drop, and getting that decision wrong turns one keystroke into a screenful of green. Here there is no decision to get wrong.
Where the scanner leans on the language, not on convention
This is the part that makes MoonBit's scanner shorter than its siblings', and the reason is one lexical rule.
A capitalised name is a type, and that is the language's rule rather than a habit. The grammar defines uident as beginning "with an ASCII uppercase letter", and only a type, a trait or an enum constructor may be spelt that way. Turbo Python has to consult PEP 8 to tell ValueError("nope") from parse("nope"); Turbo Rust has to keep a table of the constructors the language names, because Some(x) looks like a call. Here the case is the answer, so there is no table of built-in types in this repository at all — Int, StringBuilder and a type somebody wrote this morning are coloured by the same line of code.
What that costs is one thing, and it is unavoidable. An enum constructor of your own — Circle(1.0) — is coloured as a type, because nothing in the syntax separates it from a type applied to arguments. Inventing a separation would mean being wrong in both directions instead of one.
The prelude is a table, and it was read rather than remembered. println, abort, fail, ignore, inspect and the rest come out of moonbitlang/core/prelude's generated interface file. That matters more than it sounds: a table written from habit would have held print, and MoonBit has never had print. The prelude's deprecated names — dump, not, tap — are deliberately absent, because colouring them as builtins would present four things the language is retiring as its own.
The one place the grammar has to be followed exactly
1..=2 is an integer and a range operator. A scanner that swallowed any dot after a number would read the double 1. and leave .=2 behind, and every range in every file would be miscoloured.
The grammar settles it in a sentence — "before .., an integer ends first, so 1..=2 begins with 1 and ..=" — and the scanner follows it exactly: a dot joins a number only when a second one does not follow. The same discipline governs the suffixes. 42UL is one number and 42u is 42 followed by the name u, because the grammar says the suffixes are upper case, and colouring 42u as a literal would be inventing something the compiler is about to reject.
A dot has two other jobs, and both had to be written out rather than folded into punctuation. pair.0 is a tuple accessor. xs.length() is a method — and the name after the dot is looked up without the keyword table, because MoonBit's dot-identifiers "use the identifier case rules without consulting the keyword table, so .if is valid". A record with a field called type is ordinary MoonBit, and a scanner that coloured that field as a keyword would be making a claim the language contradicts.
What the scanner refuses to guess
Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet:
| Not recognised | Because |
|---|---|
The expression inside \{…} |
The grammar matches it to "the matching }", with braces inside nested literals not counting — finding the end needs the parser. A brace counter that got it wrong would end the string early, and a literal that swallows the rest of the line is the loudest way a highlighter can break. One flat run of string is the honest answer for the ordinary case, and it is the one Turbo Python gives an f-string for the same reason. Its limit is a string nested inside the interpolation — see below |
| A reserved word as a keyword | move, ref, static, unsafe, await and forty others are reserved rather than keywords: the lexer treats them as identifiers and warns. Colouring them would tell a reader they cannot write let ref = 1 when they can |
| An identifier holding non-ASCII letters | MoonBit allows CJK and several other Unicode ranges in a name. The rune predicates this scanner is built on are ASCII, so such a name is stepped over uncoloured rather than guessed at — a boundary worth knowing rather than a defect to hide |
moon.mod, moon.pkg and moon.work |
They are MoonBit's own configuration DSL rather than MoonBit. Colouring them with the MoonBit scanner would be wrong about import { … } and about every bare key, and writing a second scanner for a format that is still changing is work with a short shelf life |
The contents of a .mbt.md fence |
It is a Markdown document, and Markdown colours it. A fenced block is one colour whatever language it announces — that is turbo-core's rule, and it applies to mbt exactly as it applies to bash |
The one place that answer is visibly wrong is a string inside an interpolation. "a \{f("x")} c" is one literal to the compiler and three spans to the scanner — string, then x as an identifier, then string — because the first unescaped quote is taken as the closer. It is the price of not parsing, it is bounded (the spans stay in order and never overlap, so nothing downstream misbehaves), and demos/syntax-tour/tour.mbt has a line that shows it rather than avoiding it.
package is the one deliberate over-reach, and it is worth naming as such. In a .mbt file it is only a reserved word; in the .mbti interface files this editor also colours, it is a real keyword. One scanner serves both, and colouring it as a keyword says in a .mbt file exactly what the compiler is about to: this word is not yours to use.
The other eight languages come free
TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell are coloured by turbo-core, not here. A MoonBit project has a moon.mod, a README.md, some scripts, a CI workflow in YAML and often a Dockerfile, and an editor that coloured only the .mbt files would make you leave it for the rest.
That they are shared rather than copied is the point of the library: they were written once, for Turbo Go, and Turbo MoonBit got them by importing a package.
Completion, and why it can fail silently
Turbo MoonBit knows nothing about MoonBit's type system and does not try to. It asks moon-lsp over the Language Server Protocol and draws the answer.
Three things about that are worth knowing, because all three look like "completion is broken":
moon-lsp answers nothing until it has indexed enough of the project. jedi resolves a name by following imports outwards, which on a first request into a large dependency means reading a great deal of somebody else's code. What you see meanwhile is an empty list.
A server given the wrong root loads the wrong code, and then answers nothing at all — with no error. That is why the editor walks up from the file to the nearest moon.mod, moon.mod.json or moon.mod.json rather than using the working directory, and it is the single most confusing way completion can fail.
A server installed without its extras answers questions but never volunteers a problem. moon-lsp's linters are optional dependencies; installed bare, it completes and jumps perfectly well and publishes an empty list of diagnostics for a file that does not even parse. A blank gutter because the server has no linter and a blank gutter because the code is fine look identical. That is why the install command names the extras and why the installer checks for them.
The editor's answer to the first two is Run ▸ Language server status, which says what it found, where it started it and whether it is ready — because "nothing happened" is not something a user can act on.
Nine questions, one connection — and the two moon-lsp does not answer
Completion is the loudest thing the language server does and the least revealing. The same connection asks eight more questions, and they divide into three kinds by what comes back.
Something to read. hover — what is this? — drawn in a box.
Places in the code. definition, typeDefinition, implementation, references. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — a method used across a package has as many references as somebody cared to write, and for a long time this editor took the first and threw the rest away.
Names. documentSymbol for a file's own outline, workspace/symbol for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.
And one thing nobody asks for at all: publishDiagnostics arrives unbidden, whenever the server has an opinion, for whatever files it has loaded — which are usually more than the one in front of you. That is why Problems lists every file rather than the current one, and why the mark in the gutter appears without anything being pressed.
Two of the nine come back empty with moon-lsp, and that is the server's boundary rather than the editor's. moon-lsp advertises neither typeDefinition nor implementation, so Code ▸ Type definition and Code ▸ Find implementations report nothing found. Everything else works, including the project-wide symbol search that Turbo Python's server does not answer. This is written down rather than hidden because the alternative — greying out two menu items depending on what a server said at start-up — makes the menu a different shape on different machines, and a user who has read this page knows more than one who found a greyed item.
The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here.
How it relates to the rest
- Exactly what is recognised: Languages coloured
- Getting completion working: How to enable MoonBit completion
- Where the scanner lives and why: Architecture
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
|