| 🛟 Updated. 28d5985 k33g 19h ago | 1 | # How to add a language |
| 2 | |
| 3 | This guide shows how to teach an editor built on turbo-core to colour a language. It assumes you already have an editor — if you do not, [build one first](../tutorials/build-an-editor.md). |
| 4 | |
| 5 | turbo-core colours TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell itself. The language your editor is *for* is yours to register: that is what makes Turbo Go colour Go and Turbo Rust colour Rust. |
| 6 | |
| 7 | ## Steps |
| 8 | |
| 9 | ### 1. Decide how the language is recognised |
| 10 | |
| 11 | A file is matched by its extension first, and only failing that by its first line: |
| 12 | |
| 13 | ```go |
| 14 | syntax.Definition{ |
| 15 | Language: "zig", |
| 16 | Extensions: []string{".zig"}, // with the dot, in lower case |
| 17 | Filenames: []string{"build.zig"}, // for files with no useful extension |
| 18 | Shebangs: []string{"zig"}, // most languages need none |
| 19 | Highlight: highlightZig, |
| 20 | } |
| 21 | ``` |
| 22 | |
| 23 | `Language` is the name the language is known by everywhere else: it is what `LanguageOf` returns, and what a user writes in a snippets file's `languages` key. Keep it lower case and keep it stable. |
| 24 | |
| 25 | **A file is matched by its extension, then its name, then its first line.** `Filenames` is for a file that has no extension to go on — a `Dockerfile`, a `Makefile`. It matches on the whole name **or** on the part before the first dot, ignoring case, so listing `"Dockerfile"` also recognises `Dockerfile.dev` without naming every variant a project might invent. |
| 26 | |
| 27 | ### 2. Write the scanner |
| 28 | |
| 29 | A scanner takes the whole document and returns one slice of spans per line. `ScanLines` does the line splitting and threads whatever state crosses a line break: |
| 30 | |
| 31 | ```go |
| 32 | func highlightZig(src string) [][]syntax.Span { |
| 33 | return syntax.ScanLines(src, func(line []rune, inComment bool) ([]syntax.Span, bool) { |
| 34 | s := syntax.NewLineScanner(line) |
| 35 | // … colour the line, setting inComment as you go … |
| 36 | return s.Spans(), inComment |
| 37 | }) |
| 38 | } |
| 39 | ``` |
| 40 | |
| 41 | Inside the callback, ask `s.Peek(0)` what is there and take it: |
| 42 | |
| 43 | | To colour | Call | |
| 44 | | --- | --- | |
| 45 | | a comment running to the end of the line | `s.TakeRest(syntax.ClassComment)` | |
| 46 | | a quoted string ending on this line | `syntax.TakeQuoted(s, '"', syntax.ClassString)` | |
| 47 | | a block comment opening here | `syntax.OpenBlockComment(s, "/*", "*/", syntax.ClassComment)` | |
| 48 | | the rest of one opened earlier | `syntax.FinishBlockComment(s, "*/", syntax.ClassComment)` | |
| 49 | | a run of matching runes | `s.TakeWhile(syntax.ClassNumber, syntax.IsDigit)` | |
| 50 | | a fixed number of runes | `s.Take(2, syntax.ClassOperator)` | |
| 51 | | whitespace, uncoloured | `s.SkipSpaces()` | |
| 52 | | anything else, uncoloured | `s.Advance(1)` | |
| 53 | |
| 54 | The full list is in the [syntax reference](../reference/syntax.md). |
| 55 | |
| 56 | ### 3. Register it |
| 57 | |
| 58 | ```go |
| 59 | func Register() { |
| 60 | syntax.Register(syntax.Definition{ /* … */ }) |
| 61 | } |
| 62 | ``` |
| 63 | |
| 64 | Call that from your command's `main`, before any file is opened. Do it explicitly rather than from an `init` function, so that "this editor knows Zig" is a line somebody can read. |
| 65 | |
| 66 | ### 4. Test it |
| 67 | |
| 68 | Drive the scanner through `syntax.Highlight`, which is how the editor reaches it: |
| 69 | |
| 70 | ```go |
| 71 | func TestKeywordsAreColoured(t *testing.T) { |
| 72 | Register() |
| 73 | |
| 74 | spans := syntax.Highlight("zig", "const x = 1;") |
| 75 | |
| 76 | if spans[0][0].Class != syntax.ClassKeyword { |
| 77 | t.Errorf("const is %v, want keyword", spans[0][0].Class) |
| 78 | } |
| 79 | } |
| 80 | ``` |
| 81 | |
| 82 | Three things are worth a test of their own, because each of them is a real bug that has happened here: |
| 83 | |
| 84 | - **One entry per line.** The editor indexes the result by line number without checking, so a short result is an index-out-of-range in the middle of a redraw. |
| 85 | - **Spans in order and not overlapping.** They are drawn in order; two out of order paint over each other. |
| 86 | - **Broken input still colours.** Source under the cursor is invalid most of the time it is being typed. |
| 87 | |
| 88 | ## Variants |
| 89 | |
| 90 | ### The language has a tokeniser already |
| 91 | |
| 92 | If your language ships a lexer — Go's `go/scanner`, say — use it and convert its byte offsets with `LineIndex` instead of writing a line scanner: |
| 93 | |
| 94 | ```go |
| 95 | func highlightGo(src string) [][]syntax.Span { |
| 96 | lines := syntax.NewLineIndex(src) |
| 97 | out := make([][]syntax.Span, lines.Count()) |
| 98 | |
| 99 | for _, token := range tokenise(src) { |
| 100 | lines.AppendSpans(out, token.start, token.end, classOf(token)) |
| 101 | } |
| 102 | return out |
| 103 | } |
| 104 | ``` |
| 105 | |
| 106 | `AppendSpans` splits a range that crosses a line break into one span per line, which a span may never do. |
| 107 | |
| 108 | ### Something crosses a line break |
| 109 | |
| 110 | Put it in the state `ScanLines` threads. Use a value, not a flag, when the construct nests — Rust's block comments nest, so Turbo Rust carries a depth, and a `bool` would end a nested comment one level early. |
| 111 | |
| 112 | ### Your language needs a colour nothing else uses |
| 113 | |
| 114 | It probably does not. The classes are fixed on purpose, so that one theme colours every language an editor will ever learn. Markup needed five extra ones (`ClassHeading`, `ClassTag`, `ClassAttribute`, `ClassEmphasis`, `ClassLink`) because a heading genuinely is not a keyword; reach for those before asking for a sixth. |
| 115 | |
| 116 | ### You want to override a built-in |
| 117 | |
| 118 | Register your own definition with the same `Language`. The later registration wins, because it is the more specific statement. |
| 119 | |
| 120 | ## See also |
| 121 | |
| 122 | - Everything the scanner toolkit offers: [syntax reference](../reference/syntax.md) |
| 123 | - Why the class list is closed: [what belongs here](../explanation/what-belongs-here.md) |