How to add a language

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123

# How to add a language

This guide shows how to teach an editor built on turbo-core to colour a language. It assumes you already have an editor — if you do not, [build one first](../tutorials/build-an-editor.md).

turbo-core colours TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell itself. The language your editor is *for* is yours to register: that is what makes Turbo Go colour Go and Turbo Rust colour Rust.

## Steps

### 1. Decide how the language is recognised

A file is matched by its extension first, and only failing that by its first line:

```go
syntax.Definition{
	Language:   "zig",
	Extensions: []string{".zig"},          // with the dot, in lower case
	Filenames:  []string{"build.zig"},     // for files with no useful extension
	Shebangs:   []string{"zig"},           // most languages need none
	Highlight:  highlightZig,
}
```

`Language` is the name the language is known by everywhere else: it is what `LanguageOf` returns, and what a user writes in a snippets file's `languages` key. Keep it lower case and keep it stable.

**A file is matched by its extension, then its name, then its first line.** `Filenames` is for a file that has no extension to go on — a `Dockerfile`, a `Makefile`. It matches on the whole name **or** on the part before the first dot, ignoring case, so listing `"Dockerfile"` also recognises `Dockerfile.dev` without naming every variant a project might invent.

### 2. Write the scanner

A scanner takes the whole document and returns one slice of spans per line. `ScanLines` does the line splitting and threads whatever state crosses a line break:

```go
func highlightZig(src string) [][]syntax.Span {
	return syntax.ScanLines(src, func(line []rune, inComment bool) ([]syntax.Span, bool) {
		s := syntax.NewLineScanner(line)
		// … colour the line, setting inComment as you go …
		return s.Spans(), inComment
	})
}
```

Inside the callback, ask `s.Peek(0)` what is there and take it:

| To colour | Call |
| --- | --- |
| a comment running to the end of the line | `s.TakeRest(syntax.ClassComment)` |
| a quoted string ending on this line | `syntax.TakeQuoted(s, '"', syntax.ClassString)` |
| a block comment opening here | `syntax.OpenBlockComment(s, "/*", "*/", syntax.ClassComment)` |
| the rest of one opened earlier | `syntax.FinishBlockComment(s, "*/", syntax.ClassComment)` |
| a run of matching runes | `s.TakeWhile(syntax.ClassNumber, syntax.IsDigit)` |
| a fixed number of runes | `s.Take(2, syntax.ClassOperator)` |
| whitespace, uncoloured | `s.SkipSpaces()` |
| anything else, uncoloured | `s.Advance(1)` |

The full list is in the [syntax reference](../reference/syntax.md).

### 3. Register it

```go
func Register() {
	syntax.Register(syntax.Definition{ /* … */ })
}
```

Call that from your command's `main`, before any file is opened. Do it explicitly rather than from an `init` function, so that "this editor knows Zig" is a line somebody can read.

### 4. Test it

Drive the scanner through `syntax.Highlight`, which is how the editor reaches it:

```go
func TestKeywordsAreColoured(t *testing.T) {
	Register()

	spans := syntax.Highlight("zig", "const x = 1;")

	if spans[0][0].Class != syntax.ClassKeyword {
		t.Errorf("const is %v, want keyword", spans[0][0].Class)
	}
}
```

Three things are worth a test of their own, because each of them is a real bug that has happened here:

- **One entry per line.** The editor indexes the result by line number without checking, so a short result is an index-out-of-range in the middle of a redraw.
- **Spans in order and not overlapping.** They are drawn in order; two out of order paint over each other.
- **Broken input still colours.** Source under the cursor is invalid most of the time it is being typed.

## Variants

### The language has a tokeniser already

If your language ships a lexer — Go's `go/scanner`, say — use it and convert its byte offsets with `LineIndex` instead of writing a line scanner:

```go
func highlightGo(src string) [][]syntax.Span {
	lines := syntax.NewLineIndex(src)
	out := make([][]syntax.Span, lines.Count())

	for _, token := range tokenise(src) {
		lines.AppendSpans(out, token.start, token.end, classOf(token))
	}
	return out
}
```

`AppendSpans` splits a range that crosses a line break into one span per line, which a span may never do.

### Something crosses a line break

Put it in the state `ScanLines` threads. Use a value, not a flag, when the construct nests — Rust's block comments nest, so Turbo Rust carries a depth, and a `bool` would end a nested comment one level early.

### Your language needs a colour nothing else uses

It probably does not. The classes are fixed on purpose, so that one theme colours every language an editor will ever learn. Markup needed five extra ones (`ClassHeading`, `ClassTag`, `ClassAttribute`, `ClassEmphasis`, `ClassLink`) because a heading genuinely is not a keyword; reach for those before asking for a sixth.

### You want to override a built-in

Register your own definition with the same `Language`. The later registration wins, because it is the more specific statement.

## See also

- Everything the scanner toolkit offers: [syntax reference](../reference/syntax.md)
- Why the class list is closed: [what belongs here](../explanation/what-belongs-here.md)