Reference: the syntax extension point

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163

# Reference: the syntax extension point

> Neutral, exhaustive description of what `rickub.com/turbo-editors/turbo-core/syntax` offers a language author.

## Language

```go
type Language string
```

The name a language is known by. It is a string because it is written down outside the package: a snippets file restricts itself to `languages = ["go"]`, and a language an editor registers is not something the library could have numbered in advance.

| Constant | Value |
| --- | --- |
| `LanguageNone` | `""` |
| `LanguageTOML` | `"toml"` |
| `LanguageYAML` | `"yaml"` |
| `LanguageMarkdown` | `"markdown"` |
| `LanguageJavaScript` | `"javascript"` |
| `LanguageHTML` | `"html"` |
| `LanguageXML` | `"xml"` |
| `LanguageDockerfile` | `"dockerfile"` |
| `LanguageBash` | `"bash"` |

`Language.String()` returns the name, and `"none"` for `LanguageNone`.

## Definition

| Field | Type | Description |
| --- | --- | --- |
| `Language` | `Language` | The name; the value `LanguageOf` returns for a file of this kind. |
| `Extensions` | `[]string` | Extensions that identify it, with their dots, in lower case. |
| `Filenames` | `[]string` | Whole file names that identify it, for files with no useful extension: `"Dockerfile"`. A file matches on its whole name **or** on the part before its first dot, ignoring case — so `"Dockerfile"` also answers for `Dockerfile.dev`. A name that is all extension, such as `.gitignore`, has an empty stem and matches nothing. |
| `Shebangs` | `[]string` | Interpreter names that identify a file with no useful extension. |
| `Highlight` | `func(string) [][]Span` | Colours a whole document. May be nil, which names a language without colouring it. |

## Package functions

| Function | Description |
| --- | --- |
| `Register(Definition)` | Adds a language. Registering one already known replaces it. Not safe from two goroutines at once. |
| `Registered() []Language` | The languages that can be coloured, sorted by name. |
| `LanguageOf(path, firstLine string) Language` | The language of a file: **extension**, then **name**, then **shebang**. `LanguageNone` when nothing claims it. |
| `Highlight(l Language, src string) [][]Span` | One slice of spans per line of `src`, always exactly as many entries as there are lines. |

## Span and Class

```go
type Span struct {
	Start int // rune column, included
	End   int // rune column, excluded
	Class Class
}
```

A span may never straddle a line break. Spans on a line must be in order and must not overlap.

| Class | Used for |
| --- | --- |
| `ClassIdentifier` | the zero value: text nothing else claims |
| `ClassKeyword` | a reserved word |
| `ClassType` | a type name |
| `ClassBuiltin` | something the language itself provides |
| `ClassConstant` | a literal the language names |
| `ClassFunction` | a function name, declared or called |
| `ClassString` | a string literal |
| `ClassChar` | a character literal |
| `ClassNumber` | a numeric literal |
| `ClassComment` | a comment |
| `ClassOperator` | an operator |
| `ClassPunctuation` | structure rather than computation |
| `ClassHeading` | a Markdown heading |
| `ClassTag` | an HTML element name |
| `ClassAttribute` | an HTML attribute name, or a Rust attribute |
| `ClassEmphasis` | Markdown bold or italic |
| `ClassLink` | a Markdown link or image |

`Class.StyleKey()` returns the theme key that colours it; an out-of-range class gives the identifier key. `Class.String()` returns the name, and `"unknown"` out of range.

The set is closed. A language registered from outside colours itself with these and no others.

## ScanLines

```go
func ScanLines[State any](src string, scan func(line []rune, carry State) ([]Span, State)) [][]Span
```

Runs a per-line scanner over a document, threading whatever state crosses a line break. `State` starts at its zero value on the first line. A trailing `\r` is dropped from every line, so a CRLF file colours the same as an LF one.

## LineScanner

```go
func NewLineScanner(line []rune) *LineScanner
```

| Method | Description |
| --- | --- |
| `Spans() []Span` | The spans found so far. |
| `Len() int` | How many runes the line holds. |
| `Pos() int` | The current position, in runes. |
| `AtEnd() bool` | Whether the whole line has been consumed. |
| `Peek(offset int) rune` | The rune at an offset from the position, or `0` past either end. |
| `Advance(n int)` | Move forward without colouring, stopping at the end of the line. |
| `Emit(start, end int, class Class)` | Record a span. Empty spans are dropped. |
| `Take(n int, class Class)` | Consume `n` runes and colour them. |
| `TakeWhile(class Class, matches func(rune) bool) bool` | Consume while runes match; reports whether any were. |
| `TakeRest(class Class)` | Consume and colour everything left on the line. |
| `SkipSpaces()` | Step over spaces and tabs without colouring them. |
| `HasPrefix(offset int, want string) bool` | Whether the line reads `want` at an offset from the position. |

Because `Emit` drops empty spans, a scanner must pass a span's start in as a parameter rather than patch it onto the last span afterwards: the last one may not be the one it thinks.

## Free helpers

| Function | Description |
| --- | --- |
| `TakeQuoted(s, quote rune, class Class)` | A quoted string ending on this line, backslash escapes included. An unterminated one is coloured to the end of the line. |
| `OpenBlockComment(s, opener, closer string, class Class) bool` | A block comment starting at the position; reports whether it also ended on this line. |
| `FinishBlockComment(s, closer string, class Class) bool` | The continuation of one opened earlier; reports whether it ended here. |
| `IsOperatorRune(r rune) bool` | One of `+-*/%=<>!&\|^~?:` |
| `IsPunctuationRune(r rune) bool` | One of `()[]{},;.` |
| `IsDigit(r rune) bool` | An ASCII digit. |
| `IsLetter(r rune) bool` | An ASCII letter. |
| `IsWordRune(r rune) bool` | A letter, a digit, or an underscore. |

## LineIndex

For a scanner that works in byte offsets rather than a line at a time.

```go
func NewLineIndex(src string) *LineIndex
```

| Method | Description |
| --- | --- |
| `Count() int` | The number of lines, always at least one. |
| `AppendSpans(out [][]Span, start, end int, class Class)` | Records the byte range `[start, end)` as one span per line it covers. `out` must already have `Count()` entries. |

### Example

```go
lines := syntax.NewLineIndex(src)
out := make([][]syntax.Span, lines.Count())
lines.AppendSpans(out, token.Start, token.End, syntax.ClassKeyword)
```

## Cache

What the editor draws from: it re-scans only when the buffer's revision has moved.

| Function or method | Description |
| --- | --- |
| `NewCache(Language) *Cache` | A cache for one language. `LanguageNone` gives a disabled one. |
| `Language() Language` | The language in use. |
| `SetLanguage(Language)` | Changes it and discards what was scanned. |
| `Enabled() bool` | Whether the language is anything but `LanguageNone`. |
| `Update(src string, revision int)` | Re-scans when the revision has moved, or on the first call. |
| `Line(i int) []Span` | The spans of one line; `nil` out of range. |
| `LineCount() int` | How many lines were scanned. |

## See also

- [Add a language](../how-to/add-a-language.md)