# Reference: the syntax extension point

> Neutral, exhaustive description of what `codeberg.org/turbo-editors/turbo-core/syntax` offers a language author.

## Language

```go
type Language string
```

The name a language is known by. It is a string because it is written down outside the package: a snippets file restricts itself to `languages = ["go"]`, and a language an editor registers is not something the library could have numbered in advance.

| Constant | Value |
| --- | --- |
| `LanguageNone` | `""` |
| `LanguageTOML` | `"toml"` |
| `LanguageYAML` | `"yaml"` |
| `LanguageMarkdown` | `"markdown"` |
| `LanguageJavaScript` | `"javascript"` |
| `LanguageHTML` | `"html"` |
| `LanguageXML` | `"xml"` |
| `LanguageDockerfile` | `"dockerfile"` |
| `LanguageBash` | `"bash"` |

`Language.String()` returns the name, and `"none"` for `LanguageNone`.

## Definition

| Field | Type | Description |
| --- | --- | --- |
| `Language` | `Language` | The name; the value `LanguageOf` returns for a file of this kind. |
| `Extensions` | `[]string` | Extensions that identify it, with their dots, in lower case. |
| `Filenames` | `[]string` | Whole file names that identify it, for files with no useful extension: `"Dockerfile"`. A file matches on its whole name **or** on the part before its first dot, ignoring case — so `"Dockerfile"` also answers for `Dockerfile.dev`. A name that is all extension, such as `.gitignore`, has an empty stem and matches nothing. |
| `Shebangs` | `[]string` | Interpreter names that identify a file with no useful extension. |
| `Highlight` | `func(string) [][]Span` | Colours a whole document. May be nil, which names a language without colouring it. |

## Package functions

| Function | Description |
| --- | --- |
| `Register(Definition)` | Adds a language. Registering one already known replaces it. Not safe from two goroutines at once. |
| `Registered() []Language` | The languages that can be coloured, sorted by name. |
| `LanguageOf(path, firstLine string) Language` | The language of a file: **extension**, then **name**, then **shebang**. `LanguageNone` when nothing claims it. |
| `Highlight(l Language, src string) [][]Span` | One slice of spans per line of `src`, always exactly as many entries as there are lines. |

## Span and Class

```go
type Span struct {
	Start int // rune column, included
	End   int // rune column, excluded
	Class Class
}
```

A span may never straddle a line break. Spans on a line must be in order and must not overlap.

| Class | Used for |
| --- | --- |
| `ClassIdentifier` | the zero value: text nothing else claims |
| `ClassKeyword` | a reserved word |
| `ClassType` | a type name |
| `ClassBuiltin` | something the language itself provides |
| `ClassConstant` | a literal the language names |
| `ClassFunction` | a function name, declared or called |
| `ClassString` | a string literal |
| `ClassChar` | a character literal |
| `ClassNumber` | a numeric literal |
| `ClassComment` | a comment |
| `ClassOperator` | an operator |
| `ClassPunctuation` | structure rather than computation |
| `ClassHeading` | a Markdown heading |
| `ClassTag` | an HTML element name |
| `ClassAttribute` | an HTML attribute name, or a Rust attribute |
| `ClassEmphasis` | Markdown bold or italic |
| `ClassLink` | a Markdown link or image |

`Class.StyleKey()` returns the theme key that colours it; an out-of-range class gives the identifier key. `Class.String()` returns the name, and `"unknown"` out of range.

The set is closed. A language registered from outside colours itself with these and no others.

## ScanLines

```go
func ScanLines[State any](src string, scan func(line []rune, carry State) ([]Span, State)) [][]Span
```

Runs a per-line scanner over a document, threading whatever state crosses a line break. `State` starts at its zero value on the first line. A trailing `\r` is dropped from every line, so a CRLF file colours the same as an LF one.

## LineScanner

```go
func NewLineScanner(line []rune) *LineScanner
```

| Method | Description |
| --- | --- |
| `Spans() []Span` | The spans found so far. |
| `Len() int` | How many runes the line holds. |
| `Pos() int` | The current position, in runes. |
| `AtEnd() bool` | Whether the whole line has been consumed. |
| `Peek(offset int) rune` | The rune at an offset from the position, or `0` past either end. |
| `Advance(n int)` | Move forward without colouring, stopping at the end of the line. |
| `Emit(start, end int, class Class)` | Record a span. Empty spans are dropped. |
| `Take(n int, class Class)` | Consume `n` runes and colour them. |
| `TakeWhile(class Class, matches func(rune) bool) bool` | Consume while runes match; reports whether any were. |
| `TakeRest(class Class)` | Consume and colour everything left on the line. |
| `SkipSpaces()` | Step over spaces and tabs without colouring them. |
| `HasPrefix(offset int, want string) bool` | Whether the line reads `want` at an offset from the position. |

Because `Emit` drops empty spans, a scanner must pass a span's start in as a parameter rather than patch it onto the last span afterwards: the last one may not be the one it thinks.

## Free helpers

| Function | Description |
| --- | --- |
| `TakeQuoted(s, quote rune, class Class)` | A quoted string ending on this line, backslash escapes included. An unterminated one is coloured to the end of the line. |
| `OpenBlockComment(s, opener, closer string, class Class) bool` | A block comment starting at the position; reports whether it also ended on this line. |
| `FinishBlockComment(s, closer string, class Class) bool` | The continuation of one opened earlier; reports whether it ended here. |
| `IsOperatorRune(r rune) bool` | One of `+-*/%=<>!&\|^~?:` |
| `IsPunctuationRune(r rune) bool` | One of `()[]{},;.` |
| `IsDigit(r rune) bool` | An ASCII digit. |
| `IsLetter(r rune) bool` | An ASCII letter. |
| `IsWordRune(r rune) bool` | A letter, a digit, or an underscore. |

## LineIndex

For a scanner that works in byte offsets rather than a line at a time.

```go
func NewLineIndex(src string) *LineIndex
```

| Method | Description |
| --- | --- |
| `Count() int` | The number of lines, always at least one. |
| `AppendSpans(out [][]Span, start, end int, class Class)` | Records the byte range `[start, end)` as one span per line it covers. `out` must already have `Count()` entries. |

### Example

```go
lines := syntax.NewLineIndex(src)
out := make([][]syntax.Span, lines.Count())
lines.AppendSpans(out, token.Start, token.End, syntax.ClassKeyword)
```

## Cache

What the editor draws from: it re-scans only when the buffer's revision has moved.

| Function or method | Description |
| --- | --- |
| `NewCache(Language) *Cache` | A cache for one language. `LanguageNone` gives a disabled one. |
| `Language() Language` | The language in use. |
| `SetLanguage(Language)` | Changes it and discards what was scanned. |
| `Enabled() bool` | Whether the language is anything but `LanguageNone`. |
| `Update(src string, revision int)` | Re-scans when the revision has moved, or on the first call. |
| `Line(i int) []Span` | The spans of one line; `nil` out of range. |
| `LineCount() int` | How many lines were scanned. |

## See also

- [Add a language](../how-to/add-a-language.md)