# syntax

Colours source code, and is where an editor built on this library adds the language it is *for*.

Eight languages are coloured here — TOML, Markdown, JavaScript, HTML, shell scripts, **YAML**, **XML** and **Dockerfiles** — because every editor meets them whatever it is for: a project's configuration is TOML or YAML, its documentation Markdown, its scripts shell, its container build a Dockerfile. The language that defines an editor is not: Turbo Go registers Go, Turbo Rust registers Rust, and neither is known here.

A compose file is not a language of its own. `compose.yaml` and `docker-compose.yml` are YAML, and colouring `services:` differently from any other key would mean carrying Docker's schema here and watching it go stale.

Each scanner is one file of ordinary Go. They share `LineScanner` — a line in runes, a position in it, and the spans found so far — and nothing else: there is no pattern language and no grammar format, so adding a language means writing ordinary Go beside the others rather than learning a notation. `LineScanner` and the helpers around it are **exported**, because that is what makes writing one outside this package possible at all.

`Language` is a string, because it is written down outside this package: a snippets file restricts itself to `languages = ["go"]`. `LanguageOf(path, firstLine)` names a file by its **extension**, then by its **name**, then by a **shebang**; `LanguageNone` — anything no registered language claims — is coloured not at all rather than wrongly.

The name step is what finds a `Dockerfile`, which has neither an extension nor a shebang. A `Definition` lists whole names in `Filenames`, and a file matches on its whole name or on the part before its first dot — so `Dockerfile` also answers for `Dockerfile.dev`.

## The extension point

```go
syntax.Register(syntax.Definition{
	Language:   "rust",
	Extensions: []string{".rs"},
	Highlight:  highlightRust,
})
```

The registry is package-level state, which is the shape the standard library gives the same problem in `image.RegisterFormat`: an editor registers once at start-up, before it opens a file, and nothing ever removes an entry. Registering a language already known replaces it, so an editor can override one of the eight.

The **classes are closed**. A language registered from outside colours itself with the seventeen `Class` values and no others, which is what lets one theme colour every language an editor will ever learn.

## Every scanner guesses nothing

Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet, and each of these was a decision:

| Not recognised | Because |
| --- | --- |
| JavaScript regex literals | Telling `/x/g` from a division needs to know whether the previous token could end an expression; a wrong guess colours the rest of the line as a string |
| Shell heredocs | Carrying an arbitrary delimiter across lines, plus `<<-` and quoted spellings, for a few lines of plain text |
| JavaScript inside `<script>` | Following an element across lines and mapping another scanner's columns out — and the same argument then demands CSS |
| The language of a Markdown fence | Every scanner would have to be reachable from every other, which is the general engine this package does not have |

The exact boundary of each language is documented, in both languages, under `docs/*/reference/languages.md`.

## Multi-line state

Most constructs fit on a line. The ones that do not are carried by the scanner that owns them, through the `State` parameter of `scanLines`: a Markdown fence, a JavaScript block comment or template literal, an HTML comment, a YAML block scalar, an XML comment or CDATA section. Shell and Dockerfiles carry nothing, and say so with an empty struct.

Two of those carry more than a flag, for the same reason: a single "something is open" boolean closes on whatever delimiter comes first. XML carries *which* construct is open, so a `-->` inside a CDATA section does not end it. YAML carries the indentation its block scalar started at, because a block scalar has no closing delimiter at all — only the first line indented less than the block ends it, and a blank line inside one stays inside it.

Two traps live here, and both have bitten:

- **`emit` drops empty spans**, so a scanner must pass a span's start *in* rather than patch it onto the last one afterwards — the last one may not be the one it thinks. This produced a corrupt span in the TOML scanner and a lost line in the JavaScript one.
- **A helper that advances the position leaves nothing for `takeRest`.** `finishTemplate` looked correct and coloured nothing, because `runToBacktick` had already reached the end of the line.

## Tolerance is the point

The scanner runs in its most forgiving mode and every syntax error is discarded. Source under a cursor is broken most of the time it is being typed, and a highlighter that gives up on invalid input is a highlighter that flickers off. An unterminated string comes back as a string running to the end of the line, an unclosed `/*` as a comment running to the end of the file — which is exactly what keeps the colours steady mid-keystroke.

## Model

`Highlight(language, src)` returns `[][]Span` — **one entry per line**, always, including for `LanguageNone`, so the editor can index it by line number without a bounds check.

```go
type Span struct {
    Start int    // rune column, included
    End   int    // rune column, excluded
    Class Class
}
```

Columns are **runes**, not bytes, so an accented comment does not push the spans of the following lines out of alignment. A span never straddles a line break: a three-line block comment becomes three spans, one per line. Text no span covers — whitespace, mostly — is drawn in the editor's plain text style.

## Classes

`identifier` · `keyword` · `type` · `builtin` · `constant` · `function` · `string` · `char` · `number` · `comment` · `operator` · `punctuation` · `heading` · `tag` · `attribute` · `emphasis` · `link`

Deliberately coarser than any one language's token set: the editor needs to tell a keyword from a string, not an `ADD` from a `SUB`. Two of them are decided from context rather than from the token alone, in every scanner that has the notion:

- a name before `(` is a **function**;
- brackets, commas, dots and semicolons are **punctuation**, split out from the operators that actually compute, so a theme can quiet them down.

Names the language itself provides — `true`, `nil`, `None`, `i32`, `len` — are recognised by name, by the scanner that knows them.

`Class.StyleKey()` gives the `theme` key that colours it — the one place a token class and its colour are tied together, which is why this package depends on `theme`.

**TOML adds no classes**, and neither do JavaScript or shell: a table header names a structure so it is a `type`, `true` is a `constant`, a name before `(` is a `function`.

**The markup languages needed five.** `heading`, `tag`, `attribute`, `emphasis` and `link` have no equivalent in a programming language — a heading is not a keyword, and a theme wanting quiet headings with loud keywords could not say so otherwise. The cost falls on themes written elsewhere: one that sets none of them falls back along the dots to `syntax` and then `default`, so Markdown stays readable but undifferentiated. Every shipped theme sets all five and a test fails if one stops.

## Cache

The editor redraws far more often than the text changes — every cursor move, every scroll. `Cache` re-scans only when the buffer's revision counter has moved:

```go
cache := syntax.NewCache(syntax.LanguageOf(path, buf.Line(0)))
cache.Update(buf.Text(), buf.Revision())
for line := top; line < bottom; line++ {
    draw(line, cache.Line(line))
}
```

`NewCache(LanguageNone)`, or `SetLanguage(LanguageNone)`, gives a cache that colours nothing — what a file this package does not understand gets, without the editor needing a branch for it.

## Public API

| Function | What it does |
| --- | --- |
| `Register(Definition)` | Teaches this package a language |
| `Definition{Language, Extensions, Filenames, Shebangs, Highlight}` | How one language is recognised and coloured |
| `Registered() []Language` | What it can colour, sorted |
| `Highlight(l Language, src string) [][]Span` | Scans a whole source text, one span slice per line |
| `LanguageOf(path, firstLine string) Language` | A file's language by extension, then by name, then by shebang |
| `NewCache(l Language) *Cache` | A cache that re-scans only on a revision change |
| `(*Cache) SetLanguage(l Language)` | Change what is coloured, as Save As does |
| `(*Cache) Language() Language` | What it colours |
| `(*Cache) Update(src string, revision int)` | Re-scan if the revision moved |
| `(*Cache) Line(i int) []Span` | Spans of one line from the last scan |
| `(*Cache) LineCount() int` | Lines covered by the last scan |
| `(*Cache) Enabled() bool` | Whether it colours anything |
| `(Class) StyleKey() string` | The theme key that colours this class |
| `ScanLines(src, scan)` | Runs a per-line scanner over a document, threading its state |
| `NewLineScanner(line) *LineScanner` | The toolkit a hand-written scanner is built on |
| `NewLineIndex(src) *LineIndex` | Byte offsets into per-line spans, for a scanner built on a tokeniser |
| `TakeQuoted`, `OpenBlockComment`, `FinishBlockComment` | The constructs every language shares |
| `IsOperatorRune`, `IsPunctuationRune`, `IsDigit`, `IsLetter`, `IsWordRune` | Rune predicates the scanners here use |

The full list is in [the syntax reference](../docs/en/reference/syntax.md).

## Tests

```sh
make test
go test ./syntax/
```