syntax

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132

# syntax

Colours source code, and is where an editor built on this library adds the language it is *for*.

Eight languages are coloured here — TOML, Markdown, JavaScript, HTML, shell scripts, **YAML**, **XML** and **Dockerfiles** — because every editor meets them whatever it is for: a project's configuration is TOML or YAML, its documentation Markdown, its scripts shell, its container build a Dockerfile. The language that defines an editor is not: Turbo Go registers Go, Turbo Rust registers Rust, and neither is known here.

A compose file is not a language of its own. `compose.yaml` and `docker-compose.yml` are YAML, and colouring `services:` differently from any other key would mean carrying Docker's schema here and watching it go stale.

Each scanner is one file of ordinary Go. They share `LineScanner` — a line in runes, a position in it, and the spans found so far — and nothing else: there is no pattern language and no grammar format, so adding a language means writing ordinary Go beside the others rather than learning a notation. `LineScanner` and the helpers around it are **exported**, because that is what makes writing one outside this package possible at all.

`Language` is a string, because it is written down outside this package: a snippets file restricts itself to `languages = ["go"]`. `LanguageOf(path, firstLine)` names a file by its **extension**, then by its **name**, then by a **shebang**; `LanguageNone` — anything no registered language claims — is coloured not at all rather than wrongly.

The name step is what finds a `Dockerfile`, which has neither an extension nor a shebang. A `Definition` lists whole names in `Filenames`, and a file matches on its whole name or on the part before its first dot — so `Dockerfile` also answers for `Dockerfile.dev`.

## The extension point

```go
syntax.Register(syntax.Definition{
	Language:   "rust",
	Extensions: []string{".rs"},
	Highlight:  highlightRust,
})
```

The registry is package-level state, which is the shape the standard library gives the same problem in `image.RegisterFormat`: an editor registers once at start-up, before it opens a file, and nothing ever removes an entry. Registering a language already known replaces it, so an editor can override one of the eight.

The **classes are closed**. A language registered from outside colours itself with the seventeen `Class` values and no others, which is what lets one theme colour every language an editor will ever learn.

## Every scanner guesses nothing

Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet, and each of these was a decision:

| Not recognised | Because |
| --- | --- |
| JavaScript regex literals | Telling `/x/g` from a division needs to know whether the previous token could end an expression; a wrong guess colours the rest of the line as a string |
| Shell heredocs | Carrying an arbitrary delimiter across lines, plus `<<-` and quoted spellings, for a few lines of plain text |
| JavaScript inside `<script>` | Following an element across lines and mapping another scanner's columns out — and the same argument then demands CSS |
| The language of a Markdown fence | Every scanner would have to be reachable from every other, which is the general engine this package does not have |

The exact boundary of each language is documented, in both languages, under `docs/*/reference/languages.md`.

## Multi-line state

Most constructs fit on a line. The ones that do not are carried by the scanner that owns them, through the `State` parameter of `scanLines`: a Markdown fence, a JavaScript block comment or template literal, an HTML comment, a YAML block scalar, an XML comment or CDATA section. Shell and Dockerfiles carry nothing, and say so with an empty struct.

Two of those carry more than a flag, for the same reason: a single "something is open" boolean closes on whatever delimiter comes first. XML carries *which* construct is open, so a `-->` inside a CDATA section does not end it. YAML carries the indentation its block scalar started at, because a block scalar has no closing delimiter at all — only the first line indented less than the block ends it, and a blank line inside one stays inside it.

Two traps live here, and both have bitten:

- **`emit` drops empty spans**, so a scanner must pass a span's start *in* rather than patch it onto the last one afterwards — the last one may not be the one it thinks. This produced a corrupt span in the TOML scanner and a lost line in the JavaScript one.
- **A helper that advances the position leaves nothing for `takeRest`.** `finishTemplate` looked correct and coloured nothing, because `runToBacktick` had already reached the end of the line.

## Tolerance is the point

The scanner runs in its most forgiving mode and every syntax error is discarded. Source under a cursor is broken most of the time it is being typed, and a highlighter that gives up on invalid input is a highlighter that flickers off. An unterminated string comes back as a string running to the end of the line, an unclosed `/*` as a comment running to the end of the file — which is exactly what keeps the colours steady mid-keystroke.

## Model

`Highlight(language, src)` returns `[][]Span` — **one entry per line**, always, including for `LanguageNone`, so the editor can index it by line number without a bounds check.

```go
type Span struct {
    Start int    // rune column, included
    End   int    // rune column, excluded
    Class Class
}
```

Columns are **runes**, not bytes, so an accented comment does not push the spans of the following lines out of alignment. A span never straddles a line break: a three-line block comment becomes three spans, one per line. Text no span covers — whitespace, mostly — is drawn in the editor's plain text style.

## Classes

`identifier` · `keyword` · `type` · `builtin` · `constant` · `function` · `string` · `char` · `number` · `comment` · `operator` · `punctuation` · `heading` · `tag` · `attribute` · `emphasis` · `link`

Deliberately coarser than any one language's token set: the editor needs to tell a keyword from a string, not an `ADD` from a `SUB`. Two of them are decided from context rather than from the token alone, in every scanner that has the notion:

- a name before `(` is a **function**;
- brackets, commas, dots and semicolons are **punctuation**, split out from the operators that actually compute, so a theme can quiet them down.

Names the language itself provides — `true`, `nil`, `None`, `i32`, `len` — are recognised by name, by the scanner that knows them.

`Class.StyleKey()` gives the `theme` key that colours it — the one place a token class and its colour are tied together, which is why this package depends on `theme`.

**TOML adds no classes**, and neither do JavaScript or shell: a table header names a structure so it is a `type`, `true` is a `constant`, a name before `(` is a `function`.

**The markup languages needed five.** `heading`, `tag`, `attribute`, `emphasis` and `link` have no equivalent in a programming language — a heading is not a keyword, and a theme wanting quiet headings with loud keywords could not say so otherwise. The cost falls on themes written elsewhere: one that sets none of them falls back along the dots to `syntax` and then `default`, so Markdown stays readable but undifferentiated. Every shipped theme sets all five and a test fails if one stops.

## Cache

The editor redraws far more often than the text changes — every cursor move, every scroll. `Cache` re-scans only when the buffer's revision counter has moved:

```go
cache := syntax.NewCache(syntax.LanguageOf(path, buf.Line(0)))
cache.Update(buf.Text(), buf.Revision())
for line := top; line < bottom; line++ {
    draw(line, cache.Line(line))
}
```

`NewCache(LanguageNone)`, or `SetLanguage(LanguageNone)`, gives a cache that colours nothing — what a file this package does not understand gets, without the editor needing a branch for it.

## Public API

| Function | What it does |
| --- | --- |
| `Register(Definition)` | Teaches this package a language |
| `Definition{Language, Extensions, Filenames, Shebangs, Highlight}` | How one language is recognised and coloured |
| `Registered() []Language` | What it can colour, sorted |
| `Highlight(l Language, src string) [][]Span` | Scans a whole source text, one span slice per line |
| `LanguageOf(path, firstLine string) Language` | A file's language by extension, then by name, then by shebang |
| `NewCache(l Language) *Cache` | A cache that re-scans only on a revision change |
| `(*Cache) SetLanguage(l Language)` | Change what is coloured, as Save As does |
| `(*Cache) Language() Language` | What it colours |
| `(*Cache) Update(src string, revision int)` | Re-scan if the revision moved |
| `(*Cache) Line(i int) []Span` | Spans of one line from the last scan |
| `(*Cache) LineCount() int` | Lines covered by the last scan |
| `(*Cache) Enabled() bool` | Whether it colours anything |
| `(Class) StyleKey() string` | The theme key that colours this class |
| `ScanLines(src, scan)` | Runs a per-line scanner over a document, threading its state |
| `NewLineScanner(line) *LineScanner` | The toolkit a hand-written scanner is built on |
| `NewLineIndex(src) *LineIndex` | Byte offsets into per-line spans, for a scanner built on a tokeniser |
| `TakeQuoted`, `OpenBlockComment`, `FinishBlockComment` | The constructs every language shares |
| `IsOperatorRune`, `IsPunctuationRune`, `IsDigit`, `IsLetter`, `IsWordRune` | Rune predicates the scanners here use |

The full list is in [the syntax reference](../docs/en/reference/syntax.md).

## Tests

```sh
make test
go test ./syntax/
```