turbo-editors/turbo-core · README.md

⑂ v1.0.2

README.md · 132 lines · 9.6 KBmarkdown    Blame HistoryRaw
   1
3Colours source code, and is where an editor built on this library adds the language it is *for*.
5Eight languages are coloured here — TOML, Markdown, JavaScript, HTML, shell scripts, **YAML**, **XML** and **Dockerfiles** — because every editor meets them whatever it is for: a project's configuration is TOML or YAML, its documentation Markdown, its scripts shell, its container build a Dockerfile. The language that defines an editor is not: Turbo Go registers Go, Turbo Rust registers Rust, and neither is known here.
7A compose file is not a language of its own. `compose.yaml` and `docker-compose.yml` are YAML, and colouring `services:` differently from any other key would mean carrying Docker's schema here and watching it go stale.
9Each scanner is one file of ordinary Go. They share `LineScanner` — a line in runes, a position in it, and the spans found so far — and nothing else: there is no pattern language and no grammar format, so adding a language means writing ordinary Go beside the others rather than learning a notation. `LineScanner` and the helpers around it are **exported**, because that is what makes writing one outside this package possible at all.
11`Language` is a string, because it is written down outside this package: a snippets file restricts itself to `languages = ["go"]`. `LanguageOf(path, firstLine)` names a file by its **extension**, then by its **name**, then by a **shebang**; `LanguageNone` — anything no registered language claims — is coloured not at all rather than wrongly.
13The name step is what finds a `Dockerfile`, which has neither an extension nor a shebang. A `Definition` lists whole names in `Filenames`, and a file matches on its whole name or on the part before its first dot — so `Dockerfile` also answers for `Dockerfile.dev`.
15## The extension point
18syntax.Register(syntax.Definition{
19	Language:   "rust",
20	Extensions: []string{".rs"},
21	Highlight:  highlightRust,
25The registry is package-level state, which is the shape the standard library gives the same problem in `image.RegisterFormat`: an editor registers once at start-up, before it opens a file, and nothing ever removes an entry. Registering a language already known replaces it, so an editor can override one of the eight.
27The **classes are closed**. A language registered from outside colours itself with the seventeen `Class` values and no others, which is what lets one theme colour every language an editor will ever learn.
29## Every scanner guesses nothing
31Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet, and each of these was a decision:
33| Not recognised | Because |
34| --- | --- |
35| JavaScript regex literals | Telling `/x/g` from a division needs to know whether the previous token could end an expression; a wrong guess colours the rest of the line as a string |
36| Shell heredocs | Carrying an arbitrary delimiter across lines, plus `<<-` and quoted spellings, for a few lines of plain text |
37| JavaScript inside `<script>` | Following an element across lines and mapping another scanner's columns out — and the same argument then demands CSS |
38| The language of a Markdown fence | Every scanner would have to be reachable from every other, which is the general engine this package does not have |
40The exact boundary of each language is documented, in both languages, under `docs/*/reference/languages.md`.
42## Multi-line state
44Most constructs fit on a line. The ones that do not are carried by the scanner that owns them, through the `State` parameter of `scanLines`: a Markdown fence, a JavaScript block comment or template literal, an HTML comment, a YAML block scalar, an XML comment or CDATA section. Shell and Dockerfiles carry nothing, and say so with an empty struct.
46Two of those carry more than a flag, for the same reason: a single "something is open" boolean closes on whatever delimiter comes first. XML carries *which* construct is open, so a `-->` inside a CDATA section does not end it. YAML carries the indentation its block scalar started at, because a block scalar has no closing delimiter at all — only the first line indented less than the block ends it, and a blank line inside one stays inside it.
48Two traps live here, and both have bitten:
50- **`emit` drops empty spans**, so a scanner must pass a span's start *in* rather than patch it onto the last one afterwards — the last one may not be the one it thinks. This produced a corrupt span in the TOML scanner and a lost line in the JavaScript one.
51- **A helper that advances the position leaves nothing for `takeRest`.** `finishTemplate` looked correct and coloured nothing, because `runToBacktick` had already reached the end of the line.
53## Tolerance is the point
55The scanner runs in its most forgiving mode and every syntax error is discarded. Source under a cursor is broken most of the time it is being typed, and a highlighter that gives up on invalid input is a highlighter that flickers off. An unterminated string comes back as a string running to the end of the line, an unclosed `/*` as a comment running to the end of the file — which is exactly what keeps the colours steady mid-keystroke.
59`Highlight(language, src)` returns `[][]Span` — **one entry per line**, always, including for `LanguageNone`, so the editor can index it by line number without a bounds check.
62type Span struct {
63    Start int    // rune column, included
64    End   int    // rune column, excluded
65    Class Class
69Columns are **runes**, not bytes, so an accented comment does not push the spans of the following lines out of alignment. A span never straddles a line break: a three-line block comment becomes three spans, one per line. Text no span covers — whitespace, mostly — is drawn in the editor's plain text style.
71
73`identifier` · `keyword` · `type` · `builtin` · `constant` · `function` · `string` · `char` · `number` · `comment` · `operator` · `punctuation` · `heading` · `tag` · `attribute` · `emphasis` · `link`
75Deliberately coarser than any one language's token set: the editor needs to tell a keyword from a string, not an `ADD` from a `SUB`. Two of them are decided from context rather than from the token alone, in every scanner that has the notion:
77- a name before `(` is a **function**;
78- brackets, commas, dots and semicolons are **punctuation**, split out from the operators that actually compute, so a theme can quiet them down.
80Names the language itself provides — `true`, `nil`, `None`, `i32`, `len` — are recognised by name, by the scanner that knows them.
82`Class.StyleKey()` gives the `theme` key that colours it — the one place a token class and its colour are tied together, which is why this package depends on `theme`.
84**TOML adds no classes**, and neither do JavaScript or shell: a table header names a structure so it is a `type`, `true` is a `constant`, a name before `(` is a `function`.
86**The markup languages needed five.** `heading`, `tag`, `attribute`, `emphasis` and `link` have no equivalent in a programming language — a heading is not a keyword, and a theme wanting quiet headings with loud keywords could not say so otherwise. The cost falls on themes written elsewhere: one that sets none of them falls back along the dots to `syntax` and then `default`, so Markdown stays readable but undifferentiated. Every shipped theme sets all five and a test fails if one stops.
90The editor redraws far more often than the text changes — every cursor move, every scroll. `Cache` re-scans only when the buffer's revision counter has moved:
93cache := syntax.NewCache(syntax.LanguageOf(path, buf.Line(0)))
94cache.Update(buf.Text(), buf.Revision())
95for line := top; line < bottom; line++ {
96    draw(line, cache.Line(line))
100`NewCache(LanguageNone)`, or `SetLanguage(LanguageNone)`, gives a cache that colours nothing — what a file this package does not understand gets, without the editor needing a branch for it.
102## Public API
104| Function | What it does |
105| --- | --- |
106| `Register(Definition)` | Teaches this package a language |
107| `Definition{Language, Extensions, Filenames, Shebangs, Highlight}` | How one language is recognised and coloured |
108| `Registered() []Language` | What it can colour, sorted |
109| `Highlight(l Language, src string) [][]Span` | Scans a whole source text, one span slice per line |
110| `LanguageOf(path, firstLine string) Language` | A file's language by extension, then by name, then by shebang |
111| `NewCache(l Language) *Cache` | A cache that re-scans only on a revision change |
112| `(*Cache) SetLanguage(l Language)` | Change what is coloured, as Save As does |
113| `(*Cache) Language() Language` | What it colours |
114| `(*Cache) Update(src string, revision int)` | Re-scan if the revision moved |
115| `(*Cache) Line(i int) []Span` | Spans of one line from the last scan |
116| `(*Cache) LineCount() int` | Lines covered by the last scan |
117| `(*Cache) Enabled() bool` | Whether it colours anything |
118| `(Class) StyleKey() string` | The theme key that colours this class |
119| `ScanLines(src, scan)` | Runs a per-line scanner over a document, threading its state |
120| `NewLineScanner(line) *LineScanner` | The toolkit a hand-written scanner is built on |
121| `NewLineIndex(src) *LineIndex` | Byte offsets into per-line spans, for a scanner built on a tokeniser |
122| `TakeQuoted`, `OpenBlockComment`, `FinishBlockComment` | The constructs every language shares |
123| `IsOperatorRune`, `IsPunctuationRune`, `IsDigit`, `IsLetter`, `IsWordRune` | Rune predicates the scanners here use |
125The full list is in [the syntax reference](../docs/en/reference/syntax.md).
127
130
131go test ./syntax/

🛟 Updated. 28d5985 k33g 21h ago	1	# syntax
	2
	3	Colours source code, and is where an editor built on this library adds the language it is for.
	4
	5	Eight languages are coloured here — TOML, Markdown, JavaScript, HTML, shell scripts, YAML, XML and Dockerfiles — because every editor meets them whatever it is for: a project's configuration is TOML or YAML, its documentation Markdown, its scripts shell, its container build a Dockerfile. The language that defines an editor is not: Turbo Go registers Go, Turbo Rust registers Rust, and neither is known here.
	6
	7	A compose file is not a language of its own. `compose.yaml` and `docker-compose.yml` are YAML, and colouring `services:` differently from any other key would mean carrying Docker's schema here and watching it go stale.
	8
	9	Each scanner is one file of ordinary Go. They share `LineScanner` — a line in runes, a position in it, and the spans found so far — and nothing else: there is no pattern language and no grammar format, so adding a language means writing ordinary Go beside the others rather than learning a notation. `LineScanner` and the helpers around it are exported, because that is what makes writing one outside this package possible at all.
	10
	11	`Language` is a string, because it is written down outside this package: a snippets file restricts itself to `languages = ["go"]`. `LanguageOf(path, firstLine)` names a file by its extension, then by its name, then by a shebang; `LanguageNone` — anything no registered language claims — is coloured not at all rather than wrongly.
	12
	13	The name step is what finds a `Dockerfile`, which has neither an extension nor a shebang. A `Definition` lists whole names in `Filenames`, and a file matches on its whole name or on the part before its first dot — so `Dockerfile` also answers for `Dockerfile.dev`.
	14
	15	## The extension point
	16
	17	```go
	18	syntax.Register(syntax.Definition{
	19	Language: "rust",
	20	Extensions: []string{".rs"},
	21	Highlight: highlightRust,
	22	})
	23	```
	24
	25	The registry is package-level state, which is the shape the standard library gives the same problem in `image.RegisterFormat`: an editor registers once at start-up, before it opens a file, and nothing ever removes an entry. Registering a language already known replaces it, so an editor can override one of the eight.
	26
	27	The classes are closed. A language registered from outside colours itself with the seventeen `Class` values and no others, which is what lets one theme colour every language an editor will ever learn.
	28
	29	## Every scanner guesses nothing
	30
	31	Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet, and each of these was a decision:
	32
	33	\| Not recognised \| Because \|
	34	\| --- \| --- \|
	35	\| JavaScript regex literals \| Telling `/x/g` from a division needs to know whether the previous token could end an expression; a wrong guess colours the rest of the line as a string \|
	36	\| Shell heredocs \| Carrying an arbitrary delimiter across lines, plus `<<-` and quoted spellings, for a few lines of plain text \|
	37	\| JavaScript inside `<script>` \| Following an element across lines and mapping another scanner's columns out — and the same argument then demands CSS \|
	38	\| The language of a Markdown fence \| Every scanner would have to be reachable from every other, which is the general engine this package does not have \|
	39
	40	The exact boundary of each language is documented, in both languages, under `docs/*/reference/languages.md`.
	41
	42	## Multi-line state
	43
	44	Most constructs fit on a line. The ones that do not are carried by the scanner that owns them, through the `State` parameter of `scanLines`: a Markdown fence, a JavaScript block comment or template literal, an HTML comment, a YAML block scalar, an XML comment or CDATA section. Shell and Dockerfiles carry nothing, and say so with an empty struct.
	45
	46	Two of those carry more than a flag, for the same reason: a single "something is open" boolean closes on whatever delimiter comes first. XML carries which construct is open, so a `-->` inside a CDATA section does not end it. YAML carries the indentation its block scalar started at, because a block scalar has no closing delimiter at all — only the first line indented less than the block ends it, and a blank line inside one stays inside it.
	47
	48	Two traps live here, and both have bitten:
	49
	50	- `emit` drops empty spans, so a scanner must pass a span's start in rather than patch it onto the last one afterwards — the last one may not be the one it thinks. This produced a corrupt span in the TOML scanner and a lost line in the JavaScript one.
	51	- A helper that advances the position leaves nothing for `takeRest`. `finishTemplate` looked correct and coloured nothing, because `runToBacktick` had already reached the end of the line.
	52
	53	## Tolerance is the point
	54
	55	The scanner runs in its most forgiving mode and every syntax error is discarded. Source under a cursor is broken most of the time it is being typed, and a highlighter that gives up on invalid input is a highlighter that flickers off. An unterminated string comes back as a string running to the end of the line, an unclosed `/*` as a comment running to the end of the file — which is exactly what keeps the colours steady mid-keystroke.
	56
	57	## Model
	58
	59	`Highlight(language, src)` returns `[][]Span` — one entry per line, always, including for `LanguageNone`, so the editor can index it by line number without a bounds check.
	60
	61	```go
	62	type Span struct {
	63	Start int // rune column, included
	64	End int // rune column, excluded
65	Class Class
66	}
67	```
68
69	Columns are runes, not bytes, so an accented comment does not push the spans of the following lines out of alignment. A span never straddles a line break: a three-line block comment becomes three spans, one per line. Text no span covers — whitespace, mostly — is drawn in the editor's plain text style.
70
71	## Classes
72
73	`identifier` · `keyword` · `type` · `builtin` · `constant` · `function` · `string` · `char` · `number` · `comment` · `operator` · `punctuation` · `heading` · `tag` · `attribute` · `emphasis` · `link`
74
75	Deliberately coarser than any one language's token set: the editor needs to tell a keyword from a string, not an `ADD` from a `SUB`. Two of them are decided from context rather than from the token alone, in every scanner that has the notion:
76
77	- a name before `(` is a function;
78	- brackets, commas, dots and semicolons are punctuation, split out from the operators that actually compute, so a theme can quiet them down.
79
80	Names the language itself provides — `true`, `nil`, `None`, `i32`, `len` — are recognised by name, by the scanner that knows them.
81
82	`Class.StyleKey()` gives the `theme` key that colours it — the one place a token class and its colour are tied together, which is why this package depends on `theme`.
83
84	TOML adds no classes, and neither do JavaScript or shell: a table header names a structure so it is a `type`, `true` is a `constant`, a name before `(` is a `function`.
85
86	The markup languages needed five. `heading`, `tag`, `attribute`, `emphasis` and `link` have no equivalent in a programming language — a heading is not a keyword, and a theme wanting quiet headings with loud keywords could not say so otherwise. The cost falls on themes written elsewhere: one that sets none of them falls back along the dots to `syntax` and then `default`, so Markdown stays readable but undifferentiated. Every shipped theme sets all five and a test fails if one stops.
87
88	## Cache
89
90	The editor redraws far more often than the text changes — every cursor move, every scroll. `Cache` re-scans only when the buffer's revision counter has moved:
91
92	```go
93	cache := syntax.NewCache(syntax.LanguageOf(path, buf.Line(0)))
94	cache.Update(buf.Text(), buf.Revision())
95	for line := top; line < bottom; line++ {
96	draw(line, cache.Line(line))
97	}
98	```
99
100	`NewCache(LanguageNone)`, or `SetLanguage(LanguageNone)`, gives a cache that colours nothing — what a file this package does not understand gets, without the editor needing a branch for it.
101
102	## Public API
103
104	\| Function \| What it does \|
105	\| --- \| --- \|
106	\| `Register(Definition)` \| Teaches this package a language \|
107	\| `Definition{Language, Extensions, Filenames, Shebangs, Highlight}` \| How one language is recognised and coloured \|
108	\| `Registered() []Language` \| What it can colour, sorted \|
109	\| `Highlight(l Language, src string) [][]Span` \| Scans a whole source text, one span slice per line \|
110	\| `LanguageOf(path, firstLine string) Language` \| A file's language by extension, then by name, then by shebang \|
111	\| `NewCache(l Language) *Cache` \| A cache that re-scans only on a revision change \|
112	\| `(*Cache) SetLanguage(l Language)` \| Change what is coloured, as Save As does \|
113	\| `(*Cache) Language() Language` \| What it colours \|
114	\| `(*Cache) Update(src string, revision int)` \| Re-scan if the revision moved \|
115	\| `(*Cache) Line(i int) []Span` \| Spans of one line from the last scan \|
116	\| `(*Cache) LineCount() int` \| Lines covered by the last scan \|
117	\| `(*Cache) Enabled() bool` \| Whether it colours anything \|
118	\| `(Class) StyleKey() string` \| The theme key that colours this class \|
119	\| `ScanLines(src, scan)` \| Runs a per-line scanner over a document, threading its state \|
120	\| `NewLineScanner(line) *LineScanner` \| The toolkit a hand-written scanner is built on \|
121	\| `NewLineIndex(src) *LineIndex` \| Byte offsets into per-line spans, for a scanner built on a tokeniser \|
122	\| `TakeQuoted`, `OpenBlockComment`, `FinishBlockComment` \| The constructs every language shares \|
123	\| `IsOperatorRune`, `IsPunctuationRune`, `IsDigit`, `IsLetter`, `IsWordRune` \| Rune predicates the scanners here use \|
124
125	The full list is in [the syntax reference](../docs/en/reference/syntax.md).
126
127	## Tests
128
129	```sh
130	make test
131	go test ./syntax/
132	```