turbo-editors/turbo-core · syntax.md

⑂ main

syntax.md · 163 lines · 7.1 KBmarkdown    Blame HistoryRaw
   1# Reference: the syntax extension point
3> Neutral, exhaustive description of what `codeberg.org/turbo-editors/turbo-core/syntax` offers a language author.
5## Language
8type Language string
11The name a language is known by. It is a string because it is written down outside the package: a snippets file restricts itself to `languages = ["go"]`, and a language an editor registers is not something the library could have numbered in advance.
Constant | Value |
--- | --- |
`LanguageNone` | `""` |
`LanguageTOML` | `"toml"` |
`LanguageYAML` | `"yaml"` |
`LanguageMarkdown` | `"markdown"` |
`LanguageJavaScript` | `"javascript"` |
`LanguageHTML` | `"html"` |
`LanguageXML` | `"xml"` |
`LanguageDockerfile` | `"dockerfile"` |
`LanguageBash` | `"bash"` |
25`Language.String()` returns the name, and `"none"` for `LanguageNone`.
27## Definition
Field | Type | Description |
--- | --- | --- |
`Language` | `Language` | The name; the value `LanguageOf` returns for a file of this kind. |
`Extensions` | `[]string` | Extensions that identify it, with their dots, in lower case. |
`Filenames` | `[]string` | Whole file names that identify it, for files with no useful extension: `"Dockerfile"`. A file matches on its whole name **or** on the part before its first dot, ignoring case — so `"Dockerfile"` also answers for `Dockerfile.dev`. A name that is all extension, such as `.gitignore`, has an empty stem and matches nothing. |
`Shebangs` | `[]string` | Interpreter names that identify a file with no useful extension. |
`Highlight` | `func(string) [][]Span` | Colours a whole document. May be nil, which names a language without colouring it. |
37## Package functions
Function | Description |
--- | --- |
`Register(Definition)` | Adds a language. Registering one already known replaces it. Not safe from two goroutines at once. |
`Registered() []Language` | The languages that can be coloured, sorted by name. |
`LanguageOf(path, firstLine string) Language` | The language of a file: **extension**, then **name**, then **shebang**. `LanguageNone` when nothing claims it. |
`Highlight(l Language, src string) [][]Span` | One slice of spans per line of `src`, always exactly as many entries as there are lines. |
46## Span and Class
49type Span struct {
Start int // rune column, included
End   int // rune column, excluded
Class Class
56A span may never straddle a line break. Spans on a line must be in order and must not overlap.
Class | Used for |
--- | --- |
`ClassIdentifier` | the zero value: text nothing else claims |
`ClassKeyword` | a reserved word |
`ClassType` | a type name |
`ClassBuiltin` | something the language itself provides |
`ClassConstant` | a literal the language names |
`ClassFunction` | a function name, declared or called |
`ClassString` | a string literal |
`ClassChar` | a character literal |
`ClassNumber` | a numeric literal |
`ClassComment` | a comment |
`ClassOperator` | an operator |
`ClassPunctuation` | structure rather than computation |
`ClassHeading` | a Markdown heading |
`ClassTag` | an HTML element name |
`ClassAttribute` | an HTML attribute name, or a Rust attribute |
`ClassEmphasis` | Markdown bold or italic |
`ClassLink` | a Markdown link or image |
78`Class.StyleKey()` returns the theme key that colours it; an out-of-range class gives the identifier key. `Class.String()` returns the name, and `"unknown"` out of range.
80The set is closed. A language registered from outside colours itself with these and no others.
82## ScanLines
85func ScanLines[State any](src string, scan func(line []rune, carry State) ([]Span, State)) [][]Span
88Runs a per-line scanner over a document, threading whatever state crosses a line break. `State` starts at its zero value on the first line. A trailing `\r` is dropped from every line, so a CRLF file colours the same as an LF one.
90## LineScanner
93func NewLineScanner(line []rune) *LineScanner
Method | Description |
--- | --- |
`Spans() []Span` | The spans found so far. |
`Len() int` | How many runes the line holds. |
`Pos() int` | The current position, in runes. |
`AtEnd() bool` | Whether the whole line has been consumed. |
`Peek(offset int) rune` | The rune at an offset from the position, or `0` past either end. |
`Advance(n int)` | Move forward without colouring, stopping at the end of the line. |
`Emit(start, end int, class Class)` | Record a span. Empty spans are dropped. |
`Take(n int, class Class)` | Consume `n` runes and colour them. |
`TakeWhile(class Class, matches func(rune) bool) bool` | Consume while runes match; reports whether any were. |
`TakeRest(class Class)` | Consume and colour everything left on the line. |
`SkipSpaces()` | Step over spaces and tabs without colouring them. |
`HasPrefix(offset int, want string) bool` | Whether the line reads `want` at an offset from the position. |
111Because `Emit` drops empty spans, a scanner must pass a span's start in as a parameter rather than patch it onto the last span afterwards: the last one may not be the one it thinks.
113## Free helpers
Function | Description |
--- | --- |
`TakeQuoted(s, quote rune, class Class)` | A quoted string ending on this line, backslash escapes included. An unterminated one is coloured to the end of the line. |
`OpenBlockComment(s, opener, closer string, class Class) bool` | A block comment starting at the position; reports whether it also ended on this line. |
`FinishBlockComment(s, closer string, class Class) bool` | The continuation of one opened earlier; reports whether it ended here. |
`IsOperatorRune(r rune) bool` | One of `+-*/%=<>!&\|^~?:` |
`IsPunctuationRune(r rune) bool` | One of `()[]{},;.` |
`IsDigit(r rune) bool` | An ASCII digit. |
`IsLetter(r rune) bool` | An ASCII letter. |
`IsWordRune(r rune) bool` | A letter, a digit, or an underscore. |
126## LineIndex
128For a scanner that works in byte offsets rather than a line at a time.
131func NewLineIndex(src string) *LineIndex
Method | Description |
--- | --- |
`Count() int` | The number of lines, always at least one. |
`AppendSpans(out [][]Span, start, end int, class Class)` | Records the byte range `[start, end)` as one span per line it covers. `out` must already have `Count()` entries. |
139### Example
142lines := syntax.NewLineIndex(src)
143out := make([][]syntax.Span, lines.Count())
144lines.AppendSpans(out, token.Start, token.End, syntax.ClassKeyword)
147
149What the editor draws from: it re-scans only when the buffer's revision has moved.
Function or method | Description |
--- | --- |
`NewCache(Language) *Cache` | A cache for one language. `LanguageNone` gives a disabled one. |
`Language() Language` | The language in use. |
`SetLanguage(Language)` | Changes it and discards what was scanned. |
`Enabled() bool` | Whether the language is anything but `LanguageNone`. |
`Update(src string, revision int)` | Re-scans when the revision has moved, or on the first call. |
`Line(i int) []Span` | The spans of one line; `nil` out of range. |
`LineCount() int` | How many lines were scanned. |
161## See also
163- [Add a language](../how-to/add-a-language.md)

🛟 Updated. 28d5985 k33g 6h ago	1	# Reference: the syntax extension point
	2
	3	> Neutral, exhaustive description of what `codeberg.org/turbo-editors/turbo-core/syntax` offers a language author.
	4
	5	## Language
	6
	7	```go
	8	type Language string
	9	```
	10
	11	The name a language is known by. It is a string because it is written down outside the package: a snippets file restricts itself to `languages = ["go"]`, and a language an editor registers is not something the library could have numbered in advance.
	12
	13	\| Constant \| Value \|
	14	\| --- \| --- \|
	15	\| `LanguageNone` \| `""` \|
	16	\| `LanguageTOML` \| `"toml"` \|
	17	\| `LanguageYAML` \| `"yaml"` \|
	18	\| `LanguageMarkdown` \| `"markdown"` \|
	19	\| `LanguageJavaScript` \| `"javascript"` \|
	20	\| `LanguageHTML` \| `"html"` \|
	21	\| `LanguageXML` \| `"xml"` \|
	22	\| `LanguageDockerfile` \| `"dockerfile"` \|
	23	\| `LanguageBash` \| `"bash"` \|
	24
	25	`Language.String()` returns the name, and `"none"` for `LanguageNone`.
	26
	27	## Definition
	28
	29	\| Field \| Type \| Description \|
	30	\| --- \| --- \| --- \|
	31	\| `Language` \| `Language` \| The name; the value `LanguageOf` returns for a file of this kind. \|
	32	\| `Extensions` \| `[]string` \| Extensions that identify it, with their dots, in lower case. \|
	33	\| `Filenames` \| `[]string` \| Whole file names that identify it, for files with no useful extension: `"Dockerfile"`. A file matches on its whole name or on the part before its first dot, ignoring case — so `"Dockerfile"` also answers for `Dockerfile.dev`. A name that is all extension, such as `.gitignore`, has an empty stem and matches nothing. \|
	34	\| `Shebangs` \| `[]string` \| Interpreter names that identify a file with no useful extension. \|
	35	\| `Highlight` \| `func(string) [][]Span` \| Colours a whole document. May be nil, which names a language without colouring it. \|
	36
	37	## Package functions
	38
	39	\| Function \| Description \|
	40	\| --- \| --- \|
	41	\| `Register(Definition)` \| Adds a language. Registering one already known replaces it. Not safe from two goroutines at once. \|
	42	\| `Registered() []Language` \| The languages that can be coloured, sorted by name. \|
	43	\| `LanguageOf(path, firstLine string) Language` \| The language of a file: extension, then name, then shebang. `LanguageNone` when nothing claims it. \|
	44	\| `Highlight(l Language, src string) [][]Span` \| One slice of spans per line of `src`, always exactly as many entries as there are lines. \|
	45
	46	## Span and Class
	47
	48	```go
	49	type Span struct {
	50	Start int // rune column, included
	51	End int // rune column, excluded
	52	Class Class
	53	}
	54	```
	55
	56	A span may never straddle a line break. Spans on a line must be in order and must not overlap.
	57
	58	\| Class \| Used for \|
	59	\| --- \| --- \|
	60	\| `ClassIdentifier` \| the zero value: text nothing else claims \|
	61	\| `ClassKeyword` \| a reserved word \|
	62	\| `ClassType` \| a type name \|
	63	\| `ClassBuiltin` \| something the language itself provides \|
	64	\| `ClassConstant` \| a literal the language names \|
65	\| `ClassFunction` \| a function name, declared or called \|
66	\| `ClassString` \| a string literal \|
67	\| `ClassChar` \| a character literal \|
68	\| `ClassNumber` \| a numeric literal \|
69	\| `ClassComment` \| a comment \|
70	\| `ClassOperator` \| an operator \|
71	\| `ClassPunctuation` \| structure rather than computation \|
72	\| `ClassHeading` \| a Markdown heading \|
73	\| `ClassTag` \| an HTML element name \|
74	\| `ClassAttribute` \| an HTML attribute name, or a Rust attribute \|
75	\| `ClassEmphasis` \| Markdown bold or italic \|
76	\| `ClassLink` \| a Markdown link or image \|
77
78	`Class.StyleKey()` returns the theme key that colours it; an out-of-range class gives the identifier key. `Class.String()` returns the name, and `"unknown"` out of range.
79
80	The set is closed. A language registered from outside colours itself with these and no others.
81
82	## ScanLines
83
84	```go
85	func ScanLines[State any](src string, scan func(line []rune, carry State) ([]Span, State)) [][]Span
86	```
87
88	Runs a per-line scanner over a document, threading whatever state crosses a line break. `State` starts at its zero value on the first line. A trailing `\r` is dropped from every line, so a CRLF file colours the same as an LF one.
89
90	## LineScanner
91
92	```go
93	func NewLineScanner(line []rune) *LineScanner
94	```
95
96	\| Method \| Description \|
97	\| --- \| --- \|
98	\| `Spans() []Span` \| The spans found so far. \|
99	\| `Len() int` \| How many runes the line holds. \|
100	\| `Pos() int` \| The current position, in runes. \|
101	\| `AtEnd() bool` \| Whether the whole line has been consumed. \|
102	\| `Peek(offset int) rune` \| The rune at an offset from the position, or `0` past either end. \|
103	\| `Advance(n int)` \| Move forward without colouring, stopping at the end of the line. \|
104	\| `Emit(start, end int, class Class)` \| Record a span. Empty spans are dropped. \|
105	\| `Take(n int, class Class)` \| Consume `n` runes and colour them. \|
106	\| `TakeWhile(class Class, matches func(rune) bool) bool` \| Consume while runes match; reports whether any were. \|
107	\| `TakeRest(class Class)` \| Consume and colour everything left on the line. \|
108	\| `SkipSpaces()` \| Step over spaces and tabs without colouring them. \|
109	\| `HasPrefix(offset int, want string) bool` \| Whether the line reads `want` at an offset from the position. \|
110
111	Because `Emit` drops empty spans, a scanner must pass a span's start in as a parameter rather than patch it onto the last span afterwards: the last one may not be the one it thinks.
112
113	## Free helpers
114
115	\| Function \| Description \|
116	\| --- \| --- \|
117	\| `TakeQuoted(s, quote rune, class Class)` \| A quoted string ending on this line, backslash escapes included. An unterminated one is coloured to the end of the line. \|
118	\| `OpenBlockComment(s, opener, closer string, class Class) bool` \| A block comment starting at the position; reports whether it also ended on this line. \|
119	\| `FinishBlockComment(s, closer string, class Class) bool` \| The continuation of one opened earlier; reports whether it ended here. \|
120	\| `IsOperatorRune(r rune) bool` \| One of `+-*/%=<>!&\\|^~?:` \|
121	\| `IsPunctuationRune(r rune) bool` \| One of `()[]{},;.` \|
122	\| `IsDigit(r rune) bool` \| An ASCII digit. \|
123	\| `IsLetter(r rune) bool` \| An ASCII letter. \|
124	\| `IsWordRune(r rune) bool` \| A letter, a digit, or an underscore. \|
125
126	## LineIndex
127
128	For a scanner that works in byte offsets rather than a line at a time.
129
130	```go
131	func NewLineIndex(src string) *LineIndex
132	```
133
134	\| Method \| Description \|
135	\| --- \| --- \|
136	\| `Count() int` \| The number of lines, always at least one. \|
137	\| `AppendSpans(out [][]Span, start, end int, class Class)` \| Records the byte range `[start, end)` as one span per line it covers. `out` must already have `Count()` entries. \|
138
139	### Example
140
141	```go
142	lines := syntax.NewLineIndex(src)
143	out := make([][]syntax.Span, lines.Count())
144	lines.AppendSpans(out, token.Start, token.End, syntax.ClassKeyword)
145	```
146
147	## Cache
148
149	What the editor draws from: it re-scans only when the buffer's revision has moved.
150
151	\| Function or method \| Description \|
152	\| --- \| --- \|
153	\| `NewCache(Language) *Cache` \| A cache for one language. `LanguageNone` gives a disabled one. \|
154	\| `Language() Language` \| The language in use. \|
155	\| `SetLanguage(Language)` \| Changes it and discards what was scanned. \|
156	\| `Enabled() bool` \| Whether the language is anything but `LanguageNone`. \|
157	\| `Update(src string, revision int)` \| Re-scans when the revision has moved, or on the first call. \|
158	\| `Line(i int) []Span` \| The spans of one line; `nil` out of range. \|
159	\| `LineCount() int` \| How many lines were scanned. \|
160
161	## See also
162
163	- [Add a language](../how-to/add-a-language.md)