turbo-editors/turbo-go · 2026-08-31-new-syntaxes.md

⑂ main

2026-08-31-new-syntaxes.md · 40 lines · 4.3 KBmarkdown    Blame HistoryRaw
   1# Handoff — 2026-08-31 — Markdown, JavaScript, HTML and shell colouring
5**Tickets 0009, 0013 and 0014 are done and green**, on branch `feature/new-syntaxes`, **uncommitted**. Shell was asked for in the same breath and has no ticket.
7The editor now colours six languages. Recognition is by extension, falling back to a shebang for a shell script with no extension.
9- New scanners: `markdown.go` + `markdown_inline.go`, `javascript.go`, `html.go`, `bash.go`. A shared `scanner.go` holds `lineScanner`, `scanLines` and the helpers; **the TOML scanner was ported onto it** and its 24 tests stayed green throughout.
10- Five new classes and theme keys — `heading`, `tag`, `attribute`, `emphasis`, `link` — set in all three shipped themes.
11- `internal/syntax` at 96.0 % with 180 tests. Whole suite green under `-race`. Quality gate **PASS** after one round: 0/0/0, complexity 1389.
12- Docs: a new `reference/languages.md` per language stating each scanner's exact boundary, plus a rewritten section in `explanation/colouring-and-completion.md`.
13- **Verified in a real terminal** for all four new languages by rendering the editor through the project's own VT emulator and reading back each run's foreground colour.
15Earlier the same day, PR #4 merged the project tree.
17## In flight
19Nothing. Finished through Phase 8.
21## Next steps
231. **Commit and open the PR.** `feature/new-syntaxes` is the branch.
242. **Look at a long real file in each language.** The verification used four short files. A 500-line HTML page or a big shell script is where a scanner that is quadratic or that mis-carries state would show.
253. **Tickets.** 0002, 0003, 0007, 0009, 0013 and 0014 are all implemented and all still `state: open`.
27## Open questions / blockers
29- **The stated omissions are choices, not gaps**, and each is documented in `docs/*/reference/languages.md`: JavaScript regex literals, shell heredocs, JavaScript inside `<script>`, and the language of a Markdown fence. Any of them can be added later; all four need something the current design deliberately lacks — either a token of look-behind, or one scanner reaching into another.
30- **Third-party themes lose the five new keys.** They fall back to `default`, so Markdown is readable but its headings, emphasis and links are not distinct. Nothing warns about it. If that matters, a "theme is missing keys" report would be a small feature.
31- **CSS is not coloured**, and `<style>` therefore looks like text. It was not asked for; it would be the natural seventh.
33## Watch out for
35- **`emit` drops empty spans, so never derive a span's start from the scanner's position after a helper has moved it.** This has now bitten twice, in different shapes: the TOML scanner patched `spans[len-1].Start` and corrupted the *previous* span; `finishTemplate` called `runToBacktick`, which ran the position to the end of the line, and then asked `takeRest` to colour what was left — nothing. Pass the start in as a parameter. `takeTemplateFrom` and `finishMultilineFrom` are the shapes to copy.
36- **The theme completeness test only constrains `turbo-classic`.** `turbo-dark` and `borland-light` both inherit from it, and inheritance is resolved at parse time into the theme's own map, so `Defines` returns true for an inherited key. I removed `syntax.heading` from `turbo-dark` to check the test bit, and it passed. Delete the key from `turbo-classic` to actually test it.
37- **Colour clashes are invisible to the test suite.** `syntax.link` was set to the same lime as `syntax.string` in `turbo-classic`, making a Markdown link identical to an inline `code` span. Everything passed. Render the editor through `internal/terminal` and read back the foreground of each run — the throwaway program under `/tmp/render/main.go` in that session did it, and rebuilding it is a few minutes.
38- **A hyphen must be able to start a word in shell**, or every `-euo` is a minus followed by a command. `startsAnOption` handles it; a "simplification" that drops it breaks every script.
39- **Test names collide across scanner files.** `TestAnUnterminatedStringIsColouredToTheEndOfTheLine` and `TestABackslashDoesNotEscapeInALiteralString` both existed for TOML already. Prefix new ones with the language.
40- **`splitLines` trims a trailing `\r`**, so a CRLF file colours the same as an LF one. Do not replace it with `strings.Split(src, "\n")`.

📦 Turbo Go 3d7798b k33g 11h ago	1	# Handoff — 2026-08-31 — Markdown, JavaScript, HTML and shell colouring
	2
	3	## State
	4
	5	Tickets 0009, 0013 and 0014 are done and green, on branch `feature/new-syntaxes`, uncommitted. Shell was asked for in the same breath and has no ticket.
	6
	7	The editor now colours six languages. Recognition is by extension, falling back to a shebang for a shell script with no extension.
	8
	9	- New scanners: `markdown.go` + `markdown_inline.go`, `javascript.go`, `html.go`, `bash.go`. A shared `scanner.go` holds `lineScanner`, `scanLines` and the helpers; the TOML scanner was ported onto it and its 24 tests stayed green throughout.
	10	- Five new classes and theme keys — `heading`, `tag`, `attribute`, `emphasis`, `link` — set in all three shipped themes.
	11	- `internal/syntax` at 96.0 % with 180 tests. Whole suite green under `-race`. Quality gate PASS after one round: 0/0/0, complexity 1389.
	12	- Docs: a new `reference/languages.md` per language stating each scanner's exact boundary, plus a rewritten section in `explanation/colouring-and-completion.md`.
	13	- Verified in a real terminal for all four new languages by rendering the editor through the project's own VT emulator and reading back each run's foreground colour.
	14
	15	Earlier the same day, PR #4 merged the project tree.
	16
	17	## In flight
	18
	19	Nothing. Finished through Phase 8.
	20
	21	## Next steps
	22
	23	1. Commit and open the PR. `feature/new-syntaxes` is the branch.
	24	2. Look at a long real file in each language. The verification used four short files. A 500-line HTML page or a big shell script is where a scanner that is quadratic or that mis-carries state would show.
	25	3. Tickets. 0002, 0003, 0007, 0009, 0013 and 0014 are all implemented and all still `state: open`.
	26
	27	## Open questions / blockers
	28
	29	- The stated omissions are choices, not gaps, and each is documented in `docs/*/reference/languages.md`: JavaScript regex literals, shell heredocs, JavaScript inside `<script>`, and the language of a Markdown fence. Any of them can be added later; all four need something the current design deliberately lacks — either a token of look-behind, or one scanner reaching into another.
	30	- Third-party themes lose the five new keys. They fall back to `default`, so Markdown is readable but its headings, emphasis and links are not distinct. Nothing warns about it. If that matters, a "theme is missing keys" report would be a small feature.
	31	- CSS is not coloured, and `<style>` therefore looks like text. It was not asked for; it would be the natural seventh.
	32
	33	## Watch out for
	34
	35	- `emit` drops empty spans, so never derive a span's start from the scanner's position after a helper has moved it. This has now bitten twice, in different shapes: the TOML scanner patched `spans[len-1].Start` and corrupted the previous span; `finishTemplate` called `runToBacktick`, which ran the position to the end of the line, and then asked `takeRest` to colour what was left — nothing. Pass the start in as a parameter. `takeTemplateFrom` and `finishMultilineFrom` are the shapes to copy.
	36	- The theme completeness test only constrains `turbo-classic`. `turbo-dark` and `borland-light` both inherit from it, and inheritance is resolved at parse time into the theme's own map, so `Defines` returns true for an inherited key. I removed `syntax.heading` from `turbo-dark` to check the test bit, and it passed. Delete the key from `turbo-classic` to actually test it.
	37	- Colour clashes are invisible to the test suite. `syntax.link` was set to the same lime as `syntax.string` in `turbo-classic`, making a Markdown link identical to an inline `code` span. Everything passed. Render the editor through `internal/terminal` and read back the foreground of each run — the throwaway program under `/tmp/render/main.go` in that session did it, and rebuilding it is a few minutes.
	38	- A hyphen must be able to start a word in shell, or every `-euo` is a minus followed by a command. `startsAnOption` handles it; a "simplification" that drops it breaks every script.
	39	- Test names collide across scanner files. `TestAnUnterminatedStringIsColouredToTheEndOfTheLine` and `TestABackslashDoesNotEscapeInALiteralString` both existed for TOML already. Prefix new ones with the language.
	40	- `splitLines` trims a trailing `\r`, so a CRLF file colours the same as an LF one. Do not replace it with `strings.Split(src, "\n")`.