Handoff — 2026-08-31 — Markdown, JavaScript, HTML and shell colouring

State

Tickets 0009, 0013 and 0014 are done and green, on branch feature/new-syntaxes, uncommitted. Shell was asked for in the same breath and has no ticket.

The editor now colours six languages. Recognition is by extension, falling back to a shebang for a shell script with no extension.

New scanners: markdown.go + markdown_inline.go, javascript.go, html.go, bash.go. A shared scanner.go holds lineScanner, scanLines and the helpers; the TOML scanner was ported onto it and its 24 tests stayed green throughout.
Five new classes and theme keys — heading, tag, attribute, emphasis, link — set in all three shipped themes.
internal/syntax at 96.0 % with 180 tests. Whole suite green under -race. Quality gate PASS after one round: 0/0/0, complexity 1389.
Docs: a new reference/languages.md per language stating each scanner's exact boundary, plus a rewritten section in explanation/colouring-and-completion.md.
Verified in a real terminal for all four new languages by rendering the editor through the project's own VT emulator and reading back each run's foreground colour.

Earlier the same day, PR #4 merged the project tree.

In flight

Nothing. Finished through Phase 8.

Next steps

Commit and open the PR. feature/new-syntaxes is the branch.
Look at a long real file in each language. The verification used four short files. A 500-line HTML page or a big shell script is where a scanner that is quadratic or that mis-carries state would show.
Tickets. 0002, 0003, 0007, 0009, 0013 and 0014 are all implemented and all still state: open.

Open questions / blockers

The stated omissions are choices, not gaps, and each is documented in docs/*/reference/languages.md: JavaScript regex literals, shell heredocs, JavaScript inside <script>, and the language of a Markdown fence. Any of them can be added later; all four need something the current design deliberately lacks — either a token of look-behind, or one scanner reaching into another.
Third-party themes lose the five new keys. They fall back to default, so Markdown is readable but its headings, emphasis and links are not distinct. Nothing warns about it. If that matters, a "theme is missing keys" report would be a small feature.
CSS is not coloured, and <style> therefore looks like text. It was not asked for; it would be the natural seventh.

Watch out for

emit drops empty spans, so never derive a span's start from the scanner's position after a helper has moved it. This has now bitten twice, in different shapes: the TOML scanner patched spans[len-1].Start and corrupted the previous span; finishTemplate called runToBacktick, which ran the position to the end of the line, and then asked takeRest to colour what was left — nothing. Pass the start in as a parameter. takeTemplateFrom and finishMultilineFrom are the shapes to copy.
The theme completeness test only constrains turbo-classic. turbo-dark and borland-light both inherit from it, and inheritance is resolved at parse time into the theme's own map, so Defines returns true for an inherited key. I removed syntax.heading from turbo-dark to check the test bit, and it passed. Delete the key from turbo-classic to actually test it.
Colour clashes are invisible to the test suite. syntax.link was set to the same lime as syntax.string in turbo-classic, making a Markdown link identical to an inline code span. Everything passed. Render the editor through internal/terminal and read back the foreground of each run — the throwaway program under /tmp/render/main.go in that session did it, and rebuilding it is a few minutes.
A hyphen must be able to start a word in shell, or every -euo is a minus followed by a command. startsAnOption handles it; a "simplification" that drops it breaks every script.
Test names collide across scanner files. TestAnUnterminatedStringIsColouredToTheEndOfTheLine and TestABackslashDoesNotEscapeInALiteralString both existed for TOML already. Prefix new ones with the language.
splitLines trims a trailing \r, so a CRLF file colours the same as an LF one. Do not replace it with strings.Split(src, "\n").

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


# Handoff — 2026-08-31 — Markdown, JavaScript, HTML and shell colouring

## State

**Tickets 0009, 0013 and 0014 are done and green**, on branch `feature/new-syntaxes`, **uncommitted**. Shell was asked for in the same breath and has no ticket.

The editor now colours six languages. Recognition is by extension, falling back to a shebang for a shell script with no extension.

- New scanners: `markdown.go` + `markdown_inline.go`, `javascript.go`, `html.go`, `bash.go`. A shared `scanner.go` holds `lineScanner`, `scanLines` and the helpers; **the TOML scanner was ported onto it** and its 24 tests stayed green throughout.
- Five new classes and theme keys — `heading`, `tag`, `attribute`, `emphasis`, `link` — set in all three shipped themes.
- `internal/syntax` at 96.0 % with 180 tests. Whole suite green under `-race`. Quality gate **PASS** after one round: 0/0/0, complexity 1389.
- Docs: a new `reference/languages.md` per language stating each scanner's exact boundary, plus a rewritten section in `explanation/colouring-and-completion.md`.
- **Verified in a real terminal** for all four new languages by rendering the editor through the project's own VT emulator and reading back each run's foreground colour.

Earlier the same day, PR #4 merged the project tree.

## In flight

Nothing. Finished through Phase 8.

## Next steps

1. **Commit and open the PR.** `feature/new-syntaxes` is the branch.
2. **Look at a long real file in each language.** The verification used four short files. A 500-line HTML page or a big shell script is where a scanner that is quadratic or that mis-carries state would show.
3. **Tickets.** 0002, 0003, 0007, 0009, 0013 and 0014 are all implemented and all still `state: open`.

## Open questions / blockers

- **The stated omissions are choices, not gaps**, and each is documented in `docs/*/reference/languages.md`: JavaScript regex literals, shell heredocs, JavaScript inside `<script>`, and the language of a Markdown fence. Any of them can be added later; all four need something the current design deliberately lacks — either a token of look-behind, or one scanner reaching into another.
- **Third-party themes lose the five new keys.** They fall back to `default`, so Markdown is readable but its headings, emphasis and links are not distinct. Nothing warns about it. If that matters, a "theme is missing keys" report would be a small feature.
- **CSS is not coloured**, and `<style>` therefore looks like text. It was not asked for; it would be the natural seventh.

## Watch out for

- **`emit` drops empty spans, so never derive a span's start from the scanner's position after a helper has moved it.** This has now bitten twice, in different shapes: the TOML scanner patched `spans[len-1].Start` and corrupted the *previous* span; `finishTemplate` called `runToBacktick`, which ran the position to the end of the line, and then asked `takeRest` to colour what was left — nothing. Pass the start in as a parameter. `takeTemplateFrom` and `finishMultilineFrom` are the shapes to copy.
- **The theme completeness test only constrains `turbo-classic`.** `turbo-dark` and `borland-light` both inherit from it, and inheritance is resolved at parse time into the theme's own map, so `Defines` returns true for an inherited key. I removed `syntax.heading` from `turbo-dark` to check the test bit, and it passed. Delete the key from `turbo-classic` to actually test it.
- **Colour clashes are invisible to the test suite.** `syntax.link` was set to the same lime as `syntax.string` in `turbo-classic`, making a Markdown link identical to an inline `code` span. Everything passed. Render the editor through `internal/terminal` and read back the foreground of each run — the throwaway program under `/tmp/render/main.go` in that session did it, and rebuilding it is a few minutes.
- **A hyphen must be able to start a word in shell**, or every `-euo` is a minus followed by a command. `startsAnOption` handles it; a "simplification" that drops it breaks every script.
- **Test names collide across scanner files.** `TestAnUnterminatedStringIsColouredToTheEndOfTheLine` and `TestABackslashDoesNotEscapeInALiteralString` both existed for TOML already. Prefix new ones with the language.
- **`splitLines` trims a trailing `\r`**, so a CRLF file colours the same as an LF one. Do not replace it with `strings.Split(src, "\n")`.