turbo-editors/turbo-js · architecture.md

⑂ v1.0.0

architecture.md · 118 lines · 12.2 KBmarkdown    Blame HistoryRaw
   1# Architecture — explanation
3## What is this about?
5Turbo JS is a command, a profile and two scanners. Everything else — the editing widget, the windows, the menus, the dialogs, the themes, the terminal emulator, the file tree, the LSP client, the agent windows — is [turbo-core](https://rickub.com/turbo-editors/turbo-core), the library every Turbo editor is built on.
7This page is about that split: what is here, what is there, and why the line falls where it does.
9## What is in this repository
12main.go                  flags, the terminal, and the wiring
13internal/jslang          the whole of what makes this Turbo JS
14  jslang.go              the profile: name, menu, server, where npm puts a global binary
15  scan.go                the JavaScript scanner's dispatcher, comments, regular expressions, what crosses a line
16  literals.go            template literals and numbers
17  words.go               keywords, constants, the globals, the naming conventions
18  json.go                the JSON scanner
19  templates.go           four //go:embed declarations
20  *.toml.tmpl            the four starter files a project gets, embedded
23About twelve hundred lines counting the comments, of which some six hundred are the two scanners — under seven hundred lines of code by qlty's count. There is no `internal/app`, no `internal/ui`, no `internal/buffer` — those exist once, in the library, and all six editors use them unchanged.
25## What `main` does
27Six things, in this order:
291. Parses the flags.
302. Calls `jslang.Register()`, which teaches the library this editor's JavaScript — replacing the scanner the library ships for it — and JSON.
313. Builds `jslang.Profile()` — the value that says this editor is Turbo JS.
324. Reads `.turbo-js/settings.toml` from the working directory, if there is one.
335. Opens the terminal and hands the screen, the theme name and the profile to `app.New`.
346. Starts `typescript-language-server` in the project root — the nearest directory at or above the file being edited that holds a `package.json` — and runs the event loop.
36That is the whole command. Every decision it makes — which theme wins, which files to open, whether to start a language server — is about *this run*, not about JavaScript.
38## The profile is the seam
41profile.Profile{
42	Name:        "Turbo JS",
43	Slug:        "turbo-js",
44	Language:    "JavaScript",
45	ToolsMenu:   "~J~avaScript",
46	RootMarkers: []string{"package.json"},
47	Server:      profile.Server{Command: "typescript-language-server", Args: []string{"--stdio"}, …},
48	Templates:   profile.Templates{Settings: …, Snippets: …, Tools: …, Agents: …},
52Everything that would otherwise be a hardcoded `"turbo-js"`, `"node"` or `".js"` somewhere in eleven thousand lines is one field here. The library reads them; nothing in the library knows what any of them mean.
54`Slug` carries more than it looks. The binary is `turbo-js`, the project directory is `.turbo-js`, the user's own configuration lives in `~/.config/turbo-js`, and the environment variables that override it are `TURBO_JS_THEME_DIR` and `TURBO_JS_SNIPPET_DIR` — all derived from that one word.
56`RootMarkers` holds one file, `package.json`. It is a Node project's boundary — its name, its dependencies, its scripts — and in a workspace holding several packages the nearest one going up is the package being edited, which is the root the server should resolve imports from. The library's `ProjectRoot` does the walk; the profile only says what to look for. A directory with no `package.json` anywhere above it gets the working directory, and `typescript-language-server` infers a project from the files it is shown.
58## Why the JavaScript scanner is here when the library has one
60turbo-core colours eight languages itself: TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell. Those are the ones every editor meets whatever it is for — a project's configuration is TOML or YAML, its documentation is Markdown, its scripts are shell, its image build a Dockerfile, and a web page or a README's code fence is JavaScript.
62So JavaScript is the one language in this family that the library colours *and* an editor is for. The library's scanner is written for the Rust editor's README: it knows the keywords, the strings, the comments and the numbers, and it deliberately refuses regular expressions, because telling `/x/g` from a division needs the previous token and a wrong guess there colours the rest of a line as a string — a bad trade for a language you only meet in passing. An editor *for* JavaScript makes the opposite trade: regular expressions are on every other line of a Node program, and the guess can be bounded. It also owes its user Node's globals, the hashbang line, the name after `function`, private names and decorators.
64The library allows exactly this. `syntax.Register` replaces a language registered under the same name, and the later registration wins because it is the more specific statement. Turbo JS registers under the library's own name, `javascript`, so a snippets file saying `languages = ["javascript"]` and a ```js fence in an agent window both reach this scanner, and nothing in the library changed. The [colouring page](colouring-and-completion.md) says what the replacement adds and what it still refuses.
66## Why JSON is here too
68JSON is not one of the library's eight, and a Node project cannot be edited without it: `package.json` is the manifest and the root marker, `package-lock.json` is beside it, `tsconfig.json` and `.eslintrc.json` are common neighbours. So the editor that is for Node registers a JSON scanner, thirty lines long, that tells a key from a value and tolerates the comments `tsconfig.json` has.
70That could have gone into the library instead — a Go editor meets a `package.json` too, in a repository with a front end. It was not, for the reason that keeps the Go and Rust scanners out: the library grows a language every time somebody wants one, and "what does this editor register?" stops being the first question about a new one. If turbo-core learns JSON one day, this registration will still win by ordering, and the one here can then be removed.
72## Why the toolchain menu is `~J~avaScript` and not `~N~ode` or `npm`
74The hot key was the easy part. Ten letters are taken by the fixed menus — F, E, S, R, C, O, W, N, H and, since the Agent menu, A — which rules out the S, the C and the R of JavaScript, and the N of Node, but not J, so the hot key lands on the first letter of the word, which costs nobody a second glance. Turbo Rust had no such luck and ended up on `Rus~t~`.
76The name was the real decision, and it went the same way every sibling's did. The menu holds whatever the project put in its tools file, and that is not always npm: the starter file already runs `node`, `npm` and `npx`, and the first tools file anybody writes outgrows all three, because a project's commands include containers, databases and a `Makefile` target somebody added in 2019. A menu called **npm** holding `node main.js` is already a small lie, and one holding `docker compose up` is a large one. `JavaScript` is the language, and the language is what this editor is for.
78## Why the language server is not TypeScript 7's own
80There are two servers for JavaScript today. `typescript-language-server` wraps `tsserver`, the engine every editor's JavaScript support has run on for a decade; TypeScript 7 — the native port — ships a language server of its own inside its compiler, `tsc --lsp --stdio`, which needs nothing installed but `typescript`.
82Both were measured against the same nine end-to-end tests in this repository. The native one answers all eight requests, four times faster. It publishes **no diagnostics**: it offers them pull-style, through `textDocument/diagnostic`, which turbo-core does not ask for, so with it the gutter stays blank on a file that does not parse. An editor whose gutter is blank because the server waits to be asked looks exactly like one with nothing to report, and that is the one failure this family has learnt to design against. So the profile names the mature pair — `typescript-language-server` with TypeScript 6, the last version that ships `tsserver.js` — and says so in one line where a user reads it. The day the library learns to pull, `tsc --lsp --stdio` from a single `npm install -g typescript` is the better answer, and the change is two fields of the profile.
84## Why the tests drive the real editor
86`internal/jslang/editor_test.go` builds a whole Turbo JS on a simulated terminal — `app.New(screen, "turbo-classic", jslang.Profile())` — opens a file and checks the colouring, the menu bar and the hot keys. It uses only the library's public API.
88That is deliberate. The library's own suite proves the library works; what these tests prove is that *this editor is assembled correctly* — that `Register` was called, that the profile reached the menu bar, that a `.js` file comes out coloured by *this* scanner (a regular expression is the proof) and a `.ts` file does not, that `package.json` is JSON, and that the Agent menu offers to write the starter file. A bug where `main` forgot to register JavaScript would pass every test in turbo-core.
90The same file drives a **real `typescript-language-server`** end to end, nine times over. It writes a project with a `package.json`, opens its file, starts the server, and then:
92- **types a function declaration that exists only in the buffer**, then types its first letters inside another function and asks for a completion — the server offers every global for any file at all, so a completion holding `console` would prove nothing; one holding a function that is not on disk proves the buffer was sent;
93- asks for the **definition** of a call, the **references** to a function declared once and called twice, the **implementations** of a class with two subclasses, and the **type definition** of a variable holding an instance — four questions, one answer shape, and two of them come back as lists;
94- asks for the **hover** over a call, which comes back with the JSDoc comment written above the declaration;
95- asks for the **file's symbols** and for a **project-wide** symbol by name;
96- opens a file that **does not parse** and waits for a diagnostic to arrive unasked — the one feature whose failure looks exactly like success, because an editor with no error to show and one that cannot find the error are the same blank gutter.
98`diagram_test.go` holds `docs/diagrams/packages.drawio` to `go list`, so the diagram linked below cannot describe a package that is not there. `reference_test.go` holds every row of the [languages reference](../reference/languages.md) to the scanner.
100## Rejected alternatives
102**Forking Turbo Golo.** The obvious way to get a sixth editor, and the reason the library exists instead: six copies of eleven thousand lines drift within a month, and every fix has to be made six times by somebody who remembers there are six.
104**Keeping the library's JavaScript scanner.** It would have made this the one editor in the family with no scanner of its own, and its user the one without regular expressions coloured. Weighed above.
106**A plugin system.** Turbo JS is a Go program that imports a library. There is no dynamic loading and no ABI. Adding one would mean freezing the API of every package in turbo-core rather than of the handful a profile touches.
108**A configuration file instead of a profile.** The profile could have been TOML read at start-up, which would make a new editor a file rather than a program. It would also make the scanner inexpressible, and a half-configurable editor — everything but the colouring — is worse than either whole answer.
110**Embedding Node.** Some editors ship the runtime they edit for. Node is a hundred megabytes and the user has it already; the editor is a six-megabyte binary that talks to it.
112## How it relates to the rest
114- What each of the library's packages does: [turbo-core's package reference](https://rickub.com/turbo-editors/turbo-core/blob/main/docs/en/reference/packages.md)
115- The package diagram, checked against `go list`: [`docs/diagrams/packages.drawio`](../../diagrams/packages.drawio)
116- How the colouring works here: [Colouring and completion](colouring-and-completion.md)
117- Why the tools menu is data: [JavaScript tools](javascript-tools.md)
118- The decisions that outlived the refactoring: [Design decisions](design-decisions.md)

📦 Turbo JS 91999d1 k33g 12h ago	1	# Architecture — explanation
	2
	3	## What is this about?
	4
	5	Turbo JS is a command, a profile and two scanners. Everything else — the editing widget, the windows, the menus, the dialogs, the themes, the terminal emulator, the file tree, the LSP client, the agent windows — is [turbo-core](https://rickub.com/turbo-editors/turbo-core), the library every Turbo editor is built on.
	6
	7	This page is about that split: what is here, what is there, and why the line falls where it does.
	8
	9	## What is in this repository
	10
	11	```
	12	main.go flags, the terminal, and the wiring
	13	internal/jslang the whole of what makes this Turbo JS
	14	jslang.go the profile: name, menu, server, where npm puts a global binary
	15	scan.go the JavaScript scanner's dispatcher, comments, regular expressions, what crosses a line
	16	literals.go template literals and numbers
	17	words.go keywords, constants, the globals, the naming conventions
	18	json.go the JSON scanner
	19	templates.go four //go:embed declarations
	20	*.toml.tmpl the four starter files a project gets, embedded
	21	```
	22
	23	About twelve hundred lines counting the comments, of which some six hundred are the two scanners — under seven hundred lines of code by qlty's count. There is no `internal/app`, no `internal/ui`, no `internal/buffer` — those exist once, in the library, and all six editors use them unchanged.
	24
	25	## What `main` does
	26
	27	Six things, in this order:
	28
	29	1. Parses the flags.
	30	2. Calls `jslang.Register()`, which teaches the library this editor's JavaScript — replacing the scanner the library ships for it — and JSON.
	31	3. Builds `jslang.Profile()` — the value that says this editor is Turbo JS.
	32	4. Reads `.turbo-js/settings.toml` from the working directory, if there is one.
	33	5. Opens the terminal and hands the screen, the theme name and the profile to `app.New`.
	34	6. Starts `typescript-language-server` in the project root — the nearest directory at or above the file being edited that holds a `package.json` — and runs the event loop.
	35
	36	That is the whole command. Every decision it makes — which theme wins, which files to open, whether to start a language server — is about this run, not about JavaScript.
	37
	38	## The profile is the seam
	39
	40	```go
	41	profile.Profile{
	42	Name: "Turbo JS",
	43	Slug: "turbo-js",
	44	Language: "JavaScript",
	45	ToolsMenu: "~J~avaScript",
	46	RootMarkers: []string{"package.json"},
	47	Server: profile.Server{Command: "typescript-language-server", Args: []string{"--stdio"}, …},
	48	Templates: profile.Templates{Settings: …, Snippets: …, Tools: …, Agents: …},
	49	}
	50	```
	51
	52	Everything that would otherwise be a hardcoded `"turbo-js"`, `"node"` or `".js"` somewhere in eleven thousand lines is one field here. The library reads them; nothing in the library knows what any of them mean.
	53
	54	`Slug` carries more than it looks. The binary is `turbo-js`, the project directory is `.turbo-js`, the user's own configuration lives in `~/.config/turbo-js`, and the environment variables that override it are `TURBO_JS_THEME_DIR` and `TURBO_JS_SNIPPET_DIR` — all derived from that one word.
	55
	56	`RootMarkers` holds one file, `package.json`. It is a Node project's boundary — its name, its dependencies, its scripts — and in a workspace holding several packages the nearest one going up is the package being edited, which is the root the server should resolve imports from. The library's `ProjectRoot` does the walk; the profile only says what to look for. A directory with no `package.json` anywhere above it gets the working directory, and `typescript-language-server` infers a project from the files it is shown.
	57
	58	## Why the JavaScript scanner is here when the library has one
	59
	60	turbo-core colours eight languages itself: TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell. Those are the ones every editor meets whatever it is for — a project's configuration is TOML or YAML, its documentation is Markdown, its scripts are shell, its image build a Dockerfile, and a web page or a README's code fence is JavaScript.
	61
	62	So JavaScript is the one language in this family that the library colours and an editor is for. The library's scanner is written for the Rust editor's README: it knows the keywords, the strings, the comments and the numbers, and it deliberately refuses regular expressions, because telling `/x/g` from a division needs the previous token and a wrong guess there colours the rest of a line as a string — a bad trade for a language you only meet in passing. An editor for JavaScript makes the opposite trade: regular expressions are on every other line of a Node program, and the guess can be bounded. It also owes its user Node's globals, the hashbang line, the name after `function`, private names and decorators.
	63
	64	The library allows exactly this. `syntax.Register` replaces a language registered under the same name, and the later registration wins because it is the more specific statement. Turbo JS registers under the library's own name, `javascript`, so a snippets file saying `languages = ["javascript"]` and a ```js fence in an agent window both reach this scanner, and nothing in the library changed. The [colouring page](colouring-and-completion.md) says what the replacement adds and what it still refuses.
65
66	## Why JSON is here too
67
68	JSON is not one of the library's eight, and a Node project cannot be edited without it: `package.json` is the manifest and the root marker, `package-lock.json` is beside it, `tsconfig.json` and `.eslintrc.json` are common neighbours. So the editor that is for Node registers a JSON scanner, thirty lines long, that tells a key from a value and tolerates the comments `tsconfig.json` has.
69
70	That could have gone into the library instead — a Go editor meets a `package.json` too, in a repository with a front end. It was not, for the reason that keeps the Go and Rust scanners out: the library grows a language every time somebody wants one, and "what does this editor register?" stops being the first question about a new one. If turbo-core learns JSON one day, this registration will still win by ordering, and the one here can then be removed.
71
72	## Why the toolchain menu is `~J~avaScript` and not `~N~ode` or `npm`
73
74	The hot key was the easy part. Ten letters are taken by the fixed menus — F, E, S, R, C, O, W, N, H and, since the Agent menu, A — which rules out the S, the C and the R of JavaScript, and the N of Node, but not J, so the hot key lands on the first letter of the word, which costs nobody a second glance. Turbo Rust had no such luck and ended up on `Rus~t~`.
75
76	The name was the real decision, and it went the same way every sibling's did. The menu holds whatever the project put in its tools file, and that is not always npm: the starter file already runs `node`, `npm` and `npx`, and the first tools file anybody writes outgrows all three, because a project's commands include containers, databases and a `Makefile` target somebody added in 2019. A menu called npm holding `node main.js` is already a small lie, and one holding `docker compose up` is a large one. `JavaScript` is the language, and the language is what this editor is for.
77
78	## Why the language server is not TypeScript 7's own
79
80	There are two servers for JavaScript today. `typescript-language-server` wraps `tsserver`, the engine every editor's JavaScript support has run on for a decade; TypeScript 7 — the native port — ships a language server of its own inside its compiler, `tsc --lsp --stdio`, which needs nothing installed but `typescript`.
81
82	Both were measured against the same nine end-to-end tests in this repository. The native one answers all eight requests, four times faster. It publishes no diagnostics: it offers them pull-style, through `textDocument/diagnostic`, which turbo-core does not ask for, so with it the gutter stays blank on a file that does not parse. An editor whose gutter is blank because the server waits to be asked looks exactly like one with nothing to report, and that is the one failure this family has learnt to design against. So the profile names the mature pair — `typescript-language-server` with TypeScript 6, the last version that ships `tsserver.js` — and says so in one line where a user reads it. The day the library learns to pull, `tsc --lsp --stdio` from a single `npm install -g typescript` is the better answer, and the change is two fields of the profile.
83
84	## Why the tests drive the real editor
85
86	`internal/jslang/editor_test.go` builds a whole Turbo JS on a simulated terminal — `app.New(screen, "turbo-classic", jslang.Profile())` — opens a file and checks the colouring, the menu bar and the hot keys. It uses only the library's public API.
87
88	That is deliberate. The library's own suite proves the library works; what these tests prove is that this editor is assembled correctly — that `Register` was called, that the profile reached the menu bar, that a `.js` file comes out coloured by this scanner (a regular expression is the proof) and a `.ts` file does not, that `package.json` is JSON, and that the Agent menu offers to write the starter file. A bug where `main` forgot to register JavaScript would pass every test in turbo-core.
89
90	The same file drives a real `typescript-language-server` end to end, nine times over. It writes a project with a `package.json`, opens its file, starts the server, and then:
91
92	- types a function declaration that exists only in the buffer, then types its first letters inside another function and asks for a completion — the server offers every global for any file at all, so a completion holding `console` would prove nothing; one holding a function that is not on disk proves the buffer was sent;
93	- asks for the definition of a call, the references to a function declared once and called twice, the implementations of a class with two subclasses, and the type definition of a variable holding an instance — four questions, one answer shape, and two of them come back as lists;
94	- asks for the hover over a call, which comes back with the JSDoc comment written above the declaration;
95	- asks for the file's symbols and for a project-wide symbol by name;
96	- opens a file that does not parse and waits for a diagnostic to arrive unasked — the one feature whose failure looks exactly like success, because an editor with no error to show and one that cannot find the error are the same blank gutter.
97
98	`diagram_test.go` holds `docs/diagrams/packages.drawio` to `go list`, so the diagram linked below cannot describe a package that is not there. `reference_test.go` holds every row of the [languages reference](../reference/languages.md) to the scanner.
99
100	## Rejected alternatives
101
102	Forking Turbo Golo. The obvious way to get a sixth editor, and the reason the library exists instead: six copies of eleven thousand lines drift within a month, and every fix has to be made six times by somebody who remembers there are six.
103
104	Keeping the library's JavaScript scanner. It would have made this the one editor in the family with no scanner of its own, and its user the one without regular expressions coloured. Weighed above.
105
106	A plugin system. Turbo JS is a Go program that imports a library. There is no dynamic loading and no ABI. Adding one would mean freezing the API of every package in turbo-core rather than of the handful a profile touches.
107
108	A configuration file instead of a profile. The profile could have been TOML read at start-up, which would make a new editor a file rather than a program. It would also make the scanner inexpressible, and a half-configurable editor — everything but the colouring — is worse than either whole answer.
109
110	Embedding Node. Some editors ship the runtime they edit for. Node is a hundred megabytes and the user has it already; the editor is a six-megabyte binary that talks to it.
111
112	## How it relates to the rest
113
114	- What each of the library's packages does: [turbo-core's package reference](https://rickub.com/turbo-editors/turbo-core/blob/main/docs/en/reference/packages.md)
115	- The package diagram, checked against `go list`: [`docs/diagrams/packages.drawio`](../../diagrams/packages.drawio)
116	- How the colouring works here: [Colouring and completion](colouring-and-completion.md)
117	- Why the tools menu is data: [JavaScript tools](javascript-tools.md)
118	- The decisions that outlived the refactoring: [Design decisions](design-decisions.md)