turbo-editors/turbo-python · colouring-and-completion.md

turbo-editors/turbo-pythonpublic⑂ Fork 0

⑂ v1.0.0

colouring-and-completion.md · 103 lines · 11.5 KBmarkdown    Blame HistoryRaw
📦 Turbo Python   1# Colouring and completion — explanation
3## What is this about?
5The two features that make Turbo Python an editor *for Python* rather than a text editor that happens to open `.py` files: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.
7## Colouring is ours; completion is not
9Colouring is done here, in about six hundred lines of hand-written Go. Completion is done by python-lsp-server, and Turbo Python only asks and draws.
11That split is not an accident of effort. Colouring has to be **instant and tolerant**: it runs on every keystroke, on text that is invalid most of the time it is being typed, and a highlighter that stops to think or gives up on broken input is worse than no highlighter. Completion has to be **correct**, which for Python means following imports, resolving a name through the class hierarchy it was assigned in, and reading every installed package's public surface — and nothing that has to be instant can also be that.
13So the editor draws colours it computed itself, and shows completions somebody else computed.
15## Why Python is scanned by hand
17Go has a lexer in its standard library, and Turbo Go uses it: `go/scanner` is the same code the compiler uses, so the editor and the compiler agree about what a token is, with nothing to keep in step.
19Python has no such thing available here. CPython's tokeniser is C, `tokenize` is a Python module, and jedi's parser is a Python package. The choices were a hand-written scanner, or starting a Python process on every keystroke.
21The scanner it is. Some six hundred lines, one file each for the dispatcher, the literals and the words — and no attempt at a general engine. There is no pattern language, no grammar format and no table of regular expressions: it is ordinary Go that a reader can follow, which is the same rule the eight scanners in turbo-core follow.
23## The one thing that crosses a line break
25Almost everything in Python can be decided from the line in front of you. **A string is the only exception**, and it is an exception in two different ways — which is why what gets carried is a small structure rather than a flag.
27**A triple-quoted string runs until its three closing quotes**, however many lines away that is. Every docstring is one, so this is not an edge case; it is most of what a Python file contains that is not code.
29**A single-quoted string runs on only when the line ends with a backslash**, which escapes the newline. That is a real, if uncommon, construct — and it is the reason a single-quoted string that simply *runs out* of line gets dropped instead. Source under the cursor is unbalanced most of the time it is being typed, and an unterminated `"` carried forward paints the rest of the file green.
31**Which quote opened it is carried too.** A literal opened with three double quotes and one opened with three apostrophes are different strings, and the closer of one appearing inside the other closes nothing. A docstring that quotes anything at all — `"""say "hi" now"""` — is the case that catches a scanner carrying only "a string is open".
33**Rawness deliberately is not carried.** `r"\""` is one complete string: in a raw string the backslash stays in the value, but it still stops the quote after it from ending the literal. Termination is therefore the same rule for raw and ordinary strings, and a flag saying otherwise would be a flag nothing reads.
35## Where the scanner leans on convention
37Python's syntax leaves three questions open that its *conventions* answer, and the scanner reads the conventions rather than pretending the questions are not there.
39**A class is called exactly the way a function is.** `ValueError("nope")` and `parse("nope")` have the same shape; a parenthesis cannot tell a constructor from a call. What can is PEP 8: a class is `CapWords` and nothing else is. So a capitalised name is a type whether or not a parenthesis follows it — which is the one rule Turbo Python and Turbo Rust deliberately order differently, because in Rust a capitalised name before a parenthesis is usually a variant the language itself names.
41**A constant looks nothing like a class.** `MAX_SIZE` and `Measurement` are both "capitalised", and PEP 8 keeps them clearly apart: a module-level constant is `SCREAMING_SNAKE_CASE`. Turbo Rust has only the capital rule and documents `SCREAMING_SNAKE_CASE` constants coloured as types as a known wrong answer. Python's conventions are separated well enough that the wrong answer is worth removing rather than inheriting, so a name written wholly in capitals is a constant here. What it costs is a class named `HTTP`, which is rare enough to write down.
43**`self` is not the language's, and every reader treats it as if it were.** A method may name its first parameter anything; the compiler does not care. But a Python reader reads `self` the way a Rust reader reads `Some` — as a thing the language provides — and every other highlighter agrees. It is coloured as a builtin for that reason, and the honest cost is a plain function that happens to name a parameter `self`.
45## The one genuine ambiguity
47`match` and `case` were added to Python without being reserved. `match x:` opens a match statement; `match = re.match(pattern, text)` assigns a variable, and both are ordinary, common Python.
49Nothing about the word settles it, so the *shape of the statement* does: the word opens the line, and the line ends with the colon that opens its block. Both conditions must hold, which gets every match statement anybody writes and leaves `match` as a name everywhere else.
51It has a boundary, and the boundary is documented rather than removed. The colon is found by reading backwards from the end of the line — which is what makes the question cheap enough to ask of every word — and a trailing comment hides it, so `match value:  # dispatch` colours `match` as a name. Telling a real trailing comment from a `#` inside a string means scanning the line forwards, which is the work reading backwards exists to avoid. It is also the safe direction to be wrong in: a keyword shown as a name is a shade too plain, where a name shown as a keyword is a lie.
53## What the scanner refuses to guess
55Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet:
57| Not recognised | Because |
58| --- | --- |
59| The `{expression}` inside an f-string | Since Python 3.12 it may contain anything at all — nested quotes of the same kind, comments, another f-string. Colouring it properly means running the whole scanner inside itself; colouring it half-properly ends `f"{n:{width}}"` at the inner brace. One flat run of string is the honest answer |
60| A docstring as anything but a string | It *is* a string — `help()` reads it back as one — and the moment somebody assigns one to a name, a scanner that called it a comment is visibly wrong |
61| Whether a name is bound in this scope | Nothing here reads more than one line at a time. That is the language server's question |
63## The other eight languages come free
65TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell are coloured by turbo-core, not here. A Python project has a `pyproject.toml`, a `README.md`, some scripts, a CI workflow in YAML and often a Dockerfile, and an editor that coloured only the `.py` files would make you leave it for the rest.
67That they are shared rather than copied is the point of the library: they were written once, for Turbo Go, and Turbo Python got them by importing a package.
69## Completion, and why it can fail silently
71Turbo Python knows nothing about Python's type system and does not try to. It asks pylsp over the Language Server Protocol and draws the answer.
73Three things about that are worth knowing, because all three look like "completion is broken":
75**pylsp answers nothing until it has indexed enough of the project.** jedi resolves a name by following imports outwards, which on a first request into a large dependency means reading a great deal of somebody else's code. What you see meanwhile is an empty list.
77**A server given the wrong root loads the wrong code, and then answers nothing at all — with no error.** That is why the editor walks up from the file to the nearest `pyproject.toml`, `setup.py` or `setup.cfg` rather than using the working directory, and it is the single most confusing way completion can fail.
79**A server installed without its extras answers questions but never volunteers a problem.** pylsp's linters are optional dependencies; installed bare, it completes and jumps perfectly well and publishes an *empty* list of diagnostics for a file that does not even parse. A blank gutter because the server has no linter and a blank gutter because the code is fine look identical. That is why [the install command names the extras](../how-to/enable-completion.md) and why the installer checks for them.
81The editor's answer to the first two is [Run ▸ Language server status](../reference/menus.md), which says what it found, where it started it and whether it is ready — because "nothing happened" is not something a user can act on.
83## Nine questions, one connection — and the two pylsp does not answer
85Completion is the loudest thing the language server does and the least revealing. The same connection asks eight more questions, and they divide into three kinds by what comes back.
87**Something to read.** `hover` — what is this? — drawn in a box.
89**Places in the code.** `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — a method used across a package has as many references as somebody cared to write, and for a long time this editor took the first and threw the rest away.
91**Names.** `documentSymbol` for a file's own outline, `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.
93And one thing nobody asks for at all: **`publishDiagnostics` arrives unbidden**, whenever the server has an opinion, for whatever files it has loaded — which are usually more than the one in front of you. That is why Problems lists every file rather than the current one, and why the mark in the gutter appears without anything being pressed.
95**Two of the nine come back empty with pylsp, and that is the server's boundary rather than the editor's.** pylsp advertises neither `implementation` nor `workspace/symbol`, so **Code ▸ Find implementations** and **Code ▸ Symbol in project** report nothing found. Everything else works. This is written down rather than hidden because the alternative — greying out two menu items depending on what a server said at start-up — makes the menu a different shape on different machines, and a user who has read this page knows more than one who found a greyed item.
97The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here.
99## How it relates to the rest
101- Exactly what is recognised: [Languages coloured](../reference/languages.md)
102- Getting completion working: [How to enable Python completion](../how-to/enable-completion.md)
103- Where the scanner lives and why: [Architecture](architecture.md)

📦 Turbo Python 6fc62ea k33g 11h ago	1	# Colouring and completion — explanation
	2
	3	## What is this about?
	4
	5	The two features that make Turbo Python an editor for Python rather than a text editor that happens to open `.py` files: syntax colouring, and completion from a language server. They work quite differently, and the difference is instructive.
	6
	7	## Colouring is ours; completion is not
	8
	9	Colouring is done here, in about six hundred lines of hand-written Go. Completion is done by python-lsp-server, and Turbo Python only asks and draws.
	10
	11	That split is not an accident of effort. Colouring has to be instant and tolerant: it runs on every keystroke, on text that is invalid most of the time it is being typed, and a highlighter that stops to think or gives up on broken input is worse than no highlighter. Completion has to be correct, which for Python means following imports, resolving a name through the class hierarchy it was assigned in, and reading every installed package's public surface — and nothing that has to be instant can also be that.
	12
	13	So the editor draws colours it computed itself, and shows completions somebody else computed.
	14
	15	## Why Python is scanned by hand
	16
	17	Go has a lexer in its standard library, and Turbo Go uses it: `go/scanner` is the same code the compiler uses, so the editor and the compiler agree about what a token is, with nothing to keep in step.
	18
	19	Python has no such thing available here. CPython's tokeniser is C, `tokenize` is a Python module, and jedi's parser is a Python package. The choices were a hand-written scanner, or starting a Python process on every keystroke.
	20
	21	The scanner it is. Some six hundred lines, one file each for the dispatcher, the literals and the words — and no attempt at a general engine. There is no pattern language, no grammar format and no table of regular expressions: it is ordinary Go that a reader can follow, which is the same rule the eight scanners in turbo-core follow.
	22
	23	## The one thing that crosses a line break
	24
	25	Almost everything in Python can be decided from the line in front of you. A string is the only exception, and it is an exception in two different ways — which is why what gets carried is a small structure rather than a flag.
	26
	27	A triple-quoted string runs until its three closing quotes, however many lines away that is. Every docstring is one, so this is not an edge case; it is most of what a Python file contains that is not code.
	28
	29	A single-quoted string runs on only when the line ends with a backslash, which escapes the newline. That is a real, if uncommon, construct — and it is the reason a single-quoted string that simply runs out of line gets dropped instead. Source under the cursor is unbalanced most of the time it is being typed, and an unterminated `"` carried forward paints the rest of the file green.
	30
	31	Which quote opened it is carried too. A literal opened with three double quotes and one opened with three apostrophes are different strings, and the closer of one appearing inside the other closes nothing. A docstring that quotes anything at all — `"""say "hi" now"""` — is the case that catches a scanner carrying only "a string is open".
	32
	33	Rawness deliberately is not carried. `r"\""` is one complete string: in a raw string the backslash stays in the value, but it still stops the quote after it from ending the literal. Termination is therefore the same rule for raw and ordinary strings, and a flag saying otherwise would be a flag nothing reads.
	34
	35	## Where the scanner leans on convention
	36
	37	Python's syntax leaves three questions open that its conventions answer, and the scanner reads the conventions rather than pretending the questions are not there.
	38
	39	A class is called exactly the way a function is. `ValueError("nope")` and `parse("nope")` have the same shape; a parenthesis cannot tell a constructor from a call. What can is PEP 8: a class is `CapWords` and nothing else is. So a capitalised name is a type whether or not a parenthesis follows it — which is the one rule Turbo Python and Turbo Rust deliberately order differently, because in Rust a capitalised name before a parenthesis is usually a variant the language itself names.
	40
	41	A constant looks nothing like a class. `MAX_SIZE` and `Measurement` are both "capitalised", and PEP 8 keeps them clearly apart: a module-level constant is `SCREAMING_SNAKE_CASE`. Turbo Rust has only the capital rule and documents `SCREAMING_SNAKE_CASE` constants coloured as types as a known wrong answer. Python's conventions are separated well enough that the wrong answer is worth removing rather than inheriting, so a name written wholly in capitals is a constant here. What it costs is a class named `HTTP`, which is rare enough to write down.
	42
	43	`self` is not the language's, and every reader treats it as if it were. A method may name its first parameter anything; the compiler does not care. But a Python reader reads `self` the way a Rust reader reads `Some` — as a thing the language provides — and every other highlighter agrees. It is coloured as a builtin for that reason, and the honest cost is a plain function that happens to name a parameter `self`.
	44
	45	## The one genuine ambiguity
	46
	47	`match` and `case` were added to Python without being reserved. `match x:` opens a match statement; `match = re.match(pattern, text)` assigns a variable, and both are ordinary, common Python.
	48
	49	Nothing about the word settles it, so the shape of the statement does: the word opens the line, and the line ends with the colon that opens its block. Both conditions must hold, which gets every match statement anybody writes and leaves `match` as a name everywhere else.
	50
	51	It has a boundary, and the boundary is documented rather than removed. The colon is found by reading backwards from the end of the line — which is what makes the question cheap enough to ask of every word — and a trailing comment hides it, so `match value: # dispatch` colours `match` as a name. Telling a real trailing comment from a `#` inside a string means scanning the line forwards, which is the work reading backwards exists to avoid. It is also the safe direction to be wrong in: a keyword shown as a name is a shade too plain, where a name shown as a keyword is a lie.
	52
	53	## What the scanner refuses to guess
	54
	55	Where a construct cannot be recognised from what one line holds, it is left alone rather than approximated. A highlighter that is wrong is worse than one that is quiet:
	56
	57	\| Not recognised \| Because \|
	58	\| --- \| --- \|
	59	\| The `{expression}` inside an f-string \| Since Python 3.12 it may contain anything at all — nested quotes of the same kind, comments, another f-string. Colouring it properly means running the whole scanner inside itself; colouring it half-properly ends `f"{n:{width}}"` at the inner brace. One flat run of string is the honest answer \|
	60	\| A docstring as anything but a string \| It is a string — `help()` reads it back as one — and the moment somebody assigns one to a name, a scanner that called it a comment is visibly wrong \|
	61	\| Whether a name is bound in this scope \| Nothing here reads more than one line at a time. That is the language server's question \|
	62
	63	## The other eight languages come free
	64
65	TOML, YAML, Markdown, JavaScript, HTML, XML, Dockerfiles and shell are coloured by turbo-core, not here. A Python project has a `pyproject.toml`, a `README.md`, some scripts, a CI workflow in YAML and often a Dockerfile, and an editor that coloured only the `.py` files would make you leave it for the rest.
66
67	That they are shared rather than copied is the point of the library: they were written once, for Turbo Go, and Turbo Python got them by importing a package.
68
69	## Completion, and why it can fail silently
70
71	Turbo Python knows nothing about Python's type system and does not try to. It asks pylsp over the Language Server Protocol and draws the answer.
72
73	Three things about that are worth knowing, because all three look like "completion is broken":
74
75	pylsp answers nothing until it has indexed enough of the project. jedi resolves a name by following imports outwards, which on a first request into a large dependency means reading a great deal of somebody else's code. What you see meanwhile is an empty list.
76
77	A server given the wrong root loads the wrong code, and then answers nothing at all — with no error. That is why the editor walks up from the file to the nearest `pyproject.toml`, `setup.py` or `setup.cfg` rather than using the working directory, and it is the single most confusing way completion can fail.
78
79	A server installed without its extras answers questions but never volunteers a problem. pylsp's linters are optional dependencies; installed bare, it completes and jumps perfectly well and publishes an empty list of diagnostics for a file that does not even parse. A blank gutter because the server has no linter and a blank gutter because the code is fine look identical. That is why [the install command names the extras](../how-to/enable-completion.md) and why the installer checks for them.
80
81	The editor's answer to the first two is [Run ▸ Language server status](../reference/menus.md), which says what it found, where it started it and whether it is ready — because "nothing happened" is not something a user can act on.
82
83	## Nine questions, one connection — and the two pylsp does not answer
84
85	Completion is the loudest thing the language server does and the least revealing. The same connection asks eight more questions, and they divide into three kinds by what comes back.
86
87	Something to read. `hover` — what is this? — drawn in a box.
88
89	Places in the code. `definition`, `typeDefinition`, `implementation`, `references`. One request each, one answer shape between them, which is why they are one function underneath. A single place is opened; several are offered as a list, because a single answer is the exception rather than the rule — a method used across a package has as many references as somebody cared to write, and for a long time this editor took the first and threw the rest away.
90
91	Names. `documentSymbol` for a file's own outline, `workspace/symbol` for a search across the project. The protocol has three shapes for a symbol and the editor wants one, so the flattening is done where the answers arrive rather than where they are drawn.
92
93	And one thing nobody asks for at all: `publishDiagnostics` arrives unbidden, whenever the server has an opinion, for whatever files it has loaded — which are usually more than the one in front of you. That is why Problems lists every file rather than the current one, and why the mark in the gutter appears without anything being pressed.
94
95	Two of the nine come back empty with pylsp, and that is the server's boundary rather than the editor's. pylsp advertises neither `implementation` nor `workspace/symbol`, so Code ▸ Find implementations and Code ▸ Symbol in project report nothing found. Everything else works. This is written down rather than hidden because the alternative — greying out two menu items depending on what a server said at start-up — makes the menu a different shape on different machines, and a user who has read this page knows more than one who found a greyed item.
96
97	The editor asks for none of this until the server says it is ready, and says which of those it is when a question cannot be answered. "Nothing found" and "I have not finished loading" are the same empty answer and very different news; conflating them is the most confusing way completion has ever failed here.
98
99	## How it relates to the rest
100
101	- Exactly what is recognised: [Languages coloured](../reference/languages.md)
102	- Getting completion working: [How to enable Python completion](../how-to/enable-completion.md)
103	- Where the scanner lives and why: [Architecture](architecture.md)