nandi/frq · README.md

nandi/frqpublic⑂ Fork 0

⑂ 1d62d1a437b844c37e3e4d49efa7e255ef4deac7

README.md · 149 lines · 7.1 KBmarkdown    Blame HistoryRaw
One frontend where there were three, and a core that is not Clojure   1# The Nim core
3The portable half of frq, as a native library the Dart side calls through FFI.
5## Why this exists
7`common/` is ClojureDart, compiled into every target. That works, and the
8reason to move any of it is not that it is broken: it is that the logic under
9the screens — the IRC wire format, the atproto flows, the message signatures —
10is the part with the most rules per line and the least to do with Flutter, and
11it is the part worth having in a language with a type checker and a test runner
12that does not need a Flutter toolchain to run.
14So the plan is a seam, not a rewrite-in-place. Each module moves one at a time:
15the Nim implementation lands here with tests, the Dart binding lands in
16`flutter/src/frq/core/`, and the ClojureDart original stays until the binding
17is proven against it. Nothing is deleted on faith.
19## What is here
22src/frq_core.nim        the C ABI: every exported symbol, and nothing else
23src/frq/ircparse.nim    the IRC wire format
24tests/                  one per module, run by `just nim-test`
27`src/frq_core.nim` is the only file that knows about C. Everything under
28`src/frq/` is ordinary Nim with ordinary Nim types, so the tests test the logic
29rather than the marshalling.
31
33Strings in, strings out, and JSON where the answer is not a single string.
35That is a deliberate choice against a struct-based ABI. A struct means the Dart
36side and the Nim side have to agree on a memory layout, and every field added
37later is a version skew that segfaults instead of failing. JSON costs a parse
38per call, which is nothing against a network round trip, and it lets one side
39gain a field without the other crashing.
41Every function that returns a string returns memory the **caller must free**
42with `frq_free`. Nim's allocator is not Dart's; a `free()` from the Dart side
43on a Nim pointer is undefined. The bindings in `flutter/src/frq/core/` wrap
44that in a `try/finally` so no call site has to remember.
46`frq_init` must be called once before anything else, and calls `NimMain` to set
47up Nim's runtime. The bindings do it on first use.
49
51A native library does not load in a browser, so the web target cannot call this
52through `dart:ffi`. Nim compiles through C, so the route is emscripten to wasm
53and a JS binding rather than a second implementation — but that is not built
54yet, and until it is, **the web build must keep using the ClojureDart
55originals**. This is why the originals stay in `common/` rather than being
56deleted as each module lands: they are the web's implementation, not dead code.
A message in #test, from Nim   58## The spike
60`just nim-spike` opens a window with no ClojureDart on the path, connects to
61irc.freeq.at over TLS, joins `#test` and sends a message. Nim owns the state,
62the screens, the socket and the IRC protocol; Dart owns the pixels.
65src/frq/ui.nim              the widget tree, in the screens' own tag vocabulary
66src/frq/state.nim           the record, the reducer, and drain()
67src/frq/irc.nim             the socket, on its own thread, behind two channels
68src/frq/screens/            connect and chat, as pure functions of the state
69src/frq/trace.nim           FRQ_TRACE=1, the same switch the rest of frq uses
72Three decisions worth knowing before changing any of it:
74* **Nothing is shared with the socket thread.** Nim's ORC is thread-local for
75  ref types, so sharing the state would mean a lock per field and a heap two
76  threads both collect. The reader speaks only in channels, and `drain()` turns
77  its output into state on whichever thread Dart called in on.
78* **Dart polls; Nim never calls back.** A Dart callback from a foreign thread
79  has to be marshalled onto the main isolate — `NativeCallable`, ports, a whole
80  mechanism — and at 70µs a render a 100ms timer does the same job for free.
81* **A prop holds an event id, not a closure.** That is the one thing hiccup has
82  that a C ABI cannot, and substituting it is what makes this an architecture
83  rather than a rendering trick.
85Cost, measured: a full screen rebuild is 70–105µs, or 0.4–0.6% of a 60fps
86frame, for a 1.6KB tree. The caveat is the tree size rather than the number —
87the chat screen caps the backlog at fifty rows for exactly this reason, and
88nothing has measured what a real one costs.
The window connects   91just nim-spike        # the window; click Connect
92FRQ_TRACE=1 FRQ_AUTOCONNECT=1 just nim-spike   # ...connecting on its own
A message in #test, from Nim   93just nim-spike-test   # 7 widget tests: real taps, real widgets
94just nim-live         # connect to a real freeq and say a line
95just nim-bench        # what the boundary costs
96FRQ_TRACE=1 just nim-live    # ...and every line on the wire
The window connects   99Two switches, both for the same reason — a GUI on Wayland cannot be clicked
100from a script, so without them the only way to check the window connects is to
101sit in front of it. `FRQ_AUTOCONNECT=1` presses Connect on the first render and
102`FRQ_NICK` overrides the nickname, because two runs with the same one collide
103on the server and the second is refused.
105The GUI needs OpenSSL on its loader path, which the `flutter-desktop` shell
106provides as `FRQ_OPENSSL_LIB` and the `nim-spike` recipe prepends for the app
107alone. Not set as `LD_LIBRARY_PATH` in the shell itself: that shell also runs
108Flutter through nixGL, which does its own careful things to the loader path,
109and a blanket setting there breaks GL on some machines and not others.
One frontend where there were three, and a core that is not Clojure   111
113`frq/ircparse.nim` is ported and tested — 29 cases, `just nim-test` — and the
114ABI is exercised from C through `dlopen`, including the allocation contract
115under a hundred thousand parse/free cycles. That half is real.
The binding is Dart, and it works   117The Dart binding is real too, and is `dart/frq_core` — **plain Dart, not
118ClojureDart**. It calls the library over `dart:ffi` and is covered by 20 tests
119on the Dart VM, including the UTF-8 round trip and ten thousand calls against
120the ownership rules. `just dart-test` runs the pair of them in about a second.
122The binding was ClojureDart for one commit and should not have been: the Nim
123core exists to have less Clojure in the tree, and `lookupFunction` takes two
124type arguments, so it meant fighting generic interop to write more of the
125thing being removed. In Dart it is a typedef. See `dart/README.md`.
A message in #test, from Nim   127The spike above is wired up and runs. What is **not** done is replacing
128anything: the shipping app is still the ClojureDart one, `common/frq/irc/parse.cljc`
129is still what it runs, and the spike is a second entry point beside it
130(`lib/main_nim.dart`) rather than a replacement for `frq.main`. That step is its
The binding is Dart, and it works   131own piece of work — the Flutter app takes the package as a path dependency
132(which means a `pubspec.lock` regeneration and widening the nix build's source
133root), the library has to reach each target (`jniLibs` for the APK, beside the
134executable for the desktop bundle), and only then can a call site choose Nim
135on native and the ClojureDart original on the web.
One frontend where there were three, and a core that is not Clojure   136
137Modules still in `common/` and not yet here: `rooms`, `msgsig`, `crypto`,
138`atproto/core`, `oauth/core`, `store`, `irc/handshake`, `irc/mutate`,
139`profile`, `members`, `reactions`, `replies`, `edits`, `clock`.
141## Building
144just nim-test     # the Nim test suite
145just nim-lib      # libfrqcore.so into build/nim
148Both want `nix develop .#nim`, and re-enter it themselves if they are not
149already inside.

One frontend where there were three, and a core that is not Clojure 438b247 nandi 21h ago	1	# The Nim core
	2
	3	The portable half of frq, as a native library the Dart side calls through FFI.
	4
	5	## Why this exists
	6
	7	`common/` is ClojureDart, compiled into every target. That works, and the
	8	reason to move any of it is not that it is broken: it is that the logic under
	9	the screens — the IRC wire format, the atproto flows, the message signatures —
	10	is the part with the most rules per line and the least to do with Flutter, and
	11	it is the part worth having in a language with a type checker and a test runner
	12	that does not need a Flutter toolchain to run.
	13
	14	So the plan is a seam, not a rewrite-in-place. Each module moves one at a time:
	15	the Nim implementation lands here with tests, the Dart binding lands in
	16	`flutter/src/frq/core/`, and the ClojureDart original stays until the binding
	17	is proven against it. Nothing is deleted on faith.
	18
	19	## What is here
	20
	21	```
	22	src/frq_core.nim the C ABI: every exported symbol, and nothing else
	23	src/frq/ircparse.nim the IRC wire format
	24	tests/ one per module, run by `just nim-test`
	25	```
	26
	27	`src/frq_core.nim` is the only file that knows about C. Everything under
	28	`src/frq/` is ordinary Nim with ordinary Nim types, so the tests test the logic
	29	rather than the marshalling.
	30
	31	## The ABI
	32
	33	Strings in, strings out, and JSON where the answer is not a single string.
	34
	35	That is a deliberate choice against a struct-based ABI. A struct means the Dart
	36	side and the Nim side have to agree on a memory layout, and every field added
	37	later is a version skew that segfaults instead of failing. JSON costs a parse
	38	per call, which is nothing against a network round trip, and it lets one side
	39	gain a field without the other crashing.
	40
	41	Every function that returns a string returns memory the caller must free
	42	with `frq_free`. Nim's allocator is not Dart's; a `free()` from the Dart side
	43	on a Nim pointer is undefined. The bindings in `flutter/src/frq/core/` wrap
	44	that in a `try/finally` so no call site has to remember.
	45
	46	`frq_init` must be called once before anything else, and calls `NimMain` to set
	47	up Nim's runtime. The bindings do it on first use.
	48
	49	## The web
	50
	51	A native library does not load in a browser, so the web target cannot call this
	52	through `dart:ffi`. Nim compiles through C, so the route is emscripten to wasm
	53	and a JS binding rather than a second implementation — but that is not built
	54	yet, and until it is, **the web build must keep using the ClojureDart
	55	originals**. This is why the originals stay in `common/` rather than being
	56	deleted as each module lands: they are the web's implementation, not dead code.
	57
A message in #test, from Nim 35994d4 nandi 20h ago	58	## The spike
	59
	60	`just nim-spike` opens a window with no ClojureDart on the path, connects to
	61	irc.freeq.at over TLS, joins `#test` and sends a message. Nim owns the state,
	62	the screens, the socket and the IRC protocol; Dart owns the pixels.
	63
	64	```
	65	src/frq/ui.nim the widget tree, in the screens' own tag vocabulary
	66	src/frq/state.nim the record, the reducer, and drain()
	67	src/frq/irc.nim the socket, on its own thread, behind two channels
	68	src/frq/screens/ connect and chat, as pure functions of the state
	69	src/frq/trace.nim FRQ_TRACE=1, the same switch the rest of frq uses
	70	```
	71
	72	Three decisions worth knowing before changing any of it:
	73
	74	* Nothing is shared with the socket thread. Nim's ORC is thread-local for
	75	ref types, so sharing the state would mean a lock per field and a heap two
	76	threads both collect. The reader speaks only in channels, and `drain()` turns
	77	its output into state on whichever thread Dart called in on.
	78	* Dart polls; Nim never calls back. A Dart callback from a foreign thread
	79	has to be marshalled onto the main isolate — `NativeCallable`, ports, a whole
	80	mechanism — and at 70µs a render a 100ms timer does the same job for free.
	81	* A prop holds an event id, not a closure. That is the one thing hiccup has
	82	that a C ABI cannot, and substituting it is what makes this an architecture
	83	rather than a rendering trick.
	84
	85	Cost, measured: a full screen rebuild is 70–105µs, or 0.4–0.6% of a 60fps
	86	frame, for a 1.6KB tree. The caveat is the tree size rather than the number —
	87	the chat screen caps the backlog at fifty rows for exactly this reason, and
	88	nothing has measured what a real one costs.
	89
	90	```bash
The window connects 1d62d1a nandi 20h ago	91	just nim-spike # the window; click Connect
The window connects 1d62d1a nandi 20h ago	92	FRQ_TRACE=1 FRQ_AUTOCONNECT=1 just nim-spike # ...connecting on its own
A message in #test, from Nim 35994d4 nandi 20h ago	93	just nim-spike-test # 7 widget tests: real taps, real widgets
	94	just nim-live # connect to a real freeq and say a line
	95	just nim-bench # what the boundary costs
	96	FRQ_TRACE=1 just nim-live # ...and every line on the wire
	97	```
	98
The window connects 1d62d1a nandi 20h ago	99	Two switches, both for the same reason — a GUI on Wayland cannot be clicked
	100	from a script, so without them the only way to check the window connects is to
	101	sit in front of it. `FRQ_AUTOCONNECT=1` presses Connect on the first render and
	102	`FRQ_NICK` overrides the nickname, because two runs with the same one collide
	103	on the server and the second is refused.
	104
	105	The GUI needs OpenSSL on its loader path, which the `flutter-desktop` shell
	106	provides as `FRQ_OPENSSL_LIB` and the `nim-spike` recipe prepends for the app
	107	alone. Not set as `LD_LIBRARY_PATH` in the shell itself: that shell also runs
	108	Flutter through nixGL, which does its own careful things to the loader path,
	109	and a blanket setting there breaks GL on some machines and not others.
	110
One frontend where there were three, and a core that is not Clojure 438b247 nandi 21h ago	111	## Status
	112
	113	`frq/ircparse.nim` is ported and tested — 29 cases, `just nim-test` — and the
	114	ABI is exercised from C through `dlopen`, including the allocation contract
	115	under a hundred thousand parse/free cycles. That half is real.
	116
The binding is Dart, and it works f7aea3b nandi 20h ago	117	The Dart binding is real too, and is `dart/frq_core` — **plain Dart, not
	118	ClojureDart**. It calls the library over `dart:ffi` and is covered by 20 tests
	119	on the Dart VM, including the UTF-8 round trip and ten thousand calls against
	120	the ownership rules. `just dart-test` runs the pair of them in about a second.
	121
	122	The binding was ClojureDart for one commit and should not have been: the Nim
	123	core exists to have less Clojure in the tree, and `lookupFunction` takes two
	124	type arguments, so it meant fighting generic interop to write more of the
	125	thing being removed. In Dart it is a typedef. See `dart/README.md`.
	126
A message in #test, from Nim 35994d4 nandi 20h ago	127	The spike above is wired up and runs. What is not done is replacing
	128	anything: the shipping app is still the ClojureDart one, `common/frq/irc/parse.cljc`
	129	is still what it runs, and the spike is a second entry point beside it
	130	(`lib/main_nim.dart`) rather than a replacement for `frq.main`. That step is its
The binding is Dart, and it works f7aea3b nandi 20h ago	131	own piece of work — the Flutter app takes the package as a path dependency
	132	(which means a `pubspec.lock` regeneration and widening the nix build's source
	133	root), the library has to reach each target (`jniLibs` for the APK, beside the
	134	executable for the desktop bundle), and only then can a call site choose Nim
	135	on native and the ClojureDart original on the web.
One frontend where there were three, and a core that is not Clojure 438b247 nandi 21h ago	136
	137	Modules still in `common/` and not yet here: `rooms`, `msgsig`, `crypto`,
	138	`atproto/core`, `oauth/core`, `store`, `irc/handshake`, `irc/mutate`,
	139	`profile`, `members`, `reactions`, `replies`, `edits`, `clock`.
	140
	141	## Building
	142
	143	```bash
	144	just nim-test # the Nim test suite
	145	just nim-lib # libfrqcore.so into build/nim
	146	```
	147
	148	Both want `nix develop .#nim`, and re-enter it themselves if they are not
	149	already inside.