nandi/frq · README.md

nandi/frqpublic⑂ Fork 0

⑂ 35994d45d04d2f4d7320773e58286de34ac4085a

README.md · 136 lines · 6.3 KBmarkdown    Blame HistoryRaw
One frontend where there were three, and a core that is not Clojure   1# The Nim core
3The portable half of frq, as a native library the Dart side calls through FFI.
5## Why this exists
7`common/` is ClojureDart, compiled into every target. That works, and the
8reason to move any of it is not that it is broken: it is that the logic under
9the screens — the IRC wire format, the atproto flows, the message signatures —
10is the part with the most rules per line and the least to do with Flutter, and
11it is the part worth having in a language with a type checker and a test runner
12that does not need a Flutter toolchain to run.
14So the plan is a seam, not a rewrite-in-place. Each module moves one at a time:
15the Nim implementation lands here with tests, the Dart binding lands in
16`flutter/src/frq/core/`, and the ClojureDart original stays until the binding
17is proven against it. Nothing is deleted on faith.
19## What is here
22src/frq_core.nim        the C ABI: every exported symbol, and nothing else
23src/frq/ircparse.nim    the IRC wire format
24tests/                  one per module, run by `just nim-test`
27`src/frq_core.nim` is the only file that knows about C. Everything under
28`src/frq/` is ordinary Nim with ordinary Nim types, so the tests test the logic
29rather than the marshalling.
31
33Strings in, strings out, and JSON where the answer is not a single string.
35That is a deliberate choice against a struct-based ABI. A struct means the Dart
36side and the Nim side have to agree on a memory layout, and every field added
37later is a version skew that segfaults instead of failing. JSON costs a parse
38per call, which is nothing against a network round trip, and it lets one side
39gain a field without the other crashing.
41Every function that returns a string returns memory the **caller must free**
42with `frq_free`. Nim's allocator is not Dart's; a `free()` from the Dart side
43on a Nim pointer is undefined. The bindings in `flutter/src/frq/core/` wrap
44that in a `try/finally` so no call site has to remember.
46`frq_init` must be called once before anything else, and calls `NimMain` to set
47up Nim's runtime. The bindings do it on first use.
49
51A native library does not load in a browser, so the web target cannot call this
52through `dart:ffi`. Nim compiles through C, so the route is emscripten to wasm
53and a JS binding rather than a second implementation — but that is not built
54yet, and until it is, **the web build must keep using the ClojureDart
55originals**. This is why the originals stay in `common/` rather than being
56deleted as each module lands: they are the web's implementation, not dead code.
A message in #test, from Nim   58## The spike
60`just nim-spike` opens a window with no ClojureDart on the path, connects to
61irc.freeq.at over TLS, joins `#test` and sends a message. Nim owns the state,
62the screens, the socket and the IRC protocol; Dart owns the pixels.
65src/frq/ui.nim              the widget tree, in the screens' own tag vocabulary
66src/frq/state.nim           the record, the reducer, and drain()
67src/frq/irc.nim             the socket, on its own thread, behind two channels
68src/frq/screens/            connect and chat, as pure functions of the state
69src/frq/trace.nim           FRQ_TRACE=1, the same switch the rest of frq uses
72Three decisions worth knowing before changing any of it:
74* **Nothing is shared with the socket thread.** Nim's ORC is thread-local for
75  ref types, so sharing the state would mean a lock per field and a heap two
76  threads both collect. The reader speaks only in channels, and `drain()` turns
77  its output into state on whichever thread Dart called in on.
78* **Dart polls; Nim never calls back.** A Dart callback from a foreign thread
79  has to be marshalled onto the main isolate — `NativeCallable`, ports, a whole
80  mechanism — and at 70µs a render a 100ms timer does the same job for free.
81* **A prop holds an event id, not a closure.** That is the one thing hiccup has
82  that a C ABI cannot, and substituting it is what makes this an architecture
83  rather than a rendering trick.
85Cost, measured: a full screen rebuild is 70–105µs, or 0.4–0.6% of a 60fps
86frame, for a 1.6KB tree. The caveat is the tree size rather than the number —
87the chat screen caps the backlog at fifty rows for exactly this reason, and
88nothing has measured what a real one costs.
91just nim-spike        # the window
92just nim-spike-test   # 7 widget tests: real taps, real widgets
93just nim-live         # connect to a real freeq and say a line
94just nim-bench        # what the boundary costs
95FRQ_TRACE=1 just nim-live    # ...and every line on the wire
One frontend where there were three, and a core that is not Clojure   98
100`frq/ircparse.nim` is ported and tested — 29 cases, `just nim-test` — and the
101ABI is exercised from C through `dlopen`, including the allocation contract
102under a hundred thousand parse/free cycles. That half is real.
The binding is Dart, and it works   104The Dart binding is real too, and is `dart/frq_core` — **plain Dart, not
105ClojureDart**. It calls the library over `dart:ffi` and is covered by 20 tests
106on the Dart VM, including the UTF-8 round trip and ten thousand calls against
107the ownership rules. `just dart-test` runs the pair of them in about a second.
109The binding was ClojureDart for one commit and should not have been: the Nim
110core exists to have less Clojure in the tree, and `lookupFunction` takes two
111type arguments, so it meant fighting generic interop to write more of the
112thing being removed. In Dart it is a typedef. See `dart/README.md`.
A message in #test, from Nim   114The spike above is wired up and runs. What is **not** done is replacing
115anything: the shipping app is still the ClojureDart one, `common/frq/irc/parse.cljc`
116is still what it runs, and the spike is a second entry point beside it
117(`lib/main_nim.dart`) rather than a replacement for `frq.main`. That step is its
The binding is Dart, and it works   118own piece of work — the Flutter app takes the package as a path dependency
119(which means a `pubspec.lock` regeneration and widening the nix build's source
120root), the library has to reach each target (`jniLibs` for the APK, beside the
121executable for the desktop bundle), and only then can a call site choose Nim
122on native and the ClojureDart original on the web.
One frontend where there were three, and a core that is not Clojure   123
124Modules still in `common/` and not yet here: `rooms`, `msgsig`, `crypto`,
125`atproto/core`, `oauth/core`, `store`, `irc/handshake`, `irc/mutate`,
126`profile`, `members`, `reactions`, `replies`, `edits`, `clock`.
128## Building
131just nim-test     # the Nim test suite
132just nim-lib      # libfrqcore.so into build/nim
135Both want `nix develop .#nim`, and re-enter it themselves if they are not
136already inside.

One frontend where there were three, and a core that is not Clojure 438b247 nandi 23h ago	1	# The Nim core
	2
	3	The portable half of frq, as a native library the Dart side calls through FFI.
	4
	5	## Why this exists
	6
	7	`common/` is ClojureDart, compiled into every target. That works, and the
	8	reason to move any of it is not that it is broken: it is that the logic under
	9	the screens — the IRC wire format, the atproto flows, the message signatures —
	10	is the part with the most rules per line and the least to do with Flutter, and
	11	it is the part worth having in a language with a type checker and a test runner
	12	that does not need a Flutter toolchain to run.
	13
	14	So the plan is a seam, not a rewrite-in-place. Each module moves one at a time:
	15	the Nim implementation lands here with tests, the Dart binding lands in
	16	`flutter/src/frq/core/`, and the ClojureDart original stays until the binding
	17	is proven against it. Nothing is deleted on faith.
	18
	19	## What is here
	20
	21	```
	22	src/frq_core.nim the C ABI: every exported symbol, and nothing else
	23	src/frq/ircparse.nim the IRC wire format
	24	tests/ one per module, run by `just nim-test`
	25	```
	26
	27	`src/frq_core.nim` is the only file that knows about C. Everything under
	28	`src/frq/` is ordinary Nim with ordinary Nim types, so the tests test the logic
	29	rather than the marshalling.
	30
	31	## The ABI
	32
	33	Strings in, strings out, and JSON where the answer is not a single string.
	34
	35	That is a deliberate choice against a struct-based ABI. A struct means the Dart
	36	side and the Nim side have to agree on a memory layout, and every field added
	37	later is a version skew that segfaults instead of failing. JSON costs a parse
	38	per call, which is nothing against a network round trip, and it lets one side
	39	gain a field without the other crashing.
	40
	41	Every function that returns a string returns memory the caller must free
	42	with `frq_free`. Nim's allocator is not Dart's; a `free()` from the Dart side
	43	on a Nim pointer is undefined. The bindings in `flutter/src/frq/core/` wrap
	44	that in a `try/finally` so no call site has to remember.
	45
	46	`frq_init` must be called once before anything else, and calls `NimMain` to set
	47	up Nim's runtime. The bindings do it on first use.
	48
	49	## The web
	50
	51	A native library does not load in a browser, so the web target cannot call this
	52	through `dart:ffi`. Nim compiles through C, so the route is emscripten to wasm
	53	and a JS binding rather than a second implementation — but that is not built
	54	yet, and until it is, **the web build must keep using the ClojureDart
	55	originals**. This is why the originals stay in `common/` rather than being
	56	deleted as each module lands: they are the web's implementation, not dead code.
	57
A message in #test, from Nim 35994d4 nandi 22h ago	58	## The spike
	59
	60	`just nim-spike` opens a window with no ClojureDart on the path, connects to
	61	irc.freeq.at over TLS, joins `#test` and sends a message. Nim owns the state,
	62	the screens, the socket and the IRC protocol; Dart owns the pixels.
	63
	64	```
	65	src/frq/ui.nim the widget tree, in the screens' own tag vocabulary
	66	src/frq/state.nim the record, the reducer, and drain()
	67	src/frq/irc.nim the socket, on its own thread, behind two channels
	68	src/frq/screens/ connect and chat, as pure functions of the state
	69	src/frq/trace.nim FRQ_TRACE=1, the same switch the rest of frq uses
	70	```
	71
	72	Three decisions worth knowing before changing any of it:
	73
	74	* Nothing is shared with the socket thread. Nim's ORC is thread-local for
	75	ref types, so sharing the state would mean a lock per field and a heap two
	76	threads both collect. The reader speaks only in channels, and `drain()` turns
	77	its output into state on whichever thread Dart called in on.
	78	* Dart polls; Nim never calls back. A Dart callback from a foreign thread
	79	has to be marshalled onto the main isolate — `NativeCallable`, ports, a whole
	80	mechanism — and at 70µs a render a 100ms timer does the same job for free.
	81	* A prop holds an event id, not a closure. That is the one thing hiccup has
	82	that a C ABI cannot, and substituting it is what makes this an architecture
	83	rather than a rendering trick.
	84
	85	Cost, measured: a full screen rebuild is 70–105µs, or 0.4–0.6% of a 60fps
	86	frame, for a 1.6KB tree. The caveat is the tree size rather than the number —
	87	the chat screen caps the backlog at fifty rows for exactly this reason, and
	88	nothing has measured what a real one costs.
	89
	90	```bash
	91	just nim-spike # the window
	92	just nim-spike-test # 7 widget tests: real taps, real widgets
	93	just nim-live # connect to a real freeq and say a line
	94	just nim-bench # what the boundary costs
	95	FRQ_TRACE=1 just nim-live # ...and every line on the wire
	96	```
	97
One frontend where there were three, and a core that is not Clojure 438b247 nandi 23h ago	98	## Status
	99
	100	`frq/ircparse.nim` is ported and tested — 29 cases, `just nim-test` — and the
	101	ABI is exercised from C through `dlopen`, including the allocation contract
	102	under a hundred thousand parse/free cycles. That half is real.
	103
The binding is Dart, and it works f7aea3b nandi 23h ago	104	The Dart binding is real too, and is `dart/frq_core` — **plain Dart, not
	105	ClojureDart**. It calls the library over `dart:ffi` and is covered by 20 tests
	106	on the Dart VM, including the UTF-8 round trip and ten thousand calls against
	107	the ownership rules. `just dart-test` runs the pair of them in about a second.
	108
	109	The binding was ClojureDart for one commit and should not have been: the Nim
	110	core exists to have less Clojure in the tree, and `lookupFunction` takes two
	111	type arguments, so it meant fighting generic interop to write more of the
	112	thing being removed. In Dart it is a typedef. See `dart/README.md`.
	113
A message in #test, from Nim 35994d4 nandi 22h ago	114	The spike above is wired up and runs. What is not done is replacing
	115	anything: the shipping app is still the ClojureDart one, `common/frq/irc/parse.cljc`
	116	is still what it runs, and the spike is a second entry point beside it
	117	(`lib/main_nim.dart`) rather than a replacement for `frq.main`. That step is its
The binding is Dart, and it works f7aea3b nandi 23h ago	118	own piece of work — the Flutter app takes the package as a path dependency
	119	(which means a `pubspec.lock` regeneration and widening the nix build's source
	120	root), the library has to reach each target (`jniLibs` for the APK, beside the
	121	executable for the desktop bundle), and only then can a call site choose Nim
	122	on native and the ClojureDart original on the web.
One frontend where there were three, and a core that is not Clojure 438b247 nandi 23h ago	123
	124	Modules still in `common/` and not yet here: `rooms`, `msgsig`, `crypto`,
	125	`atproto/core`, `oauth/core`, `store`, `irc/handshake`, `irc/mutate`,
	126	`profile`, `members`, `reactions`, `replies`, `edits`, `clock`.
	127
	128	## Building
	129
	130	```bash
	131	just nim-test # the Nim test suite
	132	just nim-lib # libfrqcore.so into build/nim
	133	```
	134
	135	Both want `nix develop .#nim`, and re-enter it themselves if they are not
	136	already inside.