nandi/frq · README.md

nandi/frqpublic⑂ Fork 0

⑂ 3760746302d1370dfe997aca01cf671e5d36a553

README.md · 143 lines · 6.5 KBmarkdown    Blame HistoryRaw
One frontend where there were three, and a core that is not Clojure   1# The Nim core
3The portable half of frq, as a native library the Dart side calls through FFI.
5## Why this exists
7`common/` is ClojureDart, compiled into every target. That works, and the
8reason to move any of it is not that it is broken: it is that the logic under
9the screens — the IRC wire format, the atproto flows, the message signatures —
10is the part with the most rules per line and the least to do with Flutter, and
11it is the part worth having in a language with a type checker and a test runner
12that does not need a Flutter toolchain to run.
14So the plan is a seam, not a rewrite-in-place. Each module moves one at a time:
15the Nim implementation lands here with tests, the Dart binding lands in
16`flutter/src/frq/core/`, and the ClojureDart original stays until the binding
17is proven against it. Nothing is deleted on faith.
19## What is here
22src/frq_core.nim        the C ABI: every exported symbol, and nothing else
23src/frq/ircparse.nim    the IRC wire format
24tests/                  one per module, run by `just nim-test`
27`src/frq_core.nim` is the only file that knows about C. Everything under
28`src/frq/` is ordinary Nim with ordinary Nim types, so the tests test the logic
29rather than the marshalling.
31
33Strings in, strings out, and JSON where the answer is not a single string.
35That is a deliberate choice against a struct-based ABI. A struct means the Dart
36side and the Nim side have to agree on a memory layout, and every field added
37later is a version skew that segfaults instead of failing. JSON costs a parse
38per call, which is nothing against a network round trip, and it lets one side
39gain a field without the other crashing.
41Every function that returns a string returns memory the **caller must free**
42with `frq_free`. Nim's allocator is not Dart's; a `free()` from the Dart side
43on a Nim pointer is undefined. The bindings in `flutter/src/frq/core/` wrap
44that in a `try/finally` so no call site has to remember.
46`frq_init` must be called once before anything else, and calls `NimMain` to set
47up Nim's runtime. The bindings do it on first use.
49
51A native library does not load in a browser, so the web target cannot call this
52through `dart:ffi`. Nim compiles through C, so the route is emscripten to wasm
53and a JS binding rather than a second implementation — but that is not built
54yet, and until it is, **the web build must keep using the ClojureDart
55originals**. This is why the originals stay in `common/` rather than being
56deleted as each module lands: they are the web's implementation, not dead code.
Delete the spike that owned the screens   58## What is wired up
A message in #test, from Nim   59
Delete the spike that owned the screens   60`just nim-app run` is the real client with the Nim core as its transport. Every
61screen, cell and action is the one that was already there; `frq.main-nim` is
62`frq.main` with one line changed.
A message in #test, from Nim   63
Delete the spike that owned the screens   65src/frq/conn.nim      the socket, the TLS, the line framing — on its own thread
66src/frq/ircparse.nim  the IRC wire format
67src/frq/trace.nim     FRQ_TRACE=1, the same switch the rest of frq uses
A message in #test, from Nim   68
Delete the spike that owned the screens   70The seam is `frq.net`, which already existed with two implementations;
71`flutter/src/frq/net/nim.cljd` is a third beside `frq.net.dart` and
72`frq.net.web`. Nim owns the socket and nothing above it — the line goes to the
73existing `frq.irc.parse`, so `on-msg` receives the same map from the same
74parser and nothing upstairs can tell which transport it is on.
76Two things to know before changing `conn.nim`:
A message in #test, from Nim   77
78* **Nothing is shared with the socket thread.** Nim's ORC is thread-local for
Delete the spike that owned the screens   79  ref types, so sharing state would mean a lock per field and a heap two
80  threads both collect. It speaks only in channels.
81* **Writes drain before reads.** IRC has the client speak first, and reading
82  first deadlocked completely: CAP/NICK/USER sat queued while the loop waited
83  for a server that had nothing to say until we registered.
85There was an experiment where Nim owned the state and the screens too. It is
86gone. It meant a 43-line chat screen standing in for 1,518 — no reactions, no
87replies, no images — and the path forward from it was rewriting every screen
88in Nim and losing all of that. It is at 1d62d1a if it is ever wanted.
A message in #test, from Nim   89
Delete the spike that owned the screens   91just nim-app run     # the real client, Nim transport
92just nim-test        # the Nim suite
93just dart-test       # the Dart side of the boundary
94just nim-lib         # libfrqcore.so into build/nim
A message in #test, from Nim   95
Delete the spike that owned the screens   96FRQ_TRACE=1 just nim-app run    # every line in and out, both languages
The window connects   98
99The GUI needs OpenSSL on its loader path, which the `flutter-desktop` shell
Delete the spike that owned the screens   100provides as `FRQ_OPENSSL_LIB` and the `nim-app` recipe prepends for the app
The window connects   101alone. Not set as `LD_LIBRARY_PATH` in the shell itself: that shell also runs
Delete the spike that owned the screens   102Flutter through nixGL, which does its own careful things to the loader path.
The window connects   103
One frontend where there were three, and a core that is not Clojure   104
106`frq/ircparse.nim` is ported and tested — 29 cases, `just nim-test` — and the
107ABI is exercised from C through `dlopen`, including the allocation contract
108under a hundred thousand parse/free cycles. That half is real.
The binding is Dart, and it works   110The Dart binding is real too, and is `dart/frq_core` — **plain Dart, not
111ClojureDart**. It calls the library over `dart:ffi` and is covered by 20 tests
112on the Dart VM, including the UTF-8 round trip and ten thousand calls against
113the ownership rules. `just dart-test` runs the pair of them in about a second.
115The binding was ClojureDart for one commit and should not have been: the Nim
116core exists to have less Clojure in the tree, and `lookupFunction` takes two
117type arguments, so it meant fighting generic interop to write more of the
118thing being removed. In Dart it is a typedef. See `dart/README.md`.
Delete the spike that owned the screens   120The transport is wired up and runs. What is **not** done is replacing
121anything else: `frq.main` is untouched and still installs `frq.net.dart`, so
122the shipping app is unchanged and `frq.main-nim` is a second entry point
123beside it. `common/frq/irc/parse.cljc` is still what does the parsing on every
124target, including this one. That step is its
The binding is Dart, and it works   125own piece of work — the Flutter app takes the package as a path dependency
126(which means a `pubspec.lock` regeneration and widening the nix build's source
127root), the library has to reach each target (`jniLibs` for the APK, beside the
128executable for the desktop bundle), and only then can a call site choose Nim
129on native and the ClojureDart original on the web.
One frontend where there were three, and a core that is not Clojure   130
131Modules still in `common/` and not yet here: `rooms`, `msgsig`, `crypto`,
132`atproto/core`, `oauth/core`, `store`, `irc/handshake`, `irc/mutate`,
133`profile`, `members`, `reactions`, `replies`, `edits`, `clock`.
135## Building
138just nim-test     # the Nim test suite
139just nim-lib      # libfrqcore.so into build/nim
142Both want `nix develop .#nim`, and re-enter it themselves if they are not
143already inside.

One frontend where there were three, and a core that is not Clojure 438b247 nandi 21h ago	1	# The Nim core
	2
	3	The portable half of frq, as a native library the Dart side calls through FFI.
	4
	5	## Why this exists
	6
	7	`common/` is ClojureDart, compiled into every target. That works, and the
	8	reason to move any of it is not that it is broken: it is that the logic under
	9	the screens — the IRC wire format, the atproto flows, the message signatures —
	10	is the part with the most rules per line and the least to do with Flutter, and
	11	it is the part worth having in a language with a type checker and a test runner
	12	that does not need a Flutter toolchain to run.
	13
	14	So the plan is a seam, not a rewrite-in-place. Each module moves one at a time:
	15	the Nim implementation lands here with tests, the Dart binding lands in
	16	`flutter/src/frq/core/`, and the ClojureDart original stays until the binding
	17	is proven against it. Nothing is deleted on faith.
	18
	19	## What is here
	20
	21	```
	22	src/frq_core.nim the C ABI: every exported symbol, and nothing else
	23	src/frq/ircparse.nim the IRC wire format
	24	tests/ one per module, run by `just nim-test`
	25	```
	26
	27	`src/frq_core.nim` is the only file that knows about C. Everything under
	28	`src/frq/` is ordinary Nim with ordinary Nim types, so the tests test the logic
	29	rather than the marshalling.
	30
	31	## The ABI
	32
	33	Strings in, strings out, and JSON where the answer is not a single string.
	34
	35	That is a deliberate choice against a struct-based ABI. A struct means the Dart
	36	side and the Nim side have to agree on a memory layout, and every field added
	37	later is a version skew that segfaults instead of failing. JSON costs a parse
	38	per call, which is nothing against a network round trip, and it lets one side
	39	gain a field without the other crashing.
	40
	41	Every function that returns a string returns memory the caller must free
	42	with `frq_free`. Nim's allocator is not Dart's; a `free()` from the Dart side
	43	on a Nim pointer is undefined. The bindings in `flutter/src/frq/core/` wrap
	44	that in a `try/finally` so no call site has to remember.
	45
	46	`frq_init` must be called once before anything else, and calls `NimMain` to set
	47	up Nim's runtime. The bindings do it on first use.
	48
	49	## The web
	50
	51	A native library does not load in a browser, so the web target cannot call this
	52	through `dart:ffi`. Nim compiles through C, so the route is emscripten to wasm
	53	and a JS binding rather than a second implementation — but that is not built
	54	yet, and until it is, **the web build must keep using the ClojureDart
	55	originals**. This is why the originals stay in `common/` rather than being
	56	deleted as each module lands: they are the web's implementation, not dead code.
	57
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	58	## What is wired up
A message in #test, from Nim 35994d4 nandi 20h ago	59
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	60	`just nim-app run` is the real client with the Nim core as its transport. Every
	61	screen, cell and action is the one that was already there; `frq.main-nim` is
	62	`frq.main` with one line changed.
A message in #test, from Nim 35994d4 nandi 20h ago	63
A message in #test, from Nim 35994d4 nandi 20h ago	64	```
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	65	src/frq/conn.nim the socket, the TLS, the line framing — on its own thread
	66	src/frq/ircparse.nim the IRC wire format
	67	src/frq/trace.nim FRQ_TRACE=1, the same switch the rest of frq uses
A message in #test, from Nim 35994d4 nandi 20h ago	68	```
A message in #test, from Nim 35994d4 nandi 20h ago	69
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	70	The seam is `frq.net`, which already existed with two implementations;
	71	`flutter/src/frq/net/nim.cljd` is a third beside `frq.net.dart` and
	72	`frq.net.web`. Nim owns the socket and nothing above it — the line goes to the
	73	existing `frq.irc.parse`, so `on-msg` receives the same map from the same
	74	parser and nothing upstairs can tell which transport it is on.
	75
	76	Two things to know before changing `conn.nim`:
A message in #test, from Nim 35994d4 nandi 20h ago	77
A message in #test, from Nim 35994d4 nandi 20h ago	78	* Nothing is shared with the socket thread. Nim's ORC is thread-local for
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	79	ref types, so sharing state would mean a lock per field and a heap two
	80	threads both collect. It speaks only in channels.
	81	* Writes drain before reads. IRC has the client speak first, and reading
	82	first deadlocked completely: CAP/NICK/USER sat queued while the loop waited
	83	for a server that had nothing to say until we registered.
	84
	85	There was an experiment where Nim owned the state and the screens too. It is
	86	gone. It meant a 43-line chat screen standing in for 1,518 — no reactions, no
	87	replies, no images — and the path forward from it was rewriting every screen
	88	in Nim and losing all of that. It is at 1d62d1a if it is ever wanted.
A message in #test, from Nim 35994d4 nandi 20h ago	89
A message in #test, from Nim 35994d4 nandi 20h ago	90	```bash
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	91	just nim-app run # the real client, Nim transport
	92	just nim-test # the Nim suite
	93	just dart-test # the Dart side of the boundary
	94	just nim-lib # libfrqcore.so into build/nim
A message in #test, from Nim 35994d4 nandi 20h ago	95
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	96	FRQ_TRACE=1 just nim-app run # every line in and out, both languages
	97	```
The window connects 1d62d1a nandi 20h ago	98
The window connects 1d62d1a nandi 20h ago	99	The GUI needs OpenSSL on its loader path, which the `flutter-desktop` shell
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	100	provides as `FRQ_OPENSSL_LIB` and the `nim-app` recipe prepends for the app
The window connects 1d62d1a nandi 20h ago	101	alone. Not set as `LD_LIBRARY_PATH` in the shell itself: that shell also runs
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	102	Flutter through nixGL, which does its own careful things to the loader path.
The window connects 1d62d1a nandi 20h ago	103
One frontend where there were three, and a core that is not Clojure 438b247 nandi 21h ago	104	## Status
	105
	106	`frq/ircparse.nim` is ported and tested — 29 cases, `just nim-test` — and the
	107	ABI is exercised from C through `dlopen`, including the allocation contract
	108	under a hundred thousand parse/free cycles. That half is real.
	109
The binding is Dart, and it works f7aea3b nandi 21h ago	110	The Dart binding is real too, and is `dart/frq_core` — **plain Dart, not
	111	ClojureDart**. It calls the library over `dart:ffi` and is covered by 20 tests
	112	on the Dart VM, including the UTF-8 round trip and ten thousand calls against
	113	the ownership rules. `just dart-test` runs the pair of them in about a second.
	114
	115	The binding was ClojureDart for one commit and should not have been: the Nim
	116	core exists to have less Clojure in the tree, and `lookupFunction` takes two
	117	type arguments, so it meant fighting generic interop to write more of the
	118	thing being removed. In Dart it is a typedef. See `dart/README.md`.
	119
Delete the spike that owned the screens 1bb3f77 nandi 20h ago	120	The transport is wired up and runs. What is not done is replacing
	121	anything else: `frq.main` is untouched and still installs `frq.net.dart`, so
	122	the shipping app is unchanged and `frq.main-nim` is a second entry point
	123	beside it. `common/frq/irc/parse.cljc` is still what does the parsing on every
	124	target, including this one. That step is its
The binding is Dart, and it works f7aea3b nandi 21h ago	125	own piece of work — the Flutter app takes the package as a path dependency
	126	(which means a `pubspec.lock` regeneration and widening the nix build's source
	127	root), the library has to reach each target (`jniLibs` for the APK, beside the
	128	executable for the desktop bundle), and only then can a call site choose Nim
	129	on native and the ClojureDart original on the web.
One frontend where there were three, and a core that is not Clojure 438b247 nandi 21h ago	130
	131	Modules still in `common/` and not yet here: `rooms`, `msgsig`, `crypto`,
	132	`atproto/core`, `oauth/core`, `store`, `irc/handshake`, `irc/mutate`,
	133	`profile`, `members`, `reactions`, `replies`, `edits`, `clock`.
	134
	135	## Building
	136
	137	```bash
	138	just nim-test # the Nim test suite
	139	just nim-lib # libfrqcore.so into build/nim
	140	```
	141
	142	Both want `nix develop .#nim`, and re-enter it themselves if they are not
	143	already inside.