nandi/frq · README.md

nandi/frqpublic⑂ Fork 0

⑂ 4dfc71908cc3f12174bb0d6dd8688ff3164879c3

README.md · 144 lines · 6.6 KBmarkdown    Blame HistoryRaw
One frontend where there were three, and a core that is not Clojure   1# The Nim core
3The portable half of frq, as a native library the Dart side calls through FFI.
5## Why this exists
7`common/` is ClojureDart, compiled into every target. That works, and the
8reason to move any of it is not that it is broken: it is that the logic under
9the screens — the IRC wire format, the atproto flows, the message signatures —
10is the part with the most rules per line and the least to do with Flutter, and
11it is the part worth having in a language with a type checker and a test runner
12that does not need a Flutter toolchain to run.
14So the plan is a seam, not a rewrite-in-place. Each module moves one at a time:
15the Nim implementation lands here with tests, the Dart binding lands in
16`flutter/src/frq/core/`, and the ClojureDart original stays until the binding
17is proven against it. Nothing is deleted on faith.
19## What is here
22src/frq_core.nim        the C ABI: every exported symbol, and nothing else
23src/frq/ircparse.nim    the IRC wire format
Six verbs, and the last of the nix   24tests/                  one per module, run by `just test nim`
One frontend where there were three, and a core that is not Clojure   25
27`src/frq_core.nim` is the only file that knows about C. Everything under
28`src/frq/` is ordinary Nim with ordinary Nim types, so the tests test the logic
29rather than the marshalling.
31
33Strings in, strings out, and JSON where the answer is not a single string.
35That is a deliberate choice against a struct-based ABI. A struct means the Dart
36side and the Nim side have to agree on a memory layout, and every field added
37later is a version skew that segfaults instead of failing. JSON costs a parse
38per call, which is nothing against a network round trip, and it lets one side
39gain a field without the other crashing.
41Every function that returns a string returns memory the **caller must free**
42with `frq_free`. Nim's allocator is not Dart's; a `free()` from the Dart side
43on a Nim pointer is undefined. The bindings in `flutter/src/frq/core/` wrap
44that in a `try/finally` so no call site has to remember.
46`frq_init` must be called once before anything else, and calls `NimMain` to set
47up Nim's runtime. The bindings do it on first use.
49
51A native library does not load in a browser, so the web target cannot call this
52through `dart:ffi`. Nim compiles through C, so the route is emscripten to wasm
53and a JS binding rather than a second implementation — but that is not built
54yet, and until it is, **the web build must keep using the ClojureDart
55originals**. This is why the originals stay in `common/` rather than being
56deleted as each module lands: they are the web's implementation, not dead code.
Delete the spike that owned the screens   58## What is wired up
A message in #test, from Nim   59
Six verbs, and the last of the nix   60`just run app` is the real client with the Nim core as its transport. Every
Delete the spike that owned the screens   61screen, cell and action is the one that was already there; `frq.main-nim` is
62`frq.main` with one line changed.
A message in #test, from Nim   63
Delete the spike that owned the screens   65src/frq/conn.nim      the socket, the TLS, the line framing — on its own thread
66src/frq/ircparse.nim  the IRC wire format
67src/frq/trace.nim     FRQ_TRACE=1, the same switch the rest of frq uses
A message in #test, from Nim   68
Delete the spike that owned the screens   70The seam is `frq.net`, which already existed with two implementations;
71`flutter/src/frq/net/nim.cljd` is a third beside `frq.net.dart` and
72`frq.net.web`. Nim owns the socket and nothing above it — the line goes to the
73existing `frq.irc.parse`, so `on-msg` receives the same map from the same
74parser and nothing upstairs can tell which transport it is on.
76Two things to know before changing `conn.nim`:
A message in #test, from Nim   77
78* **Nothing is shared with the socket thread.** Nim's ORC is thread-local for
Delete the spike that owned the screens   79  ref types, so sharing state would mean a lock per field and a heap two
80  threads both collect. It speaks only in channels.
81* **Writes drain before reads.** IRC has the client speak first, and reading
82  first deadlocked completely: CAP/NICK/USER sat queued while the loop waited
83  for a server that had nothing to say until we registered.
85There was an experiment where Nim owned the state and the screens too. It is
86gone. It meant a 43-line chat screen standing in for 1,518 — no reactions, no
87replies, no images — and the path forward from it was rewriting every screen
88in Nim and losing all of that. It is at 1d62d1a if it is ever wanted.
A message in #test, from Nim   89
Six verbs, and the last of the nix   91just run app         # the real client, Nim transport
92just test nim        # the Nim suite
93just test dart       # the Dart side of the boundary
94just build lib       # libfrqcore.so into build/nim
A message in #test, from Nim   95
Six verbs, and the last of the nix   96FRQ_TRACE=1 just run app    # every line in and out, both languages
Delete the spike that owned the screens   97
The window connects   98
Six verbs, and the last of the nix   99The GUI needs OpenSSL on its loader path: Nim resolves the entry points
100through dynlib at run time, and without the library there `newContext` dies in
101a SIGSEGV that says nothing about SSL. The recipe prepends `FRQ_OPENSSL_LIB`
102for the app alone, so a host whose libssl is somewhere unusual has one variable
103to set rather than an `LD_LIBRARY_PATH` to inherit.
The window connects   104
One frontend where there were three, and a core that is not Clojure   105
Six verbs, and the last of the nix   107`frq/ircparse.nim` is ported and tested — 29 cases, `just test nim` — and the
One frontend where there were three, and a core that is not Clojure   108ABI is exercised from C through `dlopen`, including the allocation contract
109under a hundred thousand parse/free cycles. That half is real.
The binding is Dart, and it works   111The Dart binding is real too, and is `dart/frq_core` — **plain Dart, not
112ClojureDart**. It calls the library over `dart:ffi` and is covered by 20 tests
113on the Dart VM, including the UTF-8 round trip and ten thousand calls against
Six verbs, and the last of the nix   114the ownership rules. `just test dart` runs the pair of them in about a second.
The binding is Dart, and it works   115
116The binding was ClojureDart for one commit and should not have been: the Nim
117core exists to have less Clojure in the tree, and `lookupFunction` takes two
118type arguments, so it meant fighting generic interop to write more of the
119thing being removed. In Dart it is a typedef. See `dart/README.md`.
Delete the spike that owned the screens   121The transport is wired up and runs. What is **not** done is replacing
122anything else: `frq.main` is untouched and still installs `frq.net.dart`, so
123the shipping app is unchanged and `frq.main-nim` is a second entry point
124beside it. `common/frq/irc/parse.cljc` is still what does the parsing on every
125target, including this one. That step is its
The binding is Dart, and it works   126own piece of work — the Flutter app takes the package as a path dependency
Six verbs, and the last of the nix   127(which means a `pubspec.lock` regeneration), the library has to reach each target (`jniLibs` for the APK, beside the
The binding is Dart, and it works   128executable for the desktop bundle), and only then can a call site choose Nim
129on native and the ClojureDart original on the web.
One frontend where there were three, and a core that is not Clojure   130
131Modules still in `common/` and not yet here: `rooms`, `msgsig`, `crypto`,
132`atproto/core`, `oauth/core`, `store`, `irc/handshake`, `irc/mutate`,
133`profile`, `members`, `reactions`, `replies`, `edits`, `clock`.
135## Building
Six verbs, and the last of the nix   138just test nim     # the Nim test suite
139just build lib    # libfrqcore.so into build/nim
One frontend where there were three, and a core that is not Clojure   140
Six verbs, and the last of the nix   142Both run out of `.toolchain/`, which `tools/toolchain.sh` fills on first
143use. What they want from the host is a C compiler -- `nim c` shells out to one
144-- and OpenSSL.

One frontend where there were three, and a core that is not Clojure 438b247 nandi 16h ago	1	# The Nim core
	2
	3	The portable half of frq, as a native library the Dart side calls through FFI.
	4
	5	## Why this exists
	6
	7	`common/` is ClojureDart, compiled into every target. That works, and the
	8	reason to move any of it is not that it is broken: it is that the logic under
	9	the screens — the IRC wire format, the atproto flows, the message signatures —
	10	is the part with the most rules per line and the least to do with Flutter, and
	11	it is the part worth having in a language with a type checker and a test runner
	12	that does not need a Flutter toolchain to run.
	13
	14	So the plan is a seam, not a rewrite-in-place. Each module moves one at a time:
	15	the Nim implementation lands here with tests, the Dart binding lands in
	16	`flutter/src/frq/core/`, and the ClojureDart original stays until the binding
	17	is proven against it. Nothing is deleted on faith.
	18
	19	## What is here
	20
	21	```
	22	src/frq_core.nim the C ABI: every exported symbol, and nothing else
	23	src/frq/ircparse.nim the IRC wire format
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	24	tests/ one per module, run by `just test nim`
One frontend where there were three, and a core that is not Clojure 438b247 nandi 16h ago	25	```
	26
	27	`src/frq_core.nim` is the only file that knows about C. Everything under
	28	`src/frq/` is ordinary Nim with ordinary Nim types, so the tests test the logic
	29	rather than the marshalling.
	30
	31	## The ABI
	32
	33	Strings in, strings out, and JSON where the answer is not a single string.
	34
	35	That is a deliberate choice against a struct-based ABI. A struct means the Dart
	36	side and the Nim side have to agree on a memory layout, and every field added
	37	later is a version skew that segfaults instead of failing. JSON costs a parse
	38	per call, which is nothing against a network round trip, and it lets one side
	39	gain a field without the other crashing.
	40
	41	Every function that returns a string returns memory the caller must free
	42	with `frq_free`. Nim's allocator is not Dart's; a `free()` from the Dart side
	43	on a Nim pointer is undefined. The bindings in `flutter/src/frq/core/` wrap
	44	that in a `try/finally` so no call site has to remember.
	45
	46	`frq_init` must be called once before anything else, and calls `NimMain` to set
	47	up Nim's runtime. The bindings do it on first use.
	48
	49	## The web
	50
	51	A native library does not load in a browser, so the web target cannot call this
	52	through `dart:ffi`. Nim compiles through C, so the route is emscripten to wasm
	53	and a JS binding rather than a second implementation — but that is not built
	54	yet, and until it is, **the web build must keep using the ClojureDart
	55	originals**. This is why the originals stay in `common/` rather than being
	56	deleted as each module lands: they are the web's implementation, not dead code.
	57
Delete the spike that owned the screens 1bb3f77 nandi 15h ago	58	## What is wired up
A message in #test, from Nim 35994d4 nandi 15h ago	59
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	60	`just run app` is the real client with the Nim core as its transport. Every
Delete the spike that owned the screens 1bb3f77 nandi 15h ago	61	screen, cell and action is the one that was already there; `frq.main-nim` is
	62	`frq.main` with one line changed.
A message in #test, from Nim 35994d4 nandi 15h ago	63
A message in #test, from Nim 35994d4 nandi 15h ago	64	```
Delete the spike that owned the screens 1bb3f77 nandi 15h ago	65	src/frq/conn.nim the socket, the TLS, the line framing — on its own thread
	66	src/frq/ircparse.nim the IRC wire format
	67	src/frq/trace.nim FRQ_TRACE=1, the same switch the rest of frq uses
A message in #test, from Nim 35994d4 nandi 15h ago	68	```
A message in #test, from Nim 35994d4 nandi 15h ago	69
Delete the spike that owned the screens 1bb3f77 nandi 15h ago	70	The seam is `frq.net`, which already existed with two implementations;
	71	`flutter/src/frq/net/nim.cljd` is a third beside `frq.net.dart` and
	72	`frq.net.web`. Nim owns the socket and nothing above it — the line goes to the
	73	existing `frq.irc.parse`, so `on-msg` receives the same map from the same
	74	parser and nothing upstairs can tell which transport it is on.
	75
	76	Two things to know before changing `conn.nim`:
A message in #test, from Nim 35994d4 nandi 15h ago	77
A message in #test, from Nim 35994d4 nandi 15h ago	78	* Nothing is shared with the socket thread. Nim's ORC is thread-local for
Delete the spike that owned the screens 1bb3f77 nandi 15h ago	79	ref types, so sharing state would mean a lock per field and a heap two
	80	threads both collect. It speaks only in channels.
	81	* Writes drain before reads. IRC has the client speak first, and reading
	82	first deadlocked completely: CAP/NICK/USER sat queued while the loop waited
	83	for a server that had nothing to say until we registered.
	84
	85	There was an experiment where Nim owned the state and the screens too. It is
	86	gone. It meant a 43-line chat screen standing in for 1,518 — no reactions, no
	87	replies, no images — and the path forward from it was rewriting every screen
	88	in Nim and losing all of that. It is at 1d62d1a if it is ever wanted.
A message in #test, from Nim 35994d4 nandi 15h ago	89
A message in #test, from Nim 35994d4 nandi 15h ago	90	```bash
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	91	just run app # the real client, Nim transport
	92	just test nim # the Nim suite
	93	just test dart # the Dart side of the boundary
	94	just build lib # libfrqcore.so into build/nim
A message in #test, from Nim 35994d4 nandi 15h ago	95
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	96	FRQ_TRACE=1 just run app # every line in and out, both languages
Delete the spike that owned the screens 1bb3f77 nandi 15h ago	97	```
The window connects 1d62d1a nandi 15h ago	98
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	99	The GUI needs OpenSSL on its loader path: Nim resolves the entry points
	100	through dynlib at run time, and without the library there `newContext` dies in
	101	a SIGSEGV that says nothing about SSL. The recipe prepends `FRQ_OPENSSL_LIB`
	102	for the app alone, so a host whose libssl is somewhere unusual has one variable
	103	to set rather than an `LD_LIBRARY_PATH` to inherit.
The window connects 1d62d1a nandi 15h ago	104
One frontend where there were three, and a core that is not Clojure 438b247 nandi 16h ago	105	## Status
	106
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	107	`frq/ircparse.nim` is ported and tested — 29 cases, `just test nim` — and the
One frontend where there were three, and a core that is not Clojure 438b247 nandi 16h ago	108	ABI is exercised from C through `dlopen`, including the allocation contract
	109	under a hundred thousand parse/free cycles. That half is real.
	110
The binding is Dart, and it works f7aea3b nandi 16h ago	111	The Dart binding is real too, and is `dart/frq_core` — **plain Dart, not
	112	ClojureDart**. It calls the library over `dart:ffi` and is covered by 20 tests
	113	on the Dart VM, including the UTF-8 round trip and ten thousand calls against
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	114	the ownership rules. `just test dart` runs the pair of them in about a second.
The binding is Dart, and it works f7aea3b nandi 16h ago	115
	116	The binding was ClojureDart for one commit and should not have been: the Nim
	117	core exists to have less Clojure in the tree, and `lookupFunction` takes two
	118	type arguments, so it meant fighting generic interop to write more of the
	119	thing being removed. In Dart it is a typedef. See `dart/README.md`.
	120
Delete the spike that owned the screens 1bb3f77 nandi 15h ago	121	The transport is wired up and runs. What is not done is replacing
	122	anything else: `frq.main` is untouched and still installs `frq.net.dart`, so
	123	the shipping app is unchanged and `frq.main-nim` is a second entry point
	124	beside it. `common/frq/irc/parse.cljc` is still what does the parsing on every
	125	target, including this one. That step is its
The binding is Dart, and it works f7aea3b nandi 16h ago	126	own piece of work — the Flutter app takes the package as a path dependency
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	127	(which means a `pubspec.lock` regeneration), the library has to reach each target (`jniLibs` for the APK, beside the
The binding is Dart, and it works f7aea3b nandi 16h ago	128	executable for the desktop bundle), and only then can a call site choose Nim
The binding is Dart, and it works f7aea3b nandi 16h ago	129	on native and the ClojureDart original on the web.
One frontend where there were three, and a core that is not Clojure 438b247 nandi 16h ago	130
	131	Modules still in `common/` and not yet here: `rooms`, `msgsig`, `crypto`,
	132	`atproto/core`, `oauth/core`, `store`, `irc/handshake`, `irc/mutate`,
	133	`profile`, `members`, `reactions`, `replies`, `edits`, `clock`.
	134
	135	## Building
	136
	137	```bash
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	138	just test nim # the Nim test suite
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	139	just build lib # libfrqcore.so into build/nim
One frontend where there were three, and a core that is not Clojure 438b247 nandi 16h ago	140	```
	141
Six verbs, and the last of the nix 2e24e64 nandi 12h ago	142	Both run out of `.toolchain/`, which `tools/toolchain.sh` fills on first
	143	use. What they want from the host is a C compiler -- `nim c` shells out to one
	144	-- and OpenSSL.