nandi/jolt-native · README.md

⑂ 2bd393edaab22fea5afe152ea9a5a2d7b0296b30

README.md · 174 lines · 8.2 KBmarkdown    Blame HistoryRaw
Lift freeq's AV media plane out of sleek   1# jolt-native
3Native capabilities for [jolt](https://github.com/jolt-lang/jolt), one shared
4object per capability.
6A jolt library binds a `.so` through `jolt.ffi/defcfn`: it declares typed
7foreign functions and gets back integers, doubles and borrowed UTF-8 strings.
8That is the whole vocabulary. These crates are the other side of that
9boundary — the things worth writing in Rust because writing them again in
10anything else would mean writing them badly.
Move the jolt libraries under glimmer-backends   13crates/jolt-abi                 the rules every library here keeps at its edge
14crates/jolt-vidya               the retained-tree C ABI  → libvidya.so
15crates/vidya-core               the egui semantic layer behind it (theme, widgets, fonts)
16crates/jolt-tui                 the same tree ABI, painted into a terminal → libjolttui.so
17crates/jolt-moq                 freeq's AV media plane   → libjoltmoq.so
18glimmer-backends/glimmer-vidya  the jolt side of libvidya: glimmer's backend
Lift freeq's AV media plane out of sleek   19
Paint the same tree into a terminal, for the machines with no window   21All three objects land in one `target/release`, so a consumer points
Bring vidya in   22`LD_LIBRARY_PATH` at one directory and names in `:jolt/native` only the ones it
23actually wants.
Lift freeq's AV media plane out of sleek   25## Why a workspace and not one library with features
27Because cargo features are additive *within* a library. A single
28`libjoltnative.so` whose `graphics` and `moq` features selected the parts would
29link egui **and** QUIC, Opus, H.264, cpal and V4L2 into one object, so a chat
30client that is not in a call would carry the whole media plane in order to open
31a window. The feature unification that comes with it has bitten this code
32before: enabling eframe's `wayland`/`x11` features for a desktop build pulled
33them into the Android one and made the activity exit on launch.
35A workspace keeps the parts of a monorepo worth having — one CI, one release,
36shared conventions in `jolt-abi` — while each consumer links only what it asked
37for. jolt's `:jolt/native` is a vector; naming two objects there is no harder
38than naming one.
40## Building
Point the docs at the flake   42The flake is the toolchain. `nix develop` is where the compiler, the C
43toolchain, PipeWire's headers and the kernel headers bindgen wants come from,
44so **nothing needs to be installed** beyond nix itself — no `cc` on PATH, no
45rustup, no NDK.
Lift freeq's AV media plane out of sleek   47
Point the docs at the flake   48nix develop                 # then:
49just build                  # release .so files in target/release
50just test                   # the whole workspace
51just libdir                 # where to point LD_LIBRARY_PATH
Lift freeq's AV media plane out of sleek   52
Point the docs at the flake   54`just build` is the edit loop; the nix outputs are the artifact, built against
55a pinned compiler and a pinned crate graph:
Lift freeq's AV media plane out of sleek   56
Point the docs at the flake   57
58nix build                   # all three objects  → result/lib
59nix build .#android         # both device objects → result/lib/arm64-v8a
60nix flake check             # fmt, clippy, tests, all three objects
Lift freeq's AV media plane out of sleek   62
Point the docs at the flake   63Two things are worth knowing about:
65* **bindgen runs libclang directly**, so `v4l2r` needs to be pointed at
66  `linux/videodev2.h`. The flake sets `V4L2R_VIDEODEV2_H_PATH` and
67  `BINDGEN_EXTRA_CLANG_ARGS` from nixpkgs' `linuxHeaders` in both the devShell
68  and the build, which is why building inside `nix develop` rather than a bare
69  shell is the supported path.
Lift freeq's AV media plane out of sleek   70* **PipeWire** is a real system dependency — `libspa-sys` wants `pkg-config`
Point the docs at the flake   71  and PipeWire's headers, and the flake supplies both. It is behind the
72  default-on `pipewire` feature of `jolt-moq`; `just test-minimal` builds
73  without it, for a machine outside the flake. Turning it off costs device
74  *names*: only ALSA PCMs remain visible ("pipewire", "sysdefault") rather than
75  the real sources a browser would offer, which makes picking a microphone
76  guesswork.
78### Android
81just ffi-android            # nix build .#android
84Same sources and same `Cargo.lock` as the desktop objects; only the target
85configuration moves. The NDK comes from nixpkgs, so this needs nothing
86installed either. The camera wants two things from the APK that the build
87cannot supply: a `CameraCapture` class in its `classes.dex`, and a call to
88`joltmoq_android_init` — see `android/jolt_main.c`.
Lift freeq's AV media plane out of sleek   89
90## jolt-moq
92freeq's audio and video calls, extracted from
93[sleek](https://github.com/codegod100/sleek), which reached them by calling
94Rust from Rust.
96A freeq call has two halves, and only one of them is here.
98**Signaling stayed behind.** A call is opened, joined and left over IRC
99TAGMSGs — `+freeq.at/av-start` and its siblings — and the server broadcasts
100`+freeq.at/av-state` back. Any client that speaks IRC already has everything it
101needs for that, in whatever language it speaks IRC in; crossing an FFI boundary
102to send a TAGMSG would be worse than not crossing one. This library begins once
103the session id and the SFU token are known.
105**The media plane came across**, because it is not a thing to write twice: MoQ
106over QUIC, Opus, H.264, camera capture and the SFU's own dial rules come to
107some three thousand lines that would say the same thing again in another
108language and be wrong in different places.
110`crates/jolt-moq/include/joltmoq.h` is the surface, and says more about each
111call than this does. The shape of it:
113* **Nothing calls back.** Status is drained with `joltmoq_poll_status`, video
114  with `joltmoq_frame_poll`, both from whatever thread the caller paints on. A
115  callback into a foreign runtime from a tokio worker is a rule about threads
116  that the caller has to keep and that nothing can check.
117* **Frames are borrowed, not copied.** `joltmoq_frame_rgba` points into the
118  decoder's own buffer until the next poll. A frame is a megabyte or two;
119  copying it out so the caller can hand it straight to a texture upload would
120  be two copies a frame for nothing.
121* **One call at a time.** There is one microphone, so there is one session.
123### Its relationship to sleek
Let the media plane cross to the phone, camera and all   125`av.rs`, `av_media.rs`, `v4l2cam.rs`, `android_camera.rs` and `nv12_orient.rs`
126came from sleek's `android/src`. They are kept close to their originals so changes can still be moved between the
Lift freeq's AV media plane out of sleek   127two by eye. What was dropped on the way was the signaling-and-presentation
128half — `ChannelCall`, `LocalCall`, `apply_av_state`, `av_state_message` —
129which is sleek's own app state and belongs to a client, not to a media plane.
130Dropping it also dropped the `freeq-sdk` dependency entirely.
132Sleek still has its own copy. Pointing it at this crate instead is the obvious
133next step and has not been taken yet; until it is, a fix made in one place
134needs making in the other.
Let the media plane cross to the phone, camera and all   136`android_camera.rs` is the one that did not come across unchanged. Sleek is a
137single shared object, so its Camera2 bridge reads the `JavaVM` and the Activity
138straight out of the `AndroidApp` that android-activity handed it. Here the glue
139is in `libvidya.so` and the media plane is not, so the handles arrive through
140`joltmoq_android_init` instead — see `android_jni.rs` for why the usual bridge,
141`ndk_context`, cannot cross two cdylibs. Everything below that line is sleek's.
Bring vidya in   143
145The theme layer, the widget set and the retained-tree ABI that glimmer paints
146through. Imported from its own repo rather than depended on: egui is the only
147backend anyone wants now, so the indirection that let `libvidya` be swapped for
148a raylib or cimgui build of the same symbols was paying for a choice nobody was
149going to make.
151`crates/vidya-core` is the egui layer — and the 4MB under its `assets/` is not
152incidental: the Twemoji pack and the symbols font are `include_bytes!`d into
153the library, because a shared object has no host app to install a font set for
Move the jolt libraries under glimmer-backends   156`glimmer-backends/glimmer-vidya` is the other side of that ABI, and is jolt
157rather than Rust. A consumer takes it as a `:local/root` dep and `libvidya.so` on the
Bring vidya in   158loader path — frq's `deps.edn` is the worked example.
Lift freeq's AV media plane out of sleek   160## Adding a crate
162One crate, one `.so`, one capability. Depend on `jolt-abi` and keep its three
163rules — catch panics at every entry point, lend strings from a `Scratch` rather
164than handing out memory to free, and let the caller ask rather than calling it
165back. A consumer then names your object in `:jolt/native` alongside whichever
166others it wants, and pays for nothing else.
Say which licence, in the place people look for it   167
168
170MIT — the same as the projects this code came from: `crates/vidya-*` and
Move the jolt libraries under glimmer-backends   171`glimmer-backends/glimmer-vidya` from [vidya](https://tangled.org/nandi.uk/vidya),
Say which licence, in the place people look for it   172`crates/jolt-moq`'s media plane from
173[sleek](https://github.com/codegod100/sleek). `third-party/cpal` is a vendored
174pin of [cpal](https://github.com/RustAudio/cpal) and keeps its own licence.

Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	1	# jolt-native
	2
	3	Native capabilities for [jolt](https://github.com/jolt-lang/jolt), one shared
	4	object per capability.
	5
	6	A jolt library binds a `.so` through `jolt.ffi/defcfn`: it declares typed
	7	foreign functions and gets back integers, doubles and borrowed UTF-8 strings.
	8	That is the whole vocabulary. These crates are the other side of that
	9	boundary — the things worth writing in Rust because writing them again in
	10	anything else would mean writing them badly.
	11
	12	```
Move the jolt libraries under glimmer-backends 19df0d8 nandi 12d ago	13	crates/jolt-abi the rules every library here keeps at its edge
	14	crates/jolt-vidya the retained-tree C ABI → libvidya.so
	15	crates/vidya-core the egui semantic layer behind it (theme, widgets, fonts)
	16	crates/jolt-tui the same tree ABI, painted into a terminal → libjolttui.so
	17	crates/jolt-moq freeq's AV media plane → libjoltmoq.so
	18	glimmer-backends/glimmer-vidya the jolt side of libvidya: glimmer's backend
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	19	```
	20
Paint the same tree into a terminal, for the machines with no window a7f6202 nandi 17d ago	21	All three objects land in one `target/release`, so a consumer points
Bring vidya in cfd3e36 nandi 19d ago	22	`LD_LIBRARY_PATH` at one directory and names in `:jolt/native` only the ones it
	23	actually wants.
	24
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	25	## Why a workspace and not one library with features
	26
	27	Because cargo features are additive within a library. A single
	28	`libjoltnative.so` whose `graphics` and `moq` features selected the parts would
	29	link egui and QUIC, Opus, H.264, cpal and V4L2 into one object, so a chat
	30	client that is not in a call would carry the whole media plane in order to open
	31	a window. The feature unification that comes with it has bitten this code
	32	before: enabling eframe's `wayland`/`x11` features for a desktop build pulled
	33	them into the Android one and made the activity exit on launch.
	34
	35	A workspace keeps the parts of a monorepo worth having — one CI, one release,
	36	shared conventions in `jolt-abi` — while each consumer links only what it asked
	37	for. jolt's `:jolt/native` is a vector; naming two objects there is no harder
	38	than naming one.
	39
	40	## Building
	41
Point the docs at the flake d682bfa nandi 13d ago	42	The flake is the toolchain. `nix develop` is where the compiler, the C
	43	toolchain, PipeWire's headers and the kernel headers bindgen wants come from,
	44	so nothing needs to be installed beyond nix itself — no `cc` on PATH, no
	45	rustup, no NDK.
	46
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	47	```bash
Point the docs at the flake d682bfa nandi 13d ago	48	nix develop # then:
	49	just build # release .so files in target/release
	50	just test # the whole workspace
	51	just libdir # where to point LD_LIBRARY_PATH
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	52	```
	53
Point the docs at the flake d682bfa nandi 13d ago	54	`just build` is the edit loop; the nix outputs are the artifact, built against
Point the docs at the flake d682bfa nandi 13d ago	55	a pinned compiler and a pinned crate graph:
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	56
Point the docs at the flake d682bfa nandi 13d ago	57	```bash
	58	nix build # all three objects → result/lib
	59	nix build .#android # both device objects → result/lib/arm64-v8a
	60	nix flake check # fmt, clippy, tests, all three objects
	61	```
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	62
Point the docs at the flake d682bfa nandi 13d ago	63	Two things are worth knowing about:
	64
	65	* bindgen runs libclang directly, so `v4l2r` needs to be pointed at
	66	`linux/videodev2.h`. The flake sets `V4L2R_VIDEODEV2_H_PATH` and
	67	`BINDGEN_EXTRA_CLANG_ARGS` from nixpkgs' `linuxHeaders` in both the devShell
	68	and the build, which is why building inside `nix develop` rather than a bare
	69	shell is the supported path.
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	70	* PipeWire is a real system dependency — `libspa-sys` wants `pkg-config`
Point the docs at the flake d682bfa nandi 13d ago	71	and PipeWire's headers, and the flake supplies both. It is behind the
	72	default-on `pipewire` feature of `jolt-moq`; `just test-minimal` builds
	73	without it, for a machine outside the flake. Turning it off costs device
	74	names: only ALSA PCMs remain visible ("pipewire", "sysdefault") rather than
	75	the real sources a browser would offer, which makes picking a microphone
	76	guesswork.
	77
	78	### Android
	79
	80	```bash
	81	just ffi-android # nix build .#android
	82	```
	83
	84	Same sources and same `Cargo.lock` as the desktop objects; only the target
	85	configuration moves. The NDK comes from nixpkgs, so this needs nothing
	86	installed either. The camera wants two things from the APK that the build
	87	cannot supply: a `CameraCapture` class in its `classes.dex`, and a call to
	88	`joltmoq_android_init` — see `android/jolt_main.c`.
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	89
	90	## jolt-moq
	91
	92	freeq's audio and video calls, extracted from
	93	[sleek](https://github.com/codegod100/sleek), which reached them by calling
	94	Rust from Rust.
	95
	96	A freeq call has two halves, and only one of them is here.
	97
	98	Signaling stayed behind. A call is opened, joined and left over IRC
	99	TAGMSGs — `+freeq.at/av-start` and its siblings — and the server broadcasts
	100	`+freeq.at/av-state` back. Any client that speaks IRC already has everything it
	101	needs for that, in whatever language it speaks IRC in; crossing an FFI boundary
	102	to send a TAGMSG would be worse than not crossing one. This library begins once
	103	the session id and the SFU token are known.
	104
	105	The media plane came across, because it is not a thing to write twice: MoQ
	106	over QUIC, Opus, H.264, camera capture and the SFU's own dial rules come to
	107	some three thousand lines that would say the same thing again in another
	108	language and be wrong in different places.
	109
	110	`crates/jolt-moq/include/joltmoq.h` is the surface, and says more about each
	111	call than this does. The shape of it:
	112
	113	* Nothing calls back. Status is drained with `joltmoq_poll_status`, video
	114	with `joltmoq_frame_poll`, both from whatever thread the caller paints on. A
	115	callback into a foreign runtime from a tokio worker is a rule about threads
	116	that the caller has to keep and that nothing can check.
	117	* Frames are borrowed, not copied. `joltmoq_frame_rgba` points into the
	118	decoder's own buffer until the next poll. A frame is a megabyte or two;
	119	copying it out so the caller can hand it straight to a texture upload would
	120	be two copies a frame for nothing.
	121	* One call at a time. There is one microphone, so there is one session.
	122
	123	### Its relationship to sleek
	124
Let the media plane cross to the phone, camera and all fd0e21a nandi 18d ago	125	`av.rs`, `av_media.rs`, `v4l2cam.rs`, `android_camera.rs` and `nv12_orient.rs`
	126	came from sleek's `android/src`. They are kept close to their originals so changes can still be moved between the
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	127	two by eye. What was dropped on the way was the signaling-and-presentation
	128	half — `ChannelCall`, `LocalCall`, `apply_av_state`, `av_state_message` —
	129	which is sleek's own app state and belongs to a client, not to a media plane.
	130	Dropping it also dropped the `freeq-sdk` dependency entirely.
	131
	132	Sleek still has its own copy. Pointing it at this crate instead is the obvious
	133	next step and has not been taken yet; until it is, a fix made in one place
	134	needs making in the other.
	135
Let the media plane cross to the phone, camera and all fd0e21a nandi 18d ago	136	`android_camera.rs` is the one that did not come across unchanged. Sleek is a
	137	single shared object, so its Camera2 bridge reads the `JavaVM` and the Activity
	138	straight out of the `AndroidApp` that android-activity handed it. Here the glue
	139	is in `libvidya.so` and the media plane is not, so the handles arrive through
	140	`joltmoq_android_init` instead — see `android_jni.rs` for why the usual bridge,
	141	`ndk_context`, cannot cross two cdylibs. Everything below that line is sleek's.
	142
Bring vidya in cfd3e36 nandi 19d ago	143	## vidya
	144
	145	The theme layer, the widget set and the retained-tree ABI that glimmer paints
	146	through. Imported from its own repo rather than depended on: egui is the only
	147	backend anyone wants now, so the indirection that let `libvidya` be swapped for
	148	a raylib or cimgui build of the same symbols was paying for a choice nobody was
	149	going to make.
	150
	151	`crates/vidya-core` is the egui layer — and the 4MB under its `assets/` is not
	152	incidental: the Twemoji pack and the symbols font are `include_bytes!`d into
	153	the library, because a shared object has no host app to install a font set for
	154	it.
	155
Move the jolt libraries under glimmer-backends 19df0d8 nandi 12d ago	156	`glimmer-backends/glimmer-vidya` is the other side of that ABI, and is jolt
	157	rather than Rust. A consumer takes it as a `:local/root` dep and `libvidya.so` on the
Bring vidya in cfd3e36 nandi 19d ago	158	loader path — frq's `deps.edn` is the worked example.
Bring vidya in cfd3e36 nandi 19d ago	159
Lift freeq's AV media plane out of sleek 90f8b89 nandi 19d ago	160	## Adding a crate
	161
	162	One crate, one `.so`, one capability. Depend on `jolt-abi` and keep its three
	163	rules — catch panics at every entry point, lend strings from a `Scratch` rather
	164	than handing out memory to free, and let the caller ask rather than calling it
	165	back. A consumer then names your object in `:jolt/native` alongside whichever
	166	others it wants, and pays for nothing else.
Say which licence, in the place people look for it fc16d8e nandi 19d ago	167
	168	## License
	169
	170	MIT — the same as the projects this code came from: `crates/vidya-*` and
Move the jolt libraries under glimmer-backends 19df0d8 nandi 12d ago	171	`glimmer-backends/glimmer-vidya` from [vidya](https://tangled.org/nandi.uk/vidya),
Say which licence, in the place people look for it fc16d8e nandi 19d ago	172	`crates/jolt-moq`'s media plane from
	173	[sleek](https://github.com/codegod100/sleek). `third-party/cpal` is a vendored
	174	pin of [cpal](https://github.com/RustAudio/cpal) and keeps its own licence.