rickub
Sign in Get started
nandi/frqpublic⑂ Fork 0
⑂ 43ada0b
Branches
claude/add-remote-repository-47d12e main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/frq.git
git clone ssh://git@rickub.com/nandi/frq.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Carry the media plane, and the class that opens the camera

libjoltmoq crosses to the phone now, so the APK carries it: linked into
libjoltapp beside libvidya, stored rather than deflated like the others,
and staged out of a sibling jolt-native the same way.

CameraCapture is the Java half of it. Camera2 has no C API worth the name,
so the camera is opened here and NV12 planes go down to Rust over JNI —
which is also why the class names libjoltmoq in a loadLibrary: the runtime
binds a native method against libraries this class loader was told about,
and one that arrived as a DT_NEEDED of something libvidya dlopened is not
one of them.

libc++_shared travels too. openh264 is C++ and asks to be linked against
it by name; an app's linker namespace will not hand out the platform's
copy, so the APK carries one, exactly as it carries OpenSSL.

Only out of a checkout. The pinned v0.1.1 has no libjoltmoq and glue that
never mentions it, so a pinned build is consistently the APK from before
this rather than half of one. The branch in defs.bzl goes when there is a
release to pin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nandi committed 2026-08-31T21:34:42-07:00 Browse files
43ada0b parent: 6aa2a6b
modified android/AndroidManifest.xml +15 -0
@@ -14,7 +14,22 @@
1414 <uses-permission android:name="android.permission.READ_MEDIA_IMAGES" />
1515 <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"
1616 android:maxSdkVersion="32" />
17+ <!--
18+ The call. Both are runtime permissions and neither is granted by being
19+ asked for here; CameraCapture checks CAMERA before it opens anything and
20+ reports "permission denied" back to the media plane rather than throwing.
21+ Until this activity has Java to raise a dialog with, they are granted from
22+ Settings (or `adb shell pm grant`), the same as the storage reads above.
23+ -->
24+ <uses-permission android:name="android.permission.CAMERA" />
25+ <uses-permission android:name="android.permission.RECORD_AUDIO" />
26+
1727 <uses-feature android:glEsVersion="0x00030000" android:required="true" />
28+ <!--
29+ Not required: a call without a camera is audio-only and still a call, and
30+ required="true" would hide the app from tablets that have no camera.
31+ -->
32+ <uses-feature android:name="android.hardware.camera" android:required="false" />
1833
1934 <!--
2035 Not the Fullscreen theme below: under it the window owns the whole
@@ -14,7 +14,22 @@
14 <uses-permission android:name="android.permission.READ_MEDIA_IMAGES" />14 <uses-permission android:name="android.permission.READ_MEDIA_IMAGES" />
15 <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"15 <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"
16 android:maxSdkVersion="32" />16 android:maxSdkVersion="32" />
17+ <!--
18+ The call. Both are runtime permissions and neither is granted by being
19+ asked for here; CameraCapture checks CAMERA before it opens anything and
20+ reports "permission denied" back to the media plane rather than throwing.
21+ Until this activity has Java to raise a dialog with, they are granted from
22+ Settings (or `adb shell pm grant`), the same as the storage reads above.
23+ -->
24+ <uses-permission android:name="android.permission.CAMERA" />
25+ <uses-permission android:name="android.permission.RECORD_AUDIO" />
26+
17 <uses-feature android:glEsVersion="0x00030000" android:required="true" />27 <uses-feature android:glEsVersion="0x00030000" android:required="true" />
28+ <!--
29+ Not required: a call without a camera is audio-only and still a call, and
30+ required="true" would hide the app from tablets that have no camera.
31+ -->
32+ <uses-feature android:name="android.hardware.camera" android:required="false" />
18 33
19 <!--34 <!--
20 Not the Fullscreen theme below: under it the window owns the whole35 Not the Fullscreen theme below: under it the window owns the whole
modified android/BUCK +37 -5
@@ -9,7 +9,7 @@
99 # The machine-specific paths are read from .buckconfig.local, which the `buck`
1010 # recipe in the justfile writes. Nothing here is found by looking around the
1111 # machine; if a path is missing the recipe says which.
12-load(":defs.bzl", "glue", "libvidya")
12+load(":defs.bzl", "glue", "libjoltmoq", "libvidya")
1313
1414 _JOLT_NATIVE = read_root_config("frq", "jolt_native", "")
1515 _ANDROID_HOME = read_root_config("frq", "android_home", "")
@@ -70,8 +70,29 @@ export_file(
7070 # The recipe decides which, by whether that checkout exists, and says so here.
7171 libvidya(name = "libvidya")
7272
73+# The media plane, on the same terms — see the macro for why it can be absent.
74+# `_MOQ` is False for a pinned build, and every use of it below falls back to
75+# the APK this repo shipped before the media plane crossed.
76+_MOQ = libjoltmoq(name = "libjoltmoq")
77+
7378 glue(c_name = "glue-c", include_name = "glue-include")
7479
80+# The C++ runtime, out of the same NDK the glue is compiled with.
81+#
82+# openh264 is C++, and its build script asks to be linked against
83+# `libc++_shared.so` by name — so libjoltmoq carries that as a DT_NEEDED. An
84+# app's linker namespace will not hand out the platform's own copy (there is no
85+# stable one to hand out), so the APK has to carry it, exactly as it carries
86+# OpenSSL below and for the same reason.
87+#
88+# Only when the media plane is packaged: nothing else here is C++.
89+genrule(
90+ name = "libcxx",
91+ out = "libc++_shared.so",
92+ cmd = _ABS_OUT + " && " + _NDK_BIN + " && " +
93+ "cp \"$ndk\"/../sysroot/usr/lib/aarch64-linux-android/libc++_shared.so \"$out\"",
94+)
95+
7596 # --- the Jolt half ----------------------------------------------------------
7697 # The boot image: frq's Scheme, cross-compiled to arm64 by Chez. The sources
7798 # are an input so that editing one rebuilds this; the compile itself reads them
@@ -125,6 +146,7 @@ genrule(
125146 out = "libjoltapp.so",
126147 cmd = _ABS_OUT + " && lib=`mktemp -d` && " +
127148 "cp $(location :libvidya) \"$lib/libvidya.so\" && " +
149+ ("cp $(location :libjoltmoq) \"$lib/libjoltmoq.so\" && " if _MOQ else "") +
128150 _CC + " -shared -fPIC -O2 -o \"$out\" " +
129151 _GLUE_C + " " +
130152 "$(location :jolt-boot-obj) " +
@@ -133,14 +155,21 @@ genrule(
133155 "-L\"$lib\" " +
134156 _CHEZ + "/tarm64le/boot/tarm64le/libkernel.a " +
135157 _CHEZ + "/lz4/lib/liblz4.a " +
136- "-lvidya -landroid -llog -lz -ldl -lm -Wl,--no-undefined && " +
158+ "-lvidya " + ("-ljoltmoq " if _MOQ else "") +
159+ "-landroid -llog -lz -ldl -lm -Wl,--no-undefined && " +
137160 "rm -rf \"$lib\"",
138161 )
139162
140163 # --- the Java half ----------------------------------------------------------
141-# One class: the photo chooser's result has to land somewhere, and native code
142-# is not somewhere. android.jar on the class path is where every android.* type
143-# comes from; the JDK's own java.* is what is left.
164+# Two classes: the photo chooser's result has to land somewhere and native code
165+# is not somewhere, and Camera2 has no C API worth the name — CameraCapture
166+# opens the camera in Java and pushes NV12 planes down to libjoltmoq over JNI.
167+# android.jar on the class path is where every android.* type comes from; the
168+# JDK's own java.* is what is left.
169+#
170+# CameraCapture is compiled whether or not the media plane is packaged. It is a
171+# few kilobytes of dex, and a class nothing loads costs nothing; keeping it out
172+# of the `_MOQ` branch keeps the branch to the two things that actually differ.
144173 genrule(
145174 name = "classes-dex",
146175 out = "classes.dex",
@@ -166,12 +195,15 @@ genrule(
166195 "mkdir -p \"$stage/lib/arm64-v8a\" && " +
167196 "cp $(location :libvidya) \"$stage/lib/arm64-v8a/libvidya.so\" && " +
168197 "cp $(location :libjoltapp) \"$stage/lib/arm64-v8a/libjoltapp.so\" && " +
198+ ("cp $(location :libjoltmoq) \"$stage/lib/arm64-v8a/libjoltmoq.so\" && " if _MOQ else "") +
199+ ("cp $(location :libcxx) \"$stage/lib/arm64-v8a/libc++_shared.so\" && " if _MOQ else "") +
169200 "cp " + _OPENSSL + "/libssl.so " + _OPENSSL + "/libcrypto.so \"$stage/lib/arm64-v8a/\" && " +
170201 "cp $(location :classes-dex) \"$stage/classes.dex\" && " +
171202 _TOOLS + "/aapt2 link -o \"$out\" -I " + _ANDROID_JAR + " " +
172203 "--manifest $SRCS --min-sdk-version " + _API + " --target-sdk-version 36 " +
173204 "--version-code 1 --version-name 0.1.0 >&2 && " +
174205 "( cd \"$stage\" && zip -q -0 \"$out\" lib/arm64-v8a/libvidya.so " +
206+ ("lib/arm64-v8a/libjoltmoq.so lib/arm64-v8a/libc++_shared.so " if _MOQ else "") +
175207 "lib/arm64-v8a/libjoltapp.so lib/arm64-v8a/libssl.so lib/arm64-v8a/libcrypto.so && " +
176208 "zip -q \"$out\" classes.dex ) && rm -rf \"$stage\"",
177209 )
@@ -9,7 +9,7 @@
9 # The machine-specific paths are read from .buckconfig.local, which the `buck`9 # The machine-specific paths are read from .buckconfig.local, which the `buck`
10 # recipe in the justfile writes. Nothing here is found by looking around the10 # recipe in the justfile writes. Nothing here is found by looking around the
11 # machine; if a path is missing the recipe says which.11 # machine; if a path is missing the recipe says which.
12-load(":defs.bzl", "glue", "libvidya")12+load(":defs.bzl", "glue", "libjoltmoq", "libvidya")
13 13
14 _JOLT_NATIVE = read_root_config("frq", "jolt_native", "")14 _JOLT_NATIVE = read_root_config("frq", "jolt_native", "")
15 _ANDROID_HOME = read_root_config("frq", "android_home", "")15 _ANDROID_HOME = read_root_config("frq", "android_home", "")
@@ -70,8 +70,29 @@ export_file(
70 # The recipe decides which, by whether that checkout exists, and says so here.70 # The recipe decides which, by whether that checkout exists, and says so here.
71 libvidya(name = "libvidya")71 libvidya(name = "libvidya")
72 72
73+# The media plane, on the same terms — see the macro for why it can be absent.
74+# `_MOQ` is False for a pinned build, and every use of it below falls back to
75+# the APK this repo shipped before the media plane crossed.
76+_MOQ = libjoltmoq(name = "libjoltmoq")
77+
73 glue(c_name = "glue-c", include_name = "glue-include")78 glue(c_name = "glue-c", include_name = "glue-include")
74 79
80+# The C++ runtime, out of the same NDK the glue is compiled with.
81+#
82+# openh264 is C++, and its build script asks to be linked against
83+# `libc++_shared.so` by name — so libjoltmoq carries that as a DT_NEEDED. An
84+# app's linker namespace will not hand out the platform's own copy (there is no
85+# stable one to hand out), so the APK has to carry it, exactly as it carries
86+# OpenSSL below and for the same reason.
87+#
88+# Only when the media plane is packaged: nothing else here is C++.
89+genrule(
90+ name = "libcxx",
91+ out = "libc++_shared.so",
92+ cmd = _ABS_OUT + " && " + _NDK_BIN + " && " +
93+ "cp \"$ndk\"/../sysroot/usr/lib/aarch64-linux-android/libc++_shared.so \"$out\"",
94+)
95+
75 # --- the Jolt half ----------------------------------------------------------96 # --- the Jolt half ----------------------------------------------------------
76 # The boot image: frq's Scheme, cross-compiled to arm64 by Chez. The sources97 # The boot image: frq's Scheme, cross-compiled to arm64 by Chez. The sources
77 # are an input so that editing one rebuilds this; the compile itself reads them98 # are an input so that editing one rebuilds this; the compile itself reads them
@@ -125,6 +146,7 @@ genrule(
125 out = "libjoltapp.so",146 out = "libjoltapp.so",
126 cmd = _ABS_OUT + " && lib=`mktemp -d` && " +147 cmd = _ABS_OUT + " && lib=`mktemp -d` && " +
127 "cp $(location :libvidya) \"$lib/libvidya.so\" && " +148 "cp $(location :libvidya) \"$lib/libvidya.so\" && " +
149+ ("cp $(location :libjoltmoq) \"$lib/libjoltmoq.so\" && " if _MOQ else "") +
128 _CC + " -shared -fPIC -O2 -o \"$out\" " +150 _CC + " -shared -fPIC -O2 -o \"$out\" " +
129 _GLUE_C + " " +151 _GLUE_C + " " +
130 "$(location :jolt-boot-obj) " +152 "$(location :jolt-boot-obj) " +
@@ -133,14 +155,21 @@ genrule(
133 "-L\"$lib\" " +155 "-L\"$lib\" " +
134 _CHEZ + "/tarm64le/boot/tarm64le/libkernel.a " +156 _CHEZ + "/tarm64le/boot/tarm64le/libkernel.a " +
135 _CHEZ + "/lz4/lib/liblz4.a " +157 _CHEZ + "/lz4/lib/liblz4.a " +
136- "-lvidya -landroid -llog -lz -ldl -lm -Wl,--no-undefined && " +158+ "-lvidya " + ("-ljoltmoq " if _MOQ else "") +
159+ "-landroid -llog -lz -ldl -lm -Wl,--no-undefined && " +
137 "rm -rf \"$lib\"",160 "rm -rf \"$lib\"",
138 )161 )
139 162
140 # --- the Java half ----------------------------------------------------------163 # --- the Java half ----------------------------------------------------------
141-# One class: the photo chooser's result has to land somewhere, and native code164+# Two classes: the photo chooser's result has to land somewhere and native code
142-# is not somewhere. android.jar on the class path is where every android.* type165+# is not somewhere, and Camera2 has no C API worth the name — CameraCapture
143-# comes from; the JDK's own java.* is what is left.166+# opens the camera in Java and pushes NV12 planes down to libjoltmoq over JNI.
167+# android.jar on the class path is where every android.* type comes from; the
168+# JDK's own java.* is what is left.
169+#
170+# CameraCapture is compiled whether or not the media plane is packaged. It is a
171+# few kilobytes of dex, and a class nothing loads costs nothing; keeping it out
172+# of the `_MOQ` branch keeps the branch to the two things that actually differ.
144 genrule(173 genrule(
145 name = "classes-dex",174 name = "classes-dex",
146 out = "classes.dex",175 out = "classes.dex",
@@ -166,12 +195,15 @@ genrule(
166 "mkdir -p \"$stage/lib/arm64-v8a\" && " +195 "mkdir -p \"$stage/lib/arm64-v8a\" && " +
167 "cp $(location :libvidya) \"$stage/lib/arm64-v8a/libvidya.so\" && " +196 "cp $(location :libvidya) \"$stage/lib/arm64-v8a/libvidya.so\" && " +
168 "cp $(location :libjoltapp) \"$stage/lib/arm64-v8a/libjoltapp.so\" && " +197 "cp $(location :libjoltapp) \"$stage/lib/arm64-v8a/libjoltapp.so\" && " +
198+ ("cp $(location :libjoltmoq) \"$stage/lib/arm64-v8a/libjoltmoq.so\" && " if _MOQ else "") +
199+ ("cp $(location :libcxx) \"$stage/lib/arm64-v8a/libc++_shared.so\" && " if _MOQ else "") +
169 "cp " + _OPENSSL + "/libssl.so " + _OPENSSL + "/libcrypto.so \"$stage/lib/arm64-v8a/\" && " +200 "cp " + _OPENSSL + "/libssl.so " + _OPENSSL + "/libcrypto.so \"$stage/lib/arm64-v8a/\" && " +
170 "cp $(location :classes-dex) \"$stage/classes.dex\" && " +201 "cp $(location :classes-dex) \"$stage/classes.dex\" && " +
171 _TOOLS + "/aapt2 link -o \"$out\" -I " + _ANDROID_JAR + " " +202 _TOOLS + "/aapt2 link -o \"$out\" -I " + _ANDROID_JAR + " " +
172 "--manifest $SRCS --min-sdk-version " + _API + " --target-sdk-version 36 " +203 "--manifest $SRCS --min-sdk-version " + _API + " --target-sdk-version 36 " +
173 "--version-code 1 --version-name 0.1.0 >&2 && " +204 "--version-code 1 --version-name 0.1.0 >&2 && " +
174 "( cd \"$stage\" && zip -q -0 \"$out\" lib/arm64-v8a/libvidya.so " +205 "( cd \"$stage\" && zip -q -0 \"$out\" lib/arm64-v8a/libvidya.so " +
206+ ("lib/arm64-v8a/libjoltmoq.so lib/arm64-v8a/libc++_shared.so " if _MOQ else "") +
175 "lib/arm64-v8a/libjoltapp.so lib/arm64-v8a/libssl.so lib/arm64-v8a/libcrypto.so && " +207 "lib/arm64-v8a/libjoltapp.so lib/arm64-v8a/libssl.so lib/arm64-v8a/libcrypto.so && " +
176 "zip -q \"$out\" classes.dex ) && rm -rf \"$stage\"",208 "zip -q \"$out\" classes.dex ) && rm -rf \"$stage\"",
177 )209 )
modified android/defs.bzl +22 -0
@@ -27,6 +27,28 @@ def libvidya(name):
2727 cmd = "cp $(location toolchains//dist:libvidya-android)/libvidya.so \"$OUT\"",
2828 )
2929
30+def libjoltmoq(name):
31+ """The Android libjoltmoq, or nothing.
32+
33+ The media plane crosses to the phone now, but only out of a sibling
34+ jolt-native checkout: the pinned v0.1.1 release predates it and carries no
35+ such object. Returns True when the APK should package it.
36+
37+ The two halves cannot disagree, and do not: the pinned glue is the same
38+ v0.1.1 and never mentions joltmoq either, so a pinned APK is consistently
39+ without a media plane rather than half-wired. Delete the branch — and make
40+ the caller unconditional — once jolt-native cuts a release carrying
41+ libjoltmoq.so and scripts/libjoltmoq-android.dotslash pins it.
42+ """
43+ if native.read_root_config("frq", "libvidya", "pinned") != "checkout":
44+ return False
45+ native.export_file(
46+ name = name,
47+ src = "prebuilt/arm64-v8a/libjoltmoq.so",
48+ mode = "reference",
49+ )
50+ return True
51+
3052 def glue(c_name, include_name):
3153 """jolt_main.c and the ABI's headers, from wherever libvidya came from.
3254
@@ -27,6 +27,28 @@ def libvidya(name):
27 cmd = "cp $(location toolchains//dist:libvidya-android)/libvidya.so \"$OUT\"",27 cmd = "cp $(location toolchains//dist:libvidya-android)/libvidya.so \"$OUT\"",
28 )28 )
29 29
30+def libjoltmoq(name):
31+ """The Android libjoltmoq, or nothing.
32+
33+ The media plane crosses to the phone now, but only out of a sibling
34+ jolt-native checkout: the pinned v0.1.1 release predates it and carries no
35+ such object. Returns True when the APK should package it.
36+
37+ The two halves cannot disagree, and do not: the pinned glue is the same
38+ v0.1.1 and never mentions joltmoq either, so a pinned APK is consistently
39+ without a media plane rather than half-wired. Delete the branch — and make
40+ the caller unconditional — once jolt-native cuts a release carrying
41+ libjoltmoq.so and scripts/libjoltmoq-android.dotslash pins it.
42+ """
43+ if native.read_root_config("frq", "libvidya", "pinned") != "checkout":
44+ return False
45+ native.export_file(
46+ name = name,
47+ src = "prebuilt/arm64-v8a/libjoltmoq.so",
48+ mode = "reference",
49+ )
50+ return True
51+
30 def glue(c_name, include_name):52 def glue(c_name, include_name):
31 """jolt_main.c and the ABI's headers, from wherever libvidya came from.53 """jolt_main.c and the ABI's headers, from wherever libvidya came from.
32 54
added android/java/uk/nandi/frq/CameraCapture.java +610 -0
new file mode 100644
@@ -0,0 +1,610 @@
1+package uk.nandi.frq;
2+
3+import android.app.Activity;
4+import android.content.Context;
5+import android.content.pm.PackageManager;
6+import android.graphics.ImageFormat;
7+import android.hardware.camera2.CameraAccessException;
8+import android.hardware.camera2.CameraCaptureSession;
9+import android.hardware.camera2.CameraCharacteristics;
10+import android.hardware.camera2.CameraDevice;
11+import android.hardware.camera2.CameraManager;
12+import android.hardware.camera2.CaptureRequest;
13+import android.media.Image;
14+import android.media.ImageReader;
15+import android.os.Build;
16+import android.os.Handler;
17+import android.os.HandlerThread;
18+import android.util.Log;
19+import android.util.Size;
20+import android.view.Display;
21+import android.view.Surface;
22+
23+import java.nio.ByteBuffer;
24+import java.util.ArrayList;
25+import java.util.Collections;
26+import java.util.List;
27+
28+/**
29+ * Camera2 capture for NativeActivity AV calls.
30+ *
31+ * <p>Opens a camera (front preferred), streams {@link ImageFormat#YUV_420_888}
32+ * into an {@link ImageReader}, and pushes NV12 planes into Rust via JNI so
33+ * iroh-live can publish H.264 on the MoQ media plane.
34+ *
35+ * <p>ImageReader buffers are in <b>sensor</b> coordinates. Most phone sensors
36+ * sit at 90°/270°, so without a CW rotate by {@link
37+ * CameraCharacteristics#SENSOR_ORIENTATION} ± display rotation the published
38+ * video (and local preview) looks sideways. Rotation degrees are passed to
39+ * Rust, which rotates the NV12 planes before encode/preview.
40+ *
41+ * <p>Compiled into the APK's {@code classes.dex}. Rust must resolve this class
42+ * via {@code Activity.getClassLoader().loadClass} — {@code FindClass} from a
43+ * native worker thread uses the system loader and cannot see APK classes.
44+ *
45+ * <p>The native half is {@code crates/jolt-moq/src/android_camera.rs} in
46+ * jolt-native, and it needs the JavaVM and the Activity handed to it first:
47+ * {@code libjoltapp.so}'s glue calls {@code joltmoq_android_init} before Jolt
48+ * starts. Without that, {@link #start} is never reached at all.
49+ */
50+public final class CameraCapture {
51+ private static final String TAG = "FrqCamera";
52+ /** Target capture size — encoder preset is 360p; 640x480 is widely supported. */
53+ private static final int TARGET_WIDTH = 640;
54+ private static final int TARGET_HEIGHT = 480;
55+
56+ private static final Object LOCK = new Object();
57+ private static CameraDevice cameraDevice;
58+ private static CameraCaptureSession captureSession;
59+ private static ImageReader imageReader;
60+ private static HandlerThread backgroundThread;
61+ private static Handler backgroundHandler;
62+ private static volatile boolean running;
63+ private static long frameCount;
64+ /** Host activity — used to read live display rotation while capturing. */
65+ private static volatile Activity hostActivity;
66+ private static int sensorOrientationDeg = 90;
67+ private static boolean frontFacing = true;
68+
69+ static {
70+ // Required, not belt-and-braces. The runtime resolves a `native` method
71+ // against the libraries this class loader was told about through
72+ // System.loadLibrary — and libjoltmoq.so is not one of them: nothing
73+ // Java loads it. It arrives as a DT_NEEDED of libjoltapp.so, which
74+ // libvidya dlopens, and a library that reached the process that way is
75+ // invisible to JNI method binding. So it is named here, where the class
76+ // that declares the natives is initialised. Already-mapped is fine;
77+ // loadLibrary finds it and registers it rather than mapping it twice.
78+ try {
79+ System.loadLibrary("joltmoq");
80+ } catch (UnsatisfiedLinkError e) {
81+ // Nothing below can work after this: every frame and every state
82+ // change goes out through a native method.
83+ Log.e(TAG, "libjoltmoq.so not loadable; camera capture is dead", e);
84+ }
85+ }
86+
87+ private CameraCapture() {}
88+
89+ /**
90+ * Native: push one NV12 frame into the MoQ publish pipeline.
91+ *
92+ * @param rotationDegrees CW degrees (0/90/180/270) to apply so the frame
93+ * is upright for the current display — see {@link #computeRotationDegrees()}.
94+ */
95+ private static native void onNv12Frame(
96+ byte[] yData,
97+ byte[] uvData,
98+ int width,
99+ int height,
100+ int yStride,
101+ int uvStride,
102+ int rotationDegrees);
103+
104+ /** Native: camera opened successfully (or failed). */
105+ private static native void onCameraState(boolean opened, String detail);
106+
107+ /**
108+ * List available cameras as {@code "id\\tlabel"} lines (id then tab then label).
109+ * Facing preference: front first, then back, then external/other.
110+ */
111+ public static String[] listCameras(Context context) {
112+ try {
113+ CameraManager manager =
114+ (CameraManager) context.getSystemService(Context.CAMERA_SERVICE);
115+ if (manager == null) {
116+ return new String[0];
117+ }
118+ String[] ids = manager.getCameraIdList();
119+ List<String> front = new ArrayList<>();
120+ List<String> back = new ArrayList<>();
121+ List<String> other = new ArrayList<>();
122+ for (String id : ids) {
123+ CameraCharacteristics chars = manager.getCameraCharacteristics(id);
124+ Integer facing = chars.get(CameraCharacteristics.LENS_FACING);
125+ String label;
126+ if (facing != null && facing == CameraCharacteristics.LENS_FACING_FRONT) {
127+ label = "Front camera · " + id;
128+ front.add(id + "\t" + label);
129+ } else if (facing != null && facing == CameraCharacteristics.LENS_FACING_BACK) {
130+ label = "Back camera · " + id;
131+ back.add(id + "\t" + label);
132+ } else {
133+ label = "Camera · " + id;
134+ other.add(id + "\t" + label);
135+ }
136+ }
137+ List<String> out = new ArrayList<>(front.size() + back.size() + other.size());
138+ out.addAll(front);
139+ out.addAll(back);
140+ out.addAll(other);
141+ return out.toArray(new String[0]);
142+ } catch (Throwable t) {
143+ Log.w(TAG, "listCameras failed", t);
144+ return new String[0];
145+ }
146+ }
147+
148+ /**
149+ * Start capture. {@code cameraId} may be null/empty (prefer front), a device
150+ * id, or the tokens {@code "front"} / {@code "back"}.
151+ */
152+ public static void start(final Activity activity, final String cameraId) {
153+ if (activity == null) {
154+ onCameraState(false, "null activity");
155+ return;
156+ }
157+ activity.runOnUiThread(
158+ new Runnable() {
159+ @Override
160+ public void run() {
161+ startOnThread(activity, cameraId);
162+ }
163+ });
164+ }
165+
166+ /** Stop capture and release camera resources. Safe to call repeatedly. */
167+ public static void stop(final Activity activity) {
168+ Runnable r =
169+ new Runnable() {
170+ @Override
171+ public void run() {
172+ stopLocked();
173+ }
174+ };
175+ if (activity != null) {
176+ activity.runOnUiThread(r);
177+ } else {
178+ r.run();
179+ }
180+ }
181+
182+ private static void startOnThread(Activity activity, String cameraId) {
183+ synchronized (LOCK) {
184+ stopLocked();
185+ if (activity.checkSelfPermission(android.Manifest.permission.CAMERA)
186+ != PackageManager.PERMISSION_GRANTED) {
187+ Log.e(TAG, "CAMERA permission not granted");
188+ onCameraState(false, "permission denied");
189+ return;
190+ }
191+ CameraManager manager =
192+ (CameraManager) activity.getSystemService(Context.CAMERA_SERVICE);
193+ if (manager == null) {
194+ onCameraState(false, "no CameraManager");
195+ return;
196+ }
197+ String id;
198+ try {
199+ id = resolveCameraId(manager, cameraId);
200+ } catch (CameraAccessException e) {
201+ Log.e(TAG, "resolveCameraId", e);
202+ onCameraState(false, e.getMessage());
203+ return;
204+ }
205+ if (id == null) {
206+ onCameraState(false, "no camera");
207+ return;
208+ }
209+
210+ hostActivity = activity;
211+ try {
212+ CameraCharacteristics chars = manager.getCameraCharacteristics(id);
213+ Integer so = chars.get(CameraCharacteristics.SENSOR_ORIENTATION);
214+ sensorOrientationDeg = so != null ? so : 90;
215+ Integer facing = chars.get(CameraCharacteristics.LENS_FACING);
216+ frontFacing =
217+ facing != null && facing == CameraCharacteristics.LENS_FACING_FRONT;
218+ } catch (CameraAccessException e) {
219+ Log.w(TAG, "characteristics; using default orientation", e);
220+ sensorOrientationDeg = 90;
221+ frontFacing = true;
222+ }
223+
224+ Size size = chooseSize(manager, id);
225+ int width = size.getWidth();
226+ int height = size.getHeight();
227+ int rot = computeRotationDegrees();
228+ Log.i(
229+ TAG,
230+ "opening camera id="
231+ + id
232+ + " size="
233+ + width
234+ + "x"
235+ + height
236+ + " sensorOrient="
237+ + sensorOrientationDeg
238+ + " front="
239+ + frontFacing
240+ + " rot="
241+ + rot);
242+
243+ startBackgroundThread();
244+ frameCount = 0;
245+ imageReader = ImageReader.newInstance(width, height, ImageFormat.YUV_420_888, 2);
246+ imageReader.setOnImageAvailableListener(
247+ new ImageReader.OnImageAvailableListener() {
248+ @Override
249+ public void onImageAvailable(ImageReader reader) {
250+ Image image = null;
251+ try {
252+ image = reader.acquireLatestImage();
253+ if (image == null) {
254+ return;
255+ }
256+ pushNv12(image);
257+ } catch (Throwable t) {
258+ Log.w(TAG, "onImageAvailable", t);
259+ } finally {
260+ if (image != null) {
261+ image.close();
262+ }
263+ }
264+ }
265+ },
266+ backgroundHandler);
267+
268+ final String openId = id;
269+ try {
270+ manager.openCamera(
271+ openId,
272+ new CameraDevice.StateCallback() {
273+ @Override
274+ public void onOpened(CameraDevice camera) {
275+ synchronized (LOCK) {
276+ cameraDevice = camera;
277+ createSessionLocked(camera);
278+ }
279+ }
280+
281+ @Override
282+ public void onDisconnected(CameraDevice camera) {
283+ Log.w(TAG, "camera disconnected");
284+ synchronized (LOCK) {
285+ camera.close();
286+ if (cameraDevice == camera) {
287+ cameraDevice = null;
288+ }
289+ running = false;
290+ }
291+ onCameraState(false, "disconnected");
292+ }
293+
294+ @Override
295+ public void onError(CameraDevice camera, int error) {
296+ Log.e(TAG, "camera error " + error);
297+ synchronized (LOCK) {
298+ camera.close();
299+ if (cameraDevice == camera) {
300+ cameraDevice = null;
301+ }
302+ running = false;
303+ }
304+ onCameraState(false, "error " + error);
305+ }
306+ },
307+ backgroundHandler);
308+ } catch (SecurityException e) {
309+ Log.e(TAG, "openCamera security", e);
310+ stopLocked();
311+ onCameraState(false, "permission denied");
312+ } catch (CameraAccessException e) {
313+ Log.e(TAG, "openCamera", e);
314+ stopLocked();
315+ onCameraState(false, e.getMessage());
316+ }
317+ }
318+ }
319+
320+ private static void createSessionLocked(CameraDevice camera) {
321+ final ImageReader reader = imageReader;
322+ if (reader == null) {
323+ onCameraState(false, "no ImageReader");
324+ return;
325+ }
326+ final Surface surface = reader.getSurface();
327+ try {
328+ camera.createCaptureSession(
329+ Collections.singletonList(surface),
330+ new CameraCaptureSession.StateCallback() {
331+ @Override
332+ public void onConfigured(CameraCaptureSession session) {
333+ synchronized (LOCK) {
334+ if (cameraDevice == null) {
335+ return;
336+ }
337+ captureSession = session;
338+ try {
339+ CaptureRequest.Builder builder =
340+ cameraDevice.createCaptureRequest(
341+ CameraDevice.TEMPLATE_PREVIEW);
342+ builder.addTarget(surface);
343+ builder.set(
344+ CaptureRequest.CONTROL_AF_MODE,
345+ CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_VIDEO);
346+ session.setRepeatingRequest(
347+ builder.build(), null, backgroundHandler);
348+ running = true;
349+ onCameraState(true, "live");
350+ Log.i(TAG, "capture session live");
351+ } catch (CameraAccessException e) {
352+ Log.e(TAG, "setRepeatingRequest", e);
353+ onCameraState(false, e.getMessage());
354+ }
355+ }
356+ }
357+
358+ @Override
359+ public void onConfigureFailed(CameraCaptureSession session) {
360+ Log.e(TAG, "capture session configure failed");
361+ onCameraState(false, "configure failed");
362+ }
363+ },
364+ backgroundHandler);
365+ } catch (CameraAccessException e) {
366+ Log.e(TAG, "createCaptureSession", e);
367+ onCameraState(false, e.getMessage());
368+ }
369+ }
370+
371+ private static void pushNv12(Image image) {
372+ if (image.getFormat() != ImageFormat.YUV_420_888) {
373+ return;
374+ }
375+ Image.Plane[] planes = image.getPlanes();
376+ if (planes.length < 3) {
377+ return;
378+ }
379+ int width = image.getWidth();
380+ int height = image.getHeight();
381+ Image.Plane yPlane = planes[0];
382+ Image.Plane uPlane = planes[1];
383+ Image.Plane vPlane = planes[2];
384+ int yStride = yPlane.getRowStride();
385+ int uvStride = uPlane.getRowStride();
386+ int uvPixelStride = uPlane.getPixelStride();
387+ int uvHeight = height / 2;
388+
389+ ByteBuffer yBuf = yPlane.getBuffer();
390+ int ySize = yStride * height;
391+ byte[] yData = new byte[ySize];
392+ yBuf.position(0);
393+ int yCopy = Math.min(ySize, yBuf.remaining());
394+ yBuf.get(yData, 0, yCopy);
395+
396+ byte[] uvData;
397+ int outUvStride;
398+ if (uvPixelStride == 2) {
399+ // Semi-planar: U plane buffer is UVUV… (NV12-style on most devices).
400+ ByteBuffer uvBuf = uPlane.getBuffer();
401+ int uvSize = uvStride * uvHeight;
402+ uvData = new byte[uvSize];
403+ uvBuf.position(0);
404+ int uvCopy = Math.min(uvSize, uvBuf.remaining());
405+ uvBuf.get(uvData, 0, uvCopy);
406+ outUvStride = uvStride;
407+ } else {
408+ // Planar I420 → interleave U+V into NV12.
409+ ByteBuffer uBuf = uPlane.getBuffer();
410+ ByteBuffer vBuf = vPlane.getBuffer();
411+ int uvWidth = width / 2;
412+ outUvStride = width;
413+ uvData = new byte[outUvStride * uvHeight];
414+ for (int row = 0; row < uvHeight; row++) {
415+ for (int col = 0; col < uvWidth; col++) {
416+ int srcIdx = row * uPlane.getRowStride() + col * uvPixelStride;
417+ int dstIdx = row * outUvStride + col * 2;
418+ uvData[dstIdx] = uBuf.get(srcIdx);
419+ uvData[dstIdx + 1] = vBuf.get(srcIdx);
420+ }
421+ }
422+ }
423+
424+ int rotationDegrees = computeRotationDegrees();
425+ onNv12Frame(yData, uvData, width, height, yStride, outUvStride, rotationDegrees);
426+ long n = ++frameCount;
427+ if (n == 1L || n == 30L || n == 300L) {
428+ Log.i(
429+ TAG,
430+ "nv12 frame #"
431+ + n
432+ + " "
433+ + width
434+ + "x"
435+ + height
436+ + " yStride="
437+ + yStride
438+ + " uvStride="
439+ + outUvStride
440+ + " uvPixelStride="
441+ + uvPixelStride
442+ + " rot="
443+ + rotationDegrees);
444+ }
445+ }
446+
447+ /**
448+ * CW degrees to rotate sensor buffers so content is upright for the
449+ * current display orientation (Camera2 JPEG_ORIENTATION formula).
450+ */
451+ private static int computeRotationDegrees() {
452+ int device = displayRotationDegrees(hostActivity);
453+ if (frontFacing) {
454+ return (sensorOrientationDeg + device) % 360;
455+ }
456+ return (sensorOrientationDeg - device + 360) % 360;
457+ }
458+
459+ private static int displayRotationDegrees(Activity activity) {
460+ if (activity == null) {
461+ return 0;
462+ }
463+ try {
464+ int rotation;
465+ if (Build.VERSION.SDK_INT >= 30) {
466+ Display display = activity.getDisplay();
467+ rotation = display != null ? display.getRotation() : Surface.ROTATION_0;
468+ } else {
469+ @SuppressWarnings("deprecation")
470+ Display display = activity.getWindowManager().getDefaultDisplay();
471+ rotation = display.getRotation();
472+ }
473+ switch (rotation) {
474+ case Surface.ROTATION_90:
475+ return 90;
476+ case Surface.ROTATION_180:
477+ return 180;
478+ case Surface.ROTATION_270:
479+ return 270;
480+ case Surface.ROTATION_0:
481+ default:
482+ return 0;
483+ }
484+ } catch (Throwable t) {
485+ Log.w(TAG, "displayRotationDegrees", t);
486+ return 0;
487+ }
488+ }
489+
490+ private static String resolveCameraId(CameraManager manager, String requested)
491+ throws CameraAccessException {
492+ String[] ids = manager.getCameraIdList();
493+ if (ids.length == 0) {
494+ return null;
495+ }
496+ String want = requested == null ? "" : requested.trim();
497+ if (!want.isEmpty()) {
498+ for (String id : ids) {
499+ if (id.equals(want)) {
500+ return id;
501+ }
502+ }
503+ Integer facingWant = null;
504+ if ("front".equalsIgnoreCase(want)) {
505+ facingWant = CameraCharacteristics.LENS_FACING_FRONT;
506+ } else if ("back".equalsIgnoreCase(want)) {
507+ facingWant = CameraCharacteristics.LENS_FACING_BACK;
508+ }
509+ if (facingWant != null) {
510+ for (String id : ids) {
511+ Integer facing =
512+ manager.getCameraCharacteristics(id)
513+ .get(CameraCharacteristics.LENS_FACING);
514+ if (facingWant.equals(facing)) {
515+ return id;
516+ }
517+ }
518+ }
519+ }
520+ // Prefer front for self-view, then first listed.
521+ for (String id : ids) {
522+ Integer facing =
523+ manager.getCameraCharacteristics(id).get(CameraCharacteristics.LENS_FACING);
524+ if (facing != null && facing == CameraCharacteristics.LENS_FACING_FRONT) {
525+ return id;
526+ }
527+ }
528+ return ids[0];
529+ }
530+
531+ private static Size chooseSize(CameraManager manager, String cameraId) {
532+ try {
533+ CameraCharacteristics chars = manager.getCameraCharacteristics(cameraId);
534+ android.hardware.camera2.params.StreamConfigurationMap map =
535+ chars.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
536+ if (map == null) {
537+ return new Size(TARGET_WIDTH, TARGET_HEIGHT);
538+ }
539+ Size[] sizes = map.getOutputSizes(ImageFormat.YUV_420_888);
540+ if (sizes == null || sizes.length == 0) {
541+ return new Size(TARGET_WIDTH, TARGET_HEIGHT);
542+ }
543+ Size best = sizes[0];
544+ long bestScore = Long.MAX_VALUE;
545+ for (Size s : sizes) {
546+ long dw = (long) s.getWidth() - TARGET_WIDTH;
547+ long dh = (long) s.getHeight() - TARGET_HEIGHT;
548+ long score = dw * dw + dh * dh;
549+ // Prefer sizes at or above target when scores are close.
550+ if (s.getWidth() < TARGET_WIDTH / 2 || s.getHeight() < TARGET_HEIGHT / 2) {
551+ score += 10_000_000L;
552+ }
553+ if (score < bestScore) {
554+ bestScore = score;
555+ best = s;
556+ }
557+ }
558+ return best;
559+ } catch (Throwable t) {
560+ Log.w(TAG, "chooseSize fallback", t);
561+ return new Size(TARGET_WIDTH, TARGET_HEIGHT);
562+ }
563+ }
564+
565+ private static void stopLocked() {
566+ running = false;
567+ hostActivity = null;
568+ if (captureSession != null) {
569+ try {
570+ captureSession.close();
571+ } catch (Throwable ignored) {
572+ }
573+ captureSession = null;
574+ }
575+ if (cameraDevice != null) {
576+ try {
577+ cameraDevice.close();
578+ } catch (Throwable ignored) {
579+ }
580+ cameraDevice = null;
581+ }
582+ if (imageReader != null) {
583+ try {
584+ imageReader.close();
585+ } catch (Throwable ignored) {
586+ }
587+ imageReader = null;
588+ }
589+ stopBackgroundThread();
590+ }
591+
592+ private static void startBackgroundThread() {
593+ stopBackgroundThread();
594+ backgroundThread = new HandlerThread("frq-camera");
595+ backgroundThread.start();
596+ backgroundHandler = new Handler(backgroundThread.getLooper());
597+ }
598+
599+ private static void stopBackgroundThread() {
600+ if (backgroundThread != null) {
601+ backgroundThread.quitSafely();
602+ try {
603+ backgroundThread.join(1000);
604+ } catch (InterruptedException ignored) {
605+ }
606+ backgroundThread = null;
607+ backgroundHandler = null;
608+ }
609+ }
610+}
new file mode 100644
@@ -0,0 +1,610 @@
1+package uk.nandi.frq;
2+
3+import android.app.Activity;
4+import android.content.Context;
5+import android.content.pm.PackageManager;
6+import android.graphics.ImageFormat;
7+import android.hardware.camera2.CameraAccessException;
8+import android.hardware.camera2.CameraCaptureSession;
9+import android.hardware.camera2.CameraCharacteristics;
10+import android.hardware.camera2.CameraDevice;
11+import android.hardware.camera2.CameraManager;
12+import android.hardware.camera2.CaptureRequest;
13+import android.media.Image;
14+import android.media.ImageReader;
15+import android.os.Build;
16+import android.os.Handler;
17+import android.os.HandlerThread;
18+import android.util.Log;
19+import android.util.Size;
20+import android.view.Display;
21+import android.view.Surface;
22+
23+import java.nio.ByteBuffer;
24+import java.util.ArrayList;
25+import java.util.Collections;
26+import java.util.List;
27+
28+/**
29+ * Camera2 capture for NativeActivity AV calls.
30+ *
31+ * <p>Opens a camera (front preferred), streams {@link ImageFormat#YUV_420_888}
32+ * into an {@link ImageReader}, and pushes NV12 planes into Rust via JNI so
33+ * iroh-live can publish H.264 on the MoQ media plane.
34+ *
35+ * <p>ImageReader buffers are in <b>sensor</b> coordinates. Most phone sensors
36+ * sit at 90°/270°, so without a CW rotate by {@link
37+ * CameraCharacteristics#SENSOR_ORIENTATION} ± display rotation the published
38+ * video (and local preview) looks sideways. Rotation degrees are passed to
39+ * Rust, which rotates the NV12 planes before encode/preview.
40+ *
41+ * <p>Compiled into the APK's {@code classes.dex}. Rust must resolve this class
42+ * via {@code Activity.getClassLoader().loadClass} — {@code FindClass} from a
43+ * native worker thread uses the system loader and cannot see APK classes.
44+ *
45+ * <p>The native half is {@code crates/jolt-moq/src/android_camera.rs} in
46+ * jolt-native, and it needs the JavaVM and the Activity handed to it first:
47+ * {@code libjoltapp.so}'s glue calls {@code joltmoq_android_init} before Jolt
48+ * starts. Without that, {@link #start} is never reached at all.
49+ */
50+public final class CameraCapture {
51+ private static final String TAG = "FrqCamera";
52+ /** Target capture size — encoder preset is 360p; 640x480 is widely supported. */
53+ private static final int TARGET_WIDTH = 640;
54+ private static final int TARGET_HEIGHT = 480;
55+
56+ private static final Object LOCK = new Object();
57+ private static CameraDevice cameraDevice;
58+ private static CameraCaptureSession captureSession;
59+ private static ImageReader imageReader;
60+ private static HandlerThread backgroundThread;
61+ private static Handler backgroundHandler;
62+ private static volatile boolean running;
63+ private static long frameCount;
64+ /** Host activity — used to read live display rotation while capturing. */
65+ private static volatile Activity hostActivity;
66+ private static int sensorOrientationDeg = 90;
67+ private static boolean frontFacing = true;
68+
69+ static {
70+ // Required, not belt-and-braces. The runtime resolves a `native` method
71+ // against the libraries this class loader was told about through
72+ // System.loadLibrary — and libjoltmoq.so is not one of them: nothing
73+ // Java loads it. It arrives as a DT_NEEDED of libjoltapp.so, which
74+ // libvidya dlopens, and a library that reached the process that way is
75+ // invisible to JNI method binding. So it is named here, where the class
76+ // that declares the natives is initialised. Already-mapped is fine;
77+ // loadLibrary finds it and registers it rather than mapping it twice.
78+ try {
79+ System.loadLibrary("joltmoq");
80+ } catch (UnsatisfiedLinkError e) {
81+ // Nothing below can work after this: every frame and every state
82+ // change goes out through a native method.
83+ Log.e(TAG, "libjoltmoq.so not loadable; camera capture is dead", e);
84+ }
85+ }
86+
87+ private CameraCapture() {}
88+
89+ /**
90+ * Native: push one NV12 frame into the MoQ publish pipeline.
91+ *
92+ * @param rotationDegrees CW degrees (0/90/180/270) to apply so the frame
93+ * is upright for the current display — see {@link #computeRotationDegrees()}.
94+ */
95+ private static native void onNv12Frame(
96+ byte[] yData,
97+ byte[] uvData,
98+ int width,
99+ int height,
100+ int yStride,
101+ int uvStride,
102+ int rotationDegrees);
103+
104+ /** Native: camera opened successfully (or failed). */
105+ private static native void onCameraState(boolean opened, String detail);
106+
107+ /**
108+ * List available cameras as {@code "id\\tlabel"} lines (id then tab then label).
109+ * Facing preference: front first, then back, then external/other.
110+ */
111+ public static String[] listCameras(Context context) {
112+ try {
113+ CameraManager manager =
114+ (CameraManager) context.getSystemService(Context.CAMERA_SERVICE);
115+ if (manager == null) {
116+ return new String[0];
117+ }
118+ String[] ids = manager.getCameraIdList();
119+ List<String> front = new ArrayList<>();
120+ List<String> back = new ArrayList<>();
121+ List<String> other = new ArrayList<>();
122+ for (String id : ids) {
123+ CameraCharacteristics chars = manager.getCameraCharacteristics(id);
124+ Integer facing = chars.get(CameraCharacteristics.LENS_FACING);
125+ String label;
126+ if (facing != null && facing == CameraCharacteristics.LENS_FACING_FRONT) {
127+ label = "Front camera · " + id;
128+ front.add(id + "\t" + label);
129+ } else if (facing != null && facing == CameraCharacteristics.LENS_FACING_BACK) {
130+ label = "Back camera · " + id;
131+ back.add(id + "\t" + label);
132+ } else {
133+ label = "Camera · " + id;
134+ other.add(id + "\t" + label);
135+ }
136+ }
137+ List<String> out = new ArrayList<>(front.size() + back.size() + other.size());
138+ out.addAll(front);
139+ out.addAll(back);
140+ out.addAll(other);
141+ return out.toArray(new String[0]);
142+ } catch (Throwable t) {
143+ Log.w(TAG, "listCameras failed", t);
144+ return new String[0];
145+ }
146+ }
147+
148+ /**
149+ * Start capture. {@code cameraId} may be null/empty (prefer front), a device
150+ * id, or the tokens {@code "front"} / {@code "back"}.
151+ */
152+ public static void start(final Activity activity, final String cameraId) {
153+ if (activity == null) {
154+ onCameraState(false, "null activity");
155+ return;
156+ }
157+ activity.runOnUiThread(
158+ new Runnable() {
159+ @Override
160+ public void run() {
161+ startOnThread(activity, cameraId);
162+ }
163+ });
164+ }
165+
166+ /** Stop capture and release camera resources. Safe to call repeatedly. */
167+ public static void stop(final Activity activity) {
168+ Runnable r =
169+ new Runnable() {
170+ @Override
171+ public void run() {
172+ stopLocked();
173+ }
174+ };
175+ if (activity != null) {
176+ activity.runOnUiThread(r);
177+ } else {
178+ r.run();
179+ }
180+ }
181+
182+ private static void startOnThread(Activity activity, String cameraId) {
183+ synchronized (LOCK) {
184+ stopLocked();
185+ if (activity.checkSelfPermission(android.Manifest.permission.CAMERA)
186+ != PackageManager.PERMISSION_GRANTED) {
187+ Log.e(TAG, "CAMERA permission not granted");
188+ onCameraState(false, "permission denied");
189+ return;
190+ }
191+ CameraManager manager =
192+ (CameraManager) activity.getSystemService(Context.CAMERA_SERVICE);
193+ if (manager == null) {
194+ onCameraState(false, "no CameraManager");
195+ return;
196+ }
197+ String id;
198+ try {
199+ id = resolveCameraId(manager, cameraId);
200+ } catch (CameraAccessException e) {
201+ Log.e(TAG, "resolveCameraId", e);
202+ onCameraState(false, e.getMessage());
203+ return;
204+ }
205+ if (id == null) {
206+ onCameraState(false, "no camera");
207+ return;
208+ }
209+
210+ hostActivity = activity;
211+ try {
212+ CameraCharacteristics chars = manager.getCameraCharacteristics(id);
213+ Integer so = chars.get(CameraCharacteristics.SENSOR_ORIENTATION);
214+ sensorOrientationDeg = so != null ? so : 90;
215+ Integer facing = chars.get(CameraCharacteristics.LENS_FACING);
216+ frontFacing =
217+ facing != null && facing == CameraCharacteristics.LENS_FACING_FRONT;
218+ } catch (CameraAccessException e) {
219+ Log.w(TAG, "characteristics; using default orientation", e);
220+ sensorOrientationDeg = 90;
221+ frontFacing = true;
222+ }
223+
224+ Size size = chooseSize(manager, id);
225+ int width = size.getWidth();
226+ int height = size.getHeight();
227+ int rot = computeRotationDegrees();
228+ Log.i(
229+ TAG,
230+ "opening camera id="
231+ + id
232+ + " size="
233+ + width
234+ + "x"
235+ + height
236+ + " sensorOrient="
237+ + sensorOrientationDeg
238+ + " front="
239+ + frontFacing
240+ + " rot="
241+ + rot);
242+
243+ startBackgroundThread();
244+ frameCount = 0;
245+ imageReader = ImageReader.newInstance(width, height, ImageFormat.YUV_420_888, 2);
246+ imageReader.setOnImageAvailableListener(
247+ new ImageReader.OnImageAvailableListener() {
248+ @Override
249+ public void onImageAvailable(ImageReader reader) {
250+ Image image = null;
251+ try {
252+ image = reader.acquireLatestImage();
253+ if (image == null) {
254+ return;
255+ }
256+ pushNv12(image);
257+ } catch (Throwable t) {
258+ Log.w(TAG, "onImageAvailable", t);
259+ } finally {
260+ if (image != null) {
261+ image.close();
262+ }
263+ }
264+ }
265+ },
266+ backgroundHandler);
267+
268+ final String openId = id;
269+ try {
270+ manager.openCamera(
271+ openId,
272+ new CameraDevice.StateCallback() {
273+ @Override
274+ public void onOpened(CameraDevice camera) {
275+ synchronized (LOCK) {
276+ cameraDevice = camera;
277+ createSessionLocked(camera);
278+ }
279+ }
280+
281+ @Override
282+ public void onDisconnected(CameraDevice camera) {
283+ Log.w(TAG, "camera disconnected");
284+ synchronized (LOCK) {
285+ camera.close();
286+ if (cameraDevice == camera) {
287+ cameraDevice = null;
288+ }
289+ running = false;
290+ }
291+ onCameraState(false, "disconnected");
292+ }
293+
294+ @Override
295+ public void onError(CameraDevice camera, int error) {
296+ Log.e(TAG, "camera error " + error);
297+ synchronized (LOCK) {
298+ camera.close();
299+ if (cameraDevice == camera) {
300+ cameraDevice = null;
301+ }
302+ running = false;
303+ }
304+ onCameraState(false, "error " + error);
305+ }
306+ },
307+ backgroundHandler);
308+ } catch (SecurityException e) {
309+ Log.e(TAG, "openCamera security", e);
310+ stopLocked();
311+ onCameraState(false, "permission denied");
312+ } catch (CameraAccessException e) {
313+ Log.e(TAG, "openCamera", e);
314+ stopLocked();
315+ onCameraState(false, e.getMessage());
316+ }
317+ }
318+ }
319+
320+ private static void createSessionLocked(CameraDevice camera) {
321+ final ImageReader reader = imageReader;
322+ if (reader == null) {
323+ onCameraState(false, "no ImageReader");
324+ return;
325+ }
326+ final Surface surface = reader.getSurface();
327+ try {
328+ camera.createCaptureSession(
329+ Collections.singletonList(surface),
330+ new CameraCaptureSession.StateCallback() {
331+ @Override
332+ public void onConfigured(CameraCaptureSession session) {
333+ synchronized (LOCK) {
334+ if (cameraDevice == null) {
335+ return;
336+ }
337+ captureSession = session;
338+ try {
339+ CaptureRequest.Builder builder =
340+ cameraDevice.createCaptureRequest(
341+ CameraDevice.TEMPLATE_PREVIEW);
342+ builder.addTarget(surface);
343+ builder.set(
344+ CaptureRequest.CONTROL_AF_MODE,
345+ CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_VIDEO);
346+ session.setRepeatingRequest(
347+ builder.build(), null, backgroundHandler);
348+ running = true;
349+ onCameraState(true, "live");
350+ Log.i(TAG, "capture session live");
351+ } catch (CameraAccessException e) {
352+ Log.e(TAG, "setRepeatingRequest", e);
353+ onCameraState(false, e.getMessage());
354+ }
355+ }
356+ }
357+
358+ @Override
359+ public void onConfigureFailed(CameraCaptureSession session) {
360+ Log.e(TAG, "capture session configure failed");
361+ onCameraState(false, "configure failed");
362+ }
363+ },
364+ backgroundHandler);
365+ } catch (CameraAccessException e) {
366+ Log.e(TAG, "createCaptureSession", e);
367+ onCameraState(false, e.getMessage());
368+ }
369+ }
370+
371+ private static void pushNv12(Image image) {
372+ if (image.getFormat() != ImageFormat.YUV_420_888) {
373+ return;
374+ }
375+ Image.Plane[] planes = image.getPlanes();
376+ if (planes.length < 3) {
377+ return;
378+ }
379+ int width = image.getWidth();
380+ int height = image.getHeight();
381+ Image.Plane yPlane = planes[0];
382+ Image.Plane uPlane = planes[1];
383+ Image.Plane vPlane = planes[2];
384+ int yStride = yPlane.getRowStride();
385+ int uvStride = uPlane.getRowStride();
386+ int uvPixelStride = uPlane.getPixelStride();
387+ int uvHeight = height / 2;
388+
389+ ByteBuffer yBuf = yPlane.getBuffer();
390+ int ySize = yStride * height;
391+ byte[] yData = new byte[ySize];
392+ yBuf.position(0);
393+ int yCopy = Math.min(ySize, yBuf.remaining());
394+ yBuf.get(yData, 0, yCopy);
395+
396+ byte[] uvData;
397+ int outUvStride;
398+ if (uvPixelStride == 2) {
399+ // Semi-planar: U plane buffer is UVUV… (NV12-style on most devices).
400+ ByteBuffer uvBuf = uPlane.getBuffer();
401+ int uvSize = uvStride * uvHeight;
402+ uvData = new byte[uvSize];
403+ uvBuf.position(0);
404+ int uvCopy = Math.min(uvSize, uvBuf.remaining());
405+ uvBuf.get(uvData, 0, uvCopy);
406+ outUvStride = uvStride;
407+ } else {
408+ // Planar I420 → interleave U+V into NV12.
409+ ByteBuffer uBuf = uPlane.getBuffer();
410+ ByteBuffer vBuf = vPlane.getBuffer();
411+ int uvWidth = width / 2;
412+ outUvStride = width;
413+ uvData = new byte[outUvStride * uvHeight];
414+ for (int row = 0; row < uvHeight; row++) {
415+ for (int col = 0; col < uvWidth; col++) {
416+ int srcIdx = row * uPlane.getRowStride() + col * uvPixelStride;
417+ int dstIdx = row * outUvStride + col * 2;
418+ uvData[dstIdx] = uBuf.get(srcIdx);
419+ uvData[dstIdx + 1] = vBuf.get(srcIdx);
420+ }
421+ }
422+ }
423+
424+ int rotationDegrees = computeRotationDegrees();
425+ onNv12Frame(yData, uvData, width, height, yStride, outUvStride, rotationDegrees);
426+ long n = ++frameCount;
427+ if (n == 1L || n == 30L || n == 300L) {
428+ Log.i(
429+ TAG,
430+ "nv12 frame #"
431+ + n
432+ + " "
433+ + width
434+ + "x"
435+ + height
436+ + " yStride="
437+ + yStride
438+ + " uvStride="
439+ + outUvStride
440+ + " uvPixelStride="
441+ + uvPixelStride
442+ + " rot="
443+ + rotationDegrees);
444+ }
445+ }
446+
447+ /**
448+ * CW degrees to rotate sensor buffers so content is upright for the
449+ * current display orientation (Camera2 JPEG_ORIENTATION formula).
450+ */
451+ private static int computeRotationDegrees() {
452+ int device = displayRotationDegrees(hostActivity);
453+ if (frontFacing) {
454+ return (sensorOrientationDeg + device) % 360;
455+ }
456+ return (sensorOrientationDeg - device + 360) % 360;
457+ }
458+
459+ private static int displayRotationDegrees(Activity activity) {
460+ if (activity == null) {
461+ return 0;
462+ }
463+ try {
464+ int rotation;
465+ if (Build.VERSION.SDK_INT >= 30) {
466+ Display display = activity.getDisplay();
467+ rotation = display != null ? display.getRotation() : Surface.ROTATION_0;
468+ } else {
469+ @SuppressWarnings("deprecation")
470+ Display display = activity.getWindowManager().getDefaultDisplay();
471+ rotation = display.getRotation();
472+ }
473+ switch (rotation) {
474+ case Surface.ROTATION_90:
475+ return 90;
476+ case Surface.ROTATION_180:
477+ return 180;
478+ case Surface.ROTATION_270:
479+ return 270;
480+ case Surface.ROTATION_0:
481+ default:
482+ return 0;
483+ }
484+ } catch (Throwable t) {
485+ Log.w(TAG, "displayRotationDegrees", t);
486+ return 0;
487+ }
488+ }
489+
490+ private static String resolveCameraId(CameraManager manager, String requested)
491+ throws CameraAccessException {
492+ String[] ids = manager.getCameraIdList();
493+ if (ids.length == 0) {
494+ return null;
495+ }
496+ String want = requested == null ? "" : requested.trim();
497+ if (!want.isEmpty()) {
498+ for (String id : ids) {
499+ if (id.equals(want)) {
500+ return id;
501+ }
502+ }
503+ Integer facingWant = null;
504+ if ("front".equalsIgnoreCase(want)) {
505+ facingWant = CameraCharacteristics.LENS_FACING_FRONT;
506+ } else if ("back".equalsIgnoreCase(want)) {
507+ facingWant = CameraCharacteristics.LENS_FACING_BACK;
508+ }
509+ if (facingWant != null) {
510+ for (String id : ids) {
511+ Integer facing =
512+ manager.getCameraCharacteristics(id)
513+ .get(CameraCharacteristics.LENS_FACING);
514+ if (facingWant.equals(facing)) {
515+ return id;
516+ }
517+ }
518+ }
519+ }
520+ // Prefer front for self-view, then first listed.
521+ for (String id : ids) {
522+ Integer facing =
523+ manager.getCameraCharacteristics(id).get(CameraCharacteristics.LENS_FACING);
524+ if (facing != null && facing == CameraCharacteristics.LENS_FACING_FRONT) {
525+ return id;
526+ }
527+ }
528+ return ids[0];
529+ }
530+
531+ private static Size chooseSize(CameraManager manager, String cameraId) {
532+ try {
533+ CameraCharacteristics chars = manager.getCameraCharacteristics(cameraId);
534+ android.hardware.camera2.params.StreamConfigurationMap map =
535+ chars.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
536+ if (map == null) {
537+ return new Size(TARGET_WIDTH, TARGET_HEIGHT);
538+ }
539+ Size[] sizes = map.getOutputSizes(ImageFormat.YUV_420_888);
540+ if (sizes == null || sizes.length == 0) {
541+ return new Size(TARGET_WIDTH, TARGET_HEIGHT);
542+ }
543+ Size best = sizes[0];
544+ long bestScore = Long.MAX_VALUE;
545+ for (Size s : sizes) {
546+ long dw = (long) s.getWidth() - TARGET_WIDTH;
547+ long dh = (long) s.getHeight() - TARGET_HEIGHT;
548+ long score = dw * dw + dh * dh;
549+ // Prefer sizes at or above target when scores are close.
550+ if (s.getWidth() < TARGET_WIDTH / 2 || s.getHeight() < TARGET_HEIGHT / 2) {
551+ score += 10_000_000L;
552+ }
553+ if (score < bestScore) {
554+ bestScore = score;
555+ best = s;
556+ }
557+ }
558+ return best;
559+ } catch (Throwable t) {
560+ Log.w(TAG, "chooseSize fallback", t);
561+ return new Size(TARGET_WIDTH, TARGET_HEIGHT);
562+ }
563+ }
564+
565+ private static void stopLocked() {
566+ running = false;
567+ hostActivity = null;
568+ if (captureSession != null) {
569+ try {
570+ captureSession.close();
571+ } catch (Throwable ignored) {
572+ }
573+ captureSession = null;
574+ }
575+ if (cameraDevice != null) {
576+ try {
577+ cameraDevice.close();
578+ } catch (Throwable ignored) {
579+ }
580+ cameraDevice = null;
581+ }
582+ if (imageReader != null) {
583+ try {
584+ imageReader.close();
585+ } catch (Throwable ignored) {
586+ }
587+ imageReader = null;
588+ }
589+ stopBackgroundThread();
590+ }
591+
592+ private static void startBackgroundThread() {
593+ stopBackgroundThread();
594+ backgroundThread = new HandlerThread("frq-camera");
595+ backgroundThread.start();
596+ backgroundHandler = new Handler(backgroundThread.getLooper());
597+ }
598+
599+ private static void stopBackgroundThread() {
600+ if (backgroundThread != null) {
601+ backgroundThread.quitSafely();
602+ try {
603+ backgroundThread.join(1000);
604+ } catch (InterruptedException ignored) {
605+ }
606+ backgroundThread = null;
607+ backgroundHandler = null;
608+ }
609+ }
610+}
modified scripts/buck.bb +8 -1
@@ -40,6 +40,10 @@ exec "$(dirname "$0")/bb" "$0" "$@"
4040 (fs/copy (fs/path jolt-native "build" "android" "arm64-v8a" "libvidya.so")
4141 (fs/path prebuilt "arm64-v8a" "libvidya.so")
4242 {:replace-existing true})
43+ ;; The media plane, staged the same way. `just ffi-android` builds both.
44+ (fs/copy (fs/path jolt-native "build" "android" "arm64-v8a" "libjoltmoq.so")
45+ (fs/path prebuilt "arm64-v8a" "libjoltmoq.so")
46+ {:replace-existing true})
4347 ;; The glue travels with it, for the same reason: editing jolt_main.c
4448 ;; should relink libjoltapp, and it cannot if buck only knows a path.
4549 (fs/create-dirs (fs/path prebuilt "glue" "android"))
@@ -47,7 +51,10 @@ exec "$(dirname "$0")/bb" "$0" "$@"
4751 (fs/copy (fs/path jolt-native "android" "jolt_main.c")
4852 (fs/path prebuilt "glue" "android" "jolt_main.c")
4953 {:replace-existing true})
50- (doseq [h (fs/glob (fs/path jolt-native "crates" "jolt-vidya" "include") "*.h")]
54+ ;; Both ABIs' headers: jolt_main.c includes joltmoq.h now, for the
55+ ;; symbols it registers and for joltmoq_android_init.
56+ (doseq [dir ["jolt-vidya" "jolt-moq"]
57+ h (fs/glob (fs/path jolt-native "crates" dir "include") "*.h")]
5158 (fs/copy h (fs/path prebuilt "glue" "include" (fs/file-name h))
5259 {:replace-existing true}))
5360 "checkout")
@@ -40,6 +40,10 @@ exec "$(dirname "$0")/bb" "$0" "$@"
40 (fs/copy (fs/path jolt-native "build" "android" "arm64-v8a" "libvidya.so")40 (fs/copy (fs/path jolt-native "build" "android" "arm64-v8a" "libvidya.so")
41 (fs/path prebuilt "arm64-v8a" "libvidya.so")41 (fs/path prebuilt "arm64-v8a" "libvidya.so")
42 {:replace-existing true})42 {:replace-existing true})
43+ ;; The media plane, staged the same way. `just ffi-android` builds both.
44+ (fs/copy (fs/path jolt-native "build" "android" "arm64-v8a" "libjoltmoq.so")
45+ (fs/path prebuilt "arm64-v8a" "libjoltmoq.so")
46+ {:replace-existing true})
43 ;; The glue travels with it, for the same reason: editing jolt_main.c47 ;; The glue travels with it, for the same reason: editing jolt_main.c
44 ;; should relink libjoltapp, and it cannot if buck only knows a path.48 ;; should relink libjoltapp, and it cannot if buck only knows a path.
45 (fs/create-dirs (fs/path prebuilt "glue" "android"))49 (fs/create-dirs (fs/path prebuilt "glue" "android"))
@@ -47,7 +51,10 @@ exec "$(dirname "$0")/bb" "$0" "$@"
47 (fs/copy (fs/path jolt-native "android" "jolt_main.c")51 (fs/copy (fs/path jolt-native "android" "jolt_main.c")
48 (fs/path prebuilt "glue" "android" "jolt_main.c")52 (fs/path prebuilt "glue" "android" "jolt_main.c")
49 {:replace-existing true})53 {:replace-existing true})
50- (doseq [h (fs/glob (fs/path jolt-native "crates" "jolt-vidya" "include") "*.h")]54+ ;; Both ABIs' headers: jolt_main.c includes joltmoq.h now, for the
55+ ;; symbols it registers and for joltmoq_android_init.
56+ (doseq [dir ["jolt-vidya" "jolt-moq"]
57+ h (fs/glob (fs/path jolt-native "crates" dir "include") "*.h")]
51 (fs/copy h (fs/path prebuilt "glue" "include" (fs/file-name h))58 (fs/copy h (fs/path prebuilt "glue" "include" (fs/file-name h))
52 {:replace-existing true}))59 {:replace-existing true}))
53 "checkout")60 "checkout")