Hear the other people too
Audio through the plane: mic to Opus to a MoQ track, and every peer's voice decoded into a jitter buffer and summed into one stream. It rides publish_media with format opus and an OpusHead up front, because this object has no publish_audio -- that being moq-ffi's audio feature, off in every Linux artifact. Video resolves its parameters in band and audio does not, so the 19-byte header is required rather than optional, and it is little-endian where the rest of this port is big: OpusHead is Ogg's format and predates all of it. frq.av.audio is its own namespace because mixing policy is a thing worth reading on its own, and the defaults ARE the design. Depth: two frames, 40ms, accumulated before a peer plays at all -- shallower and every hiccup is a gap, deeper and the call gains latency nobody asked for. Overflow: the OLDEST frame goes, because a listener wants the most recent audio and keeping the stale end of a backlog delays everything behind it permanently. Underflow: conceal rather than stall, since waiting for a late frame would stall every other peer too -- they share the output clock. Summed and clamped rather than averaged, or every voice would get quieter as more people joined. Rings are allocated once and reused. Decoding into fresh memory every 20ms would put the allocator in the audio path, which is the one place it has no business being. The mix excludes our own ring. Writing the test found that the honest way: publishing only our own audio and waiting for it produced silence, correctly. And it found a real bug, which I had written a comment defending. The catalog walk accepted a peer only once it had a VIDEO track, so an audio-only peer waited for ever -- and because the audio track is named in the same catalog, it was never read either. Someone with their camera off would have gone silent as well as dark. Either track is enough now, and the video walk tolerates a peer that has none. What is still missing and will eventually be audible: no resampling for clock drift. The sound card's clock and the sender's are not the same, and over minutes the ring slowly fills or slowly empties. The fix is to resample by a fraction of a percent, and it wants measurement this buffer does not take. frq.av is still untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3541022 parent: 8c28876 added
src/frq/av/audio.clj +158 -0 | new file mode 100644 | ||
| @@ -0,0 +1,158 @@ | ||
| 1 | +(ns frq.av.audio | |
| 2 | + "Opus over MoQ, and the jitter buffer that makes it listenable. | |
| 3 | + | |
| 4 | + Video can afford to be simple here: a frame arrives, it is decoded, it is | |
| 5 | + painted, and if one is late the picture holds. Audio cannot. A gap in | |
| 6 | + playback is audible as a click, arriving early is as bad as arriving late, | |
| 7 | + and several peers have to be summed into ONE stream whose clock belongs to | |
| 8 | + the sound card rather than to any of them. That is what this namespace is | |
| 9 | + for, and it is why it is separate from `frq.av.plane` — mixing policy is a | |
| 10 | + thing to be able to read on its own. | |
| 11 | + | |
| 12 | + THE MODEL. Fixed 20ms frames at 48kHz — 960 samples per channel, which is | |
| 13 | + Opus's usual frame and what `frame-samples` is. Every peer decodes into a | |
| 14 | + small ring; the mixer takes one frame from each peer's ring per tick and | |
| 15 | + sums them. A peer whose ring is empty contributes Opus's own concealment | |
| 16 | + rather than silence, because a dropped packet concealed sounds like a | |
| 17 | + smudge where silence sounds like a click. | |
| 18 | + | |
| 19 | + THE THREE THINGS A JITTER BUFFER DECIDES, spelled out because the defaults | |
| 20 | + are the whole design: | |
| 21 | + | |
| 22 | + * DEPTH. `target-depth` frames are accumulated before a peer is played | |
| 23 | + at all. Too shallow and every network hiccup is a gap; too deep and | |
| 24 | + the call gains latency nobody asked for. Two frames — 40ms — is the | |
| 25 | + usual starting point for a conversation. | |
| 26 | + * OVERFLOW. Past `max-depth` the OLDEST frame is dropped, not the | |
| 27 | + newest. A listener wants the most recent audio; keeping the stale end | |
| 28 | + of a backlog just delays everything behind it permanently. | |
| 29 | + * UNDERFLOW. An empty ring conceals rather than stalls. Waiting for the | |
| 30 | + late frame would stall every OTHER peer too, since they share the | |
| 31 | + output clock. | |
| 32 | + | |
| 33 | + This is a deliberately plain buffer: fixed depth, no adaptation to | |
| 34 | + measured jitter, no clock-drift resampling. Those are real and they are | |
| 35 | + missing, and the note at `mix-into!` says what goes wrong without them." | |
| 36 | + (:require [frq.codec.opus :as opus] | |
| 37 | + [jolt.ffi :as ffi])) | |
| 38 | + | |
| 39 | +(def ^:const sample-rate 48000) | |
| 40 | +(def ^:const frame-samples 960) ; 20ms at 48kHz, per channel | |
| 41 | +(def ^:const target-depth 2) | |
| 42 | +(def ^:const max-depth 6) | |
| 43 | + | |
| 44 | +;; --- OpusHead ---------------------------------------------------------------- | |
| 45 | + | |
| 46 | +(defn opus-head! | |
| 47 | + "The 19-byte OpusHead an Opus track's catalog entry needs, into `p`. | |
| 48 | + | |
| 49 | + Little-endian, unlike everything else in this port — OpusHead is Ogg's | |
| 50 | + header format and predates any of it. An audio track will not publish | |
| 51 | + without one: video resolves its parameters in band and audio does not." | |
| 52 | + [p channels] | |
| 53 | + (let [magic [0x4f 0x70 0x75 0x73 0x48 0x65 0x61 0x64] ; "OpusHead" | |
| 54 | + pre-skip 3840] | |
| 55 | + (dotimes [i 8] (ffi/write (+ p i) :uint8 (nth magic i))) | |
| 56 | + (ffi/write (+ p 8) :uint8 1) ; version | |
| 57 | + (ffi/write (+ p 9) :uint8 channels) | |
| 58 | + (ffi/write (+ p 10) :uint8 (bit-and pre-skip 255)) | |
| 59 | + (ffi/write (+ p 11) :uint8 (bit-and (bit-shift-right pre-skip 8) 255)) | |
| 60 | + (dotimes [i 4] | |
| 61 | + (ffi/write (+ p 12 i) :uint8 | |
| 62 | + (bit-and (bit-shift-right sample-rate (* 8 i)) 255))) | |
| 63 | + (ffi/write (+ p 16) :uint8 0) ; output gain lo | |
| 64 | + (ffi/write (+ p 17) :uint8 0) ; output gain hi | |
| 65 | + (ffi/write (+ p 18) :uint8 0) ; mapping family | |
| 66 | + [p 19])) | |
| 67 | + | |
| 68 | +;; --- a peer's ring ----------------------------------------------------------- | |
| 69 | + | |
| 70 | +(defn ring | |
| 71 | + "A peer's decoded-audio ring: `max-depth` frames of foreign memory. | |
| 72 | + | |
| 73 | + Allocated once and reused. Decoding into fresh memory every 20ms would | |
| 74 | + make the allocator part of the audio path, which is the one place it has | |
| 75 | + no business being." | |
| 76 | + [channels] | |
| 77 | + {:decoder (opus/decoder sample-rate channels) | |
| 78 | + :channels channels | |
| 79 | + :slots (mapv (fn [_] (ffi/alloc (* 2 frame-samples channels))) | |
| 80 | + (range max-depth)) | |
| 81 | + :filled (atom []) ; indices holding audio, oldest first | |
| 82 | + :free (atom (vec (range max-depth))) | |
| 83 | + :started? (atom false)}) | |
| 84 | + | |
| 85 | +(defn close-ring! [r] | |
| 86 | + (opus/free-decoder! (:decoder r)) | |
| 87 | + (doseq [p (:slots r)] (ffi/free p)) | |
| 88 | + nil) | |
| 89 | + | |
| 90 | +(defn push-packet! | |
| 91 | + "Decode one Opus packet into the ring. | |
| 92 | + | |
| 93 | + Over `max-depth` the OLDEST frame goes, not this one: a listener wants the | |
| 94 | + most recent audio, and keeping the stale end of a backlog delays | |
| 95 | + everything behind it for the rest of the call." | |
| 96 | + [r ptr len] | |
| 97 | + (let [{:keys [decoder slots filled free channels]} r | |
| 98 | + i (if-let [i (first @free)] | |
| 99 | + (do (swap! free subvec 1) i) | |
| 100 | + (let [oldest (first @filled)] | |
| 101 | + (swap! filled subvec 1) | |
| 102 | + oldest)) | |
| 103 | + n (opus/decode! decoder ptr len (nth slots i) frame-samples)] | |
| 104 | + (swap! filled conj i) | |
| 105 | + (when (>= (count @filled) target-depth) (reset! (:started? r) true)) | |
| 106 | + n)) | |
| 107 | + | |
| 108 | +(defn- take-frame! | |
| 109 | + "The oldest frame in the ring, or nil while it is still filling." | |
| 110 | + [r] | |
| 111 | + (when @(:started? r) | |
| 112 | + (when-let [i (first @(:filled r))] | |
| 113 | + (swap! (:filled r) subvec 1) | |
| 114 | + (swap! (:free r) conj i) | |
| 115 | + (nth (:slots r) i)))) | |
| 116 | + | |
| 117 | +;; --- mixing ------------------------------------------------------------------ | |
| 118 | + | |
| 119 | +(defn mix-into! | |
| 120 | + "Sum one frame from every ring into `out`; answers the peak written. | |
| 121 | + | |
| 122 | + Summed and CLAMPED, not averaged. Averaging would make every voice quieter | |
| 123 | + as more people joined, which is the wrong behaviour in a meeting; clamping | |
| 124 | + only bites when several people are loud at once, which is already | |
| 125 | + unpleasant for other reasons. | |
| 126 | + | |
| 127 | + A ring with nothing in it conceals — `opus/decode!` with no packet is | |
| 128 | + Opus's own loss concealment — rather than contributing silence, because a | |
| 129 | + gap is a click and a concealed frame is a smudge. | |
| 130 | + | |
| 131 | + WHAT IS NOT HERE, and it will be audible eventually: no resampling for | |
| 132 | + clock drift. The sound card's clock and the sender's are not the same, and | |
| 133 | + over minutes one drifts against the other — the ring slowly fills or | |
| 134 | + slowly empties, and the fix is to resample by a fraction of a percent | |
| 135 | + rather than to keep dropping or concealing. That wants measurement this | |
| 136 | + buffer does not yet take." | |
| 137 | + [rings out channels] | |
| 138 | + (let [n (* frame-samples channels)] | |
| 139 | + (dotimes [i n] (ffi/write (+ out (* 2 i)) :int16 0)) | |
| 140 | + (doseq [r rings] | |
| 141 | + (let [src (or (take-frame! r) | |
| 142 | + ;; Conceal: decode nothing, which Opus turns into a | |
| 143 | + ;; plausible continuation of what it last heard. | |
| 144 | + (let [slot (nth (:slots r) 0)] | |
| 145 | + (when @(:started? r) | |
| 146 | + (opus/decode! (:decoder r) nil 0 slot frame-samples) | |
| 147 | + slot)))] | |
| 148 | + (when src | |
| 149 | + (dotimes [i n] | |
| 150 | + (let [a (ffi/read (+ out (* 2 i)) :int16) | |
| 151 | + b (ffi/read (+ src (* 2 i)) :int16) | |
| 152 | + v (+ a b)] | |
| 153 | + (ffi/write (+ out (* 2 i)) :int16 | |
| 154 | + (cond (> v 32767) 32767 (< v -32768) -32768 :else v))))))) | |
| 155 | + (loop [i 0 peak 0] | |
| 156 | + (if (= i n) | |
| 157 | + peak | |
| 158 | + (recur (inc i) (max peak (abs (ffi/read (+ out (* 2 i)) :int16)))))))) | |
| new file mode 100644 | |||
| @@ -0,0 +1,158 @@ | |||
| 1 | +(ns frq.av.audio | ||
| 2 | + "Opus over MoQ, and the jitter buffer that makes it listenable. | ||
| 3 | + | ||
| 4 | + Video can afford to be simple here: a frame arrives, it is decoded, it is | ||
| 5 | + painted, and if one is late the picture holds. Audio cannot. A gap in | ||
| 6 | + playback is audible as a click, arriving early is as bad as arriving late, | ||
| 7 | + and several peers have to be summed into ONE stream whose clock belongs to | ||
| 8 | + the sound card rather than to any of them. That is what this namespace is | ||
| 9 | + for, and it is why it is separate from `frq.av.plane` — mixing policy is a | ||
| 10 | + thing to be able to read on its own. | ||
| 11 | + | ||
| 12 | + THE MODEL. Fixed 20ms frames at 48kHz — 960 samples per channel, which is | ||
| 13 | + Opus's usual frame and what `frame-samples` is. Every peer decodes into a | ||
| 14 | + small ring; the mixer takes one frame from each peer's ring per tick and | ||
| 15 | + sums them. A peer whose ring is empty contributes Opus's own concealment | ||
| 16 | + rather than silence, because a dropped packet concealed sounds like a | ||
| 17 | + smudge where silence sounds like a click. | ||
| 18 | + | ||
| 19 | + THE THREE THINGS A JITTER BUFFER DECIDES, spelled out because the defaults | ||
| 20 | + are the whole design: | ||
| 21 | + | ||
| 22 | + * DEPTH. `target-depth` frames are accumulated before a peer is played | ||
| 23 | + at all. Too shallow and every network hiccup is a gap; too deep and | ||
| 24 | + the call gains latency nobody asked for. Two frames — 40ms — is the | ||
| 25 | + usual starting point for a conversation. | ||
| 26 | + * OVERFLOW. Past `max-depth` the OLDEST frame is dropped, not the | ||
| 27 | + newest. A listener wants the most recent audio; keeping the stale end | ||
| 28 | + of a backlog just delays everything behind it permanently. | ||
| 29 | + * UNDERFLOW. An empty ring conceals rather than stalls. Waiting for the | ||
| 30 | + late frame would stall every OTHER peer too, since they share the | ||
| 31 | + output clock. | ||
| 32 | + | ||
| 33 | + This is a deliberately plain buffer: fixed depth, no adaptation to | ||
| 34 | + measured jitter, no clock-drift resampling. Those are real and they are | ||
| 35 | + missing, and the note at `mix-into!` says what goes wrong without them." | ||
| 36 | + (:require [frq.codec.opus :as opus] | ||
| 37 | + [jolt.ffi :as ffi])) | ||
| 38 | + | ||
| 39 | +(def ^:const sample-rate 48000) | ||
| 40 | +(def ^:const frame-samples 960) ; 20ms at 48kHz, per channel | ||
| 41 | +(def ^:const target-depth 2) | ||
| 42 | +(def ^:const max-depth 6) | ||
| 43 | + | ||
| 44 | +;; --- OpusHead ---------------------------------------------------------------- | ||
| 45 | + | ||
| 46 | +(defn opus-head! | ||
| 47 | + "The 19-byte OpusHead an Opus track's catalog entry needs, into `p`. | ||
| 48 | + | ||
| 49 | + Little-endian, unlike everything else in this port — OpusHead is Ogg's | ||
| 50 | + header format and predates any of it. An audio track will not publish | ||
| 51 | + without one: video resolves its parameters in band and audio does not." | ||
| 52 | + [p channels] | ||
| 53 | + (let [magic [0x4f 0x70 0x75 0x73 0x48 0x65 0x61 0x64] ; "OpusHead" | ||
| 54 | + pre-skip 3840] | ||
| 55 | + (dotimes [i 8] (ffi/write (+ p i) :uint8 (nth magic i))) | ||
| 56 | + (ffi/write (+ p 8) :uint8 1) ; version | ||
| 57 | + (ffi/write (+ p 9) :uint8 channels) | ||
| 58 | + (ffi/write (+ p 10) :uint8 (bit-and pre-skip 255)) | ||
| 59 | + (ffi/write (+ p 11) :uint8 (bit-and (bit-shift-right pre-skip 8) 255)) | ||
| 60 | + (dotimes [i 4] | ||
| 61 | + (ffi/write (+ p 12 i) :uint8 | ||
| 62 | + (bit-and (bit-shift-right sample-rate (* 8 i)) 255))) | ||
| 63 | + (ffi/write (+ p 16) :uint8 0) ; output gain lo | ||
| 64 | + (ffi/write (+ p 17) :uint8 0) ; output gain hi | ||
| 65 | + (ffi/write (+ p 18) :uint8 0) ; mapping family | ||
| 66 | + [p 19])) | ||
| 67 | + | ||
| 68 | +;; --- a peer's ring ----------------------------------------------------------- | ||
| 69 | + | ||
| 70 | +(defn ring | ||
| 71 | + "A peer's decoded-audio ring: `max-depth` frames of foreign memory. | ||
| 72 | + | ||
| 73 | + Allocated once and reused. Decoding into fresh memory every 20ms would | ||
| 74 | + make the allocator part of the audio path, which is the one place it has | ||
| 75 | + no business being." | ||
| 76 | + [channels] | ||
| 77 | + {:decoder (opus/decoder sample-rate channels) | ||
| 78 | + :channels channels | ||
| 79 | + :slots (mapv (fn [_] (ffi/alloc (* 2 frame-samples channels))) | ||
| 80 | + (range max-depth)) | ||
| 81 | + :filled (atom []) ; indices holding audio, oldest first | ||
| 82 | + :free (atom (vec (range max-depth))) | ||
| 83 | + :started? (atom false)}) | ||
| 84 | + | ||
| 85 | +(defn close-ring! [r] | ||
| 86 | + (opus/free-decoder! (:decoder r)) | ||
| 87 | + (doseq [p (:slots r)] (ffi/free p)) | ||
| 88 | + nil) | ||
| 89 | + | ||
| 90 | +(defn push-packet! | ||
| 91 | + "Decode one Opus packet into the ring. | ||
| 92 | + | ||
| 93 | + Over `max-depth` the OLDEST frame goes, not this one: a listener wants the | ||
| 94 | + most recent audio, and keeping the stale end of a backlog delays | ||
| 95 | + everything behind it for the rest of the call." | ||
| 96 | + [r ptr len] | ||
| 97 | + (let [{:keys [decoder slots filled free channels]} r | ||
| 98 | + i (if-let [i (first @free)] | ||
| 99 | + (do (swap! free subvec 1) i) | ||
| 100 | + (let [oldest (first @filled)] | ||
| 101 | + (swap! filled subvec 1) | ||
| 102 | + oldest)) | ||
| 103 | + n (opus/decode! decoder ptr len (nth slots i) frame-samples)] | ||
| 104 | + (swap! filled conj i) | ||
| 105 | + (when (>= (count @filled) target-depth) (reset! (:started? r) true)) | ||
| 106 | + n)) | ||
| 107 | + | ||
| 108 | +(defn- take-frame! | ||
| 109 | + "The oldest frame in the ring, or nil while it is still filling." | ||
| 110 | + [r] | ||
| 111 | + (when @(:started? r) | ||
| 112 | + (when-let [i (first @(:filled r))] | ||
| 113 | + (swap! (:filled r) subvec 1) | ||
| 114 | + (swap! (:free r) conj i) | ||
| 115 | + (nth (:slots r) i)))) | ||
| 116 | + | ||
| 117 | +;; --- mixing ------------------------------------------------------------------ | ||
| 118 | + | ||
| 119 | +(defn mix-into! | ||
| 120 | + "Sum one frame from every ring into `out`; answers the peak written. | ||
| 121 | + | ||
| 122 | + Summed and CLAMPED, not averaged. Averaging would make every voice quieter | ||
| 123 | + as more people joined, which is the wrong behaviour in a meeting; clamping | ||
| 124 | + only bites when several people are loud at once, which is already | ||
| 125 | + unpleasant for other reasons. | ||
| 126 | + | ||
| 127 | + A ring with nothing in it conceals — `opus/decode!` with no packet is | ||
| 128 | + Opus's own loss concealment — rather than contributing silence, because a | ||
| 129 | + gap is a click and a concealed frame is a smudge. | ||
| 130 | + | ||
| 131 | + WHAT IS NOT HERE, and it will be audible eventually: no resampling for | ||
| 132 | + clock drift. The sound card's clock and the sender's are not the same, and | ||
| 133 | + over minutes one drifts against the other — the ring slowly fills or | ||
| 134 | + slowly empties, and the fix is to resample by a fraction of a percent | ||
| 135 | + rather than to keep dropping or concealing. That wants measurement this | ||
| 136 | + buffer does not yet take." | ||
| 137 | + [rings out channels] | ||
| 138 | + (let [n (* frame-samples channels)] | ||
| 139 | + (dotimes [i n] (ffi/write (+ out (* 2 i)) :int16 0)) | ||
| 140 | + (doseq [r rings] | ||
| 141 | + (let [src (or (take-frame! r) | ||
| 142 | + ;; Conceal: decode nothing, which Opus turns into a | ||
| 143 | + ;; plausible continuation of what it last heard. | ||
| 144 | + (let [slot (nth (:slots r) 0)] | ||
| 145 | + (when @(:started? r) | ||
| 146 | + (opus/decode! (:decoder r) nil 0 slot frame-samples) | ||
| 147 | + slot)))] | ||
| 148 | + (when src | ||
| 149 | + (dotimes [i n] | ||
| 150 | + (let [a (ffi/read (+ out (* 2 i)) :int16) | ||
| 151 | + b (ffi/read (+ src (* 2 i)) :int16) | ||
| 152 | + v (+ a b)] | ||
| 153 | + (ffi/write (+ out (* 2 i)) :int16 | ||
| 154 | + (cond (> v 32767) 32767 (< v -32768) -32768 :else v))))))) | ||
| 155 | + (loop [i 0 peak 0] | ||
| 156 | + (if (= i n) | ||
| 157 | + peak | ||
| 158 | + (recur (inc i) (max peak (abs (ffi/read (+ out (* 2 i)) :int16)))))))) | ||
modified
src/frq/av/plane.clj +126 -13 | @@ -47,9 +47,11 @@ | ||
| 47 | 47 | also how the phone will pass a Camera2 buffer in later without this |
| 48 | 48 | namespace learning about JNI." |
| 49 | 49 | (:require [clojure.string :as str] |
| 50 | + [frq.av.audio :as audio] | |
| 50 | 51 | [frq.moq.media :as media] |
| 51 | 52 | [frq.moq.uniffi :as uniffi] |
| 52 | 53 | [frq.codec.h264 :as h264] |
| 54 | + [frq.codec.opus :as opus] | |
| 53 | 55 | [jolt.ffi :as ffi])) |
| 54 | 56 | |
| 55 | 57 | ;; --- state ------------------------------------------------------------------- |
| @@ -74,13 +76,24 @@ | ||
| 74 | 76 | Everything that can fail does so HERE rather than at the first frame: the |
| 75 | 77 | encoder validates its size, the decoder opens, and the subscribe settles, |
| 76 | 78 | so a plane that comes up is one that can carry a picture." |
| 77 | - [{:keys [origin path source width height fps bitrate camera?] | |
| 78 | - :or {path "/frq" width 640 height 480 fps 30 bitrate 800000 camera? true}}] | |
| 79 | + [{:keys [origin path source mic width height fps bitrate camera? muted? | |
| 80 | + channels] | |
| 81 | + :or {path "/frq" width 640 height 480 fps 30 bitrate 800000 | |
| 82 | + camera? true muted? false channels 1}}] | |
| 79 | 83 | (stop!) |
| 80 | 84 | (let [broadcast (media/create-broadcast! origin path) |
| 81 | 85 | producer (media/publish-media! broadcast "avc3") |
| 82 | 86 | track (media/producer-name producer) |
| 83 | - consumer (media/broadcast-consumer broadcast)] | |
| 87 | + consumer (media/broadcast-consumer broadcast) | |
| 88 | + ;; The audio track rides the same publish_media as video — this | |
| 89 | + ;; object has no publish_audio, that being moq-ffi's `audio` | |
| 90 | + ;; feature. What it needs instead is an OpusHead up front: video | |
| 91 | + ;; resolves its parameters in band and audio does not. | |
| 92 | + [head-p head-n] (audio/opus-head! (ffi/alloc 19) channels) | |
| 93 | + mic-producer (when mic | |
| 94 | + (media/publish-media-bytes! broadcast "opus" | |
| 95 | + head-p head-n))] | |
| 96 | + (ffi/free head-p) | |
| 84 | 97 | (reset! plane |
| 85 | 98 | {:broadcast broadcast |
| 86 | 99 | :producer producer |
| @@ -95,6 +108,13 @@ | ||
| 95 | 108 | :peers {} |
| 96 | 109 | :encoder (h264/encoder {:width width :height height |
| 97 | 110 | :fps fps :bitrate bitrate}) |
| 111 | + :mic-producer mic-producer | |
| 112 | + :mic-encoder (when mic (opus/encoder audio/sample-rate channels :voip)) | |
| 113 | + :mic mic | |
| 114 | + :muted? muted? | |
| 115 | + :channels channels | |
| 116 | + :mix (ffi/alloc (* 2 audio/frame-samples channels)) | |
| 117 | + :mixed nil | |
| 98 | 118 | :source source |
| 99 | 119 | :size [width height] |
| 100 | 120 | :camera? camera? |
| @@ -110,7 +130,10 @@ | ||
| 110 | 130 | (when-let [p @plane] |
| 111 | 131 | (try (h264/close! (:encoder p)) (catch Exception _ nil)) |
| 112 | 132 | (doseq [[_ peer] (:peers p)] |
| 113 | - (try (h264/close-decoder! (:decoder peer)) (catch Exception _ nil))) | |
| 133 | + (try (h264/close-decoder! (:decoder peer)) (catch Exception _ nil)) | |
| 134 | + (when-let [r (:ring peer)] (try (audio/close-ring! r) (catch Exception _ nil)))) | |
| 135 | + (when-let [e (:mic-encoder p)] (try (opus/free-encoder! e) (catch Exception _ nil))) | |
| 136 | + (when-let [m (:mix p)] (try (ffi/free m) (catch Exception _ nil))) | |
| 114 | 137 | (reset! plane nil)) |
| 115 | 138 | nil) |
| 116 | 139 | |
| @@ -124,6 +147,13 @@ | ||
| 124 | 147 | (swap! plane #(when % (assoc % :camera? (boolean on?)))) |
| 125 | 148 | nil) |
| 126 | 149 | |
| 150 | +(defn set-muted! | |
| 151 | + "Stop feeding the encoder. The track stays published — a peer who saw it | |
| 152 | + vanish would have to rediscover it to hear you unmute." | |
| 153 | + [muted?] | |
| 154 | + (swap! plane #(when % (assoc % :muted? (boolean muted?)))) | |
| 155 | + nil) | |
| 156 | + | |
| 127 | 157 | (defn force-keyframe! |
| 128 | 158 | "Make the next published frame an IDR. |
| 129 | 159 | |
| @@ -197,6 +227,42 @@ | ||
| 197 | 227 | (normalise-path (:path p)))}))) |
| 198 | 228 | p))) |
| 199 | 229 | |
| 230 | +(defn- pump-peer-audio! | |
| 231 | + "Advance one peer's audio: catalog says the track, then subscribe, then | |
| 232 | + decode into that peer's ring. | |
| 233 | + | |
| 234 | + Separate from the video walk because the two are independent — a peer with | |
| 235 | + a camera off still has a voice, and blocking one on the other is how a | |
| 236 | + muted-video participant goes silent too." | |
| 237 | + [peer channels] | |
| 238 | + (cond | |
| 239 | + (nil? (get-in peer [:catalog :audio-track])) peer | |
| 240 | + | |
| 241 | + (nil? (:audio-media peer)) | |
| 242 | + (let [peer (if (:audio-subscribe peer) | |
| 243 | + peer | |
| 244 | + (assoc peer :audio-subscribe | |
| 245 | + (media/subscribe-media! (:broadcast peer) | |
| 246 | + (get-in peer [:catalog :audio-track]) | |
| 247 | + (get-in peer [:catalog :audio-container]))))] | |
| 248 | + (if-let [mc (settle (:audio-subscribe peer) nil)] | |
| 249 | + (assoc peer :audio-media mc :audio-subscribe nil | |
| 250 | + :ring (audio/ring channels)) | |
| 251 | + peer)) | |
| 252 | + | |
| 253 | + :else | |
| 254 | + (let [peer (if (:audio-pending peer) | |
| 255 | + peer | |
| 256 | + (assoc peer :audio-pending (media/next-frame! (:audio-media peer)))) | |
| 257 | + got (settle (:audio-pending peer) | |
| 258 | + #(media/lift-media-frame | |
| 259 | + % | |
| 260 | + (fn [ptr len] | |
| 261 | + (when (pos? len) | |
| 262 | + (audio/push-packet! (:ring peer) ptr len) | |
| 263 | + true))))] | |
| 264 | + (if got (assoc peer :audio-pending nil) peer)))) | |
| 265 | + | |
| 200 | 266 | (defn- pump-peer! |
| 201 | 267 | "Walk one peer from announced to a decoded picture. |
| 202 | 268 | |
| @@ -216,13 +282,21 @@ | ||
| 216 | 282 | (:consumer sub) |
| 217 | 283 | (let [cp (or (:next sub) (media/next-catalog! (:consumer sub)))] |
| 218 | 284 | (if-let [cat (settle cp media/lift-catalog)] |
| 219 | - (if-let [[track video] (first (:video cat))] | |
| 220 | - (assoc peer :catalog {:track track :container (:container video)} | |
| 221 | - :catalog-pending nil) | |
| 222 | - ;; A catalog with no video is a peer who is not sending a | |
| 223 | - ;; picture — audio only, or not yet publishing. Wait rather | |
| 224 | - ;; than treat it as an error. | |
| 225 | - (assoc peer :catalog-pending {:consumer (:consumer sub) :next nil})) | |
| 285 | + (let [[track video] (first (:video cat)) | |
| 286 | + [atrack aud] (first (:audio cat))] | |
| 287 | + ;; EITHER is enough. Requiring video here is how an audio-only | |
| 288 | + ;; peer waits for ever: with no picture coming the catalog is | |
| 289 | + ;; never accepted, so the audio track named in the same | |
| 290 | + ;; catalog is never read either, and someone with their camera | |
| 291 | + ;; off goes silent as well as dark. | |
| 292 | + (if (or track atrack) | |
| 293 | + (assoc peer :catalog {:track track | |
| 294 | + :container (:container video) | |
| 295 | + :audio-track atrack | |
| 296 | + :audio-container (:container aud)} | |
| 297 | + :catalog-pending nil) | |
| 298 | + ;; Nothing published yet. Ask again. | |
| 299 | + (assoc peer :catalog-pending {:consumer (:consumer sub) :next nil}))) | |
| 226 | 300 | (assoc peer :catalog-pending {:consumer (:consumer sub) :next cp}))) |
| 227 | 301 | |
| 228 | 302 | :else |
| @@ -230,6 +304,10 @@ | ||
| 230 | 304 | (assoc peer :catalog-pending {:consumer cc :next nil}) |
| 231 | 305 | peer))) |
| 232 | 306 | |
| 307 | + ;; No picture from this peer — audio only, or camera off. Not a state | |
| 308 | + ;; to advance out of; their audio walks on its own. | |
| 309 | + (nil? (get-in peer [:catalog :track])) peer | |
| 310 | + | |
| 233 | 311 | ;; 2. Subscribe to the track the catalog named. |
| 234 | 312 | (nil? (:media peer)) |
| 235 | 313 | (let [peer (if (:subscribe peer) |
| @@ -268,11 +346,27 @@ | ||
| 268 | 346 | (assoc peer :pending nil :frame (:payload decoded)) |
| 269 | 347 | (assoc peer :frame nil))))) |
| 270 | 348 | |
| 349 | +(defn- pump-mic! | |
| 350 | + "One 20ms frame from the microphone, encoded and published." | |
| 351 | + [{:keys [mic mic-encoder mic-producer muted? pts channels]}] | |
| 352 | + (when (and mic mic-encoder mic-producer (not muted?)) | |
| 353 | + (when-let [[pcm _] (mic)] | |
| 354 | + (ffi/with-arena [a] | |
| 355 | + (let [out (ffi/alloc a 4000) | |
| 356 | + n (opus/encode! mic-encoder pcm audio/frame-samples out 4000)] | |
| 357 | + ;; DTX is the encoder saying this frame is silence and need not be | |
| 358 | + ;; sent. Sending it anyway would be bytes for nothing. | |
| 359 | + (when-not (opus/dtx? n) | |
| 360 | + (media/write-video-frame! mic-producer out n @pts))))))) | |
| 361 | + | |
| 271 | 362 | (defn- pump-in! |
| 272 | 363 | "Discover peers, then advance every one of them." |
| 273 | 364 | [p] |
| 274 | 365 | (let [p (pump-announce! p) |
| 275 | - peers (reduce-kv (fn [m path peer] (assoc m path (pump-peer! peer))) | |
| 366 | + peers (reduce-kv (fn [m path peer] | |
| 367 | + (assoc m path (-> peer | |
| 368 | + pump-peer! | |
| 369 | + (pump-peer-audio! (:channels p))))) | |
| 276 | 370 | {} (:peers p))] |
| 277 | 371 | (assoc p |
| 278 | 372 | :peers peers |
| @@ -289,7 +383,18 @@ | ||
| 289 | 383 | [] |
| 290 | 384 | (when-let [p @plane] |
| 291 | 385 | (pump-out! p) |
| 292 | - (reset! plane (pump-in! p))) | |
| 386 | + (pump-mic! p) | |
| 387 | + (let [p' (pump-in! p) | |
| 388 | + ;; Mix everyone EXCEPT ourselves: hearing your own voice back is | |
| 389 | + ;; the thing headphones exist to prevent. | |
| 390 | + rings (keep (fn [[_ peer]] (when-not (:self? peer) (:ring peer))) | |
| 391 | + (:peers p')) | |
| 392 | + peak (when (seq rings) | |
| 393 | + (audio/mix-into! rings (:mix p') (:channels p')))] | |
| 394 | + (reset! plane (assoc p' :mixed (when peak | |
| 395 | + {:ptr (:mix p') | |
| 396 | + :samples audio/frame-samples | |
| 397 | + :peak peak}))))) | |
| 293 | 398 | nil) |
| 294 | 399 | |
| 295 | 400 | (defn poll-frames! |
| @@ -307,6 +412,14 @@ | ||
| 307 | 412 | [] |
| 308 | 413 | (:frames @plane)) |
| 309 | 414 | |
| 415 | +(defn poll-audio! | |
| 416 | + "The mixed 20ms frame from the last `pump!`, or nil. | |
| 417 | + | |
| 418 | + {:ptr :samples :peak} — interleaved int16 ready for `alsa/write!`, and | |
| 419 | + BORROWED like everything else here: the next pump mixes over it." | |
| 420 | + [] | |
| 421 | + (:mixed @plane)) | |
| 422 | + | |
| 310 | 423 | (defn peers |
| 311 | 424 | "The broadcast paths currently known, self included." |
| 312 | 425 | [] |
| @@ -47,9 +47,11 @@ | |||
| 47 | also how the phone will pass a Camera2 buffer in later without this | 47 | also how the phone will pass a Camera2 buffer in later without this |
| 48 | namespace learning about JNI." | 48 | namespace learning about JNI." |
| 49 | (:require [clojure.string :as str] | 49 | (:require [clojure.string :as str] |
| 50 | + [frq.av.audio :as audio] | ||
| 50 | [frq.moq.media :as media] | 51 | [frq.moq.media :as media] |
| 51 | [frq.moq.uniffi :as uniffi] | 52 | [frq.moq.uniffi :as uniffi] |
| 52 | [frq.codec.h264 :as h264] | 53 | [frq.codec.h264 :as h264] |
| 54 | + [frq.codec.opus :as opus] | ||
| 53 | [jolt.ffi :as ffi])) | 55 | [jolt.ffi :as ffi])) |
| 54 | 56 | ||
| 55 | ;; --- state ------------------------------------------------------------------- | 57 | ;; --- state ------------------------------------------------------------------- |
| @@ -74,13 +76,24 @@ | |||
| 74 | Everything that can fail does so HERE rather than at the first frame: the | 76 | Everything that can fail does so HERE rather than at the first frame: the |
| 75 | encoder validates its size, the decoder opens, and the subscribe settles, | 77 | encoder validates its size, the decoder opens, and the subscribe settles, |
| 76 | so a plane that comes up is one that can carry a picture." | 78 | so a plane that comes up is one that can carry a picture." |
| 77 | - [{:keys [origin path source width height fps bitrate camera?] | 79 | + [{:keys [origin path source mic width height fps bitrate camera? muted? |
| 78 | - :or {path "/frq" width 640 height 480 fps 30 bitrate 800000 camera? true}}] | 80 | + channels] |
| 81 | + :or {path "/frq" width 640 height 480 fps 30 bitrate 800000 | ||
| 82 | + camera? true muted? false channels 1}}] | ||
| 79 | (stop!) | 83 | (stop!) |
| 80 | (let [broadcast (media/create-broadcast! origin path) | 84 | (let [broadcast (media/create-broadcast! origin path) |
| 81 | producer (media/publish-media! broadcast "avc3") | 85 | producer (media/publish-media! broadcast "avc3") |
| 82 | track (media/producer-name producer) | 86 | track (media/producer-name producer) |
| 83 | - consumer (media/broadcast-consumer broadcast)] | 87 | + consumer (media/broadcast-consumer broadcast) |
| 88 | + ;; The audio track rides the same publish_media as video — this | ||
| 89 | + ;; object has no publish_audio, that being moq-ffi's `audio` | ||
| 90 | + ;; feature. What it needs instead is an OpusHead up front: video | ||
| 91 | + ;; resolves its parameters in band and audio does not. | ||
| 92 | + [head-p head-n] (audio/opus-head! (ffi/alloc 19) channels) | ||
| 93 | + mic-producer (when mic | ||
| 94 | + (media/publish-media-bytes! broadcast "opus" | ||
| 95 | + head-p head-n))] | ||
| 96 | + (ffi/free head-p) | ||
| 84 | (reset! plane | 97 | (reset! plane |
| 85 | {:broadcast broadcast | 98 | {:broadcast broadcast |
| 86 | :producer producer | 99 | :producer producer |
| @@ -95,6 +108,13 @@ | |||
| 95 | :peers {} | 108 | :peers {} |
| 96 | :encoder (h264/encoder {:width width :height height | 109 | :encoder (h264/encoder {:width width :height height |
| 97 | :fps fps :bitrate bitrate}) | 110 | :fps fps :bitrate bitrate}) |
| 111 | + :mic-producer mic-producer | ||
| 112 | + :mic-encoder (when mic (opus/encoder audio/sample-rate channels :voip)) | ||
| 113 | + :mic mic | ||
| 114 | + :muted? muted? | ||
| 115 | + :channels channels | ||
| 116 | + :mix (ffi/alloc (* 2 audio/frame-samples channels)) | ||
| 117 | + :mixed nil | ||
| 98 | :source source | 118 | :source source |
| 99 | :size [width height] | 119 | :size [width height] |
| 100 | :camera? camera? | 120 | :camera? camera? |
| @@ -110,7 +130,10 @@ | |||
| 110 | (when-let [p @plane] | 130 | (when-let [p @plane] |
| 111 | (try (h264/close! (:encoder p)) (catch Exception _ nil)) | 131 | (try (h264/close! (:encoder p)) (catch Exception _ nil)) |
| 112 | (doseq [[_ peer] (:peers p)] | 132 | (doseq [[_ peer] (:peers p)] |
| 113 | - (try (h264/close-decoder! (:decoder peer)) (catch Exception _ nil))) | 133 | + (try (h264/close-decoder! (:decoder peer)) (catch Exception _ nil)) |
| 134 | + (when-let [r (:ring peer)] (try (audio/close-ring! r) (catch Exception _ nil)))) | ||
| 135 | + (when-let [e (:mic-encoder p)] (try (opus/free-encoder! e) (catch Exception _ nil))) | ||
| 136 | + (when-let [m (:mix p)] (try (ffi/free m) (catch Exception _ nil))) | ||
| 114 | (reset! plane nil)) | 137 | (reset! plane nil)) |
| 115 | nil) | 138 | nil) |
| 116 | 139 | ||
| @@ -124,6 +147,13 @@ | |||
| 124 | (swap! plane #(when % (assoc % :camera? (boolean on?)))) | 147 | (swap! plane #(when % (assoc % :camera? (boolean on?)))) |
| 125 | nil) | 148 | nil) |
| 126 | 149 | ||
| 150 | +(defn set-muted! | ||
| 151 | + "Stop feeding the encoder. The track stays published — a peer who saw it | ||
| 152 | + vanish would have to rediscover it to hear you unmute." | ||
| 153 | + [muted?] | ||
| 154 | + (swap! plane #(when % (assoc % :muted? (boolean muted?)))) | ||
| 155 | + nil) | ||
| 156 | + | ||
| 127 | (defn force-keyframe! | 157 | (defn force-keyframe! |
| 128 | "Make the next published frame an IDR. | 158 | "Make the next published frame an IDR. |
| 129 | 159 | ||
| @@ -197,6 +227,42 @@ | |||
| 197 | (normalise-path (:path p)))}))) | 227 | (normalise-path (:path p)))}))) |
| 198 | p))) | 228 | p))) |
| 199 | 229 | ||
| 230 | +(defn- pump-peer-audio! | ||
| 231 | + "Advance one peer's audio: catalog says the track, then subscribe, then | ||
| 232 | + decode into that peer's ring. | ||
| 233 | + | ||
| 234 | + Separate from the video walk because the two are independent — a peer with | ||
| 235 | + a camera off still has a voice, and blocking one on the other is how a | ||
| 236 | + muted-video participant goes silent too." | ||
| 237 | + [peer channels] | ||
| 238 | + (cond | ||
| 239 | + (nil? (get-in peer [:catalog :audio-track])) peer | ||
| 240 | + | ||
| 241 | + (nil? (:audio-media peer)) | ||
| 242 | + (let [peer (if (:audio-subscribe peer) | ||
| 243 | + peer | ||
| 244 | + (assoc peer :audio-subscribe | ||
| 245 | + (media/subscribe-media! (:broadcast peer) | ||
| 246 | + (get-in peer [:catalog :audio-track]) | ||
| 247 | + (get-in peer [:catalog :audio-container]))))] | ||
| 248 | + (if-let [mc (settle (:audio-subscribe peer) nil)] | ||
| 249 | + (assoc peer :audio-media mc :audio-subscribe nil | ||
| 250 | + :ring (audio/ring channels)) | ||
| 251 | + peer)) | ||
| 252 | + | ||
| 253 | + :else | ||
| 254 | + (let [peer (if (:audio-pending peer) | ||
| 255 | + peer | ||
| 256 | + (assoc peer :audio-pending (media/next-frame! (:audio-media peer)))) | ||
| 257 | + got (settle (:audio-pending peer) | ||
| 258 | + #(media/lift-media-frame | ||
| 259 | + % | ||
| 260 | + (fn [ptr len] | ||
| 261 | + (when (pos? len) | ||
| 262 | + (audio/push-packet! (:ring peer) ptr len) | ||
| 263 | + true))))] | ||
| 264 | + (if got (assoc peer :audio-pending nil) peer)))) | ||
| 265 | + | ||
| 200 | (defn- pump-peer! | 266 | (defn- pump-peer! |
| 201 | "Walk one peer from announced to a decoded picture. | 267 | "Walk one peer from announced to a decoded picture. |
| 202 | 268 | ||
| @@ -216,13 +282,21 @@ | |||
| 216 | (:consumer sub) | 282 | (:consumer sub) |
| 217 | (let [cp (or (:next sub) (media/next-catalog! (:consumer sub)))] | 283 | (let [cp (or (:next sub) (media/next-catalog! (:consumer sub)))] |
| 218 | (if-let [cat (settle cp media/lift-catalog)] | 284 | (if-let [cat (settle cp media/lift-catalog)] |
| 219 | - (if-let [[track video] (first (:video cat))] | 285 | + (let [[track video] (first (:video cat)) |
| 220 | - (assoc peer :catalog {:track track :container (:container video)} | 286 | + [atrack aud] (first (:audio cat))] |
| 221 | - :catalog-pending nil) | 287 | + ;; EITHER is enough. Requiring video here is how an audio-only |
| 222 | - ;; A catalog with no video is a peer who is not sending a | 288 | + ;; peer waits for ever: with no picture coming the catalog is |
| 223 | - ;; picture — audio only, or not yet publishing. Wait rather | 289 | + ;; never accepted, so the audio track named in the same |
| 224 | - ;; than treat it as an error. | 290 | + ;; catalog is never read either, and someone with their camera |
| 225 | - (assoc peer :catalog-pending {:consumer (:consumer sub) :next nil})) | 291 | + ;; off goes silent as well as dark. |
| 292 | + (if (or track atrack) | ||
| 293 | + (assoc peer :catalog {:track track | ||
| 294 | + :container (:container video) | ||
| 295 | + :audio-track atrack | ||
| 296 | + :audio-container (:container aud)} | ||
| 297 | + :catalog-pending nil) | ||
| 298 | + ;; Nothing published yet. Ask again. | ||
| 299 | + (assoc peer :catalog-pending {:consumer (:consumer sub) :next nil}))) | ||
| 226 | (assoc peer :catalog-pending {:consumer (:consumer sub) :next cp}))) | 300 | (assoc peer :catalog-pending {:consumer (:consumer sub) :next cp}))) |
| 227 | 301 | ||
| 228 | :else | 302 | :else |
| @@ -230,6 +304,10 @@ | |||
| 230 | (assoc peer :catalog-pending {:consumer cc :next nil}) | 304 | (assoc peer :catalog-pending {:consumer cc :next nil}) |
| 231 | peer))) | 305 | peer))) |
| 232 | 306 | ||
| 307 | + ;; No picture from this peer — audio only, or camera off. Not a state | ||
| 308 | + ;; to advance out of; their audio walks on its own. | ||
| 309 | + (nil? (get-in peer [:catalog :track])) peer | ||
| 310 | + | ||
| 233 | ;; 2. Subscribe to the track the catalog named. | 311 | ;; 2. Subscribe to the track the catalog named. |
| 234 | (nil? (:media peer)) | 312 | (nil? (:media peer)) |
| 235 | (let [peer (if (:subscribe peer) | 313 | (let [peer (if (:subscribe peer) |
| @@ -268,11 +346,27 @@ | |||
| 268 | (assoc peer :pending nil :frame (:payload decoded)) | 346 | (assoc peer :pending nil :frame (:payload decoded)) |
| 269 | (assoc peer :frame nil))))) | 347 | (assoc peer :frame nil))))) |
| 270 | 348 | ||
| 349 | +(defn- pump-mic! | ||
| 350 | + "One 20ms frame from the microphone, encoded and published." | ||
| 351 | + [{:keys [mic mic-encoder mic-producer muted? pts channels]}] | ||
| 352 | + (when (and mic mic-encoder mic-producer (not muted?)) | ||
| 353 | + (when-let [[pcm _] (mic)] | ||
| 354 | + (ffi/with-arena [a] | ||
| 355 | + (let [out (ffi/alloc a 4000) | ||
| 356 | + n (opus/encode! mic-encoder pcm audio/frame-samples out 4000)] | ||
| 357 | + ;; DTX is the encoder saying this frame is silence and need not be | ||
| 358 | + ;; sent. Sending it anyway would be bytes for nothing. | ||
| 359 | + (when-not (opus/dtx? n) | ||
| 360 | + (media/write-video-frame! mic-producer out n @pts))))))) | ||
| 361 | + | ||
| 271 | (defn- pump-in! | 362 | (defn- pump-in! |
| 272 | "Discover peers, then advance every one of them." | 363 | "Discover peers, then advance every one of them." |
| 273 | [p] | 364 | [p] |
| 274 | (let [p (pump-announce! p) | 365 | (let [p (pump-announce! p) |
| 275 | - peers (reduce-kv (fn [m path peer] (assoc m path (pump-peer! peer))) | 366 | + peers (reduce-kv (fn [m path peer] |
| 367 | + (assoc m path (-> peer | ||
| 368 | + pump-peer! | ||
| 369 | + (pump-peer-audio! (:channels p))))) | ||
| 276 | {} (:peers p))] | 370 | {} (:peers p))] |
| 277 | (assoc p | 371 | (assoc p |
| 278 | :peers peers | 372 | :peers peers |
| @@ -289,7 +383,18 @@ | |||
| 289 | [] | 383 | [] |
| 290 | (when-let [p @plane] | 384 | (when-let [p @plane] |
| 291 | (pump-out! p) | 385 | (pump-out! p) |
| 292 | - (reset! plane (pump-in! p))) | 386 | + (pump-mic! p) |
| 387 | + (let [p' (pump-in! p) | ||
| 388 | + ;; Mix everyone EXCEPT ourselves: hearing your own voice back is | ||
| 389 | + ;; the thing headphones exist to prevent. | ||
| 390 | + rings (keep (fn [[_ peer]] (when-not (:self? peer) (:ring peer))) | ||
| 391 | + (:peers p')) | ||
| 392 | + peak (when (seq rings) | ||
| 393 | + (audio/mix-into! rings (:mix p') (:channels p')))] | ||
| 394 | + (reset! plane (assoc p' :mixed (when peak | ||
| 395 | + {:ptr (:mix p') | ||
| 396 | + :samples audio/frame-samples | ||
| 397 | + :peak peak}))))) | ||
| 293 | nil) | 398 | nil) |
| 294 | 399 | ||
| 295 | (defn poll-frames! | 400 | (defn poll-frames! |
| @@ -307,6 +412,14 @@ | |||
| 307 | [] | 412 | [] |
| 308 | (:frames @plane)) | 413 | (:frames @plane)) |
| 309 | 414 | ||
| 415 | +(defn poll-audio! | ||
| 416 | + "The mixed 20ms frame from the last `pump!`, or nil. | ||
| 417 | + | ||
| 418 | + {:ptr :samples :peak} — interleaved int16 ready for `alsa/write!`, and | ||
| 419 | + BORROWED like everything else here: the next pump mixes over it." | ||
| 420 | + [] | ||
| 421 | + (:mixed @plane)) | ||
| 422 | + | ||
| 310 | (defn peers | 423 | (defn peers |
| 311 | "The broadcast paths currently known, self included." | 424 | "The broadcast paths currently known, self included." |
| 312 | [] | 425 | [] |
modified
src/frq/moq/media.clj +14 -0 | @@ -231,6 +231,20 @@ | ||
| 231 | 231 | (uniffi/with-out-status |
| 232 | 232 | #(raw/method-moqbroadcastproducer-publish-media h buf %))))))) |
| 233 | 233 | |
| 234 | +(defn publish-media-bytes! | |
| 235 | + "Publish a media track whose init blob is RAW bytes. | |
| 236 | + | |
| 237 | + `publish-media!` above takes the blob as a string, which is fine for the | |
| 238 | + empty one a video track uses and wrong for an OpusHead — that is 19 bytes | |
| 239 | + of little-endian header with interior zeros, and routing it through a jolt | |
| 240 | + string would not survive." | |
| 241 | + [broadcast format ptr len] | |
| 242 | + (let [h (uniffi/with-out-status #(raw/clone-moqbroadcastproducer broadcast %))] | |
| 243 | + (lowered [[:string format] [:bytes [ptr len]] [:u8 0]] | |
| 244 | + (fn [buf] | |
| 245 | + (uniffi/with-out-status | |
| 246 | + #(raw/method-moqbroadcastproducer-publish-media h buf %)))))) | |
| 247 | + | |
| 234 | 248 | (defn producer-name |
| 235 | 249 | "The track name the object chose for this producer — what a subscriber asks |
| 236 | 250 | for by name." |
| @@ -231,6 +231,20 @@ | |||
| 231 | (uniffi/with-out-status | 231 | (uniffi/with-out-status |
| 232 | #(raw/method-moqbroadcastproducer-publish-media h buf %))))))) | 232 | #(raw/method-moqbroadcastproducer-publish-media h buf %))))))) |
| 233 | 233 | ||
| 234 | +(defn publish-media-bytes! | ||
| 235 | + "Publish a media track whose init blob is RAW bytes. | ||
| 236 | + | ||
| 237 | + `publish-media!` above takes the blob as a string, which is fine for the | ||
| 238 | + empty one a video track uses and wrong for an OpusHead — that is 19 bytes | ||
| 239 | + of little-endian header with interior zeros, and routing it through a jolt | ||
| 240 | + string would not survive." | ||
| 241 | + [broadcast format ptr len] | ||
| 242 | + (let [h (uniffi/with-out-status #(raw/clone-moqbroadcastproducer broadcast %))] | ||
| 243 | + (lowered [[:string format] [:bytes [ptr len]] [:u8 0]] | ||
| 244 | + (fn [buf] | ||
| 245 | + (uniffi/with-out-status | ||
| 246 | + #(raw/method-moqbroadcastproducer-publish-media h buf %)))))) | ||
| 247 | + | ||
| 234 | (defn producer-name | 248 | (defn producer-name |
| 235 | "The track name the object chose for this producer — what a subscriber asks | 249 | "The track name the object chose for this producer — what a subscriber asks |
| 236 | for by name." | 250 | for by name." |
modified
src/frq/moq/smoke.clj +75 -1 | @@ -41,6 +41,7 @@ | ||
| 41 | 41 | [frq.capture.v4l2 :as v4l2] |
| 42 | 42 | [frq.capture.alsa :as alsa] |
| 43 | 43 | [frq.av.plane :as plane] |
| 44 | + [frq.av.audio :as audio] | |
| 44 | 45 | [jolt.ffi :as ffi])) |
| 45 | 46 | |
| 46 | 47 | (defn- check-contract [] |
| @@ -521,6 +522,78 @@ | ||
| 521 | 522 | (h264/close! enc) |
| 522 | 523 | (plane/stop!))))))) |
| 523 | 524 | |
| 525 | +(defn- tone | |
| 526 | + "20ms of a triangle at `amp`, int16 mono, in foreign memory." | |
| 527 | + [a amp] | |
| 528 | + (let [n audio/frame-samples | |
| 529 | + p (ffi/alloc a (* 2 n))] | |
| 530 | + (dotimes [i n] | |
| 531 | + (let [phase (mod i 100) | |
| 532 | + v (if (< phase 50) | |
| 533 | + (- (* phase (quot (* 2 amp) 50)) amp) | |
| 534 | + (- amp (* (- phase 50) (quot (* 2 amp) 50))))] | |
| 535 | + (ffi/write (+ p (* 2 i)) :int16 v))) | |
| 536 | + [p n])) | |
| 537 | + | |
| 538 | +(defn- check-audio | |
| 539 | + "A voice through the plane: Opus -> MoQ -> jitter buffer -> mix. | |
| 540 | + | |
| 541 | + The voice belongs to a SECOND peer, and it has to: the mixer excludes our | |
| 542 | + own ring, because hearing your own microphone back is the thing headphones | |
| 543 | + exist to prevent. A test that published only its own audio would be | |
| 544 | + asserting on silence and calling it a bug. | |
| 545 | + | |
| 546 | + The assertion is amplitude. Opus is lossy and a triangle comes back | |
| 547 | + rounded, but silence is unmistakable — and silence is exactly what a | |
| 548 | + broken jitter buffer produces, whether it fills and never drains or drains | |
| 549 | + before it fills. | |
| 550 | + | |
| 551 | + The pump count matters too: the ring holds `target-depth` frames before it | |
| 552 | + plays anything, so a mix appearing on the first pump would mean the depth | |
| 553 | + was not being honoured." | |
| 554 | + [] | |
| 555 | + (ffi/with-arena [a] | |
| 556 | + (let [[pcm _] (tone a 8000) | |
| 557 | + origin (media/new-origin) | |
| 558 | + head-p (ffi/alloc a 19)] | |
| 559 | + (audio/opus-head! head-p 1) | |
| 560 | + (plane/start! {:origin origin :path "/us" | |
| 561 | + :source (fn [] nil) ; no camera in this check | |
| 562 | + :mic (fn [] [pcm audio/frame-samples]) | |
| 563 | + :width 64 :height 64 :channels 1}) | |
| 564 | + (let [b2 (media/create-broadcast! origin "/them") | |
| 565 | + p2 (media/publish-media-bytes! b2 "opus" head-p 19) | |
| 566 | + enc (opus/encoder audio/sample-rate 1 :voip) | |
| 567 | + out (ffi/alloc a 4000)] | |
| 568 | + (try | |
| 569 | + (let [deadline (+ (System/currentTimeMillis) 25000)] | |
| 570 | + (loop [pumps 0] | |
| 571 | + ;; The other participant keeps talking. | |
| 572 | + (let [n (opus/encode! enc pcm audio/frame-samples out 4000)] | |
| 573 | + (when-not (opus/dtx? n) | |
| 574 | + (media/write-video-frame! p2 out n (* pumps 20000)))) | |
| 575 | + (plane/pump!) | |
| 576 | + (let [mixed (plane/poll-audio!)] | |
| 577 | + (cond | |
| 578 | + (and mixed (pos? (:peak mixed))) | |
| 579 | + (do (println " mixed audio after" pumps "pumps, peak" (:peak mixed) | |
| 580 | + "over" (:samples mixed) "samples") | |
| 581 | + (when (< (:peak mixed) 500) | |
| 582 | + (throw (ex-info "the mix is effectively silent" | |
| 583 | + {:peak (:peak mixed)}))) | |
| 584 | + (when (< pumps 2) | |
| 585 | + (throw (ex-info "played before the jitter buffer filled" | |
| 586 | + {:pumps pumps}))) | |
| 587 | + true) | |
| 588 | + | |
| 589 | + (> (System/currentTimeMillis) deadline) | |
| 590 | + (throw (ex-info "no audio came back through the plane" {:pumps pumps})) | |
| 591 | + | |
| 592 | + :else (do (Thread/sleep 5) (recur (inc pumps))))))) | |
| 593 | + (finally | |
| 594 | + (opus/free-encoder! enc) | |
| 595 | + (plane/stop!))))))) | |
| 596 | + | |
| 524 | 597 | (defn -main [& _] |
| 525 | 598 | (println "libmoq_ffi smoke test") |
| 526 | 599 | (let [steps [["contract" check-contract] |
| @@ -533,7 +606,8 @@ | ||
| 533 | 606 | ["v4l2" check-v4l2-layouts] |
| 534 | 607 | ["alsa" check-alsa] |
| 535 | 608 | ["devices" check-enumeration] |
| 536 | - ["plane" check-plane]]] | |
| 609 | + ["plane" check-plane] | |
| 610 | + ["audio" check-audio]]] | |
| 537 | 611 | (doseq [[name f] steps] |
| 538 | 612 | (println (str name ":")) |
| 539 | 613 | (f)) |
| @@ -41,6 +41,7 @@ | |||
| 41 | [frq.capture.v4l2 :as v4l2] | 41 | [frq.capture.v4l2 :as v4l2] |
| 42 | [frq.capture.alsa :as alsa] | 42 | [frq.capture.alsa :as alsa] |
| 43 | [frq.av.plane :as plane] | 43 | [frq.av.plane :as plane] |
| 44 | + [frq.av.audio :as audio] | ||
| 44 | [jolt.ffi :as ffi])) | 45 | [jolt.ffi :as ffi])) |
| 45 | 46 | ||
| 46 | (defn- check-contract [] | 47 | (defn- check-contract [] |
| @@ -521,6 +522,78 @@ | |||
| 521 | (h264/close! enc) | 522 | (h264/close! enc) |
| 522 | (plane/stop!))))))) | 523 | (plane/stop!))))))) |
| 523 | 524 | ||
| 525 | +(defn- tone | ||
| 526 | + "20ms of a triangle at `amp`, int16 mono, in foreign memory." | ||
| 527 | + [a amp] | ||
| 528 | + (let [n audio/frame-samples | ||
| 529 | + p (ffi/alloc a (* 2 n))] | ||
| 530 | + (dotimes [i n] | ||
| 531 | + (let [phase (mod i 100) | ||
| 532 | + v (if (< phase 50) | ||
| 533 | + (- (* phase (quot (* 2 amp) 50)) amp) | ||
| 534 | + (- amp (* (- phase 50) (quot (* 2 amp) 50))))] | ||
| 535 | + (ffi/write (+ p (* 2 i)) :int16 v))) | ||
| 536 | + [p n])) | ||
| 537 | + | ||
| 538 | +(defn- check-audio | ||
| 539 | + "A voice through the plane: Opus -> MoQ -> jitter buffer -> mix. | ||
| 540 | + | ||
| 541 | + The voice belongs to a SECOND peer, and it has to: the mixer excludes our | ||
| 542 | + own ring, because hearing your own microphone back is the thing headphones | ||
| 543 | + exist to prevent. A test that published only its own audio would be | ||
| 544 | + asserting on silence and calling it a bug. | ||
| 545 | + | ||
| 546 | + The assertion is amplitude. Opus is lossy and a triangle comes back | ||
| 547 | + rounded, but silence is unmistakable — and silence is exactly what a | ||
| 548 | + broken jitter buffer produces, whether it fills and never drains or drains | ||
| 549 | + before it fills. | ||
| 550 | + | ||
| 551 | + The pump count matters too: the ring holds `target-depth` frames before it | ||
| 552 | + plays anything, so a mix appearing on the first pump would mean the depth | ||
| 553 | + was not being honoured." | ||
| 554 | + [] | ||
| 555 | + (ffi/with-arena [a] | ||
| 556 | + (let [[pcm _] (tone a 8000) | ||
| 557 | + origin (media/new-origin) | ||
| 558 | + head-p (ffi/alloc a 19)] | ||
| 559 | + (audio/opus-head! head-p 1) | ||
| 560 | + (plane/start! {:origin origin :path "/us" | ||
| 561 | + :source (fn [] nil) ; no camera in this check | ||
| 562 | + :mic (fn [] [pcm audio/frame-samples]) | ||
| 563 | + :width 64 :height 64 :channels 1}) | ||
| 564 | + (let [b2 (media/create-broadcast! origin "/them") | ||
| 565 | + p2 (media/publish-media-bytes! b2 "opus" head-p 19) | ||
| 566 | + enc (opus/encoder audio/sample-rate 1 :voip) | ||
| 567 | + out (ffi/alloc a 4000)] | ||
| 568 | + (try | ||
| 569 | + (let [deadline (+ (System/currentTimeMillis) 25000)] | ||
| 570 | + (loop [pumps 0] | ||
| 571 | + ;; The other participant keeps talking. | ||
| 572 | + (let [n (opus/encode! enc pcm audio/frame-samples out 4000)] | ||
| 573 | + (when-not (opus/dtx? n) | ||
| 574 | + (media/write-video-frame! p2 out n (* pumps 20000)))) | ||
| 575 | + (plane/pump!) | ||
| 576 | + (let [mixed (plane/poll-audio!)] | ||
| 577 | + (cond | ||
| 578 | + (and mixed (pos? (:peak mixed))) | ||
| 579 | + (do (println " mixed audio after" pumps "pumps, peak" (:peak mixed) | ||
| 580 | + "over" (:samples mixed) "samples") | ||
| 581 | + (when (< (:peak mixed) 500) | ||
| 582 | + (throw (ex-info "the mix is effectively silent" | ||
| 583 | + {:peak (:peak mixed)}))) | ||
| 584 | + (when (< pumps 2) | ||
| 585 | + (throw (ex-info "played before the jitter buffer filled" | ||
| 586 | + {:pumps pumps}))) | ||
| 587 | + true) | ||
| 588 | + | ||
| 589 | + (> (System/currentTimeMillis) deadline) | ||
| 590 | + (throw (ex-info "no audio came back through the plane" {:pumps pumps})) | ||
| 591 | + | ||
| 592 | + :else (do (Thread/sleep 5) (recur (inc pumps))))))) | ||
| 593 | + (finally | ||
| 594 | + (opus/free-encoder! enc) | ||
| 595 | + (plane/stop!))))))) | ||
| 596 | + | ||
| 524 | (defn -main [& _] | 597 | (defn -main [& _] |
| 525 | (println "libmoq_ffi smoke test") | 598 | (println "libmoq_ffi smoke test") |
| 526 | (let [steps [["contract" check-contract] | 599 | (let [steps [["contract" check-contract] |
| @@ -533,7 +606,8 @@ | |||
| 533 | ["v4l2" check-v4l2-layouts] | 606 | ["v4l2" check-v4l2-layouts] |
| 534 | ["alsa" check-alsa] | 607 | ["alsa" check-alsa] |
| 535 | ["devices" check-enumeration] | 608 | ["devices" check-enumeration] |
| 536 | - ["plane" check-plane]]] | 609 | + ["plane" check-plane] |
| 610 | + ["audio" check-audio]]] | ||
| 537 | (doseq [[name f] steps] | 611 | (doseq [[name f] steps] |
| 538 | (println (str name ":")) | 612 | (println (str name ":")) |
| 539 | (f)) | 613 | (f)) |