The 1080p DMA-BUF pipeline crashed the user's entire Hyprland session
tonight. Root cause identified: hyprwm/Hyprland PR #15167 (fixed
2026-06-18, first shipped v0.56.0) -- CGLRenderbuffer's destructor
unconditionally dereferences m_framebuffer, which is left null when
createEGLImage() fails to import the DMA-BUF. This machine runs
v0.55.4, a week before the fix. The crash happens inside Hyprland's
own Screenshare::CScreenshareFrame::copyDmabuf(), confirmed against
the coredump; it is not a breadcast, GStreamer, or PipeWire bug, and
it is not resolution-dependent (upstream's own reports span unrelated
triggers -- touchpad gestures, tab switching, a Discord stream -- not
capture geometry), so 1080p vs 720p was never the actual variable.
choose_capture_backend() now reads the running compositor's version
over its own IPC socket and only uses the DMA-BUF path on Hyprland
>= 0.56.0 (or non-Hyprland sessions, unaffected by this bug). Below
that, or if the version can't be determined, it falls back to the
plain system-memory/wl_shm pipeline -- slower and still subject to
xdg-desktop-portal-hyprland's separate "Out of buffers" stall bug
(also confirmed via journalctl, and also not fixed in the installed
xdpw 1.3.12), but that failure mode is a stall, not a compositor-wide
abort.
That stall used to be silent forever: xdpw stops requesting frames
after 10 failed retries and never signals PipeWire, so GStreamer's own
bus reports nothing -- no error, no EOS -- and the frame pump just
blocks. pull_encoded_frame now bails after 10s of a Playing pipeline
producing nothing, converting an indefinite silent freeze into a real,
reported session failure (still correctly distinguishing a genuine
stall from ordinary EOS/teardown, so a normal stop() doesn't trip it).
VideoParams is no longer a hand-synced constant: build_video_pipeline_for_streaming
now returns the geometry/frame-rate it actually chose alongside the
pipeline, and both breadcastd::cast_mirror and cast_stream_test thread
that straight into the OFFER instead of a separately-maintained
default. Keeping two copies in sync by hand is exactly how the
resolution mismatch bug happened earlier tonight; returning the real
value makes that class of bug unrepresentable rather than just fixed
once.
Verified without touching the real compositor: cargo build --workspace
--examples, clippy, and both new unit tests (version parsing, the
0.55.4/0.56.0 backend-selection boundary) are clean. The watchdog's
firing path and the DMA-BUF path post-Hyprland-update are not yet
validated against real hardware -- deliberately, given what the last
live test cost. Recommended order: update Hyprland
(pacman -Syu hyprland xdg-desktop-portal-hyprland gets 0.56.1 + xdpw
1.4.1, which also picks up upstream fixes for the exact copy-fence and
SHM-handling bugs hit tonight) and confirm `hyprctl version` reports
>= 0.56.0 before testing DMA-BUF again. Without updating, this commit
still helps: the wl_shm path is selected automatically and the stall
is now bounded instead of indefinite.
Every "fix" for the mirroring freezes so far has been reasoned from code
rather than measured, because the one counter that could have falsified any
of them was blind by construction: `enqueue_frame` returns as soon as a frame
is *posted* to openscreen's TaskRunner, long before `Sender::EnqueueFrame`
decides whether to accept it. The frame pump's `enqueued_fps` therefore read a
healthy 30fps through every freeze.
Add `BreadcastEnqueueStats` (new FFI accessor, no behaviour change): per-second
counts of OK / MAX_DURATION_IN_FLIGHT / REACHED_ID_SPAN_LIMIT /
PAYLOAD_TOO_LARGE, plus the in-flight window gauges and RTT sampled at the
enqueue attempt, all surfaced on the existing "frame pump rate" line as
`accepted_fps` / `rejected_*`.
Measured against the real Chromecast, that settles it: 12.2% of frames were
being rejected with MAX_DURATION_IN_FLIGHT, in 85% of all seconds -- steady,
not just during visible freezes. Since breadcast enqueues already-encoded
frames, each rejection silently breaks the H.264 reference chain rather than
merely dropping a frame.
The measurement also corrects the diagnosis. The send window is
clamp(2*RTT, kMinSenderInFlight, target_playout_delay/3); the assumption was
that a LAN pins it to the 66ms floor. It does not -- RTT to this receiver runs
42-189ms, so 2*RTT is 84-378ms and the window was pinned at the *ceiling*,
133ms at a 400ms playout delay. The ceiling was the binding constraint, so
raising the floor alone would have changed nothing.
So raise both, ceiling first: target playout delay 400ms -> 1200ms (ceiling
133ms -> 400ms) and kMinSenderInFlight 66ms -> 200ms for RTT dips. Measured
over a matched 65s steady-state window, rejections fall 12.2% -> 4.3% and
seconds containing a broken reference chain 85% -> 40%. Costs ~800ms of
added latency, which is unnoticeable for mirroring to a TV.
This is an improvement, not a cure. The residual rejections are bursts
(in-flight seen at 433ms against a 200ms window, RTT spiking to 221ms), and no
static window survives those. The real fix is the backpressure contract
sender.h documents and this facade still doesn't implement: consult
GetInFlightMediaDuration()/GetMaxInFlightMediaDuration() and throttle *before*
encoding, so a skipped frame never leaves a dangling reference behind.
Root cause (found by Opus 5 second-opinion review) of the freezing that
survived every prior fix tonight: openscreen's Sender caps in-flight
unacknowledged media at clamp(2*RTT, 66ms, 133ms) -- on a LAN that's
pinned at the 66ms floor, about two frames at 30fps. When EnqueueFrame
rejects a frame under that budget, facade.cc discarded the result
(`(void)video_sender->EnqueueFrame(frame)`) and moved on. But
vah264enc had already encoded the *next* frame as a P-slice depending
on the one that just got silently dropped -- the encoder has no idea
the drop happened, since it happens downstream of encoding, at this
FFI boundary. The receiver sees an unbroken frame-ID sequence (nothing
here told it otherwise) and decodes a P-slice against a reference
picture it never received: a stuck/corrupted frame until the next
regularly-scheduled key frame (up to ~2s, longer if that key frame is
itself dropped the same way -- explains the 20+s outlier). Zero
"receiver reported picture loss" lines across 195s of a visibly
freezing session confirms the receiver genuinely never noticed
anything was wrong, which a real decode-capability or packet-loss
problem would have triggered.
This also explains why moving to a faster network made it *worse*:
the in-flight budget is RTT-derived, not bandwidth-derived, so more
throughput doesn't raise the 66ms floor at all -- while 1080p tripled
the per-frame packet count, increasing how often frames missed that
window.
Fixed the actual corruption: both drop paths in facade.cc (the
EnqueueFrame rejection, and the pre-existing non-monotonic-capture-time
guard) now set a `frame_chain_broken` flag, consumed once by
breadcast_caststream_sender_needs_key_frame() so frame_pump_loop forces
a key frame on the very next frame instead of chaining more P-slices
onto a reference that no longer exists on the receiver. A separate flag
from the existing `needs_key_frame` atomic because SchedulePoll's 100ms
timer unconditionally overwrites that one with the Sender's own
(unrelated) NeedsKeyFrame() reading, which would have silently clobbered
this signal.
Also reverted the Cast Streaming pipeline from tonight's 1080p
experiment back to 1280x720 (build_video_pipeline_for_streaming +
VideoParams::default(), which must agree -- a mismatch there is a
separate protocol-level bug fixed earlier tonight). Not the root cause,
but a real contributing factor per the packet-count reasoning above, and
untangling it from the frame_chain_broken fix by changing both at once
would make the next test ambiguous. Kept the 6000/1500 kbps bitrate
range from earlier tonight, now actually paired with 720p for the first
time.
The deeper real fix -- raising openscreen's 66ms in-flight floor itself,
which trades latency for headroom -- is out of scope for tonight; this
targets the corruption mechanism (via the sanctioned, if awkward,
needs_key_frame signal) without touching vendored openscreen constants.
User is moving to a faster (150 Mbps) network and wants 1080p. Raised
build_video_pipeline_for_streaming and VideoParams::default() together
(they must agree -- see caststream.rs's doc comment on why a resolution
mismatch there is a protocol violation, not just soft video).
Left build_video_pipeline (the HLS/DLNA path) at 720p -- that one's
1280x720 choice is about an older Default Media Receiver's decoder
profile/level, unrelated to what's changing here.
Bitrate ceiling raised from 4-8x scaling but deliberately not straight
back up to the old 8 Mbps: real testing tonight showed the AIMD probe
pins to whatever MAX_BITRATE_KBPS is for the entire session once
estimated_bandwidth_bps() reports (unreliably -- flat ~20 Mbps most of
a session that was visibly stuttering) that there's headroom, and 8
Mbps sustained was more than the previous network+receiver could hold.
6 Mbps is a solid target for 1080p30 on its own merits. MIN_BITRATE_KBPS
bumped 1000->1500 to match (1080p needs more of a floor than 720p did
before it's a wall of blocking artifacts).
start()'s "never received an ANSWER" branch called session.stop()
directly (blocking, unbounded, not even off the async runtime's worker
thread) and capture.close().await with no timeout -- the exact same
freeze-the-whole-daemon hazard Self::stop() was just bounded against,
just at a different call site. An unresponsive receiver hits this path
by definition (that's what a negotiation timeout means), so it's not a
hypothetical: reproduced live just now, wedging the daemon for over a
minute with no way to recover short of kill -9.
Factored the bound into stop_session_bounded/close_capture_bounded so
both call sites share one implementation instead of drifting.
deadlock, and OFFER/encode resolution mismatch
Found live-testing on real hardware after the first round of fixes:
1. mDNS resolves one Chromecast on every local address it has (a private
IPv4 and a link-local IPv6 in the common case), as separate
CastDeviceFound events for the same id. The device map was a plain
HashMap::insert, so whichever address resolved last won -- and a bare
fe80:: address has no interface scope attached, so connecting to it
fails outright. This is what "failed to connect and launch the
Mirroring receiver" actually was; the message just didn't say why,
since daemon.rs was converting the anyhow::Error with to_string()
(Display, outermost .context() only) instead of "{e:#}" (full chain).
Fixed both: prefer an already-usable host over a link-local one
instead of always taking the newest resolution, and preserve the full
error chain to the GUI/bread event log.
2. The CASTV2 receive loop (run_io_loop) treated any error from
device.receive() as connection-fatal and ended the whole loop --
including a plain JSON deserialization failure on a single
MEDIA_STATUS message (rust_cast's struct requires an `images` field
this receiver didn't send). That killed the receiver-status/control
channel for the rest of every session, on the very first status
update, while the RTP stream itself kept flowing obliviously.
rust_cast::Error already distinguishes Io/Tls/Dns (actually fatal)
from Serialization/Parsing/etc (a bad message, not a dead socket) --
only end the loop on the former now.
3. CastSession::stop() blocks on an unbounded reply_rx.recv() waiting for
the io thread's device.receiver.stop_app() -- a network round trip
rust_cast gives no way to put a read timeout on. If the receiver ever
stops responding, that never returns, and since the daemon actor
processes one command at a time, a single wedged stop_cast freezes
every future IPC request too, recoverable only by killing the process
-- which is exactly what was observed live. Bounded both that wait and
join_pump's thread joins to 5s; past that, log a warning and tear down
anyway rather than hang forever. The abandoned thread(s) may leak, but
a leak beats an unrecoverable daemon.
4. VideoParams::default() advertised 1920x1080 in the OFFER while
build_video_pipeline_for_streaming actually encodes 1280x720 --
negotiated and actual resolution disagreeing is a real protocol
violation, not just soft video, and a plausible cause of a receiver
decoder corrupting or freezing outright rather than merely looking
worse. Made the OFFER match what's actually sent.
The receiver-status io loop treated any read error as a dead connection,
so one malformed/unexpected message (e.g. a MEDIA_STATUS missing a field
the struct requires) tore down the whole control channel for the rest of
the session even though the RTP stream was fine. Only end the loop on
Io/Tls/Dns errors now; log and keep going on Serialization/Parsing/etc.
mDNS resolves a single physical Cast device once per local address it
has, so the same device id can show up with a private IPv4 host and
again with a link-local IPv6 host. A bare fe80:: address has no zone id
attached, so connecting to it fails outright -- don't let it replace an
already-usable host in the device map just because it resolved more
recently.
e.to_string() on an anyhow::Error only prints the outermost .context()
message; the underlying cause (why connect_app/build_pipeline/etc.
actually failed) was being silently dropped before it ever reached the
GUI or bread event log. Use "{e:#}" instead.
Three independent problems found by reading the mirroring paths end to end.
1. Segfault on Cast session teardown. CastStreamSender::SchedulePoll
self-reschedules every 100ms with a raw `this` and was never cancelled,
so the one task that can be scheduled to run *after* an already-queued
teardown task would dereference the just-reset `environment` unique_ptr
(Environment::task_runner() dereferences a member immediately) --- a hard
null deref on openscreen's TaskRunner thread. TaskRunnerImpl's shutdown
has an explicit flushing phase and PlatformClientPosix::ShutDown()'s quit
task queues behind whatever is already pending, so this is a race the
teardown path can lose. Latch a `shutting_down` atomic before posting
teardown and check it in the poll and in every other posted task.
2. Encoder bitrate collapsing to the floor within seconds. The control loop
set `target = 0.85 * estimate` once a second unconditionally. openscreen's
BandwidthEstimator deliberately under-estimates capacity whenever the
transmit rate is below it and documents the required TCP-like response;
multiplying the target by <=0.85 every second instead walks 4000 kbps past
1500 in ~6s and pins it at the floor on a healthy LAN. Replaced with
proper AIMD (hold on a zero/unknown estimate, back off below it, probe up
10%/s otherwise), clamped to 1000..8000 kbps, with unit tests.
3. DLNA/HLS latency. Segment length is max(target-duration, GOP), so
target-duration=2 with a 2s GOP gave 2s segments, ~6s of renderer buffer,
plus 3 segments of stale video waited for before handover. 1s segments
(GOP halved to make that reachable), shorter playlist, and wait for 2
segments instead of 3.
Also hardened two paths into openscreen's fatal OSP_CHECK on strictly
increasing RTP timestamps: pull_encoded_frame no longer substitutes 0 for a
missing PTS (it skips the buffer), and facade.cc drops non-monotonic capture
times at the FFI boundary. Either could previously abort the daemon outright.
CastMirrorSession now owns the Arc<CastStreamSender> instead of leaving its
lifetime to whichever detached pump thread dropped the last clone, so the
blocking FFI destroy happens at a defined point in stop() with the pump
joins ordered around it.
Builds out the full v1 scope: a vendored+patched openscreen subset for
low-latency Cast Streaming (Mirroring receiver 0F5096E8) alongside the
existing Cast V2/HLS and new DLNA/AVTransport casting paths, breadcastd's
Idle/Casting state machine with a private IPC socket, the breadcast GTK4
popup as a thin IPC client, and bread.cast.*/bread.command.cast.* breadd
integration (device discovery, start/stop, mirroring lifecycle events).
Also adds bakery/systemd/Forgejo CI packaging.
Validated end-to-end against a real Chromecast/Google TV: negotiated
Cast Streaming session, live pipeline playback, and daemon+GUI click-to-cast/
stop through the actual popup.
