breadbug: bread-client panic hook, log forwarding and heartbeat (automatic for every app)

Breadway commented

2026-10-08 20:14:12 +08:00

Owner

Part of ECO-EP1 (see ECO-66).

"Comes with every app" means no per-app work: adopting breadbug should be a version bump of bread-client.

Requirements

bread-client installs a panic hook on connect: captures backtrace as frames (module + offset + build-id), panic location, message (scrubbed), app version, then flushes to breadd before abort. Must not panic inside the hook and must have a timeout so a dead breadd never delays exit.
Log forwarding: tracing/log output mirrored to breadd's store (rate limited, drop-oldest under backpressure).
Heartbeat from the app's main loop at a fixed interval, used by breadd for freeze detection (separate issue). For GTK apps, tick from the main context so a blocked UI thread stops the heartbeat.
Opt-out per app for apps handling sensitive data.
Works if breadd is not running (no-op, no errors).
Build-id is embedded and readable (document build flag requirements for apps).

Acceptance

Reference integration in one app (breadclip pilot, matches the earlier spinal-cord pilot) produces a store entry after an induced panic.
Overhead measured and documented.

Part of ECO-EP1 (see ECO-66). "Comes with every app" means no per-app work: adopting breadbug should be a version bump of bread-client. ## Requirements - bread-client installs a panic hook on connect: captures backtrace as frames (module + offset + build-id), panic location, message (scrubbed), app version, then flushes to breadd before abort. Must not panic inside the hook and must have a timeout so a dead breadd never delays exit. - Log forwarding: tracing/log output mirrored to breadd's store (rate limited, drop-oldest under backpressure). - Heartbeat from the app's main loop at a fixed interval, used by breadd for freeze detection (separate issue). For GTK apps, tick from the main context so a blocked UI thread stops the heartbeat. - Opt-out per app for apps handling sensitive data. - Works if breadd is not running (no-op, no errors). - Build-id is embedded and readable (document build flag requirements for apps). ## Acceptance - Reference integration in one app (breadclip pilot, matches the earlier spinal-cord pilot) produces a store entry after an induced panic. - Overhead measured and documented.

Breadway added the

breadbug

phase-1

labels

2026-10-08 20:14:12 +08:00

Breadway commented

2026-10-09 21:31:05 +08:00

Author

Owner

Implemented in bread-ecosystem PR #68 (bread-client lives in bread-utils, not the bread repo). Done: panic hook with module/offset/build-id frames and hook-frame cutting, bounded lossy log forwarding plus a log::Log backend, tick-gated heartbeat (GLib attachment under the gtk feature), per-app opt-out via BugOptions::disabled(), no-op without breadd, has_build_id(). Not done from the acceptance text: (1) the breadclip pilot integration, which will happen as part of the ECO-68 rollout once bread-utils is tagged; (2) a measured overhead number (the per-call cost of log()/tick() is a channel try_send / atomic store, but I did not benchmark it). The gtk-feature build and a release-mode run of the crash tests were not run in this session.

Breadway commented

2026-10-13 20:07:32 +08:00

Author

Owner

Independent verdict: changes_requested at exact head b2614ba0e1b48f25cd40a09e184b721f512f5408, base b52f0fd6faf10c0f595c70dbc3bfb64cb4114fa8. Full criterion mapping, commands and file/line repair checklist persisted: Breadway/bread-ecosystem#68 (comment). Findings ECO-76. Reproduced shutdown hang with retained logger (timeout3s exit124), raised SIGSEGV survives(exit0), prior disposition overwritten, raw fake PII preserved. Native glibc backtrace is not justified async-signal-safe; no prior handler/alt-stack restoration. Logs drop-newest without sustained rate limit; adoption remains manual, not automatic connect. Panic previous-hook call retained and debug/release crash fixtures pass, but do not satisfy integrated acceptance. Independent66 unit+4 integration tests,4 optimized crash fixtures,GTK check and targeted clippy pass; exact-head check/check(push) SUCCESS Actions297. Actual store PR22, scrubber and pilot CLIP-19 unmerged; store-entry/report acceptance and overhead unverified. BREAD-16/17 to todo for coordinator repair on existing PR68; preserve branch. No merge/tag/release/done. Fresh exact-head review after repair.

Independent verdict: changes_requested at exact head b2614ba0e1b48f25cd40a09e184b721f512f5408, base b52f0fd6faf10c0f595c70dbc3bfb64cb4114fa8. Full criterion mapping, commands and file/line repair checklist persisted: https://git.breadway.dev/Breadway/bread-ecosystem/pulls/68#issuecomment-4431. Findings ECO-76. Reproduced shutdown hang with retained logger (timeout3s exit124), raised SIGSEGV survives(exit0), prior disposition overwritten, raw fake PII preserved. Native glibc backtrace is not justified async-signal-safe; no prior handler/alt-stack restoration. Logs drop-newest without sustained rate limit; adoption remains manual, not automatic connect. Panic previous-hook call retained and debug/release crash fixtures pass, but do not satisfy integrated acceptance. Independent66 unit+4 integration tests,4 optimized crash fixtures,GTK check and targeted clippy pass; exact-head check/check(push) SUCCESS Actions297. Actual store PR22, scrubber and pilot CLIP-19 unmerged; store-entry/report acceptance and overhead unverified. BREAD-16/17 to todo for coordinator repair on existing PR68; preserve branch. No merge/tag/release/done. Fresh exact-head review after repair.

Breadway commented

2026-10-14 10:34:11 +08:00

Author

Owner

Independent changes_requested at exact head 63b855bce63313f789ab04ffea96f0b89fdce974, base 4f70691b33bc778294073681065665d3e2b7f559. Full mapping: Breadway/bread-ecosystem#68 (comment). Repair of comment 4431 verified for std-guard detection, SIGSEGV death (7/7 crash tests, 0 ignored), custom prior handler chaining, ucontext RIP frame, auto-connect, scrub-before-send, bounded shutdown, and measured overhead. Still failing: Queue::push (mod.rs:256-274) drops the newest line when take holds the lock, even under the 512 cap. Reproduced twice: logs_are_batched_clipped_and_leveled left 2, right 3, missing the last line. Filed ECO-77. Store entry and breadclip pilot stay unverified (CLIP-19 backlog; no tag). No merge. Move to todo for repair on existing PR 68.

Independent changes_requested at exact head 63b855bce63313f789ab04ffea96f0b89fdce974, base 4f70691b33bc778294073681065665d3e2b7f559. Full mapping: https://git.breadway.dev/Breadway/bread-ecosystem/pulls/68#issuecomment-4678. Repair of comment 4431 verified for std-guard detection, SIGSEGV death (7/7 crash tests, 0 ignored), custom prior handler chaining, ucontext RIP frame, auto-connect, scrub-before-send, bounded shutdown, and measured overhead. Still failing: Queue::push (mod.rs:256-274) drops the newest line when take holds the lock, even under the 512 cap. Reproduced twice: logs_are_batched_clipped_and_leveled left 2, right 3, missing the last line. Filed ECO-77. Store entry and breadclip pilot stay unverified (CLIP-19 backlog; no tag). No merge. Move to todo for repair on existing PR 68.

Breadway commented

2026-10-14 12:39:10 +08:00

Author

Owner

PR #68 independent re-review at 47dbe00f5e1caf06dc3fa78d904001ba978e5b9a: changes requested. Queue::push in bread-utils/src/bug/mod.rs spins 1ms then takes blocking mutex, so app-thread logging can stall without a bound if lock holder is preempted. No IO under lock does not bound acquisition. Replace with truly bounded/nonblocking enqueue preserving newest under-cap and drop-oldest at cap; add a test holding lock > retry interval and asserting prompt producer return plus retained newest item. Full verdict persisted on PR #68. Keep pilot/store and CLIP-19 gates open; do not mark done.

Breadway commented

2026-10-14 12:48:15 +08:00

Author

Owner

Independent re-review of PR68 at 711e3e8261064e066b53b8f31d0cfade94d1d8ba: ECO-77/provider implementation passes, conditional on exact-SHA CI green. Persisted verdict/evidence: Breadway/bread-ecosystem#68 (comment). Queue nonblocking/drop-oldest tests and workspace suite passed; integrated store prerequisite is present, but CLIP-19 pilot remains unmerged. This is not full BREAD-16/17 acceptance. Preserve pilot/rollout gates, keep cards in review, and do not mark done. No merge until check/check (push) is green; then re-read current head/base/reviews/mergeability and merge per authorization, no tag.

Independent re-review of PR68 at 711e3e8261064e066b53b8f31d0cfade94d1d8ba: ECO-77/provider implementation passes, conditional on exact-SHA CI green. Persisted verdict/evidence: https://git.breadway.dev/Breadway/bread-ecosystem/pulls/68#issuecomment-4795. Queue nonblocking/drop-oldest tests and workspace suite passed; integrated store prerequisite is present, but CLIP-19 pilot remains unmerged. This is not full BREAD-16/17 acceptance. Preserve pilot/rollout gates, keep cards in review, and do not mark done. No merge until check/check (push) is green; then re-read current head/base/reviews/mergeability and merge per authorization, no tag.

Breadway self-assigned this

2026-10-14 12:59:33 +08:00

Rows
Columns

breadbug: bread-client panic hook, log forwarding and heartbeat (automatic for every app) #15

Requirements

Acceptance