breadbug: freeze detection watchdog in breadd with consented live stack capture #17

Open
opened 2026-10-08 20:14:13 +08:00 by Breadway · 4 comments
Owner

Part of ECO-EP1 (see ECO-66). Depends on the bread-client heartbeat issue.

Requirements

  • breadd tracks heartbeats per app. Missed heartbeats beyond a threshold (configurable, with hysteresis to avoid false positives from suspend/resume and heavy load) mark the app as possibly frozen and record a freeze marker.
  • Emit bread.app.frozen / bread.app.recovered events and a notification: " looks frozen. Report it?" with action to launch breadbug.
  • Do not auto-kill anything.
  • Optional live stack capture of the hung process (eu-stack/gdb-style, frames only) ONLY when the user explicitly consents in the breadbug flow; never automatically.
  • Suspend/resume and monitor hotplug must not trigger false freezes (use the existing power/monitor events to pause the watchdog).

Acceptance

  • Induced main-loop block in a test app triggers a freeze marker and notification.
  • Suspend/resume test produces no false positives.
Part of ECO-EP1 (see ECO-66). Depends on the bread-client heartbeat issue. ## Requirements - breadd tracks heartbeats per app. Missed heartbeats beyond a threshold (configurable, with hysteresis to avoid false positives from suspend/resume and heavy load) mark the app as possibly frozen and record a freeze marker. - Emit `bread.app.frozen` / `bread.app.recovered` events and a notification: "<app> looks frozen. Report it?" with action to launch breadbug. - Do not auto-kill anything. - Optional live stack capture of the hung process (eu-stack/gdb-style, frames only) ONLY when the user explicitly consents in the breadbug flow; never automatically. - Suspend/resume and monitor hotplug must not trigger false freezes (use the existing power/monitor events to pause the watchdog). ## Acceptance - Induced main-loop block in a test app triggers a freeze marker and notification. - Suspend/resume test produces no false positives.
Author
Owner

Independent changes_requested at exact head c4e0277713. Complete mapping, six file/line repair items, commands, real probes and per-job exact-head CI evidence persisted: #24 (comment) . Findings BREAD-46. Real daemon emits excluded clip freeze; actual CLI/eu-stack captures unrelated Python comm=bread-unrelated even after a recovered marker and serializes absolute PCs without module/build-id. Dead frozen instances stay tracked and new PID is reported recovered. Repair capture identity/state/frame metadata, exclusions/lifecycle and broad identity-redaction bypasses on existing PR24; preserve feature/breadbug-watchdog for PR25. 51+13+29 library tests,11 watchdog,3 journal,2 real-daemon IPC tests/check-docs pass. Run146 jobcheck2479 and all individual steps green at exact head. Base advanced to e4ddf6c with PR43 privacy spec, so integrate/reconcile current main before fresh review. PR68 actually merged94ea374; full blocked-main-loop -> marker+notification, provider pilot/rollout, BOS-86 and actual suspend/resume/hotplug acceptance remain open. All original requirements retained; no merge/tag/done.

Independent changes_requested at exact head c4e02777139a3cd28cdc999c325e6c0756769f58. Complete mapping, six file/line repair items, commands, real probes and per-job exact-head CI evidence persisted: https://git.breadway.dev/Breadway/bread/pulls/24#issuecomment-4885 . Findings BREAD-46. Real daemon emits excluded clip freeze; actual CLI/eu-stack captures unrelated Python comm=bread-unrelated even after a recovered marker and serializes absolute PCs without module/build-id. Dead frozen instances stay tracked and new PID is reported recovered. Repair capture identity/state/frame metadata, exclusions/lifecycle and broad identity-redaction bypasses on existing PR24; preserve feature/breadbug-watchdog for PR25. 51+13+29 library tests,11 watchdog,3 journal,2 real-daemon IPC tests/check-docs pass. Run146 jobcheck2479 and all individual steps green at exact head. Base advanced to e4ddf6c with PR43 privacy spec, so integrate/reconcile current main before fresh review. PR68 actually merged94ea374; full blocked-main-loop -> marker+notification, provider pilot/rollout, BOS-86 and actual suspend/resume/hotplug acceptance remain open. All original requirements retained; no merge/tag/done.
Author
Owner

Independent exact-head review of PR24 at bbb12d6383 is BLOCKED, not passed: #24 (comment). The seven BREAD-46 defects verified repaired by code read plus real daemon, real eu-stack and targeted suites. Required check 'check / check (push)' (run 157) is FAILURE on this exact head with compile errors in the test step that look like runner cross-contamination (symbols that exist at the head reported missing; a concurrent run 158 held the shared build lock). Needs a clean rerun on the same SHA. PR body changed from a closing reference to 'Part of BREAD-18'. BREAD-18 stays open: blocked-GUI induction, real suspend/resume and hotplug, provider adoption, BOS-86 notification action, and bread.app.* vs bread.bug.* naming reconciliation. No merge, tag or done. Branch feature/breadbug-watchdog kept for PR25.

Independent exact-head review of PR24 at bbb12d6383e3daf1936428a36fbe0c15d111136f is BLOCKED, not passed: https://git.breadway.dev/Breadway/bread/pulls/24#issuecomment-5169. The seven BREAD-46 defects verified repaired by code read plus real daemon, real eu-stack and targeted suites. Required check 'check / check (push)' (run 157) is FAILURE on this exact head with compile errors in the test step that look like runner cross-contamination (symbols that exist at the head reported missing; a concurrent run 158 held the shared build lock). Needs a clean rerun on the same SHA. PR body changed from a closing reference to 'Part of BREAD-18'. BREAD-18 stays open: blocked-GUI induction, real suspend/resume and hotplug, provider adoption, BOS-86 notification action, and bread.app.* vs bread.bug.* naming reconciliation. No merge, tag or done. Branch feature/breadbug-watchdog kept for PR25.
Author
Owner

Independent exact-head delta review of PR24 at e018807d6b is BLOCKED, not passed: #24 (comment). Delta from reviewed bbb12d6 is only main's two CI workflow files (byte-identical to main d7add44..40ee494), so privacy, readiness and watchdog code and the seven BREAD-46 passes are unchanged. Required check 'check / check (push)' run 163 is FAILURE on this exact head: test step compile errors for symbols that exist at the head, same pattern as run 157, likely the shared cargo target cache across overlapping jobs (hypothesis, unproven). Needs a green rerun on this SHA, or a reviewed merge of main d5748d6 with green CI. PR body stays 'Part of BREAD-18'. BREAD-18 gates unchanged and unwaived: blocked-GUI induction, real suspend/resume and hotplug, provider adoption, BOS-86 notification action, event naming reconciliation. No merge, tag or done. Branch feature/breadbug-watchdog kept for PR25.

Independent exact-head delta review of PR24 at e018807d6b70707b58a245abf4ce7adcda445e8c is BLOCKED, not passed: https://git.breadway.dev/Breadway/bread/pulls/24#issuecomment-5371. Delta from reviewed bbb12d6 is only main's two CI workflow files (byte-identical to main d7add44..40ee494), so privacy, readiness and watchdog code and the seven BREAD-46 passes are unchanged. Required check 'check / check (push)' run 163 is FAILURE on this exact head: test step compile errors for symbols that exist at the head, same pattern as run 157, likely the shared cargo target cache across overlapping jobs (hypothesis, unproven). Needs a green rerun on this SHA, or a reviewed merge of main d5748d6 with green CI. PR body stays 'Part of BREAD-18'. BREAD-18 gates unchanged and unwaived: blocked-GUI induction, real suspend/resume and hotplug, provider adoption, BOS-86 notification action, event naming reconciliation. No merge, tag or done. Branch feature/breadbug-watchdog kept for PR25.
Author
Owner

PR24 prerequisite slice independently PASSed at exact head 834c07e5ed: #24 (comment) . Criterion mapping and delta proof persisted there. Main-only automatic merge verified, watchdog/privacy byte-identical to e018807. Local 312 daemon/integration plus137 privacy/CLI tests passed (including4 real eu-stack, no skips), fmt, workspace all-target clippy -D warnings, check-docs passed. Exact-head CI run166 job2606 and all steps success. Fresh head/base/formal reviews/mergeability gate passed; normal merge used documented head_commit_id condition, no force or branch deletion. Confirmed merged=true, merge SHA90cd7b4b6fae1b4dd99e2f9830d42a05e7f3c036; parents d5748d6093 and834c07e5ed8c6a498cab98a2c646bccb9f057e3a. Branch feature/breadbug-watchdog confirmed retained for PR25. PR body already Part of BREAD-18, unchanged. Full card NOT passed or done: blocked GUI marker+notification, actual suspend/resume/hotplug, provider adoption/rollout and stop PID compatibility, BOS-86 integrated notification/action, event naming reconciliation remain open. Awaiting coordinator integrated verification; set/keep review. Known breadboard sync may retain old closes linkage despite Part-of text, so any auto-done is not verification.

PR24 prerequisite slice independently PASSed at exact head 834c07e5ed8c6a498cab98a2c646bccb9f057e3a: https://git.breadway.dev/Breadway/bread/pulls/24#issuecomment-5389 . Criterion mapping and delta proof persisted there. Main-only automatic merge verified, watchdog/privacy byte-identical to e018807. Local 312 daemon/integration plus137 privacy/CLI tests passed (including4 real eu-stack, no skips), fmt, workspace all-target clippy -D warnings, check-docs passed. Exact-head CI run166 job2606 and all steps success. Fresh head/base/formal reviews/mergeability gate passed; normal merge used documented head_commit_id condition, no force or branch deletion. Confirmed merged=true, merge SHA90cd7b4b6fae1b4dd99e2f9830d42a05e7f3c036; parents d5748d6093def1c4863bd496c86af0be31f8c6f1 and834c07e5ed8c6a498cab98a2c646bccb9f057e3a. Branch feature/breadbug-watchdog confirmed retained for PR25. PR body already Part of BREAD-18, unchanged. Full card NOT passed or done: blocked GUI marker+notification, actual suspend/resume/hotplug, provider adoption/rollout and stop PID compatibility, BOS-86 integrated notification/action, event naming reconciliation remain open. Awaiting coordinator integrated verification; set/keep review. Known breadboard sync may retain old closes linkage despite Part-of text, so any auto-done is not verification.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Breadway/bread#17
No description provided.