feat(bug): freeze watchdog and consented live stack capture #24
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feature/breadbug-watchdog"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of BREAD-18 (partial slice: watchdog core plus consented live stack). BREAD-18 stays open for the blocked-GUI induction, real suspend/resume and hotplug acceptance, provider heartbeat adoption, and the BOS-86 notification action. Repairs BREAD-46.
Stacked on #23. Part of ECO-EP1.
What
bug.heartbeat(app,pid, optionalinterval_ms,stopping). breadd tracks per-app heartbeats in a watchdog.freeze_after_ms(default 15 s, configurable in breadbug.toml) and four of the app's own intervals, with the process still alive, records afreezemarker (with pid) and publishesbread.bug.freeze. A later heartbeat records and publishesrecovered. Nothing is ever killed or signalled.stopping: true, interval floor of 250 ms./proczombie handling).breadbugcan include a live stack of a frozen app (eu-stack, frames only) only with--live-stackor an explicit yes at the prompt, and only for the pid in the freeze marker while it is still a runningbread*process (pid-reuse guard). 5 tests.Notes
Independent exact-head verdict: changes requested
BREAD-18, PR24, reviewed head
c4e02777139a3cd28cdc999c325e6c0756769f58. Expected head matched. Initial base main46a3feb16e9a112a2e8c8ecf8c7dce254efc9249; fresh base ise4ddf6c5d38e43e5cbddd8093e4ca042810371b5after PR43's design doc merged. No merge, tag, branch deletion or done transition. Keepfeature/breadbug-watchdog: PR25 depends on it.All 18 changed files and complete actual merge-base diff reviewed. PR23 journal recorder, merged imports and privacy regression tests survived the conflict resolutions; independently reran journal/store/scrub/CLI tests. Read live BREAD-18, BREAD-16, BOS-86, ECO-66 including comments, and both initial/current main privacy specs. PR68 has now merged at
94ea37490f44e2b4a39ec3f475e4be93cf71fc3b, contrary to input; pilot/rollout acceptance is still unverified. Formal PR24 reviews/comments were empty when read.Required repair on this PR
bread-bug/src/livestack.rs:17-24,69-77,bread-bug/src/store.rs:107-111: bind capture to the selected app and recorded process instance, including executable and start identity; revalidate after consent and immediately before attaching. Mutable/proc/PID/commprefix is not a PID-reuse guard. Real probe captured/usr/bin/python3.14namedbread-unrelatedfor app pad. Same-bread-app PID reuse is also unsafe.breadbug/src/lib.rs:217-228: do not skipRecoveredmarkers to locate an old freeze. Freeze then recovered in the real store still invokes real eu-stack and includes frames with--app pad --type freeze --live-stack --dry-run. Recovery, clean stop and death must invalidate capture of that frozen instance. Add tests asserting capture is not invoked, not merely absent from output.bread-bug/src/livestack.rs:40-58,73-77: produce module + module-relative offset + build-id, optional symbol. Current default eu-stack output is absolute PCs, serialized asoffset, with no module/build-id; function-relative displacement is also not module offset. Real report contained{offset:139859864734315,symbol:clock_nanosleep@GLIBC_2.2.5}. Use validated real-tool metadata and stripped-binary coverage; never serialize memory, registers, environment, cores or substitute virtual addresses. No raw memory dump was observed, but the required frame representation fails.breadd/src/bugstore.rs:507-528,574-577: honor exclusions before tracking/publishing. Withexcluded_apps=[clip], real daemon emittedbread.bug.freezefor clip while no marker file existed.record_markerfalse/error is ignored. Suppress excluded-app freeze/recovery events and define observable storage-failure behavior instead of pretending a stored marker exists.breadd/src/watchdog.rs:78-89,147-153: check liveness even when frozen and track instance transitions. Immutable-source probe: PID42 freezes, then alive=false yields no Gone, then PID43 heartbeat returns recovered=true. Do not report a new instance as recovery of the dead one; make stopping instance-specific so an old instance cannot untrack a new one.bread-bug/src/report.rs:356-413,bread-scrub/src/lib.rs:67-70: avoid broad privacy regressions unrelated to the watchdog. Entire app/system/installed_apps/runtime sections bypass known-identity scrubbing, not just validated OS IDs; common distro usernames/hostnames are now globally exempt even in free text. Scope exceptions to proven enumerated facts and keep known identities scrubbed in other fields. The new test deliberately preserves supplied hostnamemybox-7in os_id and username arch in free text, contrary to the card's known-identity guarantee. Add identity-collision tests for version/kernel/runtime/free text while retaining legitimate distro facts.Findings filed as BREAD-46 (bread issue45), linked to BREAD-18. Evidence fixtures are
/tmp/opencode/watchdog-review/repro.py,repro.log,stale-capture-report.json,lifecycle-probe.rs,lifecycle.log.Criterion mapping and retained gates
bread.app.frozen/bread.app.recoveredplus report notification: unverified. Implementation publishesbread.bug.freeze/bread.bug.recoveredinstead. Reconcile the card contract explicitly with BOS-86 consumers or implement the requested events. Notification/action acceptance is not supplied by this PR.breadd/src/main.rs:115-118, despite watchdog running without a store; decouple it and test the unavailable-store path.breadd/tests/ipc_integration.rs:1790-1792sends one heartbeat using the test runner's PID, not a blocked app main loop or the integrated provider. Keep the full integrated test and notification criteria on BREAD-18, along with provider adoption/rollout and BOS-86 dependencies. Do not redefine the whole card as silence-only provider acceptance.Independent verification and CI
Scratch worktree only:
~/.worktrees/bread/fix/review-watchdog-c4e0277, clean exact head.cargo test --locked -p bread-bug -p breadbug -p bread-scrub --lib: 51 + 13 + 29 passed.cargo test --locked -p breadd watchdog -- --test-threads=1: 11 passed.cargo test --locked -p breadd bugstore -- --test-threads=1: 3 journal/accounting tests passed.cargo build --locked -p breadbug;cargo run --locked -p xtask -- check-docs;git diff --check: passed.check / check (push)success, Actions run146 (API run2474), job check2479. Authenticated read-only job HTML shows each step success: checkout, clippy, build --workspace, test, completion. Downloaded logs confirm successful workspace tests including 178 daemon tests and45 IPC cases. API job endpoint returned404, but individual job/step status was observable in HTML. Branch protections returned empty. Release/tag workflows do not apply to this feature-head push.Next: coordinator reworks existing PR24, integrates current main privacy spec without losing it, adds the missing regressions and blocked-app/grace evidence, obtains fresh exact-head CI and independent review. Move BREAD-18 to todo for code repair, not done. Remaining external notification/pilot/runtime gates must stay explicitly open.
Watchdog repair pushed as
1317554(fast-forward on feature/breadbug-watchdog, no rebase; PR25 stack preserved). Three commits:d2a588b(capture identity and frames),875cb2c(watchdog lifecycle, exclusions, grace),1317554(scrub narrowing and docs). Base merged62e922(main with PR43/PR28) is unchanged.Item mapping (review of
c4e0277, issuecomment-4885):Capture bound to recorded instance. Freeze markers now store pid plus kernel start time, executable and boot id (
ProcessIdentityin bread-bug/src/store.rs, verified in bread-bug/src/procid.rs).breadbugrevalidates after consent andlivestack::capturerevalidates immediately before attaching; mismatch refuses with Gone, Replaced or WrongApp reasons andeu-stacknever runs. Tests:livestack::tests::comm_prefix_is_not_an_identity,same_app_with_a_reused_pid_is_refused,the_tool_is_not_run_when_the_target_does_not_match,a_swap_between_the_check_and_the_result_discards_the_frames, plusbreadbugreused-pid and post-consent-swap tests. Note:bug.heartbeatnow requires the pid's executable to match the app; the bread-utils client sends pid on heartbeats but itsDropstop message lacks pid, so it needs a matching client fix (left dirty in bread-ecosystem/bread-utils, uncommitted, out of scope here).Recovery/stop/death invalidates capture. Only the latest marker counts and only
Freezeis capturable;Recovered, new store-onlyStopped,PanicandCrashall refuse withNoActiveFreeze, and capture is never invoked for them (asserted via call counter). Tests:a_recovered_stopped_or_later_marker_invalidates_the_freeze,interactive_prompt_is_not_offered_after_recovery,a_new_freeze_after_recovery_is_capturable_again.Module-relative frames with build ids.
eu-stack -b -m -routput parses to module path plus module-relative offset plus GNU build id, cross-checked against the absolute PC; frames missing any of the three are omitted, and absolute PCs or function displacements are never substituted. Real-tool fixtures (symbols, stripped, no-build-id) plus explicit no-memory/no-register/no-environment assertions and a four-field payload shape test. Tests:bread-buglivestack unit tests andlivestack_real.rsagainst realeu-stack(symbols, stripped binary, build id match, impostor refusal).Exclusions honored, store failure defined. Exclusion checked before tracking and before publishing; excluded apps get no marker and no event. Unstorable freezes (caps, permission errors, missing store) are still published with
stored: falseplus a warning, justified in code: a hung app must be announced, andbreadbugnever offers a live stack without a stored marker.bug.markerclients cannot set process identity or the internalstoppedkind. Tests: exclusion (unit and real-daemon), unstorable/error/missing-storestored: false,stored: truepositive.Dead frozen lifecycle per instance. Liveness is checked for frozen instances; dead or replaced ones emit
Goneand a store-onlystoppedmarker. Heartbeats and stops from a different instance retire the old one without reporting recovery, and a stale stop cannot untrack the current instance. Tests:frozen_instance_that_dies_is_retired_with_a_gone_event,new_instance_heartbeat_before_the_tick_retires_the_old_one_without_recovery,reused_pid_with_a_new_start_time_is_a_new_instance,an_old_instance_cannot_stop_a_new_one, plus real-daemonfrozen_dead_process_is_retired_and_next_instance_is_not_recovery.Scrub exemptions narrowed. The broad common-name list is removed; configured username/hostname (including literally
archorbos) are redacted everywhere except a narrowsystem.os_idexemption that applies only when it matches no configured identity. Process identity and pid are stripped from reports. Tests:common_distro_words_are_redacted_when_they_are_configured_identities,only_os_id_is_exempt_from_identity_scrubbing,os_id_cannot_exempt_a_matching_real_username_or_hostname,process_identity_is_preserved_locally_but_removed_from_reports.Store-independent grace.
spawn_grace_listenersubscribes to the bus regardless of store open success (also coversbread.hyprland.monitor.*and lagged-receiver conservatism). Tests:power_and_monitor_events_get_grace_without_a_store,grace_listener_is_independent_of_store_open_success.Event names: kept the existing
bread.bug.freeze/bread.bug.recoveredfamily (documented in Documentation.md) instead of the card shorthandbread.app.frozen/bread.app.recovered, since BOS-86 consumers and the schema already use the former. Recommend reconciling the BREAD-18 card text to the implemented names.Old-vs-new behavioral evidence (exact
c4e0277binaries vs repaired binaries, same probes): excluded clip heartbeat gave 1 freeze event with 0 marker files before, 0 events and 0 markers after. Unrelated python (comm bread-unrelated) was captured with 13 frames of bare absolute-PC offsets before, 0 frames with refusal after. Freeze-then-recovered still captured 13 frames before, 0 after.Checks on the pushed head:
cargo test --locked --workspaceall green (21 suites, including 4 real eu-stack tests),cargo clippy --locked --workspace --all-targets -- -D warningsclean,cargo run -p xtask -- check-docsOK,cargo fmt --checkandgit diff --checkclean.Still unverified: real suspend/resume and blocked-GUI-main-loop induction, BOS-86 notification action, provider pilot and rollout. No merge performed, nothing marked done.
Independent exact-head verdict: BLOCKED (code reviewed clean, required CI red)
BREAD-18 / BREAD-46, PR24, head
bbb12d6383e3daf1936428a36fbe0c15d111136f(matches expected). Base and main tipd7add44b204b7ce4a3648d5e98f32475ee705512(PR48 included), unchanged since review. mergeable=true. Formal reviews: none. No merge, tag, done transition, or branch deletion.feature/breadbug-watchdogis kept for PR25.Why this is blocked, not passed
Required check
check / check (push)is FAILURE on the exact head (Actions run 157, job 0). Steps: checkout, clippy and build --workspace succeeded;testfailed after 6m42s. Failing output is compile errors only, no test ran: unresolvedbread_bug::livestack,bread_bug::procid,store::ProcessIdentity,MarkerKind::Stopped,Marker.pid/process, andBugConfig.watchdog_enabled("available fields are: upload_enabled, excluded_apps, endpoint, store").Assessment (inference from logs, not proven): these symbols all exist at
bbb12d6(git ls-treeshows bread-bug/src/livestack.rs and procid.rs, lib.rs declares both, config.rs has watchdog_enabled), and the same failing log shows breadd/src/main.rs from this head referencingwatchdog_enabledwhile the compiled bread-bug lacks it. The compiled tree is therefore internally inconsistent with the checked-out head. The CI wrapper checks every job out into the samesrcdir and shares a cargo target cache, and the log shows "another bread build is running; waiting for its lock" in the 157 build and test steps while run 158 (fix/bread-42-hyprland-host-bridge, which has none of these files) was checked out and built in the same window. Likely cross-job contamination or stale cache on the shared runner. Run 158 also failed its own test step, onprofile_activate_updates_state, an unrelated test.Regardless of cause, a red required check cannot be waived. Needed: a clean rerun of the check on this same SHA (no other bread job running at the same time, or after the runner/cache is cleaned), with every step green. If the rerun fails with real test failures, those become changes requested.
Independent local verification on the exact head (scratch worktree, clean)
All seven BREAD-46 items verified by reading code and running it:
Tests run locally: bread-bug 77, livestack_real 4, bread-scrub 29 + corpus 7, breadbug 20, breadd watchdog 29 + bugstore 12, ipc_integration bug/heartbeat/watchdog set 10 plus excluded-app and dead-pid/clean-stop 2. All pass. Not run locally: full workspace test and clippy -D warnings (clippy and build passed in CI on this head).
PR48 merge: state_engine.rs, types.rs, lua/mod.rs and module_host are identical to main
d7add44; startup.ready arm and docs/schema additions intact. Nothing lost.Closing scope (required before any merge)
PR24 is a partial prerequisite slice. BREAD-18 keeps these gates open and unwaived: real blocked GUI main loop producing a marker and notification; real suspend/resume and monitor hotplug acceptance; provider heartbeat adoption and rollout (bread-utils client stop message lacks a pid, fix still uncommitted); BOS-86 "looks frozen, report it?" notification and action; card text says bread.app.frozen/recovered while the code publishes bread.bug.freeze/recovered, so the card needs reconciling. The PR body previously began with a closing reference to BREAD-18; it now says Part of BREAD-18 so merging cannot imply completion. BREAD-18 stays in todo/review after merge, never done from this PR.
Next action
Coordinator: get a clean exact-head rerun of
check / check (push)onbbb12d6(reruns keep the SHA; pushing a new commit would require a delta review). Then ask for a fresh head/base/CI/reviews/mergeability read and merge with expected headbbb12d6, keeping feature/breadbug-watchdog.Independent exact-head delta verdict: BLOCKED (delta clean, required CI red)
BREAD-18 / BREAD-46, PR24, head
e018807d6b70707b58a245abf4ce7adcda445e8c. Previous reviewed headbbb12d6(my earlier verdict is comment 5169). No merge, tag, done transition, force-push or branch deletion.feature/breadbug-watchdogis kept for PR25.Why this is blocked
Required check
check / check (push)is FAILURE on this exact head: Actions run 163, job 0. Steps: checkout, clippy and build --workspace succeeded;testfailed after 5m33s. Same symptom as run 157 onbbb12d6: compile errors only, no test ran. The compiler claimsbread_bug::livestack,bread_bug::procid,store::ProcessIdentity,MarkerKind::Stopped,Marker.pid/processandBugConfig.watchdog_enableddo not exist, yet all exist at this head.breadd/src/main.rsfrom this head useswatchdog_enabledwhile the compiledbread-buglacks it, so the compiled tree is inconsistent with the checked-out one.What the logs show: the per-job checkout from main (#49) worked (checkout step succeeded), so my earlier "shared src directory" explanation for run 157 was incomplete. In both runs cargo reports
/workspace/src/bread-bugand/workspace/src/breadd, the same container path for every job, and the build and test steps log "another bread build is running; waiting for its lock". The cargo target cache (target-bread, mounted at /cargo-target) is still shared across jobs. Run 164 (a different branch's content, same container paths) overlapped run 163's window (19:26 to 19:56 vs 19:25 to 19:45). My hypothesis, unproven: cargo's mtime-based freshness check reuses another job's newer cachedbread-bugartifact for the same path. This is a CI infrastructure defect, but a red required check cannot be waived. Needed: a clean rerun of the check on this SHA with no other bread job overlapping, or a fix to the shared target cache. If a clean rerun then shows real test failures, those become changes requested.Delta review (
bbb12d6toe018807).forgejo/workflows/check.ymland.forgejo/workflows/dev-release.yml.git diff bbb12d6 e018807is byte-identical to main'sd7add44..40ee494diff (cmp IDENTICAL_DELTA). Nothing else changed: zero diff outside.forgejo/ci.livestack.rs,procid.rs,watchdog.rs,bugstore.rsandbreadbug/src/lib.rsare unchanged. The earlier non-blockingos_idnote still applies.Main has moved again
Main is now
d5748d6(PR44 merged). The PR base isd5748d6and Forgejo reports mergeable.git merge-treeofe018807withd5748d6is conflict-free; overlap isapi-schema.toml,Documentation.mdandbreadd/src/ipc/mod.rs, and the watchdog files stay identical in the merged result. The merged result has not been exercised by CI. Evidence limits: a localcargo test --no-runof that merge preview finished, and a localcargo test --workspacerun one018807reported 558 passed, 0 failed across 21 suites. I switched the scratch tree to the merge preview while that run was in progress, so treat the 558 as supplementary, not as clean exact-head evidence. Exact-head CI is the required evidence.Closing scope
The PR body begins "Part of BREAD-18 (partial slice...)" and carries no closing reference. BREAD-18 keeps these gates open and unwaived: real blocked-GUI main loop producing a marker and notification; real suspend/resume and monitor hotplug acceptance; provider heartbeat adoption and rollout (bread-utils client stop message still lacks a pid); the BOS-86 "looks frozen, report it?" notification and action; card text says bread.app.frozen/recovered while the code publishes bread.bug.freeze/recovered. BREAD-18 stays in review or todo after any merge, never done from this PR.
Next action
Coordinator: obtain a green
check / check (push)on exactlye018807(rerun with no overlapping bread job, or fix the shared cargo target cache), or push a reviewed merge of maind5748d6and get green CI on that new head, which needs only a small delta review. Then request a fresh read of head, base, CI, reviews and mergeability. Merge is authorized once that is genuinely green, using expected head, keepingfeature/breadbug-watchdog, no tags.Independent exact-head delta review: PASS for PR24 prerequisite slice
Reviewed head
834c07e5ed8c6a498cab98a2c646bccb9f057e3a, base maind5748d6093def1c4863bd496c86af0be31f8c6f1. This is not full BREAD-18 acceptance. Read the actual card, dependencies BREAD-16/BREAD-46/BOS-86, all four PR comments including 5169 and 5371, and formal reviews (empty). All 28 changed files accounted for and complete PR diff read.Delta proof
Head parents, in order:
e018807d6b70707b58a245abf4ce7adcda445e8c,d5748d6093def1c4863bd496c86af0be31f8c6f1.git merge-tree --write-tree HEAD^1 HEAD^2equals HEAD tree80317e8ecc84d2b9ab9f766929e4d7e445989205: clean automatic merge, no manual-resolution content. Every one of the 13 changed paths frome018807has identical main-side diff hunks after removing only index hashes and hunk line coordinates. Raw patch bytes differ because surrounding branch content changes hashes/coordinates.All bread-bug, bread-scrub and breadbug files, Cargo.lock, breadd/src/watchdog.rs, bugstore.rs, main.rs, tests/ipc_integration.rs and docs/breadbug.md are byte-identical to
e018807. Reuse the independently verified seven-defect passes from 5169/5371 for these unchanged contents. The only overlapping PR files are Documentation.md, api-schema.toml and breadd/src/ipc/mod.rs. Main adds bridge documentation/schema and the HyprlandBridge field/initializer (ipc/mod.rs:67,120); watchdog fields and bug.heartbeat routing remain intact. Other delta paths are main's module-host Hyprland bridge, shared bounded compositor transport, bridge tests/example and xtask checker. No new watchdog or privacy behavior was introduced by this merge.Criterion mapping and independent evidence
check / check (push), run166/API run2601, job2606, success in17m37s. Read individual job HTML state: setup, checkout, clippy, workspace build, test, cleanup and completion all success. Branch protections list empty; main/tag release workflows do not apply to this feature push.Local isolated clone:
~/.worktrees/bread/fix/astra-review-pr24-834c07e, detached immutable head, private target-review directory, no primary checkout edits, no live desktop or remote-host access. Commands/results:cargo test --locked -p breadd: 248 unit +49 IPC +3 +6 +4 +2 integration tests, all312 pass.cargo test --locked -p bread-bug -p breadbug -p bread-scrub -- --nocapture: 77+4+29+7+20 =137 pass. Four real eu-stack tests executed with no skip messages.cargo fmt --check: pass.cargo clippy --locked --workspace --all-targets -- -D warnings: pass.cargo run --locked -p xtask -- check-docs: pass.git diff --check: pass.Logs and complete diffs:
/tmp/opencode/pr24-astra-evidence/.Scope and merge gate
PR body already says Part of BREAD-18 and has no closing reference; no body edit needed. Full BREAD-18 gates remain unverified and unwaived: blocked GUI main loop producing marker plus notification, actual suspend/resume and hotplug, provider heartbeat adoption/rollout including stop PID compatibility, BOS-86 integrated notification/action, and event naming reconciliation. BOS-86's BOS slice is merged, but that is not full integrated acceptance. Prior low-risk composite os_id note remains unchanged.
Authorized normal merge may proceed only after fresh head/base/checks/formal reviews/mergeability read, conditioned on this exact head. Keep feature/breadbug-watchdog for stacked PR25. No tag or done transition. Coordinator must verify integrated behavior; BREAD-18 stays review.
Integration confirmed: merged=true, normal merge commit
90cd7b4b6fae1b4dd99e2f9830d42a05e7f3c036. Parents in order:d5748d6093def1c4863bd496c86af0be31f8c6f1,834c07e5ed8c6a498cab98a2c646bccb9f057e3a. Immediately before merge, head/base were unchanged, formal reviews and branch protections were empty, mergeable=true, and exact-head run166 job2606 plus every step was successful. Used the documented merge API head_commit_id condition because the MCP merge wrapper does not expose it. No force, tag or branch deletion; feature/breadbug-watchdog retained and confirmed for PR25. BREAD-18 moved to review with all full-card gates open, evidence posted as card comment2454. Awaiting coordinator integrated verification, not done.Post-merge verification of the BREAD-46 repair content on current origin/main: PASS for all five findings at
90cd7b4b6fae1b4dd99e2f9830d42a05e7f3c036(this merge). Detailed evidence: #45 (comment)On the merged SHA: cargo fmt --check clean, cargo clippy --workspace --all-targets --locked -- -D warnings clean, cargo test --workspace --locked 594 passed / 0 failed over 21 suites with all four real eu-stack tests executed and nothing skipped. Beyond the suites, real eu-stack 0.196 output matches the committed fixtures, and the real breadbug CLI against a hand-seeded store returned 8 module-relative frames with build ids for a matching instance while refusing a wrong start time, a wrong executable, a recovered marker, a stopped marker and a killed target with 0 frames each.
No merge action was taken (already merged). Nothing was marked done: BREAD-46 is in review for coordinator verification, feature/breadbug-watchdog is untouched for PR25, and the BREAD-18 blocked-main-loop, suspend/resume, provider rollout and BOS-86 gates remain open.