breadbug: allowlist-based scrubber library with PII test corpus #14

Closed
opened 2026-10-08 20:14:11 +08:00 by Breadway · 7 comments
Owner

Part of ECO-EP1 (see ECO-66). This is the make-or-break piece; everything else depends on it.

Shared crate used both on ingest into the breadd store and at report time (two-pass).

Requirements

  • Allowlist model: structured fields are kept only if explicitly allowed. Free-form text (log lines, panic messages, user notes) goes through pattern redaction.
  • Redact: $HOME and username in paths, hostname, IPv4/IPv6, MACs, emails, tokens/keys (high-entropy strings, common prefixes), serials/UUIDs, SSIDs, window titles, URLs with credentials or query strings.
  • Event payloads dropped by default; per-event allowlist for safe fields.
  • Deterministic and idempotent (scrub(scrub(x)) == scrub(x)).
  • Redaction markers keep structure useful for debugging (e.g. <HOME>/.config/.., <IP>), and stable within a report so distinct values stay distinguishable (<IP:1>, <IP:2>).
  • Report-time pass also scrubs user notes.

Tests

  • Golden corpus of realistic logs, backtraces and events seeded with fake PII; test fails if any seeded value survives.
  • Fuzz test for panics and non-UTF8 input.
  • Property test for idempotence.
Part of ECO-EP1 (see ECO-66). This is the make-or-break piece; everything else depends on it. Shared crate used both on ingest into the breadd store and at report time (two-pass). ## Requirements - Allowlist model: structured fields are kept only if explicitly allowed. Free-form text (log lines, panic messages, user notes) goes through pattern redaction. - Redact: $HOME and username in paths, hostname, IPv4/IPv6, MACs, emails, tokens/keys (high-entropy strings, common prefixes), serials/UUIDs, SSIDs, window titles, URLs with credentials or query strings. - Event payloads dropped by default; per-event allowlist for safe fields. - Deterministic and idempotent (scrub(scrub(x)) == scrub(x)). - Redaction markers keep structure useful for debugging (e.g. `<HOME>/.config/..`, `<IP>`), and stable within a report so distinct values stay distinguishable (`<IP:1>`, `<IP:2>`). - Report-time pass also scrubs user notes. ## Tests - Golden corpus of realistic logs, backtraces and events seeded with fake PII; test fails if any seeded value survives. - Fuzz test for panics and non-UTF8 input. - Property test for idempotence.
Author
Owner

Independent review: changes_requested at exact head 4dec9ea9f5. Full criterion mapping, file/line checklist, commands and reproduced outputs: #21 (comment)

Acceptance failures: default-dropped event keys leak known identity/token data; generic uppercase marker bypass preserves AWS keys; valid alphabetic/punctuated IPv6 leaks; token-first URL processing leaves credential usernames; known short hostname skipped; ingest markers collide with distinct report-note values. Findings filed as BREAD-37 (bread issue #36), fix on existing PR #21. Existing 25 unit + 7 corpus/fuzz tests, targeted locked clippy, crate fmt and check-docs pass; exact-head check / check (push) success at Actions run 109. Workspace fmt fails in 18 unchanged files. Actual report-time notes/store integration is not present at this SHA and remains unverified, with consumers in open PRs #22/#23.

Move to todo for coordinator rework, preserve PR/branch and stacked dependents. Add seeded regressions and obtain fresh SHA-bound independent review. No merge authorized or performed; no done transition.

Independent review: changes_requested at exact head 4dec9ea9f52486d7d11fc660e60f2d35665056d2. Full criterion mapping, file/line checklist, commands and reproduced outputs: https://git.breadway.dev/Breadway/bread/pulls/21#issuecomment-4248 Acceptance failures: default-dropped event keys leak known identity/token data; generic uppercase marker bypass preserves AWS keys; valid alphabetic/punctuated IPv6 leaks; token-first URL processing leaves credential usernames; known short hostname skipped; ingest markers collide with distinct report-note values. Findings filed as BREAD-37 (bread issue #36), fix on existing PR #21. Existing 25 unit + 7 corpus/fuzz tests, targeted locked clippy, crate fmt and check-docs pass; exact-head check / check (push) success at Actions run 109. Workspace fmt fails in 18 unchanged files. Actual report-time notes/store integration is not present at this SHA and remains unverified, with consumers in open PRs #22/#23. Move to todo for coordinator rework, preserve PR/branch and stacked dependents. Add seeded regressions and obtain fresh SHA-bound independent review. No merge authorized or performed; no done transition.
Author
Owner

Independent exact-SHA re-review: changes_requested at 3616fcf9cc. Full criterion mapping and actionable file/line repair checklist: #21 (comment). Confidence high: compiled immutable-crate probes reproduce credential/query leaks when generated markers occur inside URLs. https://bob:@example.com/x?account=alice remains unchanged; a query containing IP:1 retains &owner=alice. This continues BREAD-37 and prior rejection 4248, no duplicate bug card needed. Existing original raw-token URL, IPv6, AWS-marker, dropped-key and short-host regressions pass; whole-report pre-reservation and bounded depth pass. 27 unit + 7 corpus/fuzz tests, crate clippy/fmt/docs pass. Multiple-ingest-scope numbering and real two-pass integration remain unverified; open PRs #22/#23 are not integrated evidence. Required exact-head CI run124 pending. Move todo for coordinator repairs on existing PR21, preserve all stacked branches. No merge or done transition. Durable session updated reworking; fresh SHA-bound review and CI required after repairs.

Independent exact-SHA re-review: changes_requested at 3616fcf9cc21914cc9cfec8f6d6a25adefe2d869. Full criterion mapping and actionable file/line repair checklist: https://git.breadway.dev/Breadway/bread/pulls/21#issuecomment-4284. Confidence high: compiled immutable-crate probes reproduce credential/query leaks when generated markers occur inside URLs. https://bob:<SECRET>@example.com/x?account=alice remains unchanged; a query containing <IP:1> retains &owner=alice. This continues BREAD-37 and prior rejection 4248, no duplicate bug card needed. Existing original raw-token URL, IPv6, AWS-marker, dropped-key and short-host regressions pass; whole-report pre-reservation and bounded depth pass. 27 unit + 7 corpus/fuzz tests, crate clippy/fmt/docs pass. Multiple-ingest-scope numbering and real two-pass integration remain unverified; open PRs #22/#23 are not integrated evidence. Required exact-head CI run124 pending. Move todo for coordinator repairs on existing PR21, preserve all stacked branches. No merge or done transition. Durable session updated reworking; fresh SHA-bound review and CI required after repairs.
Author
Owner

Independent blocked at39ff3130583485dada208fe3c470954cacb56317. Full evidence #21 (comment). URL repairs and377 workspace tests/clippy/fmt/docs pass. Clean report9117355 passes42 tests including restarted logsIP1/IP2 and notesIP3 plus complete URL redaction. However actual consumers remain unmerged/unaccepted, not integrated feature acceptance; required CI129 pending. PR22 also has reproduced strict-cap defect BREAD-40 and missing journal ingest. Leave review, no merge/retarget or done. Coordinator must reconcile prerequisite/full-issue integration scope without waiving criteria, repair consumers and obtain fresh SHA review.

Independent blocked at39ff3130583485dada208fe3c470954cacb56317. Full evidence https://git.breadway.dev/Breadway/bread/pulls/21#issuecomment-4307. URL repairs and377 workspace tests/clippy/fmt/docs pass. Clean report9117355 passes42 tests including restarted logsIP1/IP2 and notesIP3 plus complete URL redaction. However actual consumers remain unmerged/unaccepted, not integrated feature acceptance; required CI129 pending. PR22 also has reproduced strict-cap defect BREAD-40 and missing journal ingest. Leave review, no merge/retarget or done. Coordinator must reconcile prerequisite/full-issue integration scope without waiving criteria, repair consumers and obtain fresh SHA review.
Author
Owner

Independent exact-head review at 39ff313058: library checks pass, full-card verdict blocked. Persisted criterion mapping, scope decision, job-by-job CI and combined-main checks: #21 (comment) . Required check / check (push), run129/job0 attempt2, and every step SUCCESS. Live base05125ef; conflict-free combined tree builds, passes377 workspace tests, clippy and docs. Exact library28 units+7 corpus/fuzz pass. No remaining identified library code defect. Actual cards still require integrated two-pass notes and distinct markers across restarted ingest scopes, which PR21 does not supply. Reconcile actual card/closing-reference scope into a library-only prerequisite with all consumer acceptance retained downstream, then fresh scope-bound gate, or provide accepted integrated consumers/evidence. Preserve stack and review status. No merge, branch deletion or done transition.

Independent exact-head review at 39ff3130583485dada208fe3c470954cacb56317: library checks pass, full-card verdict blocked. Persisted criterion mapping, scope decision, job-by-job CI and combined-main checks: https://git.breadway.dev/Breadway/bread/pulls/21#issuecomment-4498 . Required check / check (push), run129/job0 attempt2, and every step SUCCESS. Live base05125ef; conflict-free combined tree builds, passes377 workspace tests, clippy and docs. Exact library28 units+7 corpus/fuzz pass. No remaining identified library code defect. Actual cards still require integrated two-pass notes and distinct markers across restarted ingest scopes, which PR21 does not supply. Reconcile actual card/closing-reference scope into a library-only prerequisite with all consumer acceptance retained downstream, then fresh scope-bound gate, or provide accepted integrated consumers/evidence. Preserve stack and review status. No merge, branch deletion or done transition.
Author
Owner

Scope reconciliation for the review at #21 (comment). PR21 no longer closes BREAD-15; its body now scopes it as the library half of BREAD-15 and BREAD-37. The two-pass ingest and report criteria and the cross-restart, multi-scope marker criterion stay open on both cards. They are verified on integrated main after PR22 and PR23, and neither card moves to done on this merge.

Scope reconciliation for the review at https://git.breadway.dev/Breadway/bread/pulls/21#issuecomment-4498. PR21 no longer closes BREAD-15; its body now scopes it as the library half of BREAD-15 and BREAD-37. The two-pass ingest and report criteria and the cross-restart, multi-scope marker criterion stay open on both cards. They are verified on integrated main after PR22 and PR23, and neither card moves to done on this merge.
Author
Owner

Scope-bound library prerequisite PASS at39ff3130583485dada208fe3c470954cacb56317, persisted #21 (comment) . Accepted live scope reconciliation; no closing reference remains in PR body, although breadboard retains stale closes metadata. Final head39ff313/base05125ef, exact-head required check and run129/job0 attempt2 every step success, reviews API empty and mergeability confirmed immediately before normal expected-head-conditioned merge. Actual merged=true and merge SHA4a164fd98a9c749a6afa1ffd7b70ce0a5df83b86 confirmed. Head branch preserved for PR22. Independent library35 tests and combined-tree377 workspace tests/build/clippy/docs pass as recorded4498. This is NOT full-card acceptance or done: keep review pending integrated-main two-pass ingest/report notes and cross-restart/multi-scope marker verification after accepted PR22/23. No tags or manual release.

Scope-bound library prerequisite PASS at39ff3130583485dada208fe3c470954cacb56317, persisted https://git.breadway.dev/Breadway/bread/pulls/21#issuecomment-4523 . Accepted live scope reconciliation; no closing reference remains in PR body, although breadboard retains stale closes metadata. Final head39ff313/base05125ef, exact-head required check and run129/job0 attempt2 every step success, reviews API empty and mergeability confirmed immediately before normal expected-head-conditioned merge. Actual merged=true and merge SHA4a164fd98a9c749a6afa1ffd7b70ce0a5df83b86 confirmed. Head branch preserved for PR22. Independent library35 tests and combined-tree377 workspace tests/build/clippy/docs pass as recorded4498. This is NOT full-card acceptance or done: keep review pending integrated-main two-pass ingest/report notes and cross-restart/multi-scope marker verification after accepted PR22/23. No tags or manual release.
Author
Owner

Retained consumer gate now verified on integrated main at PR23 merge 6699643099. Clean detached worktree: cargo test --locked -p bread-bug -p breadbug --lib passes 47+11. Includes scrubbed-on-disk ingest, cross-app/rotation marker reservation after reopen, report-time seeded PII absence, and reopened_store_markers_and_report_notes_remain_distinct (IP:1/IP:2 from store, IP:3 from notes with URL credentials/query removed). Library independent review remains PR21 comment 4523; composed report independent pass is PR23 comment 4672. Dev-release success at integrated merge SHA.

Retained consumer gate now verified on integrated main at PR23 merge 6699643099a82023ab1334b2e1bd0304b8e9d862. Clean detached worktree: cargo test --locked -p bread-bug -p breadbug --lib passes 47+11. Includes scrubbed-on-disk ingest, cross-app/rotation marker reservation after reopen, report-time seeded PII absence, and reopened_store_markers_and_report_notes_remain_distinct (IP:1/IP:2 from store, IP:3 from notes with URL credentials/query removed). Library independent review remains PR21 comment 4523; composed report independent pass is PR23 comment 4672. Dev-release success at integrated merge SHA.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Breadway/bread#14
No description provided.