feat: hardened intake, storage, retention and delete-by-receipt #1

Merged
Breadway merged 3 commits from feature/intake into main 2026-10-11 19:41:09 +08:00
Owner

Closes BUG-2, BUG-3. Refs BUG-1 (everything except the live deploy, which needs a person; see the deployment checklist in #8)

Part of ECO-EP1 (spec: ECO-66, bread repo docs/breadbug.md).

What

  • Public listener (:8120): POST /v1/reports, DELETE /v1/reports/{receipt}, GET /v1/reports/{receipt}/status, /healthz. Admin listener is a separate port and not routed publicly (the triage UI comes later).
  • Strict schema (schema 1), re-declared server-side: unknown fields rejected at every level, per-field length and per-list count limits, bounded depth and width for event data, created_unix sanity, app id charset.
  • Limits: body cap enforced while streaming (no decompression at all, so no bombs), per-install / per-network (/24, /48) / global sliding windows with Retry-After, bounded in-flight requests (503), data-dir size cap.
  • Privacy: install ids are HMAC'd with a server secret before storage; network buckets are HMAC'd and held in memory only; IPs are never stored or logged and request content never reaches logs.
  • Server-side re-scrub with the same bread-scrub rules. A clean client report comes back byte-for-byte unchanged; any new redaction means the client missed something, so only the scrubbed form is stored and the report is quarantined.
  • Storage: SQLite (WAL, secure_delete), migrations by user_version, idempotent resend (same install + same payload returns the original receipt), 90-day retention job, token-checked delete (wrong token and unknown receipt are indistinguishable) with WAL truncation so deleted data really leaves disk.
  • Packaging: Dockerfile and packaging/docker-compose.yml (own network and volume, non-root, read-only rootfs, no capabilities, no host mounts, memory and pid limits). docs/threat-model.md, README, CONTRIBUTING, Forgejo check workflow.

Tests

  • 21 unit tests (schema, limiter, crypto, rescrub, db) and 13 HTTP integration tests that drive the real router: a real breadbug --dry-run report is accepted and stored unchanged; bad input matrix; oversize; resend; per-install, per-network (with and without a trusted proxy) and global limits; backpressure; leaked PII is scrubbed and quarantined; hostile content stored inert; delete; status; retention.
  • clippy -D warnings clean.

Notes

  • bread-scrub is a git dependency on the bread repo's feature/breadbug-scrubber branch (bread PR #21). Repoint it to a tag/main once that merges.
  • Nothing is deployed by this PR. Bringing the container up on Hestia and adding the Aegis Caddy block (infra repo) is a separate, deliberate step (BUG-1 acceptance).
Closes BUG-2, BUG-3. Refs BUG-1 (everything except the live deploy, which needs a person; see the deployment checklist in #8) Part of ECO-EP1 (spec: ECO-66, bread repo `docs/breadbug.md`). ## What - **Public listener** (`:8120`): `POST /v1/reports`, `DELETE /v1/reports/{receipt}`, `GET /v1/reports/{receipt}/status`, `/healthz`. Admin listener is a separate port and not routed publicly (the triage UI comes later). - **Strict schema** (schema 1), re-declared server-side: unknown fields rejected at every level, per-field length and per-list count limits, bounded depth and width for event data, `created_unix` sanity, app id charset. - **Limits**: body cap enforced while streaming (no decompression at all, so no bombs), per-install / per-network (/24, /48) / global sliding windows with `Retry-After`, bounded in-flight requests (503), data-dir size cap. - **Privacy**: install ids are HMAC'd with a server secret before storage; network buckets are HMAC'd and held in memory only; IPs are never stored or logged and request content never reaches logs. - **Server-side re-scrub** with the same bread-scrub rules. A clean client report comes back byte-for-byte unchanged; any new redaction means the client missed something, so only the scrubbed form is stored and the report is quarantined. - **Storage**: SQLite (WAL, `secure_delete`), migrations by `user_version`, idempotent resend (same install + same payload returns the original receipt), 90-day retention job, token-checked delete (wrong token and unknown receipt are indistinguishable) with WAL truncation so deleted data really leaves disk. - **Packaging**: Dockerfile and `packaging/docker-compose.yml` (own network and volume, non-root, read-only rootfs, no capabilities, no host mounts, memory and pid limits). `docs/threat-model.md`, README, CONTRIBUTING, Forgejo check workflow. ## Tests - 21 unit tests (schema, limiter, crypto, rescrub, db) and 13 HTTP integration tests that drive the real router: a real `breadbug --dry-run` report is accepted and stored unchanged; bad input matrix; oversize; resend; per-install, per-network (with and without a trusted proxy) and global limits; backpressure; leaked PII is scrubbed and quarantined; hostile content stored inert; delete; status; retention. - clippy `-D warnings` clean. ## Notes - `bread-scrub` is a git dependency on the bread repo's `feature/breadbug-scrubber` branch (bread PR #21). Repoint it to a tag/main once that merges. - Nothing is deployed by this PR. Bringing the container up on Hestia and adding the Aegis Caddy block (infra repo) is a separate, deliberate step (BUG-1 acceptance).
feat: hardened intake endpoint, storage, retention and delete-by-receipt
All checks were successful
check / check (push) Successful in 28s
4809158d6a
Public listener (POST /v1/reports, DELETE /v1/reports/{receipt}, GET
status, /healthz) with a strict versioned schema, size and count limits,
per-install, per-network and global rate limits, bounded in-flight
requests, server-side re-scrub with quarantine, idempotent resends,
SQLite storage with 90-day retention and token-checked deletion. Install
ids and network buckets are HMAC'd; IPs are never stored or logged.

Includes the container packaging (isolated, non-root, read-only), the
threat model, CI check workflow and a real client report as a fixture.

Refs BUG-1, BUG-2, BUG-3
fix: admit intake streams before buffering and exercise fuzz load
All checks were successful
check / check (push) Successful in 42s
d231938dfb
fix: evict oldest raw payloads and sanitize rotating backups
All checks were successful
check / check (push) Successful in 44s
4ca69f49b2
Author
Owner

Blocked at merge. PR #1 (feature/intake into main) was reviewed and is otherwise ready: CI check is success on 4ca69f4, mergeable is true, squash is allowed, and the diff covers schema limits, rate limits, rescrub, retention, delete-by-receipt, oldest-first quota eviction, and the backup rotation contract. The Forgejo merge call failed before the API (tool error: Unable to execute forgejo_merge_pull), retried once with the same result. PR #1 stays open. Per the stack rule, PRs #2 through #16 were not touched.

Blocked at merge. PR #1 (feature/intake into main) was reviewed and is otherwise ready: CI check is success on 4ca69f4, mergeable is true, squash is allowed, and the diff covers schema limits, rate limits, rescrub, retention, delete-by-receipt, oldest-first quota eviction, and the backup rotation contract. The Forgejo merge call failed before the API (tool error: Unable to execute forgejo_merge_pull), retried once with the same result. PR #1 stays open. Per the stack rule, PRs #2 through #16 were not touched.
Breadway deleted branch feature/intake 2026-10-11 19:41:09 +08:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Breadway/breadbug-server!1
No description provided.