breadbug: release pipeline publishes debug symbols keyed by build-id #66

New issue

Open

opened 2026-10-08 23:11:39 +08:00 by Breadway · 2 comments

Breadway commented

2026-10-08 23:11:39 +08:00

Owner

Part of ECO-EP1 (see ECO-66).

Shipped binaries stay stripped; symbolication happens server-side.

Requirements

Release workflows for all bread apps (including bread-theme-fed apps and breadd) build with a stable GNU build-id and split debuginfo.
Upload debug symbol files keyed by build-id to a symbol store reachable by breadbug-server (not world-readable).
Retain symbols for every released version still supported by bakery or the pacman repo.
Document the required build flags for new apps; fail CI if a release binary lacks a build-id.
Works within the existing release train (bakery + [breadway] pacman repo, dual-remote tag rules).

Acceptance

A released app's crash frames from a user machine resolve to function names and lines on the server.

Part of ECO-EP1 (see ECO-66). Shipped binaries stay stripped; symbolication happens server-side. ## Requirements - Release workflows for all bread apps (including bread-theme-fed apps and breadd) build with a stable GNU build-id and split debuginfo. - Upload debug symbol files keyed by build-id to a symbol store reachable by breadbug-server (not world-readable). - Retain symbols for every released version still supported by bakery or the pacman repo. - Document the required build flags for new apps; fail CI if a release binary lacks a build-id. - Works within the existing release train (bakery + [breadway] pacman repo, dual-remote tag rules). ## Acceptance - A released app's crash frames from a user machine resolve to function names and lines on the server.

Breadway added the

breadbug

phase-2

labels

2026-10-08 23:11:39 +08:00

Breadway commented

2026-10-13 20:04:51 +08:00

Author

Owner

Independent exact-head review: CHANGES REQUESTED

Issue ECO-67; shared-half PR #69 / ECO-70. Reviewed head 300b3d7ff5 against base b52f0fd6fa. Expected head matches. Complete diff: 3 files, 174 additions; file and PR pagination exhausted. Read ECO-67, ECO-66 umbrella, ECO-70 and BREAD-31. Existing PR reviews/comments were empty.

Required repairs on this same PR:

scripts/publish-symbols.sh:59-68 and docs/breadbug-symbols.md:33-37: enforce/document a private store writable by CI and readable by the server, not world-readable. Normal umask 022 produces root 0755, shard 0755, artifact 0644. Not publicly HTTP-served is not the stated filesystem privacy requirement. Validate existing unsafe stores and symlink/path assumptions; do not recursively chmod arbitrary caller trees. Test default and permissive umasks, pre-existing roots and allowed private server-reader access.
scripts/publish-symbols.sh:55-68: reject inputs/outputs without usable DWARF line tables and validate existing files before calling them already published. Reproduced: publish a stripped copy first, then the original unstripped binary with the same build-id. Both exit 0; second is a no-op, addr2line yields ?? / ??:0 forever for that entry. Preserve valid append-only entries and return an actionable repair failure for corrupt/empty/wrong-ID entries. Test stripped input, bad existing entries and valid line-tables-only input.
.forgejo/workflows/check.yml:40-65 and scripts/test-publish-symbols.sh:21-31: run this shell suite in CI and assert actual function/file/line resolution plus unchanged build-id after strip. The current green Cargo check never invokes the new suite.

Criterion mapping and observed evidence:
PASS (local shared mechanics only): build-id extraction and /.debug layout, objcopy extraction on unstripped input, valid-file idempotence, missing-ID failure and mixed-run failure; shipped local stripped binary retains the same ID. scripts/test-publish-symbols.sh passes; bash -n and git diff --check pass. Independent rustc -g fixture with spaces in binary/store paths: addr2line -f on published known_fn resolves known_fn and a.rs:1; original/shipped/debug all carry 4558e2c6fb6fb1de288392f95df3e619abcd3ab3.
FAIL: private storage and fail-closed usable-symbol publication, as reproduced above. Bug recorded as ECO-75 (Forgejo #74), high confidence.
PASS (source policy only): append-only retention and documented release profile/build-id/publish-before-strip ordering, docs:9-31; real supported-release retention is UNVERIFIED.
UNVERIFIED: all-product wiring, CI enforcing IDs in actual product releases, server-readable mount, bakery/pacman release integration and released user-machine crash resolving on the server. bread#30 remains open/unmerged; no release/deployment evidence was supplied or performed. This PR changes no shared CI workflow despite that scope description.

CI: exact head context check / check (push) is success, Actions run 296/jobs/0, Successful in 2m7s. main branch API reports unprotected, no status-check contexts. Release workflows are not expected to run on this feature head. Private branch-protections endpoint returned 401 without authentication, so no stronger protection assertion is made.

Scratch worktree: ~/.worktrees/bread-ecosystem/feature/eco67-independent-review. Reproduction artifacts: /tmp/opencode/eco67-review.RRN60k. No code changes, commits, pushes, Hestia, /srv or merges. Merge not authorized for this review. Shared half is not accepted yet, and ECO-67 as a whole remains unverified. Next: fix the three checklist items on feature/breadbug-symbols, obtain fresh exact-head CI/review, then separately integrate and verify product rollout before done.

Independent exact-head review: CHANGES REQUESTED Issue ECO-67; shared-half PR #69 / ECO-70. Reviewed head 300b3d7ff5adf0f8fb5f859999b614e89ee74579 against base b52f0fd6faf10c0f595c70dbc3bfb64cb4114fa8. Expected head matches. Complete diff: 3 files, 174 additions; file and PR pagination exhausted. Read ECO-67, ECO-66 umbrella, ECO-70 and BREAD-31. Existing PR reviews/comments were empty. Required repairs on this same PR: - [ ] scripts/publish-symbols.sh:59-68 and docs/breadbug-symbols.md:33-37: enforce/document a private store writable by CI and readable by the server, not world-readable. Normal umask 022 produces root 0755, shard 0755, artifact 0644. Not publicly HTTP-served is not the stated filesystem privacy requirement. Validate existing unsafe stores and symlink/path assumptions; do not recursively chmod arbitrary caller trees. Test default and permissive umasks, pre-existing roots and allowed private server-reader access. - [ ] scripts/publish-symbols.sh:55-68: reject inputs/outputs without usable DWARF line tables and validate existing files before calling them already published. Reproduced: publish a stripped copy first, then the original unstripped binary with the same build-id. Both exit 0; second is a no-op, addr2line yields ?? / ??:0 forever for that entry. Preserve valid append-only entries and return an actionable repair failure for corrupt/empty/wrong-ID entries. Test stripped input, bad existing entries and valid line-tables-only input. - [ ] .forgejo/workflows/check.yml:40-65 and scripts/test-publish-symbols.sh:21-31: run this shell suite in CI and assert actual function/file/line resolution plus unchanged build-id after strip. The current green Cargo check never invokes the new suite. Criterion mapping and observed evidence: PASS (local shared mechanics only): build-id extraction and <xx>/<full-id>.debug layout, objcopy extraction on unstripped input, valid-file idempotence, missing-ID failure and mixed-run failure; shipped local stripped binary retains the same ID. scripts/test-publish-symbols.sh passes; bash -n and git diff --check pass. Independent rustc -g fixture with spaces in binary/store paths: addr2line -f on published known_fn resolves known_fn and a.rs:1; original/shipped/debug all carry 4558e2c6fb6fb1de288392f95df3e619abcd3ab3. FAIL: private storage and fail-closed usable-symbol publication, as reproduced above. Bug recorded as ECO-75 (Forgejo #74), high confidence. PASS (source policy only): append-only retention and documented release profile/build-id/publish-before-strip ordering, docs:9-31; real supported-release retention is UNVERIFIED. UNVERIFIED: all-product wiring, CI enforcing IDs in actual product releases, server-readable mount, bakery/pacman release integration and released user-machine crash resolving on the server. bread#30 remains open/unmerged; no release/deployment evidence was supplied or performed. This PR changes no shared CI workflow despite that scope description. CI: exact head context check / check (push) is success, Actions run 296/jobs/0, Successful in 2m7s. main branch API reports unprotected, no status-check contexts. Release workflows are not expected to run on this feature head. Private branch-protections endpoint returned 401 without authentication, so no stronger protection assertion is made. Scratch worktree: ~/.worktrees/bread-ecosystem/feature/eco67-independent-review. Reproduction artifacts: /tmp/opencode/eco67-review.RRN60k. No code changes, commits, pushes, Hestia, /srv or merges. Merge not authorized for this review. Shared half is not accepted yet, and ECO-67 as a whole remains unverified. Next: fix the three checklist items on feature/breadbug-symbols, obtain fresh exact-head CI/review, then separately integrate and verify product rollout before done.

Breadway commented

2026-10-13 21:17:07 +08:00

Author

Owner

Shared-half re-review PASS, merged. Not done.

PR #69 head ef218b88c9. Merge commit 4f70691b33. Full criterion mapping is on the PR.

Accepted for this repo only: private symbol store (0750/0640, unsafe stores refused), fail-closed usable DWARF with replacement of stale artifacts, behavioral suite actually run in check job 298 on this SHA, documented release flags and provisioning.

Still open on this card, so it stays out of done:

per-product release wiring, including bread PR 30 / BREAD-31
a real released binary whose user-machine crash resolves to function and line on breadbug-server

Left in review for coordinator verification. No tag.

Shared-half re-review PASS, merged. Not done. PR https://git.breadway.dev/Breadway/bread-ecosystem/pulls/69 head ef218b88c99b602e66fc754c1a2a4462ecbd3e39. Merge commit 4f70691b33bc778294073681065665d3e2b7f559. Full criterion mapping is on the PR. Accepted for this repo only: private symbol store (0750/0640, unsafe stores refused), fail-closed usable DWARF with replacement of stale artifacts, behavioral suite actually run in check job 298 on this SHA, documented release flags and provisioning. Still open on this card, so it stays out of done: - per-product release wiring, including bread PR 30 / BREAD-31 - a real released binary whose user-machine crash resolves to function and line on breadbug-server Left in review for coordinator verification. No tag.