Breadway/bread-ecosystem
1
0
Fork 0
Code Issues 9 Pull requests 2 Projects Releases Packages Wiki Activity Actions 1

feat(ci): publish debug symbols keyed by build-id for breadbug #69

Merged
Breadway merged 3 commits from feature/breadbug-symbols into main 2026-10-13 21:16:03 +08:00
Conversation 4 Commits 3 Files changed 5 +415
Breadway commented 2026-10-09 21:32:25 +08:00
Owner
Copy link

Part of ECO-67 (the shared half). The per-product workflow wiring is a separate PR in each repo, starting with bread.

What

  • scripts/publish-symbols.sh <binary>...: run on the unstripped release binaries. Extracts debug sections with objcopy --only-keep-debug to $BREADBUG_SYMBOL_DIR/<xx>/<build-id>.debug (default /srv/breadbug-symbols). Fails the job when a binary has no GNU build-id (REQUIRE_BUILD_ID=0 to warn instead), idempotent for the same build-id, never deletes, writes atomically, mode 0644.
  • scripts/test-publish-symbols.sh: builds real binaries with and without a build-id and checks layout, DWARF present, mode, idempotence, failure modes, mixed runs.
  • docs/breadbug-symbols.md: what a release pipeline must do (build-id, debug = "line-tables-only" in the release profile, publish before strip), where symbols go, how breadbug-server reads them.

Notes

  • Rust on Linux links with a build-id by default here; the script is the enforcement, not an assumption.
  • Products consume this script through their pinned ci/bread-ecosystem.rev, so the rev has to be bumped after this merges.
  • Retention: append-only on purpose. Disk growth is one debug file per released build; worth a look once a few hundred have accumulated.
Part of ECO-67 (the shared half). The per-product workflow wiring is a separate PR in each repo, starting with bread. ## What - `scripts/publish-symbols.sh <binary>...`: run on the **unstripped** release binaries. Extracts debug sections with `objcopy --only-keep-debug` to `$BREADBUG_SYMBOL_DIR/<xx>/<build-id>.debug` (default `/srv/breadbug-symbols`). Fails the job when a binary has no GNU build-id (`REQUIRE_BUILD_ID=0` to warn instead), idempotent for the same build-id, never deletes, writes atomically, mode 0644. - `scripts/test-publish-symbols.sh`: builds real binaries with and without a build-id and checks layout, DWARF present, mode, idempotence, failure modes, mixed runs. - `docs/breadbug-symbols.md`: what a release pipeline must do (build-id, `debug = "line-tables-only"` in the release profile, publish before strip), where symbols go, how breadbug-server reads them. ## Notes - Rust on Linux links with a build-id by default here; the script is the enforcement, not an assumption. - Products consume this script through their pinned `ci/bread-ecosystem.rev`, so the rev has to be bumped after this merges. - Retention: append-only on purpose. Disk growth is one debug file per released build; worth a look once a few hundred have accumulated.
feat(ci): publish debug symbols keyed by build-id for breadbug
All checks were successful
check / check (push) Successful in 2m7s
Details
300b3d7ff5 
scripts/publish-symbols.sh extracts the debug sections of unstripped release
binaries with objcopy and stores them at <root>/<xx>/<build-id>.debug, where
breadbug-server looks them up. Fails the job on a binary with no GNU
build-id, is idempotent, and never deletes. Tested with real binaries.
docs/breadbug-symbols.md documents what a release pipeline must do.

Refs ECO-67
Breadway commented 2026-10-13 20:04:44 +08:00
Author
Owner
Copy link

Independent exact-head review: CHANGES REQUESTED

Issue ECO-67; shared-half PR #69 / ECO-70. Reviewed head 300b3d7ff5 against base b52f0fd6fa. Expected head matches. Complete diff: 3 files, 174 additions; file and PR pagination exhausted. Read ECO-67, ECO-66 umbrella, ECO-70 and BREAD-31. Existing PR reviews/comments were empty.

Required repairs on this same PR:

  • scripts/publish-symbols.sh:59-68 and docs/breadbug-symbols.md:33-37: enforce/document a private store writable by CI and readable by the server, not world-readable. Normal umask 022 produces root 0755, shard 0755, artifact 0644. Not publicly HTTP-served is not the stated filesystem privacy requirement. Validate existing unsafe stores and symlink/path assumptions; do not recursively chmod arbitrary caller trees. Test default and permissive umasks, pre-existing roots and allowed private server-reader access.
  • scripts/publish-symbols.sh:55-68: reject inputs/outputs without usable DWARF line tables and validate existing files before calling them already published. Reproduced: publish a stripped copy first, then the original unstripped binary with the same build-id. Both exit 0; second is a no-op, addr2line yields ?? / ??:0 forever for that entry. Preserve valid append-only entries and return an actionable repair failure for corrupt/empty/wrong-ID entries. Test stripped input, bad existing entries and valid line-tables-only input.
  • .forgejo/workflows/check.yml:40-65 and scripts/test-publish-symbols.sh:21-31: run this shell suite in CI and assert actual function/file/line resolution plus unchanged build-id after strip. The current green Cargo check never invokes the new suite.

Criterion mapping and observed evidence:
PASS (local shared mechanics only): build-id extraction and /.debug layout, objcopy extraction on unstripped input, valid-file idempotence, missing-ID failure and mixed-run failure; shipped local stripped binary retains the same ID. scripts/test-publish-symbols.sh passes; bash -n and git diff --check pass. Independent rustc -g fixture with spaces in binary/store paths: addr2line -f on published known_fn resolves known_fn and a.rs:1; original/shipped/debug all carry 4558e2c6fb6fb1de288392f95df3e619abcd3ab3.
FAIL: private storage and fail-closed usable-symbol publication, as reproduced above. Bug recorded as ECO-75 (Forgejo #74), high confidence.
PASS (source policy only): append-only retention and documented release profile/build-id/publish-before-strip ordering, docs:9-31; real supported-release retention is UNVERIFIED.
UNVERIFIED: all-product wiring, CI enforcing IDs in actual product releases, server-readable mount, bakery/pacman release integration and released user-machine crash resolving on the server. bread#30 remains open/unmerged; no release/deployment evidence was supplied or performed. This PR changes no shared CI workflow despite that scope description.

CI: exact head context check / check (push) is success, Actions run 296/jobs/0, Successful in 2m7s. main branch API reports unprotected, no status-check contexts. Release workflows are not expected to run on this feature head. Private branch-protections endpoint returned 401 without authentication, so no stronger protection assertion is made.

Scratch worktree: ~/.worktrees/bread-ecosystem/feature/eco67-independent-review. Reproduction artifacts: /tmp/opencode/eco67-review.RRN60k. No code changes, commits, pushes, Hestia, /srv or merges. Merge not authorized for this review. Shared half is not accepted yet, and ECO-67 as a whole remains unverified. Next: fix the three checklist items on feature/breadbug-symbols, obtain fresh exact-head CI/review, then separately integrate and verify product rollout before done.

Independent exact-head review: CHANGES REQUESTED Issue ECO-67; shared-half PR #69 / ECO-70. Reviewed head 300b3d7ff5adf0f8fb5f859999b614e89ee74579 against base b52f0fd6faf10c0f595c70dbc3bfb64cb4114fa8. Expected head matches. Complete diff: 3 files, 174 additions; file and PR pagination exhausted. Read ECO-67, ECO-66 umbrella, ECO-70 and BREAD-31. Existing PR reviews/comments were empty. Required repairs on this same PR: - [ ] scripts/publish-symbols.sh:59-68 and docs/breadbug-symbols.md:33-37: enforce/document a private store writable by CI and readable by the server, not world-readable. Normal umask 022 produces root 0755, shard 0755, artifact 0644. Not publicly HTTP-served is not the stated filesystem privacy requirement. Validate existing unsafe stores and symlink/path assumptions; do not recursively chmod arbitrary caller trees. Test default and permissive umasks, pre-existing roots and allowed private server-reader access. - [ ] scripts/publish-symbols.sh:55-68: reject inputs/outputs without usable DWARF line tables and validate existing files before calling them already published. Reproduced: publish a stripped copy first, then the original unstripped binary with the same build-id. Both exit 0; second is a no-op, addr2line yields ?? / ??:0 forever for that entry. Preserve valid append-only entries and return an actionable repair failure for corrupt/empty/wrong-ID entries. Test stripped input, bad existing entries and valid line-tables-only input. - [ ] .forgejo/workflows/check.yml:40-65 and scripts/test-publish-symbols.sh:21-31: run this shell suite in CI and assert actual function/file/line resolution plus unchanged build-id after strip. The current green Cargo check never invokes the new suite. Criterion mapping and observed evidence: PASS (local shared mechanics only): build-id extraction and <xx>/<full-id>.debug layout, objcopy extraction on unstripped input, valid-file idempotence, missing-ID failure and mixed-run failure; shipped local stripped binary retains the same ID. scripts/test-publish-symbols.sh passes; bash -n and git diff --check pass. Independent rustc -g fixture with spaces in binary/store paths: addr2line -f on published known_fn resolves known_fn and a.rs:1; original/shipped/debug all carry 4558e2c6fb6fb1de288392f95df3e619abcd3ab3. FAIL: private storage and fail-closed usable-symbol publication, as reproduced above. Bug recorded as ECO-75 (Forgejo #74), high confidence. PASS (source policy only): append-only retention and documented release profile/build-id/publish-before-strip ordering, docs:9-31; real supported-release retention is UNVERIFIED. UNVERIFIED: all-product wiring, CI enforcing IDs in actual product releases, server-readable mount, bakery/pacman release integration and released user-machine crash resolving on the server. bread#30 remains open/unmerged; no release/deployment evidence was supplied or performed. This PR changes no shared CI workflow despite that scope description. CI: exact head context check / check (push) is success, Actions run 296/jobs/0, Successful in 2m7s. main branch API reports unprotected, no status-check contexts. Release workflows are not expected to run on this feature head. Private branch-protections endpoint returned 401 without authentication, so no stronger protection assertion is made. Scratch worktree: ~/.worktrees/bread-ecosystem/feature/eco67-independent-review. Reproduction artifacts: /tmp/opencode/eco67-review.RRN60k. No code changes, commits, pushes, Hestia, /srv or merges. Merge not authorized for this review. Shared half is not accepted yet, and ECO-67 as a whole remains unverified. Next: fix the three checklist items on feature/breadbug-symbols, obtain fresh exact-head CI/review, then separately integrate and verify product rollout before done.
Breadway commented 2026-10-13 20:04:52 +08:00
Author
Owner
Copy link

Exact-head review of 300b3d7ff5: changes requested. Full evidence persisted on PR #69 and ECO-67. Repairs: private symbol store permissions, reject unusable DWARF and invalid existing entries, run behavioral symbol tests in CI. Reproduced defects tracked by ECO-75. Shared half not accepted; full ECO-67 rollout/server acceptance unverified. Preserve PR/branch; no merge.

Exact-head review of 300b3d7ff5adf0f8fb5f859999b614e89ee74579: changes requested. Full evidence persisted on PR #69 and ECO-67. Repairs: private symbol store permissions, reject unusable DWARF and invalid existing entries, run behavioral symbol tests in CI. Reproduced defects tracked by ECO-75. Shared half not accepted; full ECO-67 rollout/server acceptance unverified. Preserve PR/branch; no merge.
publish-symbols.sh now sets umask 027 and creates the store root and shards as 0750 and artifacts as 0640 regardless of the caller umask. It refuses an existing root or shard that is a symlink, world-accessible or group-writable, without changing it. Inputs and extracted output must carry non-empty .debug_info and .debug_line, so stripped binaries fail and write nothing. A stale or unusable artifact for the same build-id is replaced atomically; a valid one is kept and has its mode corrected. docs/breadbug-symbols.md documents the provisioning contract and why 0750/0640.
ci(breadbug): run symbol publisher tests and install binutils
All checks were successful
check / check (push) Successful in 6m16s
Details
ef218b88c9 
check.yml runs scripts/test-publish-symbols.sh on feature and fix pushes. The CI image installs binutils explicitly, and the publisher checks for readelf, objcopy and objdump before running.
Breadway commented 2026-10-13 20:15:45 +08:00
Author
Owner
Copy link

Review repairs for ECO-75 (ECO-67 shared half), pushed as fast-forward ef218b8 on top of 300b3d7.

  • Private store: publish-symbols.sh sets umask 027 and creates the root and shards as 0750 and artifacts as 0640 regardless of the caller umask. Existing root or shard dirs that are symlinks, world-accessible or group-writable are refused with the exact chmod needed, never changed. Existing skipped artifacts get their mode corrected. docs/breadbug-symbols.md has the provisioning contract and the reason for 0750/0640.
  • Fail closed: input and extracted output must both have non-empty .debug_info and .debug_line. A stripped input exits non-zero and writes nothing. A stale or unusable artifact for the same build-id is replaced atomically (temp file in the shard dir, then rename). Valid artifacts stay append-only.
  • CI: scripts/test-publish-symbols.sh now covers umask 022 and 000 modes, unsafe pre-existing stores, stripped rejection, replacement of empty/stripped/wrong-id artifacts, matching build-ids across original/stripped/debug, and addr2line resolving known_fn to a.rs:1. check.yml runs it on feature and fix pushes. ci/Containerfile installs binutils explicitly.

Local results at ef218b8: test-publish-symbols.sh pass; cargo fmt --check, clippy -D warnings (default, gtk, extract), cargo test (default, gtk, extract) and cargo build --locked --features extract all pass; lint-pins and test-build-lock pass.

Not verified here: the real runner and server accounts and group membership, the reader's access from breadbug-server's container, and a rebuilt CI image containing binutils.

Review repairs for ECO-75 (ECO-67 shared half), pushed as fast-forward ef218b8 on top of 300b3d7. - Private store: publish-symbols.sh sets umask 027 and creates the root and shards as 0750 and artifacts as 0640 regardless of the caller umask. Existing root or shard dirs that are symlinks, world-accessible or group-writable are refused with the exact chmod needed, never changed. Existing skipped artifacts get their mode corrected. docs/breadbug-symbols.md has the provisioning contract and the reason for 0750/0640. - Fail closed: input and extracted output must both have non-empty .debug_info and .debug_line. A stripped input exits non-zero and writes nothing. A stale or unusable artifact for the same build-id is replaced atomically (temp file in the shard dir, then rename). Valid artifacts stay append-only. - CI: scripts/test-publish-symbols.sh now covers umask 022 and 000 modes, unsafe pre-existing stores, stripped rejection, replacement of empty/stripped/wrong-id artifacts, matching build-ids across original/stripped/debug, and addr2line resolving known_fn to a.rs:1. check.yml runs it on feature and fix pushes. ci/Containerfile installs binutils explicitly. Local results at ef218b8: test-publish-symbols.sh pass; cargo fmt --check, clippy -D warnings (default, gtk, extract), cargo test (default, gtk, extract) and cargo build --locked --features extract all pass; lint-pins and test-build-lock pass. Not verified here: the real runner and server accounts and group membership, the reader's access from breadbug-server's container, and a rebuilt CI image containing binutils.
Breadway commented 2026-10-13 21:17:10 +08:00
Author
Owner
Copy link

Independent exact-head re-review: PASS

Reviewed head ef218b88c9 against base b52f0fd6fa. Expected head matched. Complete diff: 5 files, 415 additions. Reviews API empty. Prior change request is comment 4424 at 300b3d7; repair summary is comment 4464.

Previous items, confirmed fixed:

  • Private store: umask 027, directories 0750, files 0640 regardless of caller umask. Unsafe existing roots and shards (world access, group-writable, symlinks) are refused and not changed. Local probes: umask 022 and 000, setgid root inherits the reader group, symlink root refused, shard mode 755 refused and left at 755.
  • Fail closed: stripped input exits non-zero and writes nothing. Empty, stripped, wrong-id, and symlink artifacts for the same build-id are replaced atomically. A valid republish is a no-op. A debuginfo=1 (line-tables-only) input resolves known_fn to a.rs:1.
  • CI: check.yml step "breadbug symbol publisher" ran in Actions run 298, task 2392, job check, on this exact SHA. The job log contains "publish-symbols: all tests passed" and ends "Job succeeded". ci/Containerfile installs binutils. This job used readelf, objcopy, objdump, and addr2line, so the current runner image has them. That does not by itself prove a rebuilt image from this Containerfile.

Local: scripts/test-publish-symbols.sh passed; bash -n and git diff --check passed. Required check "check / check (push)" success on ef218b88. main has no branch-protection rules.

Shared half of ECO-67 accepted and merged. Merge commit 4f70691b33 (parents b52f0fd, ef218b8), merged flag true. Head branch kept. No tag. Dev-track publish on main is expected.

ECO-67 is not done: per-product wiring (bread PR 30 / BREAD-31) and a released user-machine crash resolving on the server stay open. Cards left in review for coordinator verification.

Independent exact-head re-review: PASS Reviewed head ef218b88c99b602e66fc754c1a2a4462ecbd3e39 against base b52f0fd6faf10c0f595c70dbc3bfb64cb4114fa8. Expected head matched. Complete diff: 5 files, 415 additions. Reviews API empty. Prior change request is comment 4424 at 300b3d7; repair summary is comment 4464. Previous items, confirmed fixed: - Private store: umask 027, directories 0750, files 0640 regardless of caller umask. Unsafe existing roots and shards (world access, group-writable, symlinks) are refused and not changed. Local probes: umask 022 and 000, setgid root inherits the reader group, symlink root refused, shard mode 755 refused and left at 755. - Fail closed: stripped input exits non-zero and writes nothing. Empty, stripped, wrong-id, and symlink artifacts for the same build-id are replaced atomically. A valid republish is a no-op. A debuginfo=1 (line-tables-only) input resolves known_fn to a.rs:1. - CI: check.yml step "breadbug symbol publisher" ran in Actions run 298, task 2392, job check, on this exact SHA. The job log contains "publish-symbols: all tests passed" and ends "Job succeeded". ci/Containerfile installs binutils. This job used readelf, objcopy, objdump, and addr2line, so the current runner image has them. That does not by itself prove a rebuilt image from this Containerfile. Local: scripts/test-publish-symbols.sh passed; bash -n and git diff --check passed. Required check "check / check (push)" success on ef218b88. main has no branch-protection rules. Shared half of ECO-67 accepted and merged. Merge commit 4f70691b33bc778294073681065665d3e2b7f559 (parents b52f0fd, ef218b8), merged flag true. Head branch kept. No tag. Dev-track publish on main is expected. ECO-67 is not done: per-product wiring (bread PR 30 / BREAD-31) and a released user-machine crash resolving on the server stay open. Cards left in review for coordinator verification.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bread-theme

   
breadbug

   
bug

   
ci

   
enhancement

   
phase-2

   
release

   
rollout
No labels
bread-theme
breadbug
bug
ci
enhancement
phase-2
release
rollout
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Breadway/bread-ecosystem!69
No description provided.