Dev track builds against dependency main; releases stay on locked tag pins #38
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Cross-repo changes currently need a committed rev/branch/path pin (which leaks toward stable) or an ecosystem tag before the dependent app's dev build works. Local dev relies on a gitignored .cargo/config.toml path patch, and lockfiles regenerated under it keep breaking CI's --locked builds.
Model:
Pilot: bread-ecosystem (script + docs/release-channels.md) and breadbar (dev-release.yml, check.yml). Roll out to the other bakery-channel apps once the pilot is proven.
Script working locally. bread-ecosystem/scripts/dev-deps.sh and scripts/lint-pins.sh written and tested end to end against real clones: fresh clone, idempotent second run (refresh of owned clones), untouched-developer-checkout reuse, refuse-to-clobber a foreign .cargo/config.toml plus --force override, and a full cargo build --release that actually compiled breadbar against bread-ecosystem main and bread main via the generated path patches. Confirmed a clean-clone cargo build --release --locked still resolves to the tagged v0.7.11/v0.8.0 pins with no .cargo/config.toml present.
Pushed feature/dev-deps-main to origin in both repos and opening PRs now (bread-ecosystem first, breadbar depends on it).
Pilot progress: bread-ecosystem #39 reviewed and merged to main (merge commit
252cf7d). scripts/dev-deps.sh and scripts/lint-pins.sh, the pin-lint in check.yml, and docs/release-channels.md are now on main.breadbar #29 was failing as expected (missing scripts on bread-ecosystem main before #39 merged). Updated the branch from main and pushed to re-trigger check.yml. Watching that now.
Blocker note: breadbar's dev-release job has now failed 3 times in a row (2 on breadbar, 1 on breadbox), each after 1-3 minutes, with no error surfaced by the Forgejo API (no job-log endpoint exists). Every time, I reproduced the exact same steps locally (checkout, dev-deps.sh, dockerized cargo build via ci/build.sh) and they succeeded cleanly, including after merging a retry-hardening fix for transient git clone/fetch failures (bread-ecosystem #40). Since it recurs even after that fix, across two different repos, with identical local success, this looks like a self-hosted-runner infrastructure issue (disk, docker daemon, resource limits) that I can't diagnose or fix without direct access to the runner host. I have not been granted proactive access to Hestia for this.
Recommend the owner check the runner's disk space, docker daemon health, and resource limits directly. I'm continuing the rest of the ECO-38 rollout (check.yml/PR-gating works fine and has passed on every repo so far); I will keep watching for a dev-release success but will not keep spending cycles re-diagnosing the same unreproducible failure.
Pilot status: bread-ecosystem #39 and #40 merged. breadbar #29 and #32 merged (dev-release still pending a clean run on the runner). breadbox #11 merged (dev-release also failed once, same pattern). breadclip #9 merged.
Rollout status update. All 13 target repos now have the ECO-38 CI wiring merged: breadbox, breadclip, breadpad, breadsearch, breadmon, breadhelp, breadarr, breadcrumbs, breadpaper, breadshot, bos-settings, breadcast. bread (Breadway/bread) has no real git.breadway.dev cross-repo deps and was skipped.
Fixes made along the way:
devbranch, removed in the single-trunk migration; every dev-release run had been failing outright because of this, unrelated to ECO-38. Fixed tomainwhile touching this file anyway.Known CI gaps found that are unrelated to ECO-38 (not fixed, flagged for the owner): breadarr's cargo test needs a real ffmpeg binary the shared CI image doesn't have (check.yml has failed on this since August, confirmed via CI history) and breadarr's check.yml pull_request trigger fails instantly on GITHUB_REF_NAME resolution, both pre-existing. breadcrumbs' fake-NetworkManager D-Bus integration tests failed identically on this diff and on the untouched pre-ECO-38 baseline in a clean scratch reproduction, so also not this diff's doing.
Currently verifying each repo's first real dev-release run passes; several failed fast with no diagnosable cause (no Forgejo log endpoint) right after merging in quick succession, while local reproduction of the identical commands succeeds every time (including breadarr, breadbox, breadclip specifically checked). This looks like runner capacity contention from many first-time dev-release builds firing back to back during the rollout. Retriggering each with a follow-up empty-commit PR and confirming one at a time. Will report final status once done.
Final status for this pass.
Merged (CI wiring correct in every case, verified via clean local reproduction of the exact commands each workflow runs):
First real dev-release run confirmed green: breadhelp, bos-settings, breadarr (after a retry). breadcast's check.yml passed (its dev-release has not been retried yet).
Still red on the runner despite clean local reproduction and at least one retry each, hours apart: breadbar, breadbox, breadclip, breadpad, breadsearch, breadmon, breadcrumbs, breadpaper, breadshot. Every one of these builds and tests successfully when run locally with the identical checkout, dev-deps.sh, and build commands the workflow runs (verified directly for breadbar, breadbox, breadarr, breadclip, breadcrumbs). Failures have no diagnosable cause via the Forgejo API (no job log endpoint), fail anywhere from under a minute to several minutes in, and persist across retries spaced hours apart, so it is not simply contention from merging several PRs in quick succession. Given breadarr, breadhelp, and bos-settings did go green on the same runner using the identical mechanism, this looks like intermittent runner side flakiness (disk, docker daemon, or similar) rather than a permanent break or a defect in the CI wiring, but I cannot confirm the exact cause without direct access to the runner, which I was not given for this task.
Fixes made along the way:
Known gaps found, not fixed (pre-existing, unrelated to ECO-38, flagged for the owner): breadarr's cargo test needs a real ffmpeg binary the shared CI image does not have (failing since August per CI history); breadarr's check.yml pull_request trigger fails instantly on GITHUB_REF_NAME resolution; breadcrumbs' fake NetworkManager D-Bus integration tests fail identically on this diff and on the untouched pre-ECO-38 baseline in a clean scratch reproduction.
Moving this to review rather than done. The CI wiring itself is complete and correct, but I cannot personally confirm every repo's dev track is producing dev builds while the runner is behaving like this. Recommend a runner health check (disk space, docker daemon) on Hestia, then a final pass re-checking each red repo's next dev-release run once that is done. None of them need further code changes from what I can tell.
Root cause of the red dev-release runs found from the real logs. The forgejo-runner journal on Hestia (journalctl -u forgejo-runner, level debug) keeps full step output for every job, so no Forgejo job-log endpoint is needed. It was not disk, docker or runner flakiness (root disk 68%, docker healthy).
Cause 1 (breadbar, breadbox, breadclip, breadpad, breadsearch, breadmon, breadcrumbs, breadpaper, breadshot): the "clone bread-ecosystem (main)" step in every dev-release.yml and check.yml ran
rm -rf /tmp/bread-ecosystem-ci-*before mktemp. The runner uses the host executor, so /tmp is shared by all concurrent jobs (capacity 2) and one job deleted another's live bread-ecosystem clone. Symptoms:bash: /tmp/bread-ecosystem-ci-RI0rsb/scripts/gen-index.sh: No such file or directoryat "regenerate dev index.json" (the build itself had succeeded), orcould not lock config file /tmp/bread-ecosystem-ci-mWQCpa/.git/config: No such file or directoryinside the clone step. It explains why local repro always passed and why retries only sometimes failed. breadhelp and breadarr check.yml were already on the safe pattern, which is why they stayed green.Fix: dropped the glob and used a per-workflow mktemp prefix (bread-ecosystem-dev-release-, bread-ecosystem-check-). PRs merged: breadbar #33, breadbox #13, breadclip #11, breadpad #12, breadsearch #8, breadmon #5, breadarr #7, breadcrumbs #15, breadpaper #9, breadshot #4, bos-settings #20, breadcast #8 (breadhelp needed nothing).
Cause 2 (breadcast only): its compute dev version step lacked the
(grep -v -- '-' || true)pre-release filter every other repo has. With tag v0.1.1-rc.1 present, PA became "1-rc.1" and the run died withinvalid arithmetic operator (error token is ".1"). Fixed in breadcast #8.Cause 3 (breadcast, one run, transient): a dev-release overlapped a check job of the same product, and ci/build.sh's
cp -a /cargo-target/. target/failed on files the other job was churning in the shared ~/.cache/bread-ci/target-breadcast (cp: cannot stat ... No such file or directory). Retriggered via empty-commit PR breadcast #9 and it passed. Structural follow-up, not fixed: serialize per product (flock in bread-ecosystem ci/build.sh, then bump ci/bread-ecosystem.rev in consumers), or copy only the release artifacts.Result: latest dev-release on main is green in all 13 repos (breadcast 3f5e038, breadhelp b508940, the rest on their #33/#13/... merge commits). Pre-existing, unrelated check.yml reds remain in breadarr (ffmpeg missing, ffprobe tests) and breadcrumbs (D-Bus tests); they were merged with those known failures since the diff is workflow-only. Moving to done.
Follow-ups after the initial rollout (2026-09-29/30). All 12 wrapper-based apps are done and verified on main.
Cause of the failing dev-release runs (already noted above): concurrent jobs on the shared runner deleted each other's /tmp/bread-ecosystem-ci-* clones. Fixed per workflow with unique mktemp prefixes.
Shared build cache collision. Two jobs for one product shared target- and one failed in cp -a. bread-ecosystem #42 (
36b4ec7): ci/build.sh takes a per-product flock, BREAD_CI_LOCK_WAIT default 3600s. Tested with a fake docker. Pin bumped in all 12 apps.Clone race inside each app's ci/build.sh. The check-then-clone was not atomic and most apps had a glob rm -rf that could delete a checkout in use. Now cloned into a private temp dir, marked .ready, renamed into place under a lock, other revs pruned only after a day unused. Merged in all 12 apps. Tested with four concurrent jobs.
breadcrumbs check. 12 fake-NetworkManager tests failed because the container uid has no /etc/passwd entry, so dbus-daemon refused every connection. bread-ecosystem #44 (
7e27d78) mounts the host /etc/passwd and /etc/group read-only. breadcrumbs #19 pins it. Full suite green in CI.breadarr check. Removed the pull_request trigger (GITHUB_REF_NAME is like 10/merge there, so checkout failed at once), added ffmpeg and mkvtoolnix-cli to ci/deps.txt (#11), then #12 for the bump and clone fix. Green.
breadpaper. Its first bump PR was built on a stale clone from before its own migration to bread-theme v0.8.0 and failed for that reason only. Redone on fresh main (#14).
State: every app has the lock and the clone fix. Latest dev build is green in all 12. bos-settings has no docker wrapper and was not part of this. Superseded PRs closed and about 50 leftover branches deleted.
Optional: only breadcrumbs and breadarr are pinned to
7e27d78(with the passwd mount). The other ten are on36b4ec7and do not need it today.Not fixed: breadarr's ci/build.sh still downloads its old-glibc sysroot to a shared /tmp path, the same kind of race.