Shared CI cache path races when multiple repos release concurrently #28
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
ci/build.sh (vendored into every bread-ecosystem-pinned repo, e.g. bread/ci/build.sh) uses a fixed cache path
/tmp/bread-ecosystem-ci-${REV}and doesrm -rf /tmp/bread-ecosystem-ci-*then a fresh clone whenever that path does not already exist. Most repos pin the same rev (confirmed: bread, breadbox, breadcrumbs, breadbar, breadpaper, breadhelp all pin147cfbbf..., breadpad, breadclip, breadmon, breadshot all pin620c5a13...), so their dev/rc/release workflows all contend for the same fixed path on the shared self-hosted runner when triggered close together in time.Update: confirmed this is a real, reproducible break for bread specifically, not just a race. bread's dev-release.yml and rc-release.yml both build via
bash ci/build.sh cargo ..., which delegates to bread-ecosystem's shared ci/build.sh and ultimately shells out to docker (confirmed locally: the script fails immediately with "failed to connect to the docker API at unix:///var/run/docker.sock"). bread's v0.9.0-rc.1, rc.2, and rc.3 all failed the beta (rc) release workflow in under 2 seconds, three separate attempts, no artifact ever reached the beta index or dl.breadway.dev/beta/bread/. This is too fast to be a real build running and failing, consistent with the runner-side docker call failing immediately every time, not intermittently.By contrast, bread-ecosystem's own release-bakery.yml/release-bread-theme.yml call
cargo builddirectly with no docker wrapper, and those did publish successfully (v0.7.12 confirmed live) despite also showing a CI "failure" status, but that one is the already-known broken GH_RELEASE_TOKEN upload step, unrelated to this.So there appear to be two separate problems under this one ticket: (1) the fixed non-mktemp cache path race hypothesis (still plausible, not confirmed), and (2) a confirmed, consistent docker-connectivity failure specific to workflows that route through ci/build.sh's docker wrapper (bread confirmed broken; breadbox/breadcrumbs/breadbar/breadpaper/breadhelp/breadpad/breadclip/breadmon/breadshot all use the same wrapper and may be equally affected, not yet individually confirmed). Needs the user to check the actual runner (hestia) directly, docker daemon status and any recent load/crash around 2026-09-11 19:00-19:20 UTC+8, since this is outside what I should investigate without direct runner access.
Confirmed the exact mechanism, not just the hypothesis. Every dev/rc/release workflow's "regenerate index.json" step runs:
rm -rf /tmp/bread-ecosystem-ci-* 2>/dev/null || true
ECOSYSTEM_CI_DIR="$(mktemp -d /tmp/bread-ecosystem-ci-XXXXXX)"
The rm -rf wildcard deletes every directory matching that prefix, including other concurrently-running jobs' own mktemp'd directories, not just stale leftovers from itself. Caught breadbar's v0.4.0-rc.1 failing on exactly this: "error: could not lock config file /tmp/bread-ecosystem-ci-4AXOdG/.git/config: No such file or directory" while another job's index-regen step was mid-clone into that same directory and got rm -rf'd out from under it by a sibling job.
Fix: drop the rm -rf entirely (mktemp already guarantees a fresh, unique path, there is nothing to clean first) or scope the rm to something you would never share with a concurrent job.
Separately, also confirmed a real bug in bos-settings specifically (not part of this race): its rc-release.yml was missing the "build frontend" and "build" steps entirely that dev-release.yml and release.yml both have, so every beta release failed instantly trying to cp a binary that was never built. Fixed directly (PR #14, merged) since it was unrelated to the race and blocking this repo's promotion.
One more finding: retagging v0.9.0-rc.2 at the same commit as the original failed rc.1 (after merging the rc-release.yml fix to main) still failed with the identical "cp: cannot stat" error, even though main's rc-release.yml had the fix. This means the docs/release-channels.md claim that tag-triggered workflows always resolve their YAML from the default branch does not hold in practice, at least not reliably. Had to retag rc.3 pointing directly at a commit that itself includes the fix for it to actually take effect. Worth another look separately: either the doc is wrong, or there is a caching/resolution edge case around re-pushing a tag at an old commit after main moved on.
Fresh main confirms the confirmed wildcard race is outside this repo now: the bread-ecosystem
ci/build.shand its own index workflows no longer contain/tmp/bread-ecosystem-ci-*cleanup. The currentBreadway/breaddev-release workflow still runsrm -rf /tmp/bread-ecosystem-ci-*immediately before creating its uniquemktempdirectory (index regeneration step, lines 87-90). That wildcard can still delete sibling jobs' active directories. Removing the wildcard and cleaning up only the invocation's own temp directory is the direct fix, but the affected YAML is copied across separate consumer repos, so one PR to bread-ecosystem will not fix them all. The Docker socket failure in the issue body is separate and needs runner-side diagnosis. Please choose whether to split the workflow cleanup into per-repo issues/PRs, or track only the runner Docker outage here. Leaving this in TODO for direction.I’m choosing to split the confirmed wildcard cleanup race into per-repository issues and PRs so each copied workflow gets its own fix and CI. The Docker socket outage is a separate runner-side blocker; this task will not modify Hestia or runner configuration. I’ll keep this umbrella issue open until all confirmed consumer copies are fixed or have an explicit blocker.
Queue status update: all currently accessible consumer workflow fixes are merged, but PAD-16 remains blocked because this runtime cannot fetch current Breadpad main (
socket: Operation not permittedto Forgejo). There is no active ECO-28 implementation work until that fetch/publish blocker changes, so the umbrella is moving to TODO. Keep it open until PAD-16 and any remaining consumer copies are resolved; the separate Docker socket runner outage remains outside scope.The final known consumer cleanup, PAD-16, is now implemented at Breadway/breadpad#17 (b4be88667f18f0743e0e464f986dcd8949cfee2b). Forgejo check CI succeeded in 5m31s. The two tag release copies use invocation-specific mktemp paths and EXIT cleanup; isolated concurrent success/failure tests preserve sibling directories. Prior comments record the other consumer fixes as merged.
Moving this umbrella to REVIEW alongside its final linked consumer PR. Keep it open until independent review accepts PAD-16 and the reviewer confirms the consumer coverage. The historical runner Docker failure was separated from the confirmed wildcard cleanup race in prior scope decisions; no runner or user configuration was changed.
Independent consumer coverage review found an unresolved mainline copy of the destructive wildcard cleanup. PAD-16 PR #17 at b4be886 is correctly scoped and green, but it is not the final ecosystem consumer fix.
I fetched Breadway/bread's configured authoritative origin. Current main is
47745812c41a29b2b2f2329f85fe86af33c5c82b. It still executesrm -rf /tmp/bread-ecosystem-ci-*in:ci/build.sh:16.forgejo/workflows/dev-release.yml:87.forgejo/workflows/rc-release.yml:58.forgejo/workflows/release.yml:51Acceptance interpretation: consumer helpers and release steps must not delete another concurrently active checkout. These four paths still violate that criterion.
Rework checklist:
Root and the implementation owner have been notified. This umbrella remains open and returns to TODO for the newly confirmed missing coverage; PAD-16 can remain REVIEW independently. No code or live configuration was changed.
The remaining bread consumer copy is now tracked as BREAD-8 with PR Breadway/bread#9. The helper and all three release index steps were fixed in that PR. Isolated concurrent clone and success/failure checks passed; Forgejo CI and independent review remain pending. ECO-28 stays TODO until consumer coverage is confirmed and the child PR is accepted.
Authoritative mainline coverage check after opening BREAD-8 found additional rc-release.yml wildcard cleanup in breadbar, breadclip, breadmon, and breadshot. Breadpad also still has the wildcard on main while PAD-16 PR #17 awaits review. I checked ci/build.sh plus dev, rc, and stable release workflows for breadbox, breadcrumbs, breadbar, breadpaper, breadhelp, breadpad, breadclip, breadmon, and breadshot via Forgejo. The four newly confirmed rc copies need follow-up child issues/PRs; the umbrella remains TODO. BREAD-8 PR #9 has only a pending check as of this comment.
Mainline coverage recheck for the confirmed wildcard cleanup race:
.forgejo/workflows/rc-release.yml:47. PAD-16 / PR #17 is green and mergeable, but needs its linked board card synchronized through review.rm -rf /tmp/bread-ecosystem-ci-*in their mainrc-release.ymlfiles. The wildcard is immediately before a distinctmktemp -d /tmp/bread-ecosystem-ci-XXXXXX, so it can remove a sibling job's active checkout.The bread-ecosystem shared helper and its own index workflows do not contain this wildcard now. ECO-28 remains TODO because the four remaining consumer workflows still need reviewable fixes. Please track and fix each affected consumer on its own branch, then recheck the umbrella after those changes land. The separate runner Docker issue is not part of this remaining source finding.
Correction to the PAD-16 status note above: current Breadboard state shows both PAD-16 and synced PR card PAD-17 already in REVIEW. PR #17 is green and mergeable. The remaining mainline wildcard copies are still present in breadbar, breadclip, breadmon and breadshot RC workflows; PAD-16's PR has not landed, so its mainline copy remains until that review is completed.
Coverage checkpoint: the newly confirmed consumer copies are tracked under ECO-56, linked to this umbrella. BAR-50 PR #51 is independently reviewed, merged, and DONE. BREAD-8/#9 and PAD-16/#17 are in REVIEW with green checks. CLIP-17/#18, MON-11/#12, and SHOT-10/#11 have separate scoped PRs with local acceptance passing; their Forgejo checks are queued, so the cards remain DOING. ECO-28 stays TODO until the remaining PRs pass CI, receive independent review, and consumer coverage is confirmed.
The known consumer wildcard-cleanup copies are now addressed: bread, breadpad, breadbar, breadclip, breadmon, and breadshot. BAR-50 is merged, BREAD-8 and PAD-16 have green CI, and CLIP-17, MON-11, and SHOT-10 have scoped PRs with local acceptance and queued CI. Moving this umbrella into REVIEW for coverage review. If any child CI fails, return that child to TODO with the failure recorded.
The cache-path race this issue tracks is fixed and independently reviewed across all consumers.
On the docker-connectivity observation recorded in the body: it was never independently reproduced, and the runner's docker route is demonstrably working now. breadbar's check run for #55 went through ci/build.sh's docker wrapper and succeeded in 10m57s today (2026-10-02). The rc.1/2/3 failures from 2026-09-11 were never attributed with logs, so if they recur that should be a fresh issue with runner evidence rather than an open question here.
Closing as done: every shared temp-path copy is now scoped to its own invocation and reviewed.