Breadway/bread-ecosystem
1
0
Fork 0
Code Issues 8 Pull requests 1 Projects Releases Packages Wiki Activity Actions

ci: serialize builds per product in the shared build.sh #42

Merged
Breadway merged 1 commit from fix/ci-serialize-builds-per-product into main 2026-09-29 08:40:12 +08:00
Conversation 0 Commits 1 Files changed 1 +30 -5
Breadway commented 2026-09-29 08:38:18 +08:00
Owner
Copy link

Follow-up to ECO-38.

Two jobs for the same product (a dev-release and a check, say) share the target-<product> cache directory. While one job's cargo rewrites incremental/, the other's final cp -a /cargo-target/. target/ fails with cannot stat '/cargo-target/./debug/incremental/...'. This was seen on breadcast during the ECO-38 rollout, and it is why one run there needed a retrigger.

ci/build.sh now takes an flock on target-<product>.lock before the docker build and holds it until the script exits.

  • Different products never contend.
  • It waits up to BREAD_CI_LOCK_WAIT seconds (default 3600), then fails with a clear message.
  • It warns and continues unserialized if flock is not installed.
  • CACHE_ROOT and its directories are now created before the lock, since the lock file lives there.

Tested with a fake docker on PATH: two builds of one product ran back to back while a build of another product ran alongside them, and the timeout path exits 1 without starting a build.

Consumers pick this up when their ci/bread-ecosystem.rev pin is bumped to the merged commit.

Follow-up to ECO-38. Two jobs for the same product (a dev-release and a check, say) share the `target-<product>` cache directory. While one job's cargo rewrites `incremental/`, the other's final `cp -a /cargo-target/. target/` fails with `cannot stat '/cargo-target/./debug/incremental/...'`. This was seen on breadcast during the ECO-38 rollout, and it is why one run there needed a retrigger. `ci/build.sh` now takes an `flock` on `target-<product>.lock` before the docker build and holds it until the script exits. - Different products never contend. - It waits up to `BREAD_CI_LOCK_WAIT` seconds (default 3600), then fails with a clear message. - It warns and continues unserialized if `flock` is not installed. - `CACHE_ROOT` and its directories are now created before the lock, since the lock file lives there. Tested with a fake `docker` on PATH: two builds of one product ran back to back while a build of another product ran alongside them, and the timeout path exits 1 without starting a build. Consumers pick this up when their `ci/bread-ecosystem.rev` pin is bumped to the merged commit.
ci: serialize builds per product in the shared build.sh
All checks were successful
check / check (push) Successful in 1m39s
Details
0f04d88408 
Two jobs for the same product (a dev-release and a check, say) share
the target-<product> cache directory under CACHE_ROOT. While one job's
cargo rewrites incremental/, the other's final `cp -a /cargo-target/.
target/` fails with "cannot stat '/cargo-target/./debug/incremental/...'"
(seen on breadcast during the ECO-38 rollout).

build.sh now takes an flock on target-<product>.lock before the docker
build and holds it until the script exits. Different products never
contend. It waits up to BREAD_CI_LOCK_WAIT seconds (default 3600) and
then fails with a clear message, and it warns and carries on unserialized
if flock is not installed. CACHE_ROOT and its directories are now created
before the lock, since the lock file lives there.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bread-theme

   
breadbug

   
bug

   
ci

   
enhancement

   
phase-2

   
release

   
rollout
No labels
bread-theme
breadbug
bug
ci
enhancement
phase-2
release
rollout
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Breadway/bread-ecosystem!42
No description provided.