CI consumer checkout clones every job into the same src path, so concurrent jobs swap the tree under a running build #78

New issue

Closed

opened 2026-10-16 16:26:18 +08:00 by Breadway · 2 comments

Breadway commented

2026-10-16 16:26:18 +08:00

Owner

Found while diagnosing bread PR24 (run 157, feature/breadbug-watchdog at bbb12d6383) and PR44 (run 158, fix/bread-42-hyprland-host-bridge at 71c469b452).

Symptom: run 157's test compile failed with unresolved bread_bug::livestack, bread_bug::procid, store::ProcessIdentity and MarkerKind::Stopped. bbb12d6383 defines all of them. Clippy and the build step in the same run passed.

Confidence: high that the consumer checkout step is unsafe when jobs share a runner. Reproduced locally with two branch fixtures cloned into one shared src path: the second clone replaced the tree under the first job. Medium-high that this caused run 157: its failing compile saw lib.rs without the livestack/procid modules (matches 71c469b452) but bread-bug/tests/livestack_real.rs present (only in bbb12d6383). No single commit has that mix. The runner workspace is not readable from hermes, so the exact interleaving is not confirmed.

Mechanism:

Each consumer check.yml (bread, breadbar, breadbox, breadcrumbs, breadpad, ...) runs rm -rf src && mkdir src && git clone --branch "$GITHUB_REF_NAME" --depth 1 ... src as a step before ci/build.sh.
Jobs on the same self-hosted runner clone to the same relative path, so the workspace is shared.
The per-product flock added under ECO-54 covers only the docker build and run inside ci/build.sh. It does not cover the checkout, so it cannot stop the swap.
The clone is by branch name, not by SHA, so the tree is whatever the branch tip is when the step runs.
Timeline on the runner: run 157 checked out at 06:41:12 and passed check/build at 06:41:20. Run 158 checked out at 06:43:41 while run 157 waited for the build lock (06:44:17). Run 157's test compile ran after its lock wait and failed at 07:01:35.

Not a duplicate of ECO-54, ECO-28 or BREAD-41: those cover the build.sh lock and cache. This is the checkout step before the lock.

Proposed fix (prerequisite: a shared helper in bread-ecosystem, since the pattern is copied across consumer repos):

Clone each job into its own directory (for example mktemp -d) and fetch and check out the event SHA rather than the branch name.
Keep ci/build.sh's per-product lock unchanged so same-product serialization and cross-product parallelism stay as they are.

Regression: two concurrent fixture jobs on different branches must each see their own tree; the existing ci/test-build-lock.sh must still pass.

Evidence: https://git.breadway.dev/Breadway/bread/actions/runs/157 and /158 (job logs at .../jobs/0/attempt/1/logs).

Found while diagnosing bread PR24 (run 157, feature/breadbug-watchdog at bbb12d6383) and PR44 (run 158, fix/bread-42-hyprland-host-bridge at 71c469b452). Symptom: run 157's test compile failed with unresolved bread_bug::livestack, bread_bug::procid, store::ProcessIdentity and MarkerKind::Stopped. bbb12d6383 defines all of them. Clippy and the build step in the same run passed. Confidence: high that the consumer checkout step is unsafe when jobs share a runner. Reproduced locally with two branch fixtures cloned into one shared src path: the second clone replaced the tree under the first job. Medium-high that this caused run 157: its failing compile saw lib.rs without the livestack/procid modules (matches 71c469b452) but bread-bug/tests/livestack_real.rs present (only in bbb12d6383). No single commit has that mix. The runner workspace is not readable from hermes, so the exact interleaving is not confirmed. Mechanism: - Each consumer check.yml (bread, breadbar, breadbox, breadcrumbs, breadpad, ...) runs `rm -rf src && mkdir src && git clone --branch "$GITHUB_REF_NAME" --depth 1 ... src` as a step before ci/build.sh. - Jobs on the same self-hosted runner clone to the same relative path, so the workspace is shared. - The per-product flock added under ECO-54 covers only the docker build and run inside ci/build.sh. It does not cover the checkout, so it cannot stop the swap. - The clone is by branch name, not by SHA, so the tree is whatever the branch tip is when the step runs. - Timeline on the runner: run 157 checked out at 06:41:12 and passed check/build at 06:41:20. Run 158 checked out at 06:43:41 while run 157 waited for the build lock (06:44:17). Run 157's test compile ran after its lock wait and failed at 07:01:35. Not a duplicate of ECO-54, ECO-28 or BREAD-41: those cover the build.sh lock and cache. This is the checkout step before the lock. Proposed fix (prerequisite: a shared helper in bread-ecosystem, since the pattern is copied across consumer repos): - Clone each job into its own directory (for example mktemp -d) and fetch and check out the event SHA rather than the branch name. - Keep ci/build.sh's per-product lock unchanged so same-product serialization and cross-product parallelism stay as they are. Regression: two concurrent fixture jobs on different branches must each see their own tree; the existing ci/test-build-lock.sh must still pass. Evidence: https://git.breadway.dev/Breadway/bread/actions/runs/157 and /158 (job logs at .../jobs/0/attempt/1/logs).

Breadway added the

label

2026-10-16 16:26:18 +08:00

Breadway commented

2026-10-16 16:53:16 +08:00

Author

Owner

Independent review of PR 79 at de66355a16: CHANGES REQUESTED, not merged. Full checklist (5 required items, non-blocking notes) is on the PR: #79 . Summary: local regression, build-lock and pin-lint pass and the exact-head CI check is green, but (1) test-ci-checkout.sh is not run by check.yml, (2) token-only URL userinfo is accepted, (3) workflow inlines the clone without the helper's HEAD verification, (4) no documented consumer path to obtain the helper, (5) vacuous ssh test case.

Independent review of PR 79 at de66355a16f532eb8419f683df09b7c02ee6dee0: CHANGES REQUESTED, not merged. Full checklist (5 required items, non-blocking notes) is on the PR: https://git.breadway.dev/Breadway/bread-ecosystem/pulls/79 . Summary: local regression, build-lock and pin-lint pass and the exact-head CI check is green, but (1) test-ci-checkout.sh is not run by check.yml, (2) token-only URL userinfo is accepted, (3) workflow inlines the clone without the helper's HEAD verification, (4) no documented consumer path to obtain the helper, (5) vacuous ssh test case.

Breadway commented

2026-10-17 14:03:52 +08:00

Author

Owner

Coordinator verification of the accepted fix. Shared helper PR79 independently passed (comment 5354) and merged c6442cabb6; bread-ecosystem main check on that merge is success (4m46s). Bread consumer PR49 accepted (comment 5308) and merged 40ee494aef39981943c6141c547c104968d93b03; its post-merge dev-release run 162 is success, and the watchdog branch now builds green on the same tree (PR24 head 834c07e check success 17m37s). So the reproducing consumer (bread) plus the shared helper are both fixed and green. Residual, not silently claimed: other consumers (breadbar, breadbox, breadcrumbs, breadpad and bread-ecosystem's own main-trigger workflows) still use the old fixed src path; ECO-81 and ECO-82 track two of them and a new card tracks the rest.

Coordinator verification of the accepted fix. Shared helper PR79 independently passed (comment 5354) and merged c6442cabb6c02acd1380dcd0d17189c4e22def03; bread-ecosystem main check on that merge is success (4m46s). Bread consumer PR49 accepted (comment 5308) and merged 40ee494aef39981943c6141c547c104968d93b03; its post-merge dev-release run 162 is success, and the watchdog branch now builds green on the same tree (PR24 head 834c07e check success 17m37s). So the reproducing consumer (bread) plus the shared helper are both fixed and green. Residual, not silently claimed: other consumers (breadbar, breadbox, breadcrumbs, breadpad and bread-ecosystem's own main-trigger workflows) still use the old fixed src path; ECO-81 and ECO-82 track two of them and a new card tracks the rest.

Breadway closed this issue

2026-10-17 14:03:52 +08:00