shutdown test still flakes: sigterm_exits_promptly_and_closes_sse_streams misses the 5s startup budget on a loaded runner #102

New issue

Open

opened 2026-10-20 17:47:34 +08:00 by Breadway · 0 comments

Breadway commented

2026-10-20 17:47:34 +08:00

Owner

How it was found

While reviewing PR 101 (BB-114) the required check failed twice on unrelated heads (261a59d6 and an empty re-run commit acec0189). The user supplied the CI log. Clippy, the full workspace tests, check-docs, scripts/test-publish-bakery-stage.sh and the web build all passed locally on the same code, and main passed at 2a538f08.

What the log shows

tests/shutdown.rs, test sigterm_exits_promptly_and_closes_sse_streams: "server did not become healthy within 5s (elapsed 5.001s): child never logged its bound address". The child printed "starting breadboard-server" and "db: first connection open (WAL set) elapsed_ms=235" and then nothing before the deadline. The other seven tests in that file passed. Failing step took 3m42s overall.

Confidence

Medium-high that this is the same cause BB-104 and BB-105 described (slow cold SQLite startup: WAL plus six migrations is about 21 fsyncs before bind) and not a regression in PR 101: that PR changes only crates/breadboard-server/src/import.rs and its test file. Not proven for these runs because the log has no stage lines after the first connection. The hestia host load average was 11 to 17 on 12 cores at the time, with a qemu guest and other services running, which would make fsync-heavy startup slow.

Explanation

BB-105 made the test deadline-checked and diagnosable but kept the 5s budget, so a slow runner still fails the required check. Each failure blocks merges that are otherwise green. Candidates, not decided: log the later startup stages ("migrations done", "seed checked") so the stalled stage is visible in CI, and give the startup budget headroom or make it configurable for CI, without changing the 2s SIGTERM and 1s SSE budgets that the test actually asserts.

Related: BB-104, BB-105 (both done).

## How it was found While reviewing PR 101 (BB-114) the required check failed twice on unrelated heads (261a59d6 and an empty re-run commit acec0189). The user supplied the CI log. Clippy, the full workspace tests, check-docs, scripts/test-publish-bakery-stage.sh and the web build all passed locally on the same code, and main passed at 2a538f08. ## What the log shows tests/shutdown.rs, test sigterm_exits_promptly_and_closes_sse_streams: "server did not become healthy within 5s (elapsed 5.001s): child never logged its bound address". The child printed "starting breadboard-server" and "db: first connection open (WAL set) elapsed_ms=235" and then nothing before the deadline. The other seven tests in that file passed. Failing step took 3m42s overall. ## Confidence Medium-high that this is the same cause BB-104 and BB-105 described (slow cold SQLite startup: WAL plus six migrations is about 21 fsyncs before bind) and not a regression in PR 101: that PR changes only crates/breadboard-server/src/import.rs and its test file. Not proven for these runs because the log has no stage lines after the first connection. The hestia host load average was 11 to 17 on 12 cores at the time, with a qemu guest and other services running, which would make fsync-heavy startup slow. ## Explanation BB-105 made the test deadline-checked and diagnosable but kept the 5s budget, so a slow runner still fails the required check. Each failure blocks merges that are otherwise green. Candidates, not decided: log the later startup stages ("migrations done", "seed checked") so the stalled stage is visible in CI, and give the startup budget headroom or make it configurable for CI, without changing the 2s SIGTERM and 1s SSE budgets that the test actually asserts. Related: BB-104, BB-105 (both done).

Breadway added the

bug

labels

2026-10-20 17:47:34 +08:00