Server ignores SIGTERM, so every container stop waits 10s and ends in SIGKILL #78

New issue

Closed

opened 2026-10-01 16:22:06 +08:00 by Breadway · 2 comments

Breadway commented

2026-10-01 16:22:06 +08:00

Owner

How found: reading crates/breadboard-server/src/main.rs while planning a low-downtime deploy of PR #76.

Confidence: high for the SIGTERM part (code is unambiguous and Docker's PID 1 semantics are well known); medium for the SSE hang (follows from axum graceful shutdown semantics, not measured).

Explanation: shutdown_signal() only awaits tokio::signal::ctrl_c() (SIGINT). docker stop / docker compose up recreation sends SIGTERM. The server runs as PID 1 in the container, and PID 1 has no default SIGTERM action, so the signal is ignored. Docker then waits its 10s timeout and SIGKILLs. Result: every deploy or restart is about 10s of downtime instead of under 1s. Separately, even with SIGTERM handled, with_graceful_shutdown waits for in-flight responses, and /events SSE streams never end on their own, so shutdown would still hang until the kill while any browser tab is open.

Fix idea: also listen for SignalKind::terminate(), and on shutdown end the SSE streams (e.g. a shutdown watch/CancellationToken that the events stream selects on), so open tabs disconnect immediately and EventSource reconnects to the new container. Optionally add stop_grace_period / STOPSIGNAL in the Containerfile/compose.

Workaround used meanwhile: docker compose up -d --timeout 1 (SQLite WAL is crash-safe).

**How found:** reading `crates/breadboard-server/src/main.rs` while planning a low-downtime deploy of PR #76. **Confidence:** high for the SIGTERM part (code is unambiguous and Docker's PID 1 semantics are well known); medium for the SSE hang (follows from axum graceful shutdown semantics, not measured). **Explanation:** `shutdown_signal()` only awaits `tokio::signal::ctrl_c()` (SIGINT). `docker stop` / `docker compose up` recreation sends SIGTERM. The server runs as PID 1 in the container, and PID 1 has no default SIGTERM action, so the signal is ignored. Docker then waits its 10s timeout and SIGKILLs. Result: every deploy or restart is about 10s of downtime instead of under 1s. Separately, even with SIGTERM handled, `with_graceful_shutdown` waits for in-flight responses, and `/events` SSE streams never end on their own, so shutdown would still hang until the kill while any browser tab is open. **Fix idea:** also listen for `SignalKind::terminate()`, and on shutdown end the SSE streams (e.g. a shutdown `watch`/`CancellationToken` that the events stream selects on), so open tabs disconnect immediately and EventSource reconnects to the new container. Optionally add `stop_grace_period` / `STOPSIGNAL` in the Containerfile/compose. **Workaround used meanwhile:** `docker compose up -d --timeout 1` (SQLite WAL is crash-safe).

Breadway added the

deploy

bug

labels

2026-10-01 16:22:07 +08:00

Breadway referenced this issue from a pull request that will close it,

2026-10-01 17:50:05 +08:00

Server ignores SIGTERM, so every container stop waits 10s and ends in SIGKILL #81

Breadway commented

2026-10-01 17:50:25 +08:00

Author

Owner

Implemented in PR #81: #81. The server now handles SIGTERM and closes active SSE streams during graceful shutdown. cargo build --workspace, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, cargo run -p xtask -- check-docs, and git diff --check pass. cargo fmt --all -- --check reports pre-existing formatting drift in untouched files.

Implemented in PR #81: https://git.breadway.dev/Breadway/breadboard/pulls/81. The server now handles SIGTERM and closes active SSE streams during graceful shutdown. `cargo build --workspace`, `cargo test --workspace`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo run -p xtask -- check-docs`, and `git diff --check` pass. `cargo fmt --all -- --check` reports pre-existing formatting drift in untouched files.

Breadway commented

2026-10-08 19:46:33 +08:00

Author

Owner

Board cleanup 2026-10-08: rebased PR #81 onto main (707b758). One conflict in http.rs: main's BB-90 refactor moved SSE mapping into event_stream(), so the shutdown take_until now wraps event_stream(...) and keeps the lag-ends-stream behaviour. After the rebase the SIGTERM integration test got a 401 because BB-93 made /events require auth. Fixed by minting a bearer token in the server DB once it is healthy. Workspace tests and server clippy pass. The remaining cargo fmt --check drift is pre-existing on main and not from this PR. Reviewed the signal handling: SIGTERM and SIGINT both trigger shutdown, the watch flag closes open SSE streams so graceful shutdown no longer hangs, and the non-unix fallback is intact. Ready to merge once CI is green.

Board cleanup 2026-10-08: rebased PR #81 onto main (707b758). One conflict in `http.rs`: main's BB-90 refactor moved SSE mapping into `event_stream()`, so the shutdown `take_until` now wraps `event_stream(...)` and keeps the lag-ends-stream behaviour. After the rebase the SIGTERM integration test got a 401 because BB-93 made `/events` require auth. Fixed by minting a bearer token in the server DB once it is healthy. Workspace tests and server clippy pass. The remaining `cargo fmt --check` drift is pre-existing on main and not from this PR. Reviewed the signal handling: SIGTERM and SIGINT both trigger shutdown, the watch flag closes open SSE streams so graceful shutdown no longer hangs, and the non-unix fallback is intact. Ready to merge once CI is green.

Breadway closed this issue

2026-10-08 19:54:27 +08:00