Fix test harness process leak: graceful SIGTERM shutdown + disable podman adapter in test fixtures
All checks were successful
dev release / build (push) Successful in 1m3s

The integration test harness (ipc_integration.rs and
module_host_sandbox.rs) tore down spawned breadd instances via
Child::kill() (SIGKILL). Since breadd's PodmanAdapter is enabled by
default and the test breadd.toml fixtures didn't disable it, every
test-spawned daemon on a machine with podman installed ran a real
`podman events --format json` child. SIGKILL gives breadd zero chance
to run its own graceful shutdown path (wait_for_shutdown -> adapter
tasks dropping -> kill_on_drop firing on that child), so it was
orphaned and reparented to init on every single test run instead.

A day of repeated `cargo test --workspace` runs during today's
workstream development left 1,559 such orphaned processes consuming
76.8GB of RSS system-wide, found and cleaned up separately.

Fix is two-layered: disable the podman adapter in both harnesses'
breadd.toml fixtures (tests don't need real container-lifecycle
watching, matching how hyprland/udev/power/network are already
disabled there), and change TestHarness::drop to send SIGTERM with a
bounded wait before falling back to SIGKILL, so breadd's own cleanup
gets a real chance to run - defense in depth against any future
adapter that spawns a subprocess.

Verified: two full `cargo test --workspace` runs (282 tests) produce
zero new podman events processes and zero leftover breadd/
bread-module-host processes.
This commit is contained in:
Breadway 2026-08-05 06:45:20 +08:00
parent d26861697c
commit 50ff425d2a
2 changed files with 59 additions and 0 deletions

View file

@ -1448,6 +1448,9 @@ enabled = false
[adapters.network] [adapters.network]
enabled = false enabled = false
[adapters.podman]
enabled = false
"#, "#,
)?; )?;
@ -1524,6 +1527,9 @@ enabled = false
[adapters.network] [adapters.network]
enabled = false enabled = false
[adapters.podman]
enabled = false
"#, "#,
)?; )?;
@ -1678,6 +1684,34 @@ impl Drop for TestHarness {
/// failure into cascading slowdowns/timeouts across the whole suite /// failure into cascading slowdowns/timeouts across the whole suite
/// (observed directly while developing Workstream G's tests). /// (observed directly while developing Workstream G's tests).
fn drop(&mut self) { fn drop(&mut self) {
// SIGTERM first, not straight to SIGKILL: `breadd`'s own
// `wait_for_shutdown()` (src/main.rs) listens for SIGTERM and runs
// a graceful shutdown path — `lua_runtime.shutdown()`,
// `module_host_registry.shutdown_all()`, and, as `main()` returns,
// every adapter task getting dropped, which is what actually fires
// `kill_on_drop` on a still-running child process like
// `PodmanAdapter`'s `podman events` watcher. SIGKILL bypasses all of
// that — the process is torn down before any of its own code, Drop
// impls included, ever runs. That gap is exactly how a single day
// of repeated `cargo test --workspace` runs left 1,559 orphaned
// `podman events` processes system-wide (found and cleaned up
// separately; see also the `[adapters.podman] enabled = false`
// fixtures this file now sets, which close the same hole from the
// other side). Bounded wait, then SIGKILL as a fallback so a
// genuinely wedged `breadd` doesn't hang the test suite.
unsafe {
libc::kill(self.child.id() as libc::pid_t, libc::SIGTERM);
}
let deadline = Instant::now() + Duration::from_secs(2);
loop {
match self.child.try_wait() {
Ok(Some(_)) => return,
Ok(None) if Instant::now() < deadline => {
std::thread::sleep(Duration::from_millis(20));
}
_ => break,
}
}
let _ = self.child.kill(); let _ = self.child.kill();
let _ = self.child.wait(); let _ = self.child.wait();
} }

View file

@ -106,6 +106,9 @@ enabled = false
[adapters.network] [adapters.network]
enabled = false enabled = false
[adapters.podman]
enabled = false
"#, "#,
)?; )?;
@ -255,6 +258,28 @@ impl Drop for TestHarness {
/// `shutdown()`, is what keeps a failed test run from leaving orphaned /// `shutdown()`, is what keeps a failed test run from leaving orphaned
/// sandboxed processes behind for the next run to trip over. /// sandboxed processes behind for the next run to trip over.
fn drop(&mut self) { fn drop(&mut self) {
// SIGTERM first, not straight to SIGKILL — see the matching comment
// in `ipc_integration.rs`'s `TestHarness::drop`. SIGKILL prevents
// `breadd` from ever running its own graceful shutdown path, which
// is what actually fires `kill_on_drop` on adapter-spawned child
// processes (e.g. `PodmanAdapter`'s `podman events` watcher) —
// exactly how a day of repeated `cargo test --workspace` runs left
// 1,559 orphaned `podman events` processes system-wide. Bounded
// wait, then SIGKILL as a fallback so a genuinely wedged `breadd`
// doesn't hang the test suite.
unsafe {
libc::kill(self.child.id() as libc::pid_t, libc::SIGTERM);
}
let deadline = Instant::now() + Duration::from_secs(2);
loop {
match self.child.try_wait() {
Ok(Some(_)) => return,
Ok(None) if Instant::now() < deadline => {
std::thread::sleep(Duration::from_millis(20));
}
_ => break,
}
}
let _ = self.child.kill(); let _ = self.child.kill();
let _ = self.child.wait(); let _ = self.child.wait();
} }