Fix a real production incident: unbounded concurrent transcode encodes

The daemon's steady-state transcode ticker and a manually-launched
transcode-library backfill process were both independently claiming
pending jobs from the same queue with no shared concurrency
awareness, each capping only at its own parallelism_max. Combined
with large files easily outlasting the 30s poll interval, this
stacked to 20+ simultaneous GPU encode/decode sessions and triggered
the kernel OOM killer on a shared 15GB host running a dozen+ other
containers (confirmed via dmesg; no services were lost, no library
files were touched — the original-file-safety design held under the
crash).

Fixes: claim_pending_jobs now treats its limit as a total concurrency
cap (subtracting already-running jobs, wrapped in a BEGIN IMMEDIATE
transaction so this is correct across concurrent processes touching
the same database, not just within one). Added reset_orphaned_running_jobs,
called on real daemon startup, so a crash never permanently strands
job slots in 'running'. Lowered the default parallelism_max from 4
to 2 given the observed real-world memory pressure.
This commit is contained in:
Breadway 2026-07-25 00:20:05 +08:00
parent d533301880
commit 576aad3bfe
3 changed files with 188 additions and 31 deletions

View file

@ -364,7 +364,13 @@ fn default_parallelism_min() -> usize {
}
fn default_parallelism_max() -> usize {
4
// Conservative on purpose: a real incident on a shared 15GB host
// running a dozen+ other containers showed concurrent 1080p/4K
// decode+encode sessions can push memory pressure into swap fast
// enough to trigger the OOM killer well before the GPU itself is the
// bottleneck. Raise this deliberately, per-deployment, once you've
// watched `free -h` under real load at the current setting.
2
}
fn default_reference_bitrate_kbps() -> u32 {