Fix a real production incident: unbounded concurrent transcode encodes
The daemon's steady-state transcode ticker and a manually-launched transcode-library backfill process were both independently claiming pending jobs from the same queue with no shared concurrency awareness, each capping only at its own parallelism_max. Combined with large files easily outlasting the 30s poll interval, this stacked to 20+ simultaneous GPU encode/decode sessions and triggered the kernel OOM killer on a shared 15GB host running a dozen+ other containers (confirmed via dmesg; no services were lost, no library files were touched — the original-file-safety design held under the crash). Fixes: claim_pending_jobs now treats its limit as a total concurrency cap (subtracting already-running jobs, wrapped in a BEGIN IMMEDIATE transaction so this is correct across concurrent processes touching the same database, not just within one). Added reset_orphaned_running_jobs, called on real daemon startup, so a crash never permanently strands job slots in 'running'. Lowered the default parallelism_max from 4 to 2 given the observed real-world memory pressure.
This commit is contained in:
parent
d533301880
commit
576aad3bfe
3 changed files with 188 additions and 31 deletions
|
|
@ -364,7 +364,13 @@ fn default_parallelism_min() -> usize {
|
|||
}
|
||||
|
||||
fn default_parallelism_max() -> usize {
|
||||
4
|
||||
// Conservative on purpose: a real incident on a shared 15GB host
|
||||
// running a dozen+ other containers showed concurrent 1080p/4K
|
||||
// decode+encode sessions can push memory pressure into swap fast
|
||||
// enough to trigger the OOM killer well before the GPU itself is the
|
||||
// bottleneck. Raise this deliberately, per-deployment, once you've
|
||||
// watched `free -h` under real load at the current setting.
|
||||
2
|
||||
}
|
||||
|
||||
fn default_reference_bitrate_kbps() -> u32 {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue