Breadway/breadarr
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix a real production incident: unbounded concurrent transcode encodes

Browse source 
The daemon's steady-state transcode ticker and a manually-launched
transcode-library backfill process were both independently claiming
pending jobs from the same queue with no shared concurrency
awareness, each capping only at its own parallelism_max. Combined
with large files easily outlasting the 30s poll interval, this
stacked to 20+ simultaneous GPU encode/decode sessions and triggered
the kernel OOM killer on a shared 15GB host running a dozen+ other
containers (confirmed via dmesg; no services were lost, no library
files were touched — the original-file-safety design held under the
crash).

Fixes: claim_pending_jobs now treats its limit as a total concurrency
cap (subtracting already-running jobs, wrapped in a BEGIN IMMEDIATE
transaction so this is correct across concurrent processes touching
the same database, not just within one). Added reset_orphaned_running_jobs,
called on real daemon startup, so a crash never permanently strands
job slots in 'running'. Lowered the default parallelism_max from 4
to 2 given the observed real-world memory pressure.
This commit is contained in:
Breadway 2026-07-25 00:20:05 +08:00
parent d533301880
commit 576aad3bfe
3 changed files with 188 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

15
breadarrd/src/main.rs
View file

@ -142,6 +142,11 @@ async fn run_daemon(config: Config) -> Result<()> {
let conn = Connection::open(config.db_path())?;
db::init(&conn)?;
info!(path = %config.db_path().display(), "database ready");
match transcode::reset_orphaned_running_jobs(&conn) {
Ok(0) => {}
Ok(n) => info!(n, "reset orphaned 'running' transcode jobs left over from a previous crash"),
Err(e) => tracing::warn!(error = %e, "failed to reset orphaned transcode jobs"),
}
// A second, independent connection for the HTTP API rather than sharing
// `background_loop`'s. Both point at the same on-disk (WAL-mode)
@ -1209,6 +1214,16 @@ async fn probe_library_cmd(config: &Config) -> Result<()> {
/// (`transcode::run_cycle`) until nothing is left pending. Shares that one
/// code path deliberately — there is exactly one place that actually runs
/// an encode, whether triggered by a backlog sweep or a fresh grab.
// Deliberately does NOT call `transcode::reset_orphaned_running_jobs` on
// startup the way `run_daemon` does — from this CLI's vantage point a
// `running` row could belong to the actual daemon's own ticker legitimately
// working on it right now (see the safety writeup on `claim_pending_jobs`:
// running this backfill alongside a live daemon with transcode enabled is
// intentionally supported, the two now share one atomic, count-aware
// concurrency cap), and there's no reliable way to tell that apart from a
// genuinely orphaned row from a ago-crashed run of this same command.
// If *this* command itself is killed mid-run, its claimed jobs stay
// `running` until the daemon is next restarted (which does its own reset).
async fn transcode_library_cmd(config: &Config) -> Result<()> {
if !config.transcode.enabled {
bail!("transcode.enabled is false in config — enable it before running a backfill");