breadsearch adapter treats a mill under short repeated queries as idle and unloads it #49
Breadway
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by the RAM-30 provider contention benchmark (scripts/bench/bench-contention.sh on feature/ram-30-benchmarks). Confidence: high that the mechanism is as described (code inspection plus 3 of 3 live reproductions); not yet measured that query latency is the only factor.
Two disposable breadmill instances run under one private daemon in automatic mode with unload_idle_model as the only allowed action and bounded hog pressure. mill-a received a query every 3 s for 90 s; mill-b stayed quiet. In 3 of 3 runs the journal shows one confirmed unload on mill-a (the busy mill) as well as one on mill-b. The old pass check only looked at end-of-run readiness, which hid this because a query reloads an unloaded model on demand.
Mechanism: BreadsearchAdapter derives idle_since_ms only from polled in_flight, pending and loading (adapter/breadsearch.rs refresh and note_socket_result). A query holds in_flight for milliseconds, and the daemon polls every settle_ms (2 s in the bench, 3 s default), so a poll almost never lands inside one. StatusInfo in breadsearch-shared exposes no last-query timestamp or request counter, so the adapter cannot tell a constantly queried mill from an unused one. After an unload the next query reloads the model, so the user sees repeated cold-start latency while the card contract says an actively serving provider is never unloaded.
Impact: serving protection for breadsearch holds only for queries long enough to be caught in flight. The earlier single-mill bench (bench-breadsearch-daemon.sh) reported no second unload in a 45 s query window. That is not evidence of protection: the contention runs unloaded the busy mill once within a 90 s window, so 45 s may simply be too short. The reason it showed none has not been established (idle gate timing, unload cooldown and poll phase are all candidates), and that protection claim is withdrawn until the window is lengthened and the result explained.
Expected: either breadmill reports activity the adapter can trust (a monotonic last_query_age or a completed-query counter, an additive optional field like the existing ones), with the adapter treating any change since the previous poll as activity, or the adapter documents that only in-flight work counts and the contention gate is recorded as a known failure. Reproduce: BREADMILL_BIN= sh scripts/bench/bench-contention.sh and read busy_mill_unloads in the output. Related: RAM-79 (llama.cpp never reports idle) is the opposite failure on a different adapter.
Review of breadram PR 50 at head
532fb5d055: changes requested, two small items.Logic is correct and CI plus local test, clippy and fmt are green on that SHA. Mutating shows_work to ignore the counter fails 3 of the 4 new tests.
A. The note_socket_result path (executor confirmation reads) has no test: replacing its previous-status argument with None passes the whole workspace suite. A test that passes now and fails under that mutation is in the PR comment; add it.
B. The PR body cites contention-fixed-{1,2,3}.jsonl on feature/ram-30-benchmarks, but they are not on that branch on the remote (only untracked in the local ram30 worktree). Values read there match the claim (0 busy unloads, 1 quiet unload each). Commit and push them or reword the body.
Full review: #50 (comment). Not merged.