llama.cpp loaded model never reports idle so daemon policy can never unload it #46
Breadway
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by code inspection while building the RAM-30 daemon-driven provider test. Confidence: high for the mechanism, medium for whether the intended design is request-lease tracking (ollama Complete mode) or a llama-specific idle signal. LlamaCppAdapter::activity returns Unknown whenever any model is Loaded, Loading, Downloading or Unknown (holding_anything in crates/breadram-core/src/adapter/llamacpp.rs), and idle_since_ms is cleared in that case. The planner requires is_confirmed_idle with MIN_IDLE_MS of 30s before planning UnloadIdleModel, and AdapterExecutor::resolve denies the write on the same predicate. So a loaded router model can never satisfy the idle gate through this adapter, even when no client has used it for hours. By contrast ollama with complete request tracking reports resident-but-unused as idle from leases, and breadsearch reports Active only while queries run or files await indexing, otherwise Idle. The gateway tracks in-flight leases but the daemon only folds them in as Active evidence, never as Idle, so gateway idleness does not fill the gap. Docs/LLAMACPP.md says the adapter can unload one loaded router model when automation is enabled; that path is unreachable through daemon policy as written. Expected: either llama request visibility (gateway-routed or server-side activity) feeding Idle, or docs stating llama unload is manual/API only and RAM-30 recording it as a limitation. Reproduce: configure a llama_cpp provider owning a disposable router scope with a loaded model, run breadramd in automatic mode, wait past MIN_IDLE_MS, and read explain output: the scope reports needs confirmed idleness and none is proven, skipped.
Review of PR 54 at
8816d98960: changes requested, not merged. docs/LLAMACPP.md and the registry note correctly say automatic loaded-model unload is unavailable, and the idle gates plus low-level unload API are intact. docs/AGENT-HANDOFF.md:79-84 still says enabling automation can unload one loaded router model. Persisted review: #54 (comment) (Forgejo review id 1; a formal REQUEST_CHANGES event was refused because the token owns the PR). Card moved back to todo for that doc fix. Do not weaken planner or executor gates to clear it.Rereview of PR 54 at
de0268f6f5: pass. The AGENT-HANDOFF claim is corrected, idle gates are unchanged, and the low-level unload API plus Unknown-activity pins remain. Required CI was green. Merged as455d3c45a6. Not marked done: coordinator still verifies the integrated result. Persisted review is on the PR.