breadmill queries_served does not count a query that triggers a demand reload #17
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by the independent review of breadram PR 50 (RAM-82). Confidence: high on the code path, low on practical impact. In feature/status-queries-served (commit
7e5eef9), state.queries_served is incremented in dispatch only after the model-present check. A query that arrives while the model is unloaded starts a background reload and returns "model loading, retry shortly" before the counter moves. The reload sets loading, which the breadram adapter already treats as work, but if a poll misses the load window (a fast CPU reload takes about 3 s against a 2 to 3 s poll interval) that demand-reload query leaves no trace and the mill reads as long idle again straight after the reload, so it could be unloaded again and thrash. Expected: count the query at dispatch entry, before the model check, so every arriving query is visible. That changes the documented meaning from queries that entered an embedding call to queries received, which should be stated in the field doc. Reproduce: unload the mill, send one query, poll status once the load finished, and read queries_served: it is unchanged.