Correct llama.cpp observation freshness and idle-safe probes #22

New issue

Closed

opened 2026-10-11 18:04:36 +08:00 by Breadway · 7 comments

Breadway commented 2026-10-11 18:04:36 +08:00

Owner

Copy link

Discovered while reviewing the unpublished RAM-13 commit c2439ce against the upstream server README. Confidence: high, reproduced by inspection of discover(), observe(), and parse_models(). The adapter probes /v1/models despite claiming all probes are idle-timer exempt; upstream only lists /health, /props, /models, and /metrics. Unauthorized or absent endpoints and props-only answers can mark stale inventory current, missing status becomes unloaded, and router mode persists across refreshes. Loaded residency is labeled Active without measured request activity. This can disturb native sleep and manufacture idle/current evidence. Fix within the existing RAM-13 branch; no separate implementation PR. Router unload is documented and per-model ID/status can confirm logical state, but the existing substring confirmation contract cannot safely correlate those fields. Memory release still requires separate measurements.

Discovered while reviewing the unpublished RAM-13 commit c2439ce against the upstream server README. Confidence: high, reproduced by inspection of discover(), observe(), and parse_models(). The adapter probes /v1/models despite claiming all probes are idle-timer exempt; upstream only lists /health, /props, /models, and /metrics. Unauthorized or absent endpoints and props-only answers can mark stale inventory current, missing status becomes unloaded, and router mode persists across refreshes. Loaded residency is labeled Active without measured request activity. This can disturb native sleep and manufacture idle/current evidence. Fix within the existing RAM-13 branch; no separate implementation PR. Router unload is documented and per-model ID/status can confirm logical state, but the existing substring confirmation contract cannot safely correlate those fields. Memory release still requires separate measurements.

Breadway commented 2026-10-11 20:49:53 +08:00

Author

Owner

Copy link

The correction this card asks for is contained in the RAM-13 branch: commit "fix: keep llama.cpp observation idle-safe and conservative" removes /v1/models from discovery, replaces stale observations each pass, requires a valid inventory rather than unauthorized/absent/props-only answers, preserves a missing status as unknown, and re-evaluates mode per pass.

It ships with the RAM-13 PR: #31

No separate implementation PR, per this card.

The correction this card asks for is contained in the RAM-13 branch: commit "fix: keep llama.cpp observation idle-safe and conservative" removes /v1/models from discovery, replaces stale observations each pass, requires a valid inventory rather than unauthorized/absent/props-only answers, preserves a missing status as unknown, and re-evaluates mode per pass. It ships with the RAM-13 PR: https://git.breadway.dev/Breadway/breadram/pulls/31 No separate implementation PR, per this card.

Breadway commented 2026-10-11 21:09:34 +08:00

Author

Owner

Copy link

Reviewed, fix verified, left in review.

The RAM-55 correction on branch feature/ram-13-llamacpp-adapter (PR #31) is complete and correct, checked against the card point by point:

• /v1/models is no longer probed; discovery builds only /health, /models and /props, and the test it_probes_only_idle_timer_exempt_endpoints (crates/breadram-core/src/adapter/llamacpp.rs:655) pins the exact URL set.
• A failed, unauthorized, malformed or props-only answer can no longer make stale inventory current: each refresh replaces the observation, and inventory_is_current follows it (llamacpp.rs:461, tests absent_unauthorized_and_props_only_cannot_refresh_router_inventory at llamacpp.rs:909 and a_read_failure_clears_a_previous_success at llamacpp.rs:841).
• Loaded residency now reports unknown rather than Active, and a missing or unrecognized status stays unknown rather than becoming unloaded (activity at llamacpp.rs:465, tests a_router_listing_an_unknown_status_is_not_an_idle_model at llamacpp.rs:762 and an_unknown_status_is_not_failed_or_idle at llamacpp.rs:899).
• Router mode is re-evaluated every pass instead of persisting (test router_can_change_to_sleeping_single_model_without_sticky_mode at llamacpp.rs:961).
• The confirmation limitation the card names is documented rather than papered over in docs/LLAMACPP.md.

I am not moving this card to done yet, and it is not a problem with the fix: RAM-55 has no branch of its own, so it rides on PR #31, which I have sent back to todo as a partial delivery of RAM-13 (the router administration and measurement criteria). When that branch lands, this fix lands with it. No action needed from the implementer on this card; keep the correction commits intact.

**Reviewed, fix verified, left in review.** The RAM-55 correction on branch feature/ram-13-llamacpp-adapter (PR #31) is complete and correct, checked against the card point by point: - /v1/models is no longer probed; discovery builds only /health, /models and /props, and the test it_probes_only_idle_timer_exempt_endpoints (crates/breadram-core/src/adapter/llamacpp.rs:655) pins the exact URL set. - A failed, unauthorized, malformed or props-only answer can no longer make stale inventory current: each refresh replaces the observation, and inventory_is_current follows it (llamacpp.rs:461, tests absent_unauthorized_and_props_only_cannot_refresh_router_inventory at llamacpp.rs:909 and a_read_failure_clears_a_previous_success at llamacpp.rs:841). - Loaded residency now reports unknown rather than Active, and a missing or unrecognized status stays unknown rather than becoming unloaded (activity at llamacpp.rs:465, tests a_router_listing_an_unknown_status_is_not_an_idle_model at llamacpp.rs:762 and an_unknown_status_is_not_failed_or_idle at llamacpp.rs:899). - Router mode is re-evaluated every pass instead of persisting (test router_can_change_to_sleeping_single_model_without_sticky_mode at llamacpp.rs:961). - The confirmation limitation the card names is documented rather than papered over in docs/LLAMACPP.md. I am not moving this card to done yet, and it is not a problem with the fix: RAM-55 has no branch of its own, so it rides on PR #31, which I have sent back to todo as a partial delivery of RAM-13 (the router administration and measurement criteria). When that branch lands, this fix lands with it. No action needed from the implementer on this card; keep the correction commits intact.

Breadway commented 2026-10-11 22:50:59 +08:00

Author

Owner

Copy link

Re-verified at the current PR #31 head (8069135), which added router unload confirmation after the previous review. The RAM-55 correction is intact: discover() still probes only /health, /models and /props, with /v1/models deliberately unused; a failed, unauthorized, absent, or props-only answer cannot refresh stale inventory (inventory_is_current requires a fresh read); a resident model reports unknown activity rather than Active; a missing or unrecognized status stays unknown; and router mode is re-evaluated per pass. Left in review: the fix rides on PR #31, which is still a partial delivery of RAM-13, so it lands with that card. No action needed on this card.

Re-verified at the current PR #31 head (8069135), which added router unload confirmation after the previous review. The RAM-55 correction is intact: `discover()` still probes only /health, /models and /props, with /v1/models deliberately unused; a failed, unauthorized, absent, or props-only answer cannot refresh stale inventory (`inventory_is_current` requires a fresh read); a resident model reports unknown activity rather than Active; a missing or unrecognized status stays unknown; and router mode is re-evaluated per pass. Left in review: the fix rides on PR #31, which is still a partial delivery of RAM-13, so it lands with that card. No action needed on this card.

Breadway commented 2026-10-12 09:27:46 +08:00

Author

Owner

Copy link

Re-verified at PR #31 head 7bc9e64 (rebased onto main 2d0965e, CI success, mergeable true). The RAM-55 correction is intact:

• discover() probes only GET /health, GET /models, GET /props (crates/breadram-core/src/adapter/llamacpp.rs:281), with /v1/models deliberately unused and pinned by it_probes_only_idle_timer_exempt_endpoints.
• Each refresh replaces the observation and inventory_is_current requires a fresh read, so failed, unauthorized, malformed, missing, or props-only answers cannot refresh stale inventory (tests absent_unauthorized_and_props_only_cannot_refresh_router_inventory and a_read_failure_clears_a_previous_success).
• Loaded residency reports Unknown activity, missing or unrecognized status stays Unknown (tests a_router_listing_an_unknown_status_is_not_an_idle_model and an_unknown_status_is_not_failed_or_idle).
• Router mode is re-evaluated per pass (test router_can_change_to_sleeping_single_model_without_sticky_mode).
• Structured unload confirmation (exact id plus status in the same /models entry) is intact, with the sibling-substring non-confirmation test.

What changed this pass: the stale docs that contradicted the unload path are fixed (module header plus docs/LLAMACPP.md now state the router POST /models/unload with structured confirmation), the host llama-server revision is recorded (0.5.0-dev build 11146, no model file so no real relief measured), and the workload limit_bytes fixture is fixed. No regression in the RAM-55 scope.

Left in review: this fix rides on PR #31, which is a partial delivery of RAM-13 (PR body says Part of RAM-13), so it lands with that card. No action needed on this card. Parent RAM-13 next change: exercise a real supported build with a loaded model, verify inference after relief, and report separate host RAM and device RAM measurements.

Re-verified at PR #31 head 7bc9e64 (rebased onto main 2d0965e, CI success, mergeable true). The RAM-55 correction is intact: - discover() probes only GET /health, GET /models, GET /props (crates/breadram-core/src/adapter/llamacpp.rs:281), with /v1/models deliberately unused and pinned by it_probes_only_idle_timer_exempt_endpoints. - Each refresh replaces the observation and inventory_is_current requires a fresh read, so failed, unauthorized, malformed, missing, or props-only answers cannot refresh stale inventory (tests absent_unauthorized_and_props_only_cannot_refresh_router_inventory and a_read_failure_clears_a_previous_success). - Loaded residency reports Unknown activity, missing or unrecognized status stays Unknown (tests a_router_listing_an_unknown_status_is_not_an_idle_model and an_unknown_status_is_not_failed_or_idle). - Router mode is re-evaluated per pass (test router_can_change_to_sleeping_single_model_without_sticky_mode). - Structured unload confirmation (exact id plus status in the same /models entry) is intact, with the sibling-substring non-confirmation test. What changed this pass: the stale docs that contradicted the unload path are fixed (module header plus docs/LLAMACPP.md now state the router POST /models/unload with structured confirmation), the host llama-server revision is recorded (0.5.0-dev build 11146, no model file so no real relief measured), and the workload limit_bytes fixture is fixed. No regression in the RAM-55 scope. Left in review: this fix rides on PR #31, which is a partial delivery of RAM-13 (PR body says Part of RAM-13), so it lands with that card. No action needed on this card. Parent RAM-13 next change: exercise a real supported build with a loaded model, verify inference after relief, and report separate host RAM and device RAM measurements.

Breadway commented 2026-10-12 16:21:48 +08:00

Author

Owner

Copy link

Re-verified at PR #31 head 34d2b9e (rebased onto main 44a23a5, CI success, mergeable true). The RAM-55 correction is intact, byte for byte against the last verified head 7bc9e64: the branch files (llamacpp.rs, hook.rs, mod.rs, docs/LLAMACPP.md) are identical, and the only differences are main side changes from the rebase.

• discover() probes only GET /health, GET /models, GET /props (llamacpp.rs:286), with /v1/models deliberately unused (line 385) and pinned by it_probes_only_idle_timer_exempt_endpoints (line 766).
• Each refresh replaces the observation (line 339) and inventory_is_current requires a fresh read (lines 421, 466), so failed, unauthorized, malformed, missing, or props-only answers cannot refresh stale inventory (tests absent_unauthorized_and_props_only_cannot_refresh_router_inventory and a_read_failure_clears_a_previous_success).
• Loaded residency reports Unknown activity (lines 470 and following), missing or unrecognized status stays Unknown (line 606, test at line 873, plus an_unknown_status_is_not_failed_or_idle).
• Router mode is re-evaluated per pass (test router_can_change_to_sleeping_single_model_without_sticky_mode at line 1072).
• Structured unload confirmation (exact id plus status in the same /models entry, line 531) is intact, with the sibling-substring non-confirmation test.

Left in review: this fix rides on PR #31, which is a partial delivery of RAM-13, so it lands with that card. No action needed on this card.

Re-verified at PR #31 head 34d2b9e (rebased onto main 44a23a5, CI success, mergeable true). The RAM-55 correction is intact, byte for byte against the last verified head 7bc9e64: the branch files (llamacpp.rs, hook.rs, mod.rs, docs/LLAMACPP.md) are identical, and the only differences are main side changes from the rebase. - discover() probes only GET /health, GET /models, GET /props (llamacpp.rs:286), with /v1/models deliberately unused (line 385) and pinned by it_probes_only_idle_timer_exempt_endpoints (line 766). - Each refresh replaces the observation (line 339) and inventory_is_current requires a fresh read (lines 421, 466), so failed, unauthorized, malformed, missing, or props-only answers cannot refresh stale inventory (tests absent_unauthorized_and_props_only_cannot_refresh_router_inventory and a_read_failure_clears_a_previous_success). - Loaded residency reports Unknown activity (lines 470 and following), missing or unrecognized status stays Unknown (line 606, test at line 873, plus an_unknown_status_is_not_failed_or_idle). - Router mode is re-evaluated per pass (test router_can_change_to_sleeping_single_model_without_sticky_mode at line 1072). - Structured unload confirmation (exact id plus status in the same /models entry, line 531) is intact, with the sibling-substring non-confirmation test. Left in review: this fix rides on PR #31, which is a partial delivery of RAM-13, so it lands with that card. No action needed on this card.

Breadway commented 2026-10-12 19:28:27 +08:00

Author

Owner

Copy link

Re-verified at PR #31 head 3bd0e64 (base main 2d18cac, CI success, mergeable true). The RAM-55 correction is intact:

• discover() probes only GET /health, GET /models, GET /props (crates/breadram-core/src/adapter/llamacpp.rs:287-292), with /v1/models deliberately unused and pinned by it_probes_only_idle_timer_exempt_endpoints.
• Each refresh replaces the observation and inventory_is_current requires a fresh read, so failed, unauthorized, malformed, missing, or props-only answers cannot refresh stale inventory (tests absent_unauthorized_and_props_only_cannot_refresh_router_inventory, a_read_failure_clears_a_previous_success, missing_status_and_missing_list_are_not_empty_idle_inventory).
• Loaded residency reports Unknown activity, missing or unrecognized status stays Unknown (tests a_router_listing_an_unknown_status_is_not_an_idle_model, an_unknown_status_is_not_failed_or_idle).
• Router mode is re-evaluated per pass (test router_can_change_to_sleeping_single_model_without_sticky_mode).
• Structured unload confirmation (exact id plus status in the same /models entry, llamacpp.rs:532-536 with Confirmation::model_status) is intact, with the sibling-substring non-confirmation tests in both llamacpp.rs and mod.rs.

Left in review: this fix rides on PR #31, which I have sent back to todo on RAM-13 for a router-guard correctness issue (operation() at llamacpp.rs:496-515 needs a state.router check so an awake single-model fallback entry is never offered POST /models/unload) plus two stale doc lines. No action needed on this card; keep the correction commits intact. It lands with RAM-13.

Re-verified at PR #31 head 3bd0e64 (base main 2d18cac, CI success, mergeable true). The RAM-55 correction is intact: - discover() probes only GET /health, GET /models, GET /props (crates/breadram-core/src/adapter/llamacpp.rs:287-292), with /v1/models deliberately unused and pinned by it_probes_only_idle_timer_exempt_endpoints. - Each refresh replaces the observation and inventory_is_current requires a fresh read, so failed, unauthorized, malformed, missing, or props-only answers cannot refresh stale inventory (tests absent_unauthorized_and_props_only_cannot_refresh_router_inventory, a_read_failure_clears_a_previous_success, missing_status_and_missing_list_are_not_empty_idle_inventory). - Loaded residency reports Unknown activity, missing or unrecognized status stays Unknown (tests a_router_listing_an_unknown_status_is_not_an_idle_model, an_unknown_status_is_not_failed_or_idle). - Router mode is re-evaluated per pass (test router_can_change_to_sleeping_single_model_without_sticky_mode). - Structured unload confirmation (exact id plus status in the same /models entry, llamacpp.rs:532-536 with Confirmation::model_status) is intact, with the sibling-substring non-confirmation tests in both llamacpp.rs and mod.rs. Left in review: this fix rides on PR #31, which I have sent back to todo on RAM-13 for a router-guard correctness issue (operation() at llamacpp.rs:496-515 needs a state.router check so an awake single-model fallback entry is never offered POST /models/unload) plus two stale doc lines. No action needed on this card; keep the correction commits intact. It lands with RAM-13.

Breadway commented 2026-10-12 19:35:23 +08:00

Author

Owner

Copy link

Re-verified at the merged head a1adebf and landed with PR #31 (squashed as 19ab3dc, branch deleted). The correction is intact: discover() probes only GET /health, GET /models, GET /props with /v1/models deliberately unused and pinned by it_probes_only_idle_timer_exempt_endpoints; each refresh replaces the observation and inventory_is_current requires a fresh read; loaded residency reports Unknown and missing status stays Unknown; router mode is re-evaluated per pass; structured unload confirmation by exact id plus status is intact with the sibling-substring tests. No action was needed on this card; it rode on the RAM-13 branch as specified.

Re-verified at the merged head a1adebf and landed with PR #31 (squashed as 19ab3dc, branch deleted). The correction is intact: discover() probes only GET /health, GET /models, GET /props with /v1/models deliberately unused and pinned by it_probes_only_idle_timer_exempt_endpoints; each refresh replaces the observation and inventory_is_current requires a fresh read; loaded residency reports Unknown and missing status stays Unknown; router mode is re-evaluated per pass; structured unload confirmation by exact id plus status is intact with the sibling-substring tests. No action was needed on this card; it rode on the RAM-13 branch as specified.

Breadway closed this issue 2026-10-12 19:35:23 +08:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

feature/bakery-breadram

feature/ram-30-benchmarks

fix/ram-79-honest-llama-capability

fix/ram-84-provider-rearm

fix/ram-78-pressure-sample-clock

feature/ram-14-vllm-adapter

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Breadway/breadram#22

No description provided.