Add a llama.cpp server and router adapter

Breadway commented

2026-10-11 20:49:51 +08:00

Owner

llama.cpp server and router adapter with idle-safe observation, router unload, and real-server verification. Closes breadboard RAM-13. Carries the RAM-55 correction.

What this adds

crates/breadram-core/src/adapter/llamacpp.rs: discovery sends only GET /health, GET /models, and GET /props, which are on the upstream README's idle-timer exemption list. /v1/models and /slots are deliberately not probed.
Router IDs and statuses come from a parseable /models response; a properties response with a nonempty model_path and explicit is_sleeping supplies single-model inventory. Each refresh replaces the previous observation, and failed, unauthorized, malformed, missing, or props-only answers cannot make old inventory current.
Resident models report unknown activity, not Active. Missing or unrecognized statuses report unknown, not unloaded. A complete current list of only unloaded or sleeping models reports idle.
Router unload: when automation enables UnloadModel, operation drives the documented POST /models/unload for one loaded router model, confirmed by the exact model id plus its status in the same /models listing (structured predicate, so a sibling sharing a substring cannot cross-confirm). Single-model servers expose no documented manual unload and are never written to. The unloaded status is a logical state, not a measured memory delta.
Registry notes state whether router unload is available or the endpoint is observation only. docs/LLAMACPP.md records the contract, the boundary, and the measured verification.

RAM-55 fix

This branch contains the RAM-55 correction: /v1/models is no longer probed, stale observations are replaced each pass, inventory must be valid rather than unauthorized/absent/props-only, missing status is preserved as unknown, and mode is re-evaluated per pass.

Verified against a real server

llama-server 0.5.0-dev (build 11146) in router mode on loopback with Qwen3 8B Q4_K_M (5.2GB, local Ollama blob store), offloaded to the host iGPU:

The adapter's exact probe set answered as specified; inference completes and marks the model loaded.
POST /models/unload with the adapter's exact body answers success:true, flips the same entry to unloaded, and reaps the model subprocess: host RSS delta about 60MB against 9.8GB mapped, iGPU shared memory about 8.7GB reclaimed per cycle.
POST /models/load answers success:true through loading to loaded.
A completion issued at unloaded state reloads and succeeds: native wake-on-request preserved end to end.
Inventory still reports host and device memory as unknown for router models: the server API supplies neither figure, and mmap plus shared memory make file size a false measurement. Unknown stays unknown.

Verification

Run on this branch rebased onto main (2d18cac):

cargo fmt --all --check
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo build --workspace --locked
cargo test --workspace --locked

All pass, plus the live server evidence above.

Closes RAM-13.

llama.cpp server and router adapter with idle-safe observation, router unload, and real-server verification. Closes breadboard RAM-13. Carries the RAM-55 correction. ## What this adds - crates/breadram-core/src/adapter/llamacpp.rs: discovery sends only GET /health, GET /models, and GET /props, which are on the upstream README's idle-timer exemption list. /v1/models and /slots are deliberately not probed. - Router IDs and statuses come from a parseable /models response; a properties response with a nonempty model_path and explicit is_sleeping supplies single-model inventory. Each refresh replaces the previous observation, and failed, unauthorized, malformed, missing, or props-only answers cannot make old inventory current. - Resident models report unknown activity, not Active. Missing or unrecognized statuses report unknown, not unloaded. A complete current list of only unloaded or sleeping models reports idle. - Router unload: when automation enables UnloadModel, operation drives the documented POST /models/unload for one loaded router model, confirmed by the exact model id plus its status in the same /models listing (structured predicate, so a sibling sharing a substring cannot cross-confirm). Single-model servers expose no documented manual unload and are never written to. The unloaded status is a logical state, not a measured memory delta. - Registry notes state whether router unload is available or the endpoint is observation only. docs/LLAMACPP.md records the contract, the boundary, and the measured verification. ## RAM-55 fix This branch contains the RAM-55 correction: /v1/models is no longer probed, stale observations are replaced each pass, inventory must be valid rather than unauthorized/absent/props-only, missing status is preserved as unknown, and mode is re-evaluated per pass. ## Verified against a real server llama-server 0.5.0-dev (build 11146) in router mode on loopback with Qwen3 8B Q4_K_M (5.2GB, local Ollama blob store), offloaded to the host iGPU: - The adapter's exact probe set answered as specified; inference completes and marks the model loaded. - POST /models/unload with the adapter's exact body answers success:true, flips the same entry to unloaded, and reaps the model subprocess: host RSS delta about 60MB against 9.8GB mapped, iGPU shared memory about 8.7GB reclaimed per cycle. - POST /models/load answers success:true through loading to loaded. - A completion issued at unloaded state reloads and succeeds: native wake-on-request preserved end to end. - Inventory still reports host and device memory as unknown for router models: the server API supplies neither figure, and mmap plus shared memory make file size a false measurement. Unknown stays unknown. ## Verification Run on this branch rebased onto main (2d18cac): - cargo fmt --all --check - cargo clippy --workspace --all-targets --locked -- -D warnings - cargo build --workspace --locked - cargo test --workspace --locked All pass, plus the live server evidence above. Closes RAM-13.

Breadway added 2 commits

2026-10-11 20:49:51 +08:00

feat: add the llama.cpp server and router adapter fc6b487ee8

Observes configured llama-server deployments in both modes and never
writes. The router's model list, its per-model status, and the
server's own sleeping state are reported as inventory and activity
evidence; single-model servers fall back to the model info endpoint when
the router surface is absent. No release operation exists in either mode,
because llama.cpp documents no manual unload or sleep, and the router's
unload leaves the model listed, so an unload cannot be confirmed.

Closes RAM-13

fix: keep llama.cpp observation idle-safe and conservative

CPU checks / check (push) Successful in 58s

Details

4efc4e2f03

Breadway referenced this pull request

2026-10-11 20:49:53 +08:00

Correct llama.cpp observation freshness and idle-safe probes #22

Breadway referenced this pull request

2026-10-11 20:51:24 +08:00

Add the optional local inference gateway for request tracking #13

Breadway referenced this pull request

2026-10-11 21:09:34 +08:00

Correct llama.cpp observation freshness and idle-safe probes #22

Breadway added 1 commit

2026-10-11 22:31:04 +08:00

feat: confirm a router unload by model id and status, and drive it

CPU checks / check (push) Successful in 53s

Details

80691354e2

Criterion 2 was unmet: operation() returned None in every mode. A loaded router
model can now be unloaded through the documented endpoint, and the effect is
confirmed structurally rather than by a body substring.

- The adapter confirmation gains a structured model status check (exact id plus
  expected status), preferred over the substring checks, so two models sharing a
  substring cannot cross-confirm. The llama.cpp router nests status under a value
  field, which the check reads.
- LlamaCppAdapter::operation drives POST /models/unload for one loaded router
  model when automation enables the UnloadModel capability, and confirms with
  the model's own status in the next /models listing. Single-model servers, with
  no documented manual unload, are never touched.
- Registry notes state whether llama.cpp may unload or is observation only.
- Tests: a router unload is confirmed by the exact id and status and not by a
  sibling substring; a single-model server offers no unload; the confirmation
  contract cannot be fooled by a shared substring.

Breadway referenced this pull request

2026-10-11 22:50:59 +08:00

Correct llama.cpp observation freshness and idle-safe probes #22

Breadway force-pushed feature/ram-13-llamacpp-adapter from 80691354e2

CPU checks / check (push) Successful in 53s

Details

to 7bc9e64966

CPU checks / check (push) Successful in 35s

Details

2026-10-12 09:18:49 +08:00

Compare

Breadway referenced this pull request