Add the optional local inference gateway for request tracking

Breadway commented

2026-10-11 15:15:01 +08:00

Owner

Split out of RAM-11 so the coordination core could land verified (PR #12) without holding the card open on a second project.

The serving coordination core is in breadram-core/src/serving: request leases and the drain, sleep, wake lifecycle, with the contract in docs/SERVING-COORDINATION.md. Nothing feeds it from real traffic yet, so the only activity evidence an adapter can offer today is the provider's own answer, and a backend whose idleness cannot be proven is reported unsupported.

Scope

An optional, locally configured HTTP gateway that fronts one or more model backends, preserves native and OpenAI-compatible request and response semantics (including streaming, cancellation, and retries), and admits every request through a lease before dispatch.
Lease lifecycle on the wire: a stream holds its lease for its whole duration, a client disconnect cancels and releases it, a retry with the same idempotency key joins or replays instead of double counting, and a queued request holds its lease until it completes.
Demand-driven wake on the request path: a request that arrives at a sleeping backend triggers the wake, the request is refused with a retry hint until health is verified, and the wake is reported back to the lifecycle.
Provider calls for sleep and wake, driven by SleepDecision, DrainOutcome, and WakeHealth rather than by the gateway's own timer.
No prompt or completion is logged, and none can be: the gateway passes bodies through and holds only leases, which have nowhere to store content.

Acceptance criteria

Pointing an OpenAI-compatible client at the gateway produces the same responses as pointing it at the backend directly, for both streamed and non-streamed requests.
With the gateway in the path, a quiet backend is put to sleep, and the next request wakes it and completes after a bounded delay rather than failing.
A client that disconnects mid-stream releases its lease and the backend becomes idle.
The gateway refuses to sleep a backend it is not tracking unless the operator has explicitly allowed it.

Depends on the coordination core (merged in PR #12) and the provider adapters for sleep and wake (RAM-12, RAM-13, RAM-14).

Split out of RAM-11 so the coordination core could land verified (PR #12) without holding the card open on a second project. The serving coordination core is in `breadram-core/src/serving`: request leases and the drain, sleep, wake lifecycle, with the contract in `docs/SERVING-COORDINATION.md`. Nothing feeds it from real traffic yet, so the only activity evidence an adapter can offer today is the provider's own answer, and a backend whose idleness cannot be proven is reported unsupported. ### Scope - An optional, locally configured HTTP gateway that fronts one or more model backends, preserves native and OpenAI-compatible request and response semantics (including streaming, cancellation, and retries), and admits every request through a lease before dispatch. - Lease lifecycle on the wire: a stream holds its lease for its whole duration, a client disconnect cancels and releases it, a retry with the same idempotency key joins or replays instead of double counting, and a queued request holds its lease until it completes. - Demand-driven wake on the request path: a request that arrives at a sleeping backend triggers the wake, the request is refused with a retry hint until health is verified, and the wake is reported back to the lifecycle. - Provider calls for sleep and wake, driven by `SleepDecision`, `DrainOutcome`, and `WakeHealth` rather than by the gateway's own timer. - No prompt or completion is logged, and none can be: the gateway passes bodies through and holds only leases, which have nowhere to store content. ### Acceptance criteria - Pointing an OpenAI-compatible client at the gateway produces the same responses as pointing it at the backend directly, for both streamed and non-streamed requests. - With the gateway in the path, a quiet backend is put to sleep, and the next request wakes it and completes after a bounded delay rather than failing. - A client that disconnects mid-stream releases its lease and the backend becomes idle. - The gateway refuses to sleep a backend it is not tracking unless the operator has explicitly allowed it. Depends on the coordination core (merged in PR #12) and the provider adapters for sleep and wake (RAM-12, RAM-13, RAM-14).

Breadway commented

2026-10-11 16:08:57 +08:00

Author

Owner

Dependency note from RAM-48 (provider execution), which is now implemented and in review.

Provider relief can now be carried out, but it cannot fire in a real deployment until request visibility exists, because of how idleness is proven:

OllamaAdapter::activity returns Idle from request leases only when TrackingCompleteness::Complete was declared with with_tracking. Otherwise a loaded model reports Active, whatever its expiry says, which is the correct conservative direction.
Nothing in the daemon declares complete tracking today, because the only honest source of real requests is the gateway this card is about. AdapterRegistry::from_config therefore leaves every adapter at Unknown.

So the current behaviour is deliberate and, stated plainly: a resident model is never released until a gateway reports the requests that reach it (or a deployment explicitly declares tracking through a documented configuration field, which does not exist yet). Whoever picks this up should decide which of those two it is, rather than having the release path stay unreachable by accident.

Dependency note from RAM-48 (provider execution), which is now implemented and in review. Provider relief can now be carried out, but it cannot fire in a real deployment until request visibility exists, because of how idleness is proven: - `OllamaAdapter::activity` returns `Idle` from request leases only when `TrackingCompleteness::Complete` was declared with `with_tracking`. Otherwise a *loaded* model reports `Active`, whatever its expiry says, which is the correct conservative direction. - Nothing in the daemon declares complete tracking today, because the only honest source of real requests is the gateway this card is about. `AdapterRegistry::from_config` therefore leaves every adapter at `Unknown`. So the current behaviour is deliberate and, stated plainly: a resident model is never released until a gateway reports the requests that reach it (or a deployment explicitly declares tracking through a documented configuration field, which does not exist yet). Whoever picks this up should decide which of those two it is, rather than having the release path stay unreachable by accident.

Breadway commented

2026-10-11 17:08:05 +08:00

Author

Owner

Triage (backlog sweep): not starting yet. The gateway core is well specified and the coordination core it needs is merged, but the card depends on the sleep/wake adapters RAM-13 and RAM-14, which are still unstarted. Starting the gateway now would force stubbing the exact provider calls it must drive. Proposed order: implement RAM-13 and RAM-14 first, then build the gateway against the real SleepDecision, DrainOutcome, and WakeHealth paths. Leaving in backlog.

Breadway commented

2026-10-11 20:51:24 +08:00

Author

Owner

Triage during the breadram blocker sweep (session ses_ed5126ae3ffeORjTfoQf89tltt). Not implemented here.

State: the serving coordination core this card needs is merged (PR #12), and the sleep and wake provider calls it must drive now have PRs: llama.cpp as RAM-13 (#31) and vLLM as RAM-14 (#29). Neither is verified or merged yet, so the exact provider calls the gateway would drive are still in review.

Per the implementer rule against speculative PRs, this card is left in backlog. Once RAM-13 and RAM-14 are verified and merged, the gateway's scope (fronting one or more backends, admitting every request through a lease, streaming and cancellation and retry semantics, demand-driven wake, and no prompt or completion logging) can be built against a settled provider contract rather than stubs. Recommend starting it immediately after those two land.

Triage during the breadram blocker sweep (session ses_ed5126ae3ffeORjTfoQf89tltt). Not implemented here. State: the serving coordination core this card needs is merged (PR #12), and the sleep and wake provider calls it must drive now have PRs: llama.cpp as RAM-13 (#31) and vLLM as RAM-14 (#29). Neither is verified or merged yet, so the exact provider calls the gateway would drive are still in review. Per the implementer rule against speculative PRs, this card is left in backlog. Once RAM-13 and RAM-14 are verified and merged, the gateway's scope (fronting one or more backends, admitting every request through a lease, streaming and cancellation and retry semantics, demand-driven wake, and no prompt or completion logging) can be built against a settled provider contract rather than stubs. Recommend starting it immediately after those two land.

Breadway commented

2026-10-11 22:59:18 +08:00

Author

Owner

Started the gateway. First piece published on feature/ram-45-gateway-core.

PR: #33

What landed: the request-path routing core (breadram-core/src/serving/gateway.rs), which admits every request through a lease before dispatch, begins a demand-driven wake when a request arrives at a sleeping backend and refuses it with a retry hint, releases a stream's lease on disconnect, joins or replays an idempotent retry without double counting, and refuses an untracked backend rather than serving through a path that could be slept from under it. It stores no body. Tests and docs are included.

Still to come: the socket server preserving native and OpenAI-compatible request and response semantics including streaming and cancellation end to end, wiring the provider sleep and wake commands to SleepDecision, DrainOutcome, and WakeHealth, and the operator configuration. The PR body says Part of RAM-45, so the card stays open.

Started the gateway. First piece published on `feature/ram-45-gateway-core`. PR: https://git.breadway.dev/Breadway/breadram/pulls/33 What landed: the request-path routing core (`breadram-core/src/serving/gateway.rs`), which admits every request through a lease before dispatch, begins a demand-driven wake when a request arrives at a sleeping backend and refuses it with a retry hint, releases a stream's lease on disconnect, joins or replays an idempotent retry without double counting, and refuses an untracked backend rather than serving through a path that could be slept from under it. It stores no body. Tests and docs are included. Still to come: the socket server preserving native and OpenAI-compatible request and response semantics including streaming and cancellation end to end, wiring the provider sleep and wake commands to SleepDecision, DrainOutcome, and WakeHealth, and the operator configuration. The PR body says Part of RAM-45, so the card stays open.

Breadway self-assigned this

2026-10-11 22:59:19 +08:00

Breadway commented

2026-10-12 00:19:16 +08:00

Author

Owner

More gateway core on the same branch (PR #33 head 4be2dc8):

The gateway now implements the policy's ActivitySource: each backend's leases are its activity evidence for the workload it serves. A live lease reports Active, so a workload in use is never released, and an unknown workload reports Unknown, never idle. That is the bridge the card asks for, and it is tested.
Previously on this branch: header-only request classification and the HTTP disposition (proxy, retry with the Retry-After hint, refuse), and a bounded HTTP request-head parser.

Still to come: the socket server that preserves native and OpenAI-compatible request and response semantics including streaming and cancellation end to end, wiring the provider sleep and wake commands into the lifecycle on the request path, and the operator configuration that points a client at the gateway.

More gateway core on the same branch (PR #33 head 4be2dc8): - The gateway now implements the policy's ActivitySource: each backend's leases are its activity evidence for the workload it serves. A live lease reports Active, so a workload in use is never released, and an unknown workload reports Unknown, never idle. That is the bridge the card asks for, and it is tested. - Previously on this branch: header-only request classification and the HTTP disposition (proxy, retry with the Retry-After hint, refuse), and a bounded HTTP request-head parser. Still to come: the socket server that preserves native and OpenAI-compatible request and response semantics including streaming and cancellation end to end, wiring the provider sleep and wake commands into the lifecycle on the request path, and the operator configuration that points a client at the gateway.

Breadway commented

2026-10-12 16:14:32 +08:00

Author

Owner

More gateway on the same branch (PR #33 head 4e751e0, rebased onto main 44a23a5).

What landed this pass is the socket server the card asks for (serving/server.rs): every request admitted through a lease before dispatch over loopback TCP, heads parsed for routing metadata only with bodies relayed byte for byte, streams relayed until either side closes, disconnects releasing leases at once including while waiting on a slow backend, and 503 plus Retry-After / 502 / 404 / 400 / 413 semantics. Plus a response-head parser in http.rs (a status line is not a request line; found because the first proxy test returned 502).

Still to come: performing the provider wake behind WakeThenRetry through the adapter sleep/wake commands, sharing one in-flight response between joined requests, and the operator configuration that points a client at the gateway. The PR body says Part of RAM-45, so the card stays open.

Verification: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo build --workspace --locked, cargo test --workspace --locked all pass (24 suites ok).

More gateway on the same branch (PR #33 head 4e751e0, rebased onto main 44a23a5). What landed this pass is the socket server the card asks for (serving/server.rs): every request admitted through a lease before dispatch over loopback TCP, heads parsed for routing metadata only with bodies relayed byte for byte, streams relayed until either side closes, disconnects releasing leases at once including while waiting on a slow backend, and 503 plus Retry-After / 502 / 404 / 400 / 413 semantics. Plus a response-head parser in http.rs (a status line is not a request line; found because the first proxy test returned 502). Still to come: performing the provider wake behind WakeThenRetry through the adapter sleep/wake commands, sharing one in-flight response between joined requests, and the operator configuration that points a client at the gateway. The PR body says Part of RAM-45, so the card stays open. Verification: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo build --workspace --locked, cargo test --workspace --locked all pass (24 suites ok).

Breadway commented

2026-10-12 16:20:17 +08:00

Author

Owner

More gateway on the same branch (PR #33 head 9cec265, main at 44a23a5).

What landed this pass is the operator configuration that points a client at the gateway:

[gateway] config section with GatewayPolicy/GatewayRoute/GatewayTracking: enabled plus a required loopback listen address and longest-prefix routes with loopback backend addresses, validated at load (non-loopback, bad scope keys, duplicate prefixes, and missing listen/routes all rejected; routes validated even while disabled).
The daemon serves the gateway for as long as it runs when enabled, on its own loopback listener.
Tests: config validation cases plus a live daemon test that serves a fake backend through a configured gateway and gets byte-identical responses.
docs/SERVING-COORDINATION.md documents the operator surface with an example.

Still to come: performing the provider wake behind WakeThenRetry through the adapter sleep/wake commands, and sharing one in-flight response between joined requests. The PR body says Part of RAM-45, so the card stays open.

Verification: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo build --workspace --locked, cargo test --workspace --locked all pass (24 suites ok), with zero /tmp leftovers across consecutive runs.

More gateway on the same branch (PR #33 head 9cec265, main at 44a23a5). What landed this pass is the operator configuration that points a client at the gateway: - [gateway] config section with GatewayPolicy/GatewayRoute/GatewayTracking: enabled plus a required loopback listen address and longest-prefix routes with loopback backend addresses, validated at load (non-loopback, bad scope keys, duplicate prefixes, and missing listen/routes all rejected; routes validated even while disabled). - The daemon serves the gateway for as long as it runs when enabled, on its own loopback listener. - Tests: config validation cases plus a live daemon test that serves a fake backend through a configured gateway and gets byte-identical responses. - docs/SERVING-COORDINATION.md documents the operator surface with an example. Still to come: performing the provider wake behind WakeThenRetry through the adapter sleep/wake commands, and sharing one in-flight response between joined requests. The PR body says Part of RAM-45, so the card stays open. Verification: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo build --workspace --locked, cargo test --workspace --locked all pass (24 suites ok), with zero /tmp leftovers across consecutive runs.

Breadway commented

2026-10-12 20:43:44 +08:00

Author

Owner

More gateway on the same branch (PR #33 head fdf8133, rebased onto main 19ab3dc).

What landed this pass is the provider wake on the request path, which the card names as wiring SleepDecision, DrainOutcome, and WakeHealth:

A route may carry a wake spec (wake_model plus optional load/health paths). An arrival at a sleeping backend begins the wake in the lifecycle and spawns the provider call, answering the client 503 plus Retry-After immediately.
The driver POSTs the load with the exact model id, then polls the listing until that exact id carries the loaded status (structured, never a substring match), bounded by the lifecycle's own wake_timeout_ms. A timeout returns the backend to a state a later request can wake from; once max_wake_attempts is spent the backend is failed and arrivals are refused with the reason.
Routing wakes from a confirmed sleep AND from an unverified backend. The tests caught that gap: a failed wake left the backend unverified, and routing only woke from Sleeping, so one failed attempt stranded the backend with no state a request could move. Fixed and pinned by a unit test plus an end-to-end one.
Config: gateway.routes gains wake_model, wake_load_path, wake_health_path with validation.
Four new end-to-end tests with a stateful router fake: wake-then-serve-on-retry, a wake that never completes returning to a wakeable state, waking without a spec still timing out with no provider call, and exhausted attempts failing open with a reason. Full unit suite 370 pass.

The sleep side stays provider work by design: the gateway only answers arrivals and never decides to sleep a backend, so SleepDecision and DrainOutcome remain the adapters' to drive. The PR body says Part of RAM-45.

Verification: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo build --workspace --locked, cargo test --workspace --locked all pass.

More gateway on the same branch (PR #33 head fdf8133, rebased onto main 19ab3dc). What landed this pass is the provider wake on the request path, which the card names as wiring SleepDecision, DrainOutcome, and WakeHealth: - A route may carry a wake spec (wake_model plus optional load/health paths). An arrival at a sleeping backend begins the wake in the lifecycle and spawns the provider call, answering the client 503 plus Retry-After immediately. - The driver POSTs the load with the exact model id, then polls the listing until that exact id carries the loaded status (structured, never a substring match), bounded by the lifecycle's own wake_timeout_ms. A timeout returns the backend to a state a later request can wake from; once max_wake_attempts is spent the backend is failed and arrivals are refused with the reason. - Routing wakes from a confirmed sleep AND from an unverified backend. The tests caught that gap: a failed wake left the backend unverified, and routing only woke from Sleeping, so one failed attempt stranded the backend with no state a request could move. Fixed and pinned by a unit test plus an end-to-end one. - Config: [[gateway.routes]] gains wake_model, wake_load_path, wake_health_path with validation. - Four new end-to-end tests with a stateful router fake: wake-then-serve-on-retry, a wake that never completes returning to a wakeable state, waking without a spec still timing out with no provider call, and exhausted attempts failing open with a reason. Full unit suite 370 pass. The sleep side stays provider work by design: the gateway only answers arrivals and never decides to sleep a backend, so SleepDecision and DrainOutcome remain the adapters' to drive. The PR body says Part of RAM-45. Verification: cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo build --workspace --locked, cargo test --workspace --locked all pass.

Breadway commented

2026-10-12 21:08:34 +08:00

Author

Owner

Follow-up on the same branch (PR #33 head 92b2653).

A live probe caught a real gap the tests missed: the daemon seeds every backend Awake, and nothing ever asked the provider what it actually was. Pointing breadramd at a router whose model was already unloaded forwarded the first request straight through (200) instead of waking it, because routing only wakes a backend it believes is asleep.

Fixed: before the listener opens, and only then, each route with a wake spec is probed once through the same structured listing the wake verifies against, and the backend is reconciled. loaded means serving, any other reported status means asleep, unverified when the provider cannot be read. Reconcile drops leases, which is why it runs before serving and never again. Pinned by startup_reconcile_finds_a_backend_that_is_already_asleep.

Verified live against llama-server build 11146 in router mode with Qwen3 8B, context capped at 4096: first request to the gateway answered 503 with the wake reason, the gateway called POST /models/load itself, the next request answered 200 with the model loaded. Device memory about 6.3GB loaded against 0.4GB after unload. The operator's own server on port 8080 was not touched.

Full workspace suite green.

Follow-up on the same branch (PR #33 head 92b2653). A live probe caught a real gap the tests missed: the daemon seeds every backend Awake, and nothing ever asked the provider what it actually was. Pointing breadramd at a router whose model was already unloaded forwarded the first request straight through (200) instead of waking it, because routing only wakes a backend it believes is asleep. Fixed: before the listener opens, and only then, each route with a wake spec is probed once through the same structured listing the wake verifies against, and the backend is reconciled. loaded means serving, any other reported status means asleep, unverified when the provider cannot be read. Reconcile drops leases, which is why it runs before serving and never again. Pinned by startup_reconcile_finds_a_backend_that_is_already_asleep. Verified live against llama-server build 11146 in router mode with Qwen3 8B, context capped at 4096: first request to the gateway answered 503 with the wake reason, the gateway called POST /models/load itself, the next request answered 200 with the model loaded. Device memory about 6.3GB loaded against 0.4GB after unload. The operator's own server on port 8080 was not touched. Full workspace suite green.

Breadway commented

2026-10-12 21:16:12 +08:00

Author

Owner

Follow-up on the same branch (PR #33 head 67ca1e5).

The gateway tracked leases, but the planner never saw them: the monitor pass built activity only from the adapter registry, and the gateway task shared nothing with it. A request in flight therefore could not stop an unload.

Fixed: the gateway task publishes the scopes that currently hold a lease, and the monitor pass treats a published scope as active evidence on its own, ahead of whatever the provider last answered. A request in flight now keeps its backend loaded, and the planner still never learns that a gateway exists. Pinned by a_lease_in_flight_is_reported_as_an_active_scope, which checks the scope appears while a lease is held and disappears once it finishes.

Full workspace suite green.

Follow-up on the same branch (PR #33 head 67ca1e5). The gateway tracked leases, but the planner never saw them: the monitor pass built activity only from the adapter registry, and the gateway task shared nothing with it. A request in flight therefore could not stop an unload. Fixed: the gateway task publishes the scopes that currently hold a lease, and the monitor pass treats a published scope as active evidence on its own, ahead of whatever the provider last answered. A request in flight now keeps its backend loaded, and the planner still never learns that a gateway exists. Pinned by a_lease_in_flight_is_reported_as_an_active_scope, which checks the scope appears while a lease is held and disappears once it finishes. Full workspace suite green.

Breadway commented

2026-10-12 21:19:49 +08:00

Author

Owner

The sleep half of the loop, on the same branch (PR #33 head 5318955).

The daemon already unloads a confirmed-idle model through the adapter, but the gateway never learned it had happened, so the next request was forwarded into a backend that was no longer serving.

Fixed: a confirmed UnloadIdleModel is handed to the gateway, which marks the backend asleep with Backend::unloaded, and the next arrival wakes it on the request path. An unload that races a live lease loses: the request finishes and the backend stays awake, pinned by an_unload_that_races_a_request_does_not_strand_it. an_unload_of_an_idle_backend_puts_it_to_sleep checks the backend then wakes on the next request.

That completes the loop the card asks for: a quiet backend is unloaded, the gateway records the sleep, and the next request wakes it and is served after the retry hint rather than failing. Full workspace suite green. Moving to review.

The sleep half of the loop, on the same branch (PR #33 head 5318955). The daemon already unloads a confirmed-idle model through the adapter, but the gateway never learned it had happened, so the next request was forwarded into a backend that was no longer serving. Fixed: a confirmed UnloadIdleModel is handed to the gateway, which marks the backend asleep with Backend::unloaded, and the next arrival wakes it on the request path. An unload that races a live lease loses: the request finishes and the backend stays awake, pinned by an_unload_that_races_a_request_does_not_strand_it. an_unload_of_an_idle_backend_puts_it_to_sleep checks the backend then wakes on the next request. That completes the loop the card asks for: a quiet backend is unloaded, the gateway records the sleep, and the next request wakes it and is served after the retry hint rather than failing. Full workspace suite green. Moving to review.

Breadway commented

2026-10-12 21:20:53 +08:00

Author

Owner

Reviewed and merged PR #33 (head 5318955, merge 58bc71b). The gateway admits every request through a lease, wakes a sleeping or unverified backend on the request path with an exact model-id match, reconciles an already-asleep backend before the listener opens, publishes in-flight leases as active evidence, and records a confirmed idle unload so the next request wakes it. CI was green and the PR was mergeable.

Reviewed and merged PR #33 (head 5318955, merge 58bc71b). The gateway admits every request through a lease, wakes a sleeping or unverified backend on the request path with an exact model-id match, reconciles an already-asleep backend before the listener opens, publishes in-flight leases as active evidence, and records a confirmed idle unload so the next request wakes it. CI was green and the PR was mergeable.

Breadway closed this issue

2026-10-12 21:20:53 +08:00

Rows
Columns

Add the optional local inference gateway for request tracking #13

Scope

Acceptance criteria