Add the optional local inference gateway for request tracking

Breadway commented

2026-10-11 22:59:17 +08:00

Owner

The optional local inference gateway: request-path routing, the socket server, and the operator configuration. Part of RAM-45 (provider sleep/wake wiring remains).

What this adds

breadram-core/src/serving/gateway.rs: Gateway::route maps a request and a backend state to a decision with no I/O of its own: Forward holds a lease for the request's whole duration (Started, Joined, or Replayed); WakeThenRetry begins the wake on the request path and tells the client when to retry; Retry for draining, waking, or unverified backends with the policy hint; Refused for unsupported, failed, or unknown backends. An untracked backend is Unsupported, so routing refuses rather than serving through a path that could be slept from under it.
breadram-core/src/serving/server.rs: the socket server over the routing core. One request per connection; heads parsed for routing metadata only; bodies relayed byte for byte both ways, including streams until either side closes. A disconnect releases the client's leases at once, including while waiting on a slow backend. 503 plus Retry-After for retryable backends, 502 for backends that cannot serve, 404 for unrouted paths, 400/413 for unusable heads and bodies. Bounds: 64 KiB heads, 64 MiB non-streamed bodies, head/dial timeouts.
breadram-core/src/serving/http.rs: a response-head parser beside the request parser (a status line is not a request line), sharing one header-block parser and bound.
Operator configuration: [gateway] with a required loopback listen address plus longest-prefix routes with loopback backend addresses, validated at load; the daemon serves it while it runs. Tracking per route defaults to complete and may be bypassable or unknown.
The gateway presents backend leases as policy activity evidence: a live lease reports Active, an unknown workload reports Unknown.
docs/SERVING-COORDINATION.md documents the server, the operator surface with an example, and the boundaries.
Nothing stores a body: outcomes carry only a client label, a request kind, streaming, and an idempotency key.

Boundaries, stated not dropped

Every Forward outcome is proxied, so a joined or replayed request duplicates backend work that response sharing would avoid; the core still counts it once.
WakeThenRetry records the wake in the lifecycle and serves the hint; performing the provider wake through SleepDecision, DrainOutcome, and WakeHealth belongs to the adapter work (RAM-13/RAM-14).

Verification

Run on this branch rebased onto main (44a23a5):

cargo fmt --all --check
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo build --workspace --locked
cargo test --workspace --locked

All pass (24 suites ok). New tests prove byte-identical proxying against a direct hit, verbatim stream and chunked relay, lease release on disconnect, 503 plus Retry-After with no dispatch to a sleeping backend, 502 for untracked backends, 404/400/413 refusals, gateway config validation, and a live daemon serving a fake backend through its configured gateway.

Part of RAM-45.

The optional local inference gateway: request-path routing, the socket server, and the operator configuration. Part of RAM-45 (provider sleep/wake wiring remains). ## What this adds - breadram-core/src/serving/gateway.rs: Gateway::route maps a request and a backend state to a decision with no I/O of its own: Forward holds a lease for the request's whole duration (Started, Joined, or Replayed); WakeThenRetry begins the wake on the request path and tells the client when to retry; Retry for draining, waking, or unverified backends with the policy hint; Refused for unsupported, failed, or unknown backends. An untracked backend is Unsupported, so routing refuses rather than serving through a path that could be slept from under it. - breadram-core/src/serving/server.rs: the socket server over the routing core. One request per connection; heads parsed for routing metadata only; bodies relayed byte for byte both ways, including streams until either side closes. A disconnect releases the client's leases at once, including while waiting on a slow backend. 503 plus Retry-After for retryable backends, 502 for backends that cannot serve, 404 for unrouted paths, 400/413 for unusable heads and bodies. Bounds: 64 KiB heads, 64 MiB non-streamed bodies, head/dial timeouts. - breadram-core/src/serving/http.rs: a response-head parser beside the request parser (a status line is not a request line), sharing one header-block parser and bound. - Operator configuration: [gateway] with a required loopback listen address plus longest-prefix routes with loopback backend addresses, validated at load; the daemon serves it while it runs. Tracking per route defaults to complete and may be bypassable or unknown. - The gateway presents backend leases as policy activity evidence: a live lease reports Active, an unknown workload reports Unknown. - docs/SERVING-COORDINATION.md documents the server, the operator surface with an example, and the boundaries. - Nothing stores a body: outcomes carry only a client label, a request kind, streaming, and an idempotency key. ## Boundaries, stated not dropped - Every Forward outcome is proxied, so a joined or replayed request duplicates backend work that response sharing would avoid; the core still counts it once. - WakeThenRetry records the wake in the lifecycle and serves the hint; performing the provider wake through SleepDecision, DrainOutcome, and WakeHealth belongs to the adapter work (RAM-13/RAM-14). ## Verification Run on this branch rebased onto main (44a23a5): - cargo fmt --all --check - cargo clippy --workspace --all-targets --locked -- -D warnings - cargo build --workspace --locked - cargo test --workspace --locked All pass (24 suites ok). New tests prove byte-identical proxying against a direct hit, verbatim stream and chunked relay, lease release on disconnect, 503 plus Retry-After with no dispatch to a sleeping backend, 502 for untracked backends, 404/400/413 refusals, gateway config validation, and a live daemon serving a fake backend through its configured gateway. Part of RAM-45.

Breadway added 1 commit

2026-10-11 22:59:17 +08:00

feat: add the gateway request-path routing core

CPU checks / check (push) Successful in 51s

Details

a4421ae1bf

First piece of the optional local inference gateway: the part a socket server
drives, with no I/O of its own, so it is fully testable against a hand-advanced
clock.

- Gateway::route maps a request and a backend state to Forward (holding a lease),
  WakeThenRetry (a demand-driven wake begun on the request path, with a retry
  hint), Retry (draining, waking, or unverified), or Refused (unsupported,
  failed, or unknown workload).
- A stream holds its lease until finish; a disconnect releases every lease of the
  client; an idempotent retry joins a running request or is recognized as
  replayed rather than counted twice.
- An untracked backend is Unsupported, so routing refuses rather than serving
  through a path that could be slept from under it.
- Nothing stores a body, so a gateway built on this cannot log a prompt or a
  completion.
- Tests cover forwarding, idempotent join, stream hold plus disconnect release,
  demand-driven wake then retry, the untracked refusal, and replay after finish.
- docs/SERVING-COORDINATION.md records the core and what remains.

Breadway referenced this pull request

2026-10-11 22:59:18 +08:00

Add the optional local inference gateway for request tracking #13

Breadway added 1 commit

2026-10-11 23:25:09 +08:00

feat: classify inbound requests and map outcomes to dispositions

CPU checks / check (push) Successful in 1m3s

Details

8375f7973f

Next piece of the gateway: the header-only request classification and the HTTP
disposition, so a socket server has what it needs without reading a body.

- classify reads a client label, an idempotency key, and whether the client
  asked for a stream, from headers only. No body is read, so a prompt or
  completion cannot be captured.
- disposition maps a routing outcome to Proxy (streaming when asked), Retry with
  the hint, or Refuse, which is what the server turns into 200/503/501.
- Tests cover header classification with safe defaults and each disposition.

Breadway added 1 commit

2026-10-11 23:29:58 +08:00

feat: add the bounded HTTP request-head parser for the gateway

CPU checks / check (push) Successful in 58s

Details

126dc4c30a

The socket server reads a request line and headers and passes the body through,
so the parser is pure, bounded, and never touches content.

- parse_head reads the method, target, and headers from a buffer, returning
  Incomplete until the blank line is present, TooLarge past 64 KiB or 128
  headers, and Malformed for a non HTTP/1.x line or a bad content length.
- RequestHead exposes case-insensitive header lookup and wants_stream, and
  reports chunked transfer and content length rather than guessing.
- Tests cover a complete head, incomplete and oversized heads, malformed lines,
  and the chunked and content-length cases.

Breadway added 1 commit

2026-10-12 00:18:59 +08:00

feat: present gateway leases as policy activity evidence

CPU checks / check (push) Successful in 59s

Details

4be2dc8452

The gateway now implements the policy's ActivitySource: each backend's leases are
its activity evidence for the workload it serves. A live lease reports Active, so
a workload in use is never released, and an unknown workload reports Unknown,
never idle. This is the bridge the card asks for: the policy sees activity
without learning that a gateway or a lease exists.

Breadway referenced this pull request

2026-10-12 00:19:16 +08:00

Add the optional local inference gateway for request tracking #13

Breadway force-pushed feature/ram-45-gateway-core from 4be2dc8452

CPU checks / check (push) Successful in 59s

Details

to e2466766c5

CPU checks / check (push) Successful in 42s

Details

2026-10-12 09:24:32 +08:00

Compare

Breadway force-pushed feature/ram-45-gateway-core from e2466766c5