Coordinate demand-driven model sleep and wake #12

Merged

Breadway merged 1 commit from feature/ram-11-serving-coordination into main

2026-10-11 15:14:52 +08:00

Breadway commented

2026-10-11 15:14:46 +08:00

Owner

Request leases (`serving/lease.rs`)

A lease records what is in flight and nothing else: a client label, a request kind, whether the response is streamed, an optional idempotency key, and monotonic timestamps. There is nowhere to put a prompt, a completion, a response body, or a credential, so "the gateway never logs prompts" is checkable by reading the type instead of auditing call sites.

A retry with the same key while the request is in flight joins it and is counted once; a retry after it finished is replayed rather than dispatched; a cancelled or disconnected request releases its lease and is not remembered as finished, so a retry actually runs.
Queued requests hold a lease, streams hold theirs for their whole duration, and the retry memory is bounded so a busy gateway does not grow without bound.
Idleness is reported as the ActivityEvidence the planner already consumes: Active while anything is in flight, Idle { since_ms } from the moment the last lease ended. A workload with a live lease is never released, even when its processes are quiet.

Lifecycle (`serving/lifecycle.rs`)

One state machine per backend: awake, draining, sleeping, waking, unverified, unsupported, failed. It owns no clock and no socket, so it decides what is allowed and the caller performs the provider call, which is what makes the races testable.

An arrival during a drain is refused with a retry hint, or cancels the drain, depending on operator policy. A drain that cannot finish inside its deadline is abandoned and the backend stays awake rather than sitting in limbo.
Only a verified healthy wake returns a backend to serving. An unhealthy wake or a timeout spends an attempt and returns the backend to a state a later demand can wake from; after the bounded budget it becomes failed and the client error names the reason and the attempt count.
Unprovable idleness, because clients can bypass tracking and the provider cannot prove it has no work in flight, marks automatic unload unsupported with a reportable reason, unless the operator explicitly allows it with allow_untracked_unload.
A restart reconciles from what the provider reports: sleeping means a request demands a wake, serving means serve, and no answer means health is verified before serving rather than guessed.

Verification

cargo fmt --all -- --check clean, cargo clippy --workspace --all-targets -- -D warnings clean, cargo test --workspace 294 tests pass, including 18 new ones that drive a hand advanced clock through busy rejection, the idle minimum, arrival during a drain, an operator that prefers the client, a long stream, a mid-stream disconnect, queued work, retries, a demand-driven wake, an unhealthy wake, three failed attempts, a timed-out wake, an out-of-order sleep confirmation, unprovable idleness, a restart with and without a provider answer, policy clamping, and the evidence the planner sees.

docs/SERVING-COORDINATION.md documents the contract, including what is not implemented yet.

Part of RAM-11. This is the coordination core: request leases and the drain, sleep, and wake lifecycle. The optional local inference gateway that would feed it from real traffic is not in this change, so RAM-11 stays open and this pull request deliberately has no closing keyword. ## Request leases (`serving/lease.rs`) A lease records what is in flight and nothing else: a client label, a request kind, whether the response is streamed, an optional idempotency key, and monotonic timestamps. There is nowhere to put a prompt, a completion, a response body, or a credential, so "the gateway never logs prompts" is checkable by reading the type instead of auditing call sites. - A retry with the same key while the request is in flight joins it and is counted once; a retry after it finished is replayed rather than dispatched; a cancelled or disconnected request releases its lease and is not remembered as finished, so a retry actually runs. - Queued requests hold a lease, streams hold theirs for their whole duration, and the retry memory is bounded so a busy gateway does not grow without bound. - Idleness is reported as the `ActivityEvidence` the planner already consumes: `Active` while anything is in flight, `Idle { since_ms }` from the moment the last lease ended. A workload with a live lease is never released, even when its processes are quiet. ## Lifecycle (`serving/lifecycle.rs`) One state machine per backend: awake, draining, sleeping, waking, unverified, unsupported, failed. It owns no clock and no socket, so it decides what is allowed and the caller performs the provider call, which is what makes the races testable. - An arrival during a drain is refused with a retry hint, or cancels the drain, depending on operator policy. A drain that cannot finish inside its deadline is abandoned and the backend stays awake rather than sitting in limbo. - Only a verified healthy wake returns a backend to serving. An unhealthy wake or a timeout spends an attempt and returns the backend to a state a later demand can wake from; after the bounded budget it becomes `failed` and the client error names the reason and the attempt count. - Unprovable idleness, because clients can bypass tracking and the provider cannot prove it has no work in flight, marks automatic unload `unsupported` with a reportable reason, unless the operator explicitly allows it with `allow_untracked_unload`. - A restart reconciles from what the provider reports: sleeping means a request demands a wake, serving means serve, and no answer means health is verified before serving rather than guessed. ## Verification `cargo fmt --all -- --check` clean, `cargo clippy --workspace --all-targets -- -D warnings` clean, `cargo test --workspace` 294 tests pass, including 18 new ones that drive a hand advanced clock through busy rejection, the idle minimum, arrival during a drain, an operator that prefers the client, a long stream, a mid-stream disconnect, queued work, retries, a demand-driven wake, an unhealthy wake, three failed attempts, a timed-out wake, an out-of-order sleep confirmation, unprovable idleness, a restart with and without a provider answer, policy clamping, and the evidence the planner sees. `docs/SERVING-COORDINATION.md` documents the contract, including what is not implemented yet.

Breadway added 1 commit

2026-10-11 15:14:46 +08:00

feat: coordinate demand-driven model sleep and wake d233e84ddb

Sleeping a model backend frees the most memory of anything breadram does and
is the only action that can break a client, so the decision needs evidence and
a state machine rather than a timer.

Request leases record what is in flight and nothing else: a client label, a
request kind, whether the response is streamed, an optional idempotency key,
and monotonic timestamps, so a gateway built on top cannot log a prompt or a
completion through these types. A retry joins a request that is still running
instead of counting twice, a retry after it finished is replayed rather than
dispatched, and a cancelled or disconnected request releases its lease so a
client that vanishes mid stream cannot leave a backend permanently busy.

The lifecycle decides what is allowed and reports why; the caller performs the
provider call. A drain that cannot finish in time is abandoned rather than
holding the backend in limbo, nothing is admitted without a verified healthy
wake, and a wake that keeps failing spends a bounded budget and then becomes an
actionable error. Idleness that cannot be proven, because clients can bypass
tracking and the provider cannot prove it has no work in flight, marks
automatic unload unsupported unless an operator explicitly allows it.

The module owns no clock and no socket, so the races that are hardest to
reproduce against a live provider are covered by tests against a hand advanced
clock: arrival during a drain, a stream outliving an idle check, a drain that
times out, and a wake that fails three times.

Breadway merged commit 856dc7dde2 into main

2026-10-11 15:14:52 +08:00

Breadway deleted branch feature/ram-11-serving-coordination

2026-10-11 15:14:52 +08:00

Breadway referenced this pull request from a commit

2026-10-11 15:14:52 +08:00

Merge pull request 'Coordinate demand-driven model sleep and wake' (#12) from feature/ram-11-serving-coordination into main

Breadway referenced this pull request

2026-10-11 15:15:01 +08:00

Add the optional local inference gateway for request tracking #13

Breadway referenced this pull request

2026-10-11 20:51:24 +08:00