Add model unload and load verbs with demand reload

Breadway commented

2026-10-13 19:44:21 +08:00

Owner

Companion to breadram RAM-18 (breadsearch model-release adapter, breadram PR 42, merged). breadram can only release the embedding session through a supported integration point, and none existed: the session loaded once at startup with no unload path, no in-flight counter, and reload only by restart.

This adds unload_model (refused while queries run or a load runs, otherwise drops the session; config, model files, and indexed chunks untouched), load_model (idempotent background load, readiness polled through status), and query-triggered demand reload (a search that finds the model gone starts loading and answers that it is loading). Status gains in_flight, loading, model_bytes, and version, all optional so older clients still parse. Verified against a real breadmill on CPU: unload freed about 400 MB resident, demand reload completed in under 20 s, and a query embeds after it. Default-feature build note: breadmill currently needs --features breadmill/full against the pinned bread-onnx v0.7.2 (filed separately).

Head: bc4b2d9

Companion to breadram RAM-18 (breadsearch model-release adapter, breadram PR 42, merged). breadram can only release the embedding session through a supported integration point, and none existed: the session loaded once at startup with no unload path, no in-flight counter, and reload only by restart. This adds unload_model (refused while queries run or a load runs, otherwise drops the session; config, model files, and indexed chunks untouched), load_model (idempotent background load, readiness polled through status), and query-triggered demand reload (a search that finds the model gone starts loading and answers that it is loading). Status gains in_flight, loading, model_bytes, and version, all optional so older clients still parse. Verified against a real breadmill on CPU: unload freed about 400 MB resident, demand reload completed in under 20 s, and a query embeds after it. Default-feature build note: breadmill currently needs --features breadmill/full against the pinned bread-onnx v0.7.2 (filed separately). Head: bc4b2d9

Breadway added 1 commit

2026-10-13 19:44:21 +08:00

feat: add model unload and load verbs with demand reload

check / check (push) Successful in 5m22s

Details

bc4b2d98d2

The embedder lives behind the shared mutex as an Option, so dropping the
session is always safe. UnloadModel refuses while queries run or a load
runs and otherwise drops the session, freeing backend allocations while
configuration, model files, and indexed chunks stay put. LoadModel starts
the same background load the daemon uses at startup, idempotent while one
runs. A query that finds the model gone starts a reload and answers that
the model is loading, so the triggering search retries and the next finds
a warm model. Status gains in_flight, loading, model_bytes, and version,
all optional so older clients still parse.

Breadway added 2 commits

2026-10-20 08:15:34 +08:00

feat: report queries served in the breadmill status

check / check (push) Successful in 2m18s

Details

7e5eef9b2e

Merge pull request 'Report queries served in the breadmill status' (#16 ) from feature/status-queries-served into feature/model-unload-verbs

check / check (push) Successful in 23s

Details

287ea872b2