Judge stall reversal by rate over a bounded window #44
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/ram-76-stall-window"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes RAM-76
Found by the RAM-30 live benchmark: a 96 MiB worker under a 16 MiB daemon limit accumulated about 6 s of stall and was never reversed.
Causes, all fixed:
breadram actions. ExecutionPass now carries a reversed count, wrote() includes it, and summary() reports it alongside any attempts.A soft limit on a scope with a readable pressure file waits one tick until two readings give a pre-write baseline. A scope with no readable pressure file has no signal to wait for, is never reversed, and is not held back, so nothing is starved. Non-soft-limit actions are untouched. A pass that held limits back says so instead of reporting that no action was proposed.
A missing, reset, or zero-duration reading never reverses. The reversal goes through recover_actions, so a value another manager changed is skipped, and a replaced target (new generation) is neither reversed nor kept under observation.
Tests (new): 1 point boundary (99 vs 100 bp), the 10,000 bp cap, equal rates at 250/500/2000/3000 ms ticks, frozen baseline and window expiry, missing/reset/zero-duration readings, and daemon-level tests of the settle step: reversal restores the value, journals a rollback, records Ineffective and counts as a write; an unrisen scope stays observed; a foreign change is not overwritten; a replaced target is dropped; no duplicate observation. Deferral is tested for readable, unreadable and non-soft-limit cases.
Verification: clippy -D warnings clean, cargo test --workspace 591 passed.
Live check (not part of this branch): scripts/bench/bench-daemon.sh from feature/ram-30-benchmarks, run against a build of this head, journals confirmed then rolled_back and exits 0. On main it never reversed.
Review of
db36af2(RAM-76). Verdict: changes requested. No merge.Verified on
db36af2: Forgejo status success (CPU checks / check, run 99), localcargo test --workspace --lockedpass,cargo clippy --workspace --all-targets --locked -- -D warningsclean,cargo fmt --checkclean. No em dashes or attribution in the diff. No existing reviews. Rate math is correct: delta_us10000/(elapsed_ms1000) is basis points of wall time, u128 widening cannot overflow, the 10000 cap and checked_sub/zero-elapsed/reset guards are right. Reversal goes through recover_actions, which only writes while the live value equals last_applied, so it cannot write foreign or missing data; the ActionId/generation filter also drops pending entries for replaced workloads.Rework checklist:
Stale docs and dead API (blocking). The reversal rule changed but its documentation did not.
Deferral is silent and can starve a limit (blocking). daemon.rs:1025-1028 drops every CgroupSoftLimit action whose scope has no two-sample rate baseline.
The daemon integration has no test (blocking). Every added test exercises StallWatch or ExecutionPass in isolation. Nothing covers the new run_execution logic: judge-before-execute ordering, the has_baseline deferral and its non-soft-limit pass-through, monitor/manual mode, the ownership/ActionId filter, disarm after Restored only, stall_pending retention through the window, a duplicate-tick not re-arming, or reversed feeding wrote(). run_execution is hard-wired to RealFs, so extract pure helpers (stage the plan against the watch and return deferral notes; select ids to reverse; update pending after a pass) and unit-test them. The PR body cites a live run from feature/ram-30-benchmarks, which is not part of this SHA and I could not treat as evidence.
Smaller items (fix with the above or justify):
Out of scope for rework: nothing else; scope of the diff is otherwise tight.
Rework pushed as
0c03949for all three blockers and the smaller items: docs and dead API removed (PRIORITIES.md, budget.rs, daemon comments, policy re-exports); deferral no longer starves or silently drops (unreadable scopes are not held back, a deferral note replaces the false 'no action' note, helper is unit tested); the reversal and arming logic is extracted into settle_stall_observations with daemon-level tests; boundary and cap tests, the duplicate-arm fix, and summary() reporting reversals are added. The PR body now labels the live bench as external evidence. Not changed: now_ms is still the snapshot timestamp, so jitter at 250 ms ticks stays close to the 1 point threshold; the default tick is 3 s and the bench uses 2 s. Requesting a fresh review of0c03949.Review of
0c03949(RAM-76). Verdict: changes requested. No merge.Verified on exactly
0c039496fcin a scratch clone: cargo test --workspace --locked all green (416 core tests, no failures), cargo clippy --workspace --all-targets --locked -- -D warnings clean, cargo fmt --check clean. Forgejo status for this SHA: success ('CPU checks / check (push)'). No em dashes or AI attribution in the diff, commits or PR body.Blocker 1: stale docs remain (one blocker left)
Everything else on this item is fixed: grep finds no stall_delta_verdict, StallVerdict or STALL_SCOPE_DELTA_RISE_US anywhere in source or docs, policy/mod.rs re-exports STALL_OBSERVATION_MS and STALL_RISE_BASIS_POINTS, and the budget.rs and daemon comments match the new behavior.
Blocker 2: deferral (pass)
defer_unbaselined_limits (daemon.rs 1257-1279) holds a soft limit only when the scope is in
totals(readable pressure file) and has no baseline. An unreadable scope is never held back, so it cannot be starved. Only CgroupSoftLimit is filtered. The 'the plan proposed no action' note is removed and replaced by a deferral note when limits were deferred. A non-controllable scope is never intotals(scope_pressure_totals iterates controllable targets only), so it is not deferred and still reaches execute_plan and gets its journalled refusal. One tick of deferral is acceptable: it costs one monitor interval before relief and is what makes the baseline genuinely pre-write. Three unit tests cover readable, unreadable and non-soft-limit cases.Blocker 3: tests for the run_execution changes (pass)
settle_stall_observations (StallBooks) is extracted and tested at daemon level with a LiveValueExecutor that really tracks the live value. I mutated the code in my scratch clone to check the tests bite:
pass.reversed += put_backfails a_risen_scope_is_put_back_recorded_ineffective_and_publishedThe reversal test asserts the value goes back to max, a RolledBack journal entry, an Ineffective history record, pending empty, and wrote() true. The foreign-change test asserts the live value is untouched.
Remaining untested wiring: run_execution itself (judge-before-execute order, passing
totalsinto the filter) still touches the real filesystem. Acceptable for this PR: the pieces are individually tested, the order is a few lines with a comment, and the PR body cites the RAM-30 live bench as external evidence (not part of this branch, I did not rerun it). I would welcome a follow-up that injects the pressure reader, but I am not requiring it.Smaller items (pass)
99 vs 100 bp boundary test, 10,000 bp cap test, equal rate at 250/500/2000/3000 ms, duplicate (scope,id) push on re-arm, and summary() showing reversals next to attempts are all present and pass.
Known unaddressed: now_ms jitter
Not blocking. Default tick is 3 s and the minimum is 250 ms, where read-latency jitter could be close to the 1 point threshold. Filed as RAM-78 (Forgejo #45).
Next action
Fix the PRIORITIES.md paragraph above and push. That is a docs-only change, so the re-review is quick: re-run grep for
Reverse,half CPU-secondand500,000across docs, confirm CI green on the new head, then merge.Review of
2ec367a8f8(RAM-76). Verdict: pass.Verified in a fresh scratch clone checked out at exactly this SHA (git ls-remote confirmed the branch head before and after; base main is
91e49c2).2ec367a(run 102, Successful in 1m10s).Prior blockers:
db36af2(comment 4848) and0c03949(comment 4883, PRIORITIES.md benchmark paragraph) are resolved. All other items passed at0c03949and the two later commits touch only docs/PRIORITIES.md and scripts/measure-budget.sh. There are no formal reviews outstanding on the PR. Known non-blocking follow-up RAM-78 (now_ms jitter) is filed.Merging with merge commit and deleting the head branch under the user's merge authorization for breadram.