training_reduces_loss_and_is_seeded_deterministic failed once in a full workspace run

Breadway commented

2026-10-08 23:14:30 +08:00

Owner

How it was found: running cargo test --release --workspace on branch feature/model-v1.1 (hermes, other CPU load from an unrelated build running at the time). omnia-training test training_reduces_loss_and_is_seeded_deterministic panicked at crates/omnia-training/tests/training.rs:96 (the seeded-runs-diverged assertion).

What happens: the test did not reproduce. It passed alone, in 3 further runs of the training test file, and in a second full workspace run.

Confidence: that the failure happened is certain; the cause is not diagnosed. Roughly 55% that this is pre-existing flakiness, because the v1.1 changes leave the v1 numeric path untouched (new modules are None, the cat mask is skipped), and the test compares two runs with a 1e-4 tolerance. My guess is the data-parallel gradient summation order in ndarray under CPU contention, which is also not bit-deterministic (the same effect is noted for Novacana's trainer). I did not run the test against the base commit repeatedly.

Suggested fix: loop the test 50 times under artificial load on the base commit to confirm it is pre-existing, then loosen the tolerance or make the reduction order deterministic.

**How it was found:** running `cargo test --release --workspace` on branch feature/model-v1.1 (hermes, other CPU load from an unrelated build running at the time). `omnia-training` test `training_reduces_loss_and_is_seeded_deterministic` panicked at `crates/omnia-training/tests/training.rs:96` (the seeded-runs-diverged assertion). **What happens:** the test did not reproduce. It passed alone, in 3 further runs of the `training` test file, and in a second full workspace run. **Confidence:** that the failure happened is certain; the cause is not diagnosed. Roughly 55% that this is pre-existing flakiness, because the v1.1 changes leave the v1 numeric path untouched (new modules are `None`, the cat mask is skipped), and the test compares two runs with a 1e-4 tolerance. My guess is the data-parallel gradient summation order in ndarray under CPU contention, which is also not bit-deterministic (the same effect is noted for Novacana's trainer). I did not run the test against the base commit repeatedly. **Suggested fix:** loop the test 50 times under artificial load on the base commit to confirm it is pre-existing, then loosen the tolerance or make the reduction order deterministic.

Breadway added the

bug

label

2026-10-08 23:14:30 +08:00

Breadway self-assigned this

2026-10-15 14:38:13 +08:00

Breadway commented

2026-10-20 09:26:59 +08:00

Author

Owner

Reproduced the seeded divergence and the lock patch removes it under the same soak. Unpatched release binary, --test-threads=16, eight sha256sum /dev/zero loops: 54 pass / 1 fail in 55 runs (training.rs:96, 2.0957823 vs 2.1332889). Patched: 47 pass / 0 fail in 47 runs. Tolerance 1e-4 unchanged.

PR: #31
Head: 9561c6bd4c

Card left in doing for independent review. Not merged.

Reproduced the seeded divergence and the lock patch removes it under the same soak. Unpatched release binary, --test-threads=16, eight sha256sum /dev/zero loops: 54 pass / 1 fail in 55 runs (training.rs:96, 2.0957823 vs 2.1332889). Patched: 47 pass / 0 fail in 47 runs. Tolerance 1e-4 unchanged. PR: https://git.breadway.dev/Breadway/omniastra/pulls/31 Head: 9561c6bd4ccf499b17187bd74b80c8e61a9669ae Card left in doing for independent review. Not merged.

Rows
Columns

training_reduces_loss_and_is_seeded_deterministic failed once in a full workspace run #29