Breadway/breadsearch
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
breadsearch/docs/error-reports/2026-06-24-oom-minified-json-chunking.md
Breadway 2778f14574 Initial commit
2026-07-02 20:59:18 +08:00

8.1 KiB
Raw Blame History

Error Report: breadmill OOM loop on minified JSON .txt file

Date: 2026-06-24
Reporter: breadway
Component: breadmill (semantic search indexer)
Version: 0.1.0 (/home/breadway/Projects/breadsearch/breadmill)
Severity: Critical — repeated kernel OOM kills, ~28 GB RAM consumption, system instability

Summary

breadmill enters a crash loop when indexing a 705 KB minified JSON game save file exposed as .txt. Word-based chunking produces ~486–616 KB chunks that are passed unbounded to the ONNX embedding model. ONNX Runtime then attempts multi-terabyte allocations in attention MatMul, the process balloons to ~28 GB RSS, and the kernel OOM killer terminates it. Restart=on-failure causes an immediate restart and the cycle repeats.

On 2026-06-24 this produced 40 embed errors and 76 OOM kills in journal logs within ~45 minutes.

Environment

Item Value
Host Lenovo Yoga Slim 7 14AKP10 (83JY)
OS BOS (Arch-based), kernel 7.0.12-arch1-1
RAM 32 GB
Swap 4 GB zram
Model nomic-embed-text-v1.5 (768-dim, ONNX via ort 2.0.0-rc.12)
Service breadmill.service (user systemd unit)
Config ~/.config/breadsearch/config.toml

Service unit

[Service]
Type=simple
ExecStart=%h/.cargo/bin/breadmill
Restart=on-failure
RestartSec=5
# No MemoryMax set (unlimited)
# OOMScoreAdjust=200 (default for user services — preferential OOM victim)

Relevant config

[index]
roots = ["~/Documents", "~/.config/breadpad"]
extensions = ["md", "txt", "org", "pdf", "odt", "docx"]
max_file_mb = 5.0

Triggering file

Path:
~/Documents/Gaming/Games/Spaceflight Simulator Game/Saving/Worlds/Bread/Persistent/Rockets.txt

Property Value
Size 721,601 bytes (705 KB)
Type JSON array (11 rocket objects), stored as .txt
Lines 1 (single-line minified JSON)
Whitespace-delimited "words" 601
Longest "word" 68,068 characters

The file is under max_file_mb = 5.0 and has extension .txt, so it is indexed.

Root cause

1. Word-based chunking breaks on minified JSON

chunk::chunk_text() splits on whitespace only (chunk.rs). For minified JSON, most structure is packed into very long tokens with few spaces.

Chunking with words_per_chunk=400, overlap_words=80 (indexer.rs:268) yields:

Chunk Word range Character length
0 0–399 486,347
1 320–600 616,582

A "400-word chunk" is not ~400 natural-language words; it is hundreds of kilobytes of dense JSON.

2. No token/character limit before embedding

embed::OrtEmbedder::embed_with_prefix() tokenizes the full chunk and runs ONNX inference with no max_length truncation (embed.rs). The nomic model supports ~8192 tokens; these chunks are orders of magnitude larger when tokenized.

3. ONNX attention allocation explodes

Embedding fails inside ONNX Runtime:

Non-zero status code returned while running FusedMatMul node.
Name: '/encoder/layers.0/attn/MatMul/MatMulScaleFusion/'
Status Message: ... BFCArena::AllocateRawInternal ...
Failed to allocate memory for requested buffer of size 5525780084992

Requested buffer: ~5.1 TiB (5,525,780,084,992 bytes).

Kernel also logs repeated allocation attempts before OOM:

__vm_enough_memory: pid: NNNN, comm: breadmill,
  bytes: 8796093026304 not enough memory for the allocation   (~8.0 TiB)
  bytes: 7916483514368 not enough memory for the allocation   (~7.2 TiB)
  bytes: 7124834848768 not enough memory for the allocation   (~6.5 TiB)

4. Restart loop amplifies damage

Restart=on-failure + RestartSec=5 restarts breadmill immediately after each OOM kill. Each restart reloads the model, rescans ~/Documents, hits the same file, and OOMs again. Peak memory per attempt: ~27–28 GB RSS (anon-rss:27867432kB).

Log excerpts

Embed error (repeats on every restart)

breadmill: embed error for .../Rockets.txt: Non-zero status code returned while running FusedMatMul node.
Name:'/encoder/layers.0/attn/MatMul/MatMulScaleFusion/' Status Message:
  ... Failed to allocate memory for requested buffer of size 5525780084992

OOM kill

oom-kill: ... task_memcg=.../breadmill.service, task=breadmill, pid=4515, uid=1000
Out of memory: Killed process 4515 (breadmill)
  total-vm:64081848kB, anon-rss:27867432kB, ... oom_score_adj:200
systemd[1649]: breadmill.service: Failed with result 'oom-kill'.
systemd[1649]: breadmill.service: Consumed ... 27.9G memory peak, 2.1G memory swap peak.
systemd[1649]: breadmill.service: Scheduled restart job, restart counter is at 8.

Secondary noise (non-fatal)

breadmill: extract .../Novacana Info.txt: stream did not contain valid UTF-8

Reproduction

  1. Place a single-line minified JSON file (>400 KB, .txt extension) under a configured index root.
  2. Start breadmill serve (or systemctl --user start breadmill.service).
  3. Wait for initial scan to reach the file.
  4. Observe embed errors, climbing RSS, and OOM kills in journalctl --user -u breadmill.service -f.

Minimal reproducer characteristics:

  • Extension in extensions list (e.g. txt)
  • File size < max_file_mb
  • Content is minified JSON or other whitespace-sparse text on one line
  • Results in chunks >> model max_seq_len when tokenized

Impact

  • breadmill unusable while the file is present in index roots
  • System-wide memory pressure: swap exhaustion, unrelated process kills, journal/cache pressure
  • Can be mistaken for unrelated storage or suspend issues when swap write errors appear in kernel log

Suggested fixes

Required (correctness)

  1. Truncate before tokenization — cap input to model max_seq_len (8192 tokens for nomic-embed-text-v1.5) in embed.rs, with explicit logging when truncation occurs.

  2. Character-based chunk limits — add max_chunk_chars (e.g. 8_000–32_000) independent of word count; split oversized chunks before embedding.

  3. Sanity-check chunk size — refuse to embed chunks above a byte/token threshold; log and skip rather than calling ONNX.

Recommended (resilience)

  1. Per-chunk error isolation — on embed failure for one chunk, skip that chunk but do not retry the entire file in a tight loop; mark file as failed in SQLite.

  2. Service memory limit — set MemoryMax=4G (or similar) on breadmill.service so a runaway embed cannot take the whole machine.

  3. Lower OOMScoreAdjust — use 0 or negative so breadmill is not preferentially killed while still allowing limits.

  4. Backoff on repeated OOM — RestartSec=exponential or stop after N OOM kills per file.

Optional (UX)

  1. Exclude patterns — config option for glob excludes (e.g. **/Saving/**, **/*.json even if renamed .txt).

  2. Detect minified JSON — if serde_json::from_str succeeds on .txt, skip or pretty-print/chunk differently.

  3. --version flag — aids bug reports (currently unknown command: --version).

Workaround (immediate)

Exclude the game save directory from index roots in ~/.config/breadsearch/config.toml:

[index]
roots = [
    "~/Documents",
    "~/.config/breadpad",
]
# Then add an exclude mechanism when available, OR temporarily narrow roots:
# roots = ["~/Documents/Creative", "~/.config/breadpad"]

Or stop the service until a fix is deployed:

systemctl --user stop breadmill.service

Files involved

File Role
breadmill/src/chunk.rs Whitespace word chunking — no char/token cap
breadmill/src/indexer.rs:268 chunk_text(&text, 400, 80) — hardcoded params
breadmill/src/embed.rs No truncation before tokenizer.encode() / ONNX run
breadmill/src/extract.rs Treats .txt as raw UTF-8 (JSON passes through)
~/.config/breadsearch/config.toml max_file_mb only; no chunk/token limits

Related

  • ONNX node: /encoder/layers.0/attn/MatMul/MatMulScaleFusion/
  • Model: nomic-embed-text-v1.5 (max sequence length 8192)