Breadway/breadsearch
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial commit

Browse source
This commit is contained in:
Breadway 2026-07-02 20:59:18 +08:00
commit 2778f14574
29 changed files with 7110 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

31
.gitignore vendored Normal file
View file

@ -0,0 +1,31 @@
# Rust build artifacts
target/
# Editor and IDE files
.vscode/
.idea/
*.swp
*.swo
*~
# OS artifacts
.DS_Store
Thumbs.db
desktop.ini
# Environment and secrets
.env
.env.*
*.env
*.pem
*.key
*.p12
secrets/
# Log files
*.log
logs/
# Runtime files
*.sock
*.pid

3722
Cargo.lock generated Normal file
View file

File diff suppressed because it is too large Load diff

3
Cargo.toml Normal file
View file

@ -0,0 +1,3 @@
[workspace]
members = ["breadsearch-shared", "breadmill", "breadsearch"]
resolver = "2"

82
DESIGN.md Normal file
View file

@ -0,0 +1,82 @@
# breadsearch + breadmill — semantic system-wide search for BOS
## Context
BOS/bread has no content search — only breadbox's app launcher (exact/fuzzy over `.desktop` files). The goal is a **semantic** "find anything by meaning" engine: a flagship, differentiating BOS feature that's also the *right* workload for the AMD XDNA NPU (small encoder model, compute-bound single forward pass, always-on background embedding — none of the bandwidth-bound problems that make LLMs a bad NPU fit).
Two components, mirroring the breadpad/breadman split and the breadbox/breadbox-sync precedent (GUI + background helper + shared lib in one repo):
- **breadmill** — always-on daemon: walks files → extracts text → chunks → embeds → vector index; answers queries over a Unix socket. ("mill grain into flour.")
- **breadsearch** — standalone GTK4 GUI, *forked from breadbox's UI*, that queries breadmill and shows ranked hits. ("sift the flour.")
breadbox stays a pure app launcher, unchanged.
## Decisions (confirmed with user)
- **Index scope (v1):** curated roots — `~/Documents`, `~/Projects` (notes/docs, not code yet), `~/.config/breadpad`. Extract `md, txt, org, pdf, odt, docx`. Skip binaries/images/build dirs/`.git`.
- **Repo layout:** one cargo workspace at `~/Projects/breadsearch/`.
- **Embedding model:** `nomic-embed-text-v1.5` (768-dim ONNX, ~550MB). Requires task prefixes: `search_document: ` for indexed chunks, `search_query: ` for queries; mean-pool + L2-normalize.
- **Compute:** CPU-first; NPU (XDNA via ONNX Runtime VitisAI EP) is a later backend swap, not a v1 dependency.
## Workspace layout
```
~/Projects/breadsearch/
Cargo.toml # [workspace] members = breadsearch-shared, breadmill, breadsearch
bakery.toml # bread package manifest (binaries: breadsearch, breadmill)
config.example.toml # ~/.config/breadsearch/config.toml template
README.md
breadsearch-shared/ # lib: XDG paths, config, IPC types + socket client
breadmill/ # daemon bin
breadsearch/ # GUI bin (forked breadbox UI)
```
## Component: breadsearch-shared (lib)
Model on `breadbox/breadbox-shared/src/lib.rs` (XDG helpers + serde/toml config).
- **Paths:** `config_dir()` → `~/.config/breadsearch`; `state_dir()` → `~/.local/state/breadsearch` (index); `cache_dir()` → `~/.cache/breadsearch` (models); `socket_path()` → `$XDG_RUNTIME_DIR/breadmill.sock`.
- **Config** (serde + `toml`): `[index] roots, extensions, max_file_mb`; `[search] limit, snippet_len`; `[model] name, dim`.
- **IPC types** (serde_json, newline-delimited JSON over the Unix socket):
- Request: `Query { query: String, limit: usize }`, `Status`, `Reindex`.
- Response: `Hits(Vec<Hit>)` where `Hit { title, path, snippet, score }`; `StatusInfo { indexed, pending, model_ready }`.
- **Socket client** helper used by the GUI (connect, send, read one response).
## Component: breadmill (daemon)
Pipeline, isolated behind small traits so each stage is swappable:
1. **Walk** — `ignore` crate (parallel, respects `.gitignore`) over configured roots; filter by extension + size.
2. **Extract** — `md/txt/org`: read directly; `pdf`: `pdf-extract`; `docx/odt`: unzip + strip XML (`zip` + `quick-xml`), best-effort.
3. **Chunk** — ~512-token windows with overlap; keep byte offsets for snippets.
4. **Embed** — `Embedder` trait. v1 impl: `ort` (ONNX Runtime 2.x, CPU EP) + `tokenizers` (HF) running nomic-embed-text-v1.5. Apply `search_document:`/`search_query:` prefixes, mean-pool, normalize.
5. **Store** — `rusqlite` for metadata (path, mtime, content-hash, chunk text/offsets) keyed by rowid + `usearch` (HNSW, 768-dim, cosine) for vectors keyed by the same id. Both persisted under `state_dir()`.
6. **Incremental** — on start, diff roots against sqlite (mtime+hash): embed new/changed, drop deleted. Then live-watch with `notify` (debounced) to re-embed on change.
7. **Serve** — `tokio` (or std threads) Unix-socket listener: `Query` → embed query → usearch top-k → join sqlite metadata → `Hits`. Also `Status`/`Reindex`.
- **Model fetch:** first run downloads `model.onnx` + `tokenizer.json` from HF into `cache_dir()/models/` (needs network once); `breadmill --fetch-model` to pre-fetch. Log clearly if absent.
- **Lifecycle:** systemd **user** service `breadmill.service` (pattern from `breadbox-sync.service` / breadd), `WantedBy=default.target`.
## Component: breadsearch (GUI) — fork of breadbox
Start from `breadbox/breadbox/src/main.rs`. **Reuse verbatim:** the gtk4-layer-shell overlay window (rename namespace/app-id to `breadsearch` / `com.breadway.breadsearch`), `SearchEntry` + `ScrolledWindow` + `ListBox`, ↑/↓/Enter/Esc handling, click-outside-to-close, PID-toggle (`breadsearch.pid`), and the theming path: `bread_theme::gtk::apply_shared()` + `apply_app_css(|| build_css(&load_palette()))` + user `style.css`. Pin `bread-theme` git tag `v0.2.8`, feature `gtk` (same as breadbox).
**Swap:**
- **Result source:** delete `load_sorted_entries`/`fuzzy_*`/`DesktopEntry`. On `search.connect_changed`, **debounce ~150ms** (`glib::timeout_add_local`) then query breadmill **off the UI thread** (`std::thread` + `glib::MainContext::channel`), clear the `ListBox`, append a row per `Hit`.
- **Row content:** title (filename/heading) + muted path + snippet line; filetype icon via `gio::content_type_guess` → `Image::from_gicon`. Extend `build_css` with a `.hit-snippet` class.
- **Action:** replace `do_launch` with open-file — `Enter`/row-activated → `xdg-open <path>`; `Ctrl+Enter` → open containing folder. Then close.
## Key crates
`ort` (ONNX Runtime), `tokenizers`, `usearch`, `rusqlite`, `ignore`, `notify`, `pdf-extract`, `zip`+`quick-xml`, `serde`/`serde_json`/`toml`, `gtk4` 0.11 + `gtk4-layer-shell` 0.8, `bread-theme` (git tag v0.2.8).
## Packaging & BOS integration (last phase — post-1.0, per earlier decision)
- `bakery.toml` (model on `breadbox/bakery.toml`): `binaries = ["breadsearch","breadmill"]`, system_deps for onnxruntime/gtk; `[[service]] unit="breadmill.service" enable=true`; `[config] dir="~/.config/breadsearch"`.
- BOS: add `breadsearch`+`breadmill` to `build-local.sh` `BREAD_BINS`; autostart `breadmill.service`; Hyprland keybind (e.g. `SUPER+slash`) → `breadsearch` in the skel `hyprland.lua`.
- Release: dual remotes (origin GitHub + forgejo), bakery index regen — per the bread release train.
## Phasing (de-risked: ship CPU, NPU later)
1. Scaffold workspace + `breadsearch-shared` (paths, config, IPC types, socket client).
2. `breadmill` CPU pipeline end-to-end (walk→extract→chunk→embed→store→serve) + `--reindex`/`--fetch-model` + systemd unit.
3. `breadsearch` GUI fork (socket query + xdg-open + theme).
4. Packaging (bakery, config.example, README) + BOS wiring.
5. **Later:** NPU `Embedder` impl (ort VitisAI/XDNA EP) — the go/no-go POC; pure backend swap.
## Verification
- `breadmill --fetch-model` then `--reindex` over a small test corpus; log embedded-chunk count; confirm `state_dir` index persists across restart.
- Query the socket directly (a `breadmill query "..."` subcommand or `socat`) and confirm semantically-relevant hits with sane scores for a concept query (not keyword).
- Launch `breadsearch`, type a *concept* (e.g. "tax stuff", "that suspend bug fix"), see relevant files ranked, `Enter` opens via xdg-open, `Ctrl+Enter` reveals folder, `Esc` closes; theme matches breadbox; hot-reloads on `bread-theme reload`.
- Edit/add/delete a file in a root → `notify` re-index → new content findable within seconds.
## Notes / risks
- nomic prefixes + mean-pool + normalize must match between index and query or recall collapses.
- `ort` linking: prefer the crate's downloaded/bundled ONNX Runtime to avoid version skew with Arch's `onnxruntime`.
- Office formats (docx/odt) are best-effort in v1; md/txt/org/pdf are the reliable path.

21
LICENSE Normal file
View file

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Breadway
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

134
README.md Normal file
View file

@ -0,0 +1,134 @@
# breadsearch
Semantic document search for Bread OS. Type a concept — not a keyword — and get ranked hits from your documents.
Two binaries in one Cargo workspace:
- **breadmill** — background daemon. Walks configured directories, extracts text, chunks and embeds documents with [nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) (768-dim ONNX), stores vectors in an HNSW index (usearch) backed by SQLite metadata, and serves queries over a Unix socket. Watches for filesystem changes and re-indexes incrementally.
- **breadsearch** — GTK4 overlay GUI. Queries breadmill via the Unix socket and shows ranked results. Press Enter to open a file, Ctrl+Enter to reveal its folder, Esc to close. Bind it to a hotkey (e.g. Super+/) and invoke it as a toggle.
## Build
System dependencies: `gtk4`, `gtk4-layer-shell`, `librsvg`.
```
cargo build --release
```
The `ort` crate downloads a bundled ONNX Runtime at build time (no system `onnxruntime` needed for the default CPU build).
Optional features:
| Feature | What it adds |
|---------|-------------|
| `npu` | AMD XDNA NPU via VitisAI ONNX Runtime EP (requires Ryzen AI SDK) |
| `rocm` | AMD iGPU via ROCm ONNX Runtime EP |
```
# NPU build
cargo build --release -p breadmill --features npu
```
## Setup
**1. Fetch the embedding model** (~550 MB, downloaded once from Hugging Face):
```
breadmill fetch-model
```
Model files are stored in `~/.cache/breadsearch/models/`.
**2. Enable the systemd user service:**
```
cp packaging/breadmill.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now breadmill
```
Or run it directly: `breadmill serve` (or just `breadmill`).
**3. Copy the example config** (optional — built-in defaults are used otherwise):
```
mkdir -p ~/.config/breadsearch
cp config.example.toml ~/.config/breadsearch/config.toml
```
## breadsearch (GUI)
```
breadsearch
```
Invoking again while running closes the window (PID-toggle). Bind to a hotkey in your compositor config.
Results show: filename/title, full path, and a snippet from the matching chunk. Score is cosine similarity as a percentage.
User CSS overrides go in `~/.config/breadsearch/style.css`.
## breadmill (daemon / CLI)
```
# Start the daemon (also the default when called with no arguments)
breadmill serve
# Force a full re-index from scratch
breadmill reindex
# Download model files
breadmill fetch-model
# Query from the terminal
breadmill query "tax stuff"
# Show daemon status (chunks indexed, pending, model ready)
breadmill status
# Backend flags (requires the matching Cargo feature)
breadmill --npu
breadmill --rocm
```
## Config
`~/.config/breadsearch/config.toml` — all keys are optional; built-in defaults are shown.
```toml
[index]
roots = ["~/Documents", "~/Projects", "~/.config/breadpad"]
extensions = ["md", "txt", "org", "pdf", "odt", "docx"]
excludes = [] # paths to skip (prefix match)
max_file_mb = 10.0
[search]
limit = 10 # max results per query
snippet_len = 200 # max characters in result snippet
[model]
name = "nomic-embed-text-v1.5"
dim = 768
backend = "cpu" # "cpu", "npu", or "rocm"
```
`roots` and `excludes` support `~/` expansion. The index respects `.gitignore` files found during the walk.
### NPU backend
Set `backend = "npu"` in config (or pass `--npu`) when running a build compiled with `--features npu`. breadmill looks for the VitisAI EP config file in this order:
1. `$VAIP_CONFIG`
2. `~/.config/breadsearch/vaip_config.json`
3. `~/.local/share/ryzen-ai-1.7.1/voe-4.0-linux_x86_64/vaip_config.json`
4. `/etc/vaip_config.json`
5. `/opt/xilinx/vaip_config.json`
## Runtime paths
| Purpose | Path |
|---------|------|
| Config | `~/.config/breadsearch/` |
| Index (SQLite + HNSW) | `~/.local/state/breadsearch/` |
| Model cache | `~/.cache/breadsearch/models/` |
| Unix socket | `$XDG_RUNTIME_DIR/breadmill.sock` |

18
bakery.toml Normal file
View file

@ -0,0 +1,18 @@
name = "breadsearch"
description = "Semantic system-wide search for BOS"
binaries = ["breadsearch", "breadmill"]
system_deps = ["gtk4", "gtk4-layer-shell", "librsvg"]
bread_deps = []
[[service]]
unit = "breadmill.service"
enable = true
[config]
dir = "~/.config/breadsearch"
example = "config.example.toml"
[install]
post_install = [
"systemctl --user start breadmill.service 2>/dev/null || breadmill &",
]

96
benchmark_embed.py Normal file
View file

@ -0,0 +1,96 @@
#!/usr/bin/env python3
"""
Benchmark CPU (float32) vs CPU (int8 quantized) vs VitisAI (int8 quantized static).
"""
import os
import time
import numpy as np
from pathlib import Path
FLOAT32_MODEL = Path.home() / ".cache/breadsearch/models/model.onnx"
QUANT_MODEL = Path.home() / ".cache/breadsearch/models/model_quantized.onnx"
STATIC_MODEL = Path.home() / ".cache/breadsearch/models/model_quantized_static.onnx"
CACHE_DIR = Path.home() / ".cache/breadsearch/npu/nomic-quantized-static"
VAIP_CONFIG = Path.home() / ".config/breadsearch/vaip_config.json"
RYZEN_AI_LIB = Path.home() / ".local/share/ryzen-ai-1.7.1/lib"
os.environ["RYZEN_AI_INSTALLATION_PATH"] = str(RYZEN_AI_LIB)
os.environ["LD_LIBRARY_PATH"] = str(RYZEN_AI_LIB) + ":" + os.environ.get("LD_LIBRARY_PATH", "")
import onnxruntime as ort
WARMUP = 2
RUNS = 10
SEQ_FLOAT = 512 # dynamic model accepts any seq len
SEQ_STATIC = 512 # static model locked to this
def make_input(seq_len: int):
ids = np.ones((1, seq_len), dtype=np.int64)
mask = np.ones((1, seq_len), dtype=np.int64)
types = np.zeros((1, seq_len), dtype=np.int64)
return {"input_ids": ids, "token_type_ids": types, "attention_mask": mask}
def time_session(sess: ort.InferenceSession, feed: dict, n: int) -> list[float]:
times = []
for _ in range(n):
t0 = time.perf_counter()
sess.run(None, feed)
times.append(time.perf_counter() - t0)
return times
def stats(times):
arr = np.array(times)
return arr.mean(), arr.min(), arr.max()
print("=" * 60)
print("BENCHMARK: nomic-embed-text-v1.5 embedding speed")
print("=" * 60)
# ── 1. float32 CPU ────────────────────────────────────────────
print("\n[1] float32 model — CPU EP")
sess = ort.InferenceSession(str(FLOAT32_MODEL), providers=["CPUExecutionProvider"])
feed = make_input(SEQ_FLOAT)
for _ in range(WARMUP): sess.run(None, feed)
times = time_session(sess, feed, RUNS)
mean, lo, hi = stats(times)
print(f" seq={SEQ_FLOAT} mean={mean*1000:.0f}ms min={lo*1000:.0f}ms max={hi*1000:.0f}ms ({RUNS} runs)")
# ── 2. int8 quantized CPU (dynamic) ───────────────────────────
print("\n[2] int8 quantized model (dynamic shapes) — CPU EP")
sess = ort.InferenceSession(str(QUANT_MODEL), providers=["CPUExecutionProvider"])
feed = make_input(SEQ_FLOAT)
for _ in range(WARMUP): sess.run(None, feed)
times = time_session(sess, feed, RUNS)
mean2, lo2, hi2 = stats(times)
print(f" seq={SEQ_FLOAT} mean={mean2*1000:.0f}ms min={lo2*1000:.0f}ms max={hi2*1000:.0f}ms ({RUNS} runs)")
print(f" Speedup vs float32: {mean/mean2:.2f}x")
# ── 3. int8 quantized static — VitisAI EP ─────────────────────
print("\n[3] int8 quantized model (static shapes) — VitisAI EP (NPU+CPU)")
providers = [
("VitisAIExecutionProvider", {
"config_file": str(VAIP_CONFIG),
"cacheDir": str(CACHE_DIR),
"cacheKey": "nomic-quantized-static",
}),
"CPUExecutionProvider",
]
print(" Loading session (should be fast — already compiled)...")
t_load = time.perf_counter()
sess_npu = ort.InferenceSession(str(STATIC_MODEL), providers=providers)
print(f" Load time: {time.perf_counter()-t_load:.1f}s")
feed_static = make_input(SEQ_STATIC)
for _ in range(WARMUP): sess_npu.run(None, feed_static)
times_npu = time_session(sess_npu, feed_static, RUNS)
mean3, lo3, hi3 = stats(times_npu)
print(f" seq={SEQ_STATIC} mean={mean3*1000:.0f}ms min={lo3*1000:.0f}ms max={hi3*1000:.0f}ms ({RUNS} runs)")
print(f" Speedup vs float32: {mean/mean3:.2f}x")
print(f" Speedup vs int8 CPU: {mean2/mean3:.2f}x")
print("\n" + "=" * 60)
print("SUMMARY")
print(f" float32 CPU : {mean*1000:.0f}ms/inference")
print(f" int8 CPU : {mean2*1000:.0f}ms/inference ({mean/mean2:.2f}x)")
print(f" int8 VitisAI : {mean3*1000:.0f}ms/inference ({mean/mean3:.2f}x)")
print("=" * 60)

55
breadmill/Cargo.toml Normal file
View file

@ -0,0 +1,55 @@
[package]
name = "breadmill"
version = "0.1.0"
edition = "2021"
license = "MIT"
[features]
default = []
npu = ["ort/vitis", "ort/load-dynamic"]
rocm = ["ort/rocm", "ort/load-dynamic"]
[[bin]]
name = "breadmill"
path = "src/main.rs"
[dependencies]
breadsearch-shared = { path = "../breadsearch-shared" }
# Embedding: ONNX Runtime + HF tokenizers
# download-binaries: fetches the MLAS-optimized ORT 1.24.x at build time (CPU default).
# api-23: compatible with both the downloaded ORT 1.24.x and the Ryzen AI SDK ORT 1.23.3;
# ORT 1.24 is backwards-compatible and honours GetApi(23) requests.
# npu feature adds load-dynamic + vitis: dlopen at runtime lets ORT_DYLIB_PATH redirect
# to the Ryzen AI SDK ORT; rpath from download-binaries means no ORT_DYLIB_PATH
# needed for the plain CPU path even in the npu build.
ort = { version = "2.0.0-rc.12", default-features = false, features = ["std", "tracing", "download-binaries", "tls-native", "copy-dylibs", "api-23"] }
tokenizers = "0"
# Vector index
usearch = "2"
# Metadata store
rusqlite = { version = "0", features = ["bundled"] }
# File walking (respects .gitignore)
ignore = "0"
# Live filesystem watching
notify = "6"
# Text extraction
pdf-extract = "0"
zip = "2"
quick-xml = { version = "0", features = ["serialize"] }
# Hashing
sha2 = "0"
hex = "0"
# HTTP for model download
ureq = "2"
# Serialization
serde = { version = "1", features = ["derive"] }
serde_json = "1"

142
breadmill/src/chunk.rs Normal file
View file

@ -0,0 +1,142 @@
pub struct Chunk {
pub text: String,
pub start: usize,
pub end: usize,
}
/// Split `text` into overlapping word-based windows, then enforce `max_chunk_chars`.
///
/// Any word-window that exceeds `max_chunk_chars` characters is split further at
/// character boundaries so that no chunk passed to the embedder is pathologically
/// large (e.g. minified JSON where a single "word" is hundreds of KB).
///
/// Set `max_chunk_chars = 0` to skip the character cap.
pub fn chunk_text(text: &str, words_per_chunk: usize, overlap_words: usize, max_chunk_chars: usize) -> Vec<Chunk> {
let word_chunks = chunk_by_words(text, words_per_chunk, overlap_words);
if max_chunk_chars == 0 {
return word_chunks;
}
let mut result = Vec::new();
for chunk in word_chunks {
if chunk.text.len() <= max_chunk_chars {
result.push(chunk);
} else {
result.extend(split_by_chars(chunk, max_chunk_chars));
}
}
result
}
fn chunk_by_words(text: &str, words_per_chunk: usize, overlap_words: usize) -> Vec<Chunk> {
let mut positions: Vec<(usize, usize)> = Vec::new();
let mut in_word = false;
let mut word_start = 0;
for (i, c) in text.char_indices() {
if c.is_whitespace() {
if in_word {
positions.push((word_start, i));
in_word = false;
}
} else if !in_word {
word_start = i;
in_word = true;
}
}
if in_word {
positions.push((word_start, text.len()));
}
if positions.is_empty() {
return Vec::new();
}
let step = words_per_chunk.saturating_sub(overlap_words).max(1);
let mut chunks = Vec::new();
let mut i = 0;
while i < positions.len() {
let last = (i + words_per_chunk - 1).min(positions.len() - 1);
let start = positions[i].0;
let end = positions[last].1;
chunks.push(Chunk {
text: text[start..end].to_string(),
start,
end,
});
if last == positions.len() - 1 {
break;
}
i += step;
}
chunks
}
fn split_by_chars(chunk: Chunk, max_chars: usize) -> Vec<Chunk> {
let text = &chunk.text;
let mut result = Vec::new();
let mut seg_start = 0usize;
let mut count = 0usize;
for (byte_idx, _) in text.char_indices() {
if count > 0 && count % max_chars == 0 {
result.push(Chunk {
text: text[seg_start..byte_idx].to_string(),
start: chunk.start + seg_start,
end: chunk.start + byte_idx,
});
seg_start = byte_idx;
}
count += 1;
}
if seg_start < text.len() {
result.push(Chunk {
text: text[seg_start..].to_string(),
start: chunk.start + seg_start,
end: chunk.end,
});
}
result
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn basic_chunk() {
let text = "one two three four five six seven eight nine ten";
let chunks = chunk_text(text, 4, 1, 0);
assert!(!chunks.is_empty());
for c in &chunks {
assert!(!c.text.is_empty());
assert!(c.start <= c.end);
assert_eq!(&text[c.start..c.end], c.text);
}
}
#[test]
fn char_cap_splits_large_chunks() {
// Simulate a "word" that is 200 chars long — exceeds cap of 50.
let text = "a".repeat(200);
let chunks = chunk_text(&text, 1, 0, 50);
assert_eq!(chunks.len(), 4);
for c in &chunks {
assert!(c.text.len() <= 50);
assert_eq!(&text[c.start..c.end], c.text);
}
}
#[test]
fn char_cap_disabled() {
let text = "a".repeat(200);
let chunks = chunk_text(&text, 1, 0, 0);
assert_eq!(chunks.len(), 1);
assert_eq!(chunks[0].text.len(), 200);
}
}

258
breadmill/src/embed.rs Normal file
View file

@ -0,0 +1,258 @@
use std::path::{Path, PathBuf};
use ort::{
session::{Session, builder::{GraphOptimizationLevel, SessionBuilder}},
value::Tensor,
};
use tokenizers::Tokenizer;
const DOCUMENT_PREFIX: &str = "search_document: ";
const QUERY_PREFIX: &str = "search_query: ";
/// Hard token cap for nomic-embed-text-v1.5 (8192-token context window).
/// Sequences longer than this are truncated before ONNX inference to prevent
/// quadratic attention memory blowup.
pub const MAX_SEQ_LEN: usize = 8192;
pub enum Backend {
Cpu,
/// AMD XDNA NPU via the VitisAI ONNX Runtime execution provider.
/// `cache_dir` is used to store the compiled NPU model between runs.
Npu { cache_dir: PathBuf },
/// AMD iGPU via the ROCm ONNX Runtime execution provider.
Rocm,
}
pub struct OrtEmbedder {
session: Session,
tokenizer: Tokenizer,
dim: usize,
}
impl OrtEmbedder {
pub fn load(model_path: &Path, tokenizer_path: &Path, dim: usize, backend: Backend) -> Result<Self, String> {
let builder = Session::builder()
.map_err(|e| e.to_string())?
.with_optimization_level(GraphOptimizationLevel::All)
.map_err(|e| e.to_string())?;
let mut builder = configure_eps(builder, &backend)?;
let session = builder.commit_from_file(model_path).map_err(|e| e.to_string())?;
let tokenizer = Tokenizer::from_file(tokenizer_path).map_err(|e| e.to_string())?;
Ok(Self { session, tokenizer, dim })
}
pub fn embed_document(&mut self, text: &str) -> Result<Vec<f32>, String> {
self.embed_with_prefix(text, DOCUMENT_PREFIX)
}
pub fn embed_query(&mut self, text: &str) -> Result<Vec<f32>, String> {
self.embed_with_prefix(text, QUERY_PREFIX)
}
fn embed_with_prefix(&mut self, text: &str, prefix: &str) -> Result<Vec<f32>, String> {
let input = format!("{}{}", prefix, text);
let encoding = self
.tokenizer
.encode(input, true)
.map_err(|e| e.to_string())?;
let mut ids: Vec<i64> = encoding.get_ids().iter().map(|&x| x as i64).collect();
let mut mask: Vec<i64> = encoding
.get_attention_mask()
.iter()
.map(|&x| x as i64)
.collect();
let mut type_ids: Vec<i64> = encoding
.get_type_ids()
.iter()
.map(|&x| x as i64)
.collect();
if ids.len() > MAX_SEQ_LEN {
eprintln!(
"breadmill: truncating {} tokens to {} (chunk too large)",
ids.len(),
MAX_SEQ_LEN
);
ids.truncate(MAX_SEQ_LEN);
mask.truncate(MAX_SEQ_LEN);
type_ids.truncate(MAX_SEQ_LEN);
}
let seq_len = ids.len() as i64;
let id_tensor =
Tensor::<i64>::from_array((vec![1i64, seq_len], ids.clone())).map_err(|e| e.to_string())?;
let mask_tensor =
Tensor::<i64>::from_array((vec![1i64, seq_len], mask.clone())).map_err(|e| e.to_string())?;
let type_tensor =
Tensor::<i64>::from_array((vec![1i64, seq_len], type_ids)).map_err(|e| e.to_string())?;
let outputs = self
.session
.run(ort::inputs! {
"input_ids" => id_tensor,
"attention_mask" => mask_tensor,
"token_type_ids" => type_tensor,
})
.map_err(|e| e.to_string())?;
// last_hidden_state: shape [1, seq_len, dim]
let (shape, data) = outputs["last_hidden_state"]
.try_extract_tensor::<f32>()
.map_err(|e| e.to_string())?;
let actual_seq = shape[1] as usize;
let actual_dim = shape[2] as usize;
// Mean-pool over non-padding positions
let mut result = vec![0.0f32; actual_dim];
let mut count = 0usize;
for t in 0..actual_seq {
if mask[t] > 0 {
for d in 0..actual_dim {
result[d] += data[t * actual_dim + d];
}
count += 1;
}
}
if count > 0 {
for x in &mut result {
*x /= count as f32;
}
}
l2_normalize(&mut result);
// Clamp/pad to configured dim
result.truncate(self.dim);
while result.len() < self.dim {
result.push(0.0);
}
Ok(result)
}
}
fn l2_normalize(v: &mut Vec<f32>) {
let norm: f32 = v.iter().map(|x| x * x).sum::<f32>().sqrt();
if norm > 1e-10 {
for x in v.iter_mut() {
*x /= norm;
}
}
}
// ---- Execution provider selection -------------------------------------------
fn configure_eps(builder: SessionBuilder, backend: &Backend) -> Result<SessionBuilder, String> {
match backend {
Backend::Cpu => Ok(builder),
Backend::Npu { cache_dir } => npu_session(builder, cache_dir),
Backend::Rocm => rocm_session(builder),
}
}
#[cfg(feature = "npu")]
fn npu_session(builder: SessionBuilder, cache_dir: &Path) -> Result<SessionBuilder, String> {
let vitis_ep = build_vitis_ep(cache_dir)?;
eprintln!("breadmill: using NPU (VitisAI) execution provider");
if std::env::var("ORT_DYLIB_PATH").is_err() {
eprintln!(
"breadmill: hint — set ORT_DYLIB_PATH to the Ryzen AI SDK ORT, e.g.:\n \
ORT_DYLIB_PATH=~/.local/share/ryzen-ai-1.7.1/lib/libonnxruntime.so"
);
}
builder
.with_execution_providers([vitis_ep, ort::ep::CPU::default().build()])
.map_err(|e| e.to_string())
}
#[cfg(not(feature = "npu"))]
fn npu_session(builder: SessionBuilder, _cache_dir: &Path) -> Result<SessionBuilder, String> {
eprintln!("breadmill: NPU backend requested but not compiled in (rebuild with --features npu); using CPU");
Ok(builder)
}
// ---- VitisAI EP (NPU) -------------------------------------------------------
#[cfg(feature = "npu")]
fn build_vitis_ep(cache_dir: &Path) -> Result<ort::ep::ExecutionProviderDispatch, String> {
let vaip_config = find_vaip_config()?;
let npu_cache = cache_dir.join("npu");
std::fs::create_dir_all(&npu_cache).map_err(|e| e.to_string())?;
eprintln!("breadmill: vaip_config: {}", vaip_config.display());
eprintln!("breadmill: NPU model cache: {}", npu_cache.display());
Ok(ort::ep::Vitis::default()
.with_config_file(vaip_config.to_string_lossy())
.with_cache_dir(npu_cache.to_string_lossy())
.with_cache_key("nomic-embed-text-v1.5")
.build())
}
// ---- ROCm EP (AMD iGPU) -----------------------------------------------------
#[cfg(feature = "rocm")]
fn rocm_session(builder: SessionBuilder) -> Result<SessionBuilder, String> {
eprintln!("breadmill: using ROCm execution provider (device 0)");
builder
.with_execution_providers([
ort::execution_providers::ROCmExecutionProvider::default().build(),
ort::ep::CPU::default().build(),
])
.map_err(|e| e.to_string())
}
#[cfg(not(feature = "rocm"))]
fn rocm_session(builder: SessionBuilder) -> Result<SessionBuilder, String> {
eprintln!("breadmill: ROCm backend requested but not compiled in (rebuild with --features rocm); using CPU");
Ok(builder)
}
/// Locate the VitisAI EP config file required by the AMD Ryzen AI SDK.
///
/// Search order:
/// 1. `VAIP_CONFIG` environment variable
/// 2. `~/.config/breadsearch/vaip_config.json`
/// 3. `/etc/vaip_config.json`
/// 4. `/opt/xilinx/vaip_config.json`
#[cfg(feature = "npu")]
fn find_vaip_config() -> Result<PathBuf, String> {
if let Ok(p) = std::env::var("VAIP_CONFIG") {
let path = PathBuf::from(&p);
if path.exists() {
return Ok(path);
}
return Err(format!("VAIP_CONFIG={p} does not exist"));
}
let user_path = breadsearch_shared::config_dir().join("vaip_config.json");
if user_path.exists() {
return Ok(user_path);
}
// Standard system / SDK paths (checked in priority order)
let home = std::env::var("HOME").unwrap_or_default();
let sdk_paths = [
format!("{home}/.local/share/ryzen-ai-1.7.1/voe-4.0-linux_x86_64/vaip_config.json"),
"/etc/vaip_config.json".into(),
"/opt/xilinx/vaip_config.json".into(),
];
for p in &sdk_paths {
let path = Path::new(p.as_str());
if path.exists() {
return Ok(path.to_path_buf());
}
}
Err(
"vaip_config.json not found; set VAIP_CONFIG=/path/to/vaip_config.json, \
copy to ~/.config/breadsearch/vaip_config.json, or install the AMD Ryzen AI SDK"
.into(),
)
}

84
breadmill/src/extract.rs Normal file
View file

@ -0,0 +1,84 @@
use std::{
fs, io::Read, path::Path,
};
pub type ExtractResult = Result<String, String>;
/// Extract plain text from a file based on its extension.
/// Returns Err on hard failures; Err with message if format unsupported.
pub fn extract(path: &Path) -> ExtractResult {
let ext = path
.extension()
.and_then(|e| e.to_str())
.unwrap_or("")
.to_lowercase();
match ext.as_str() {
"md" | "txt" | "org" => read_text(path),
"pdf" => extract_pdf(path),
"docx" => extract_docx(path),
"odt" => extract_odt(path),
other => Err(format!("unsupported extension: {}", other)),
}
}
fn read_text(path: &Path) -> ExtractResult {
fs::read_to_string(path).map_err(|e| e.to_string())
}
fn extract_pdf(path: &Path) -> ExtractResult {
// pdf-extract panics on some malformed PDFs; catch_unwind prevents indexer thread death.
let path = path.to_path_buf();
match std::panic::catch_unwind(|| pdf_extract::extract_text(&path)) {
Ok(result) => result.map_err(|e| e.to_string()),
Err(_) => Err("pdf-extract panicked on malformed content stream".into()),
}
}
fn extract_docx(path: &Path) -> ExtractResult {
extract_office_xml(path, "word/document.xml", "w:t")
}
fn extract_odt(path: &Path) -> ExtractResult {
extract_office_xml(path, "content.xml", "text:p")
}
/// Open a zip-based office format and concatenate text from the named XML entry.
/// We grab all Text events as a best-effort extraction.
fn extract_office_xml(path: &Path, xml_entry: &str, _tag_hint: &str) -> ExtractResult {
let file = fs::File::open(path).map_err(|e| e.to_string())?;
let mut archive = zip::ZipArchive::new(file).map_err(|e| e.to_string())?;
let mut xml_bytes = Vec::new();
archive
.by_name(xml_entry)
.map_err(|e| format!("entry '{}' not found: {}", xml_entry, e))?
.read_to_end(&mut xml_bytes)
.map_err(|e| e.to_string())?;
let xml_str = String::from_utf8_lossy(&xml_bytes);
let mut reader = quick_xml::Reader::from_str(&xml_str);
reader.config_mut().trim_text(true);
let mut text = String::new();
let mut buf = Vec::new();
loop {
match reader.read_event_into(&mut buf) {
Ok(quick_xml::events::Event::Text(e)) => {
if let Ok(s) = e.decode() {
if !text.is_empty() {
text.push(' ');
}
text.push_str(&s);
}
}
Ok(quick_xml::events::Event::Eof) => break,
Err(e) => return Err(e.to_string()),
_ => {}
}
buf.clear();
}
Ok(text)
}

414
breadmill/src/indexer.rs Normal file
View file

@ -0,0 +1,414 @@
use std::{
collections::{HashMap, HashSet},
fs,
path::{Path, PathBuf},
sync::{Arc, Mutex, atomic::{AtomicBool, AtomicUsize, Ordering}},
time::{Duration, Instant, UNIX_EPOCH},
};
use ignore::WalkBuilder;
use notify::{RecommendedWatcher, RecursiveMode, Watcher, EventKind};
use sha2::{Digest, Sha256};
use crate::{embed::OrtEmbedder, extract, chunk, power, store::Store};
pub struct SharedState {
pub store: Mutex<Store>,
pub embedder: Mutex<Option<OrtEmbedder>>,
pub model_ready: AtomicBool,
pub indexed: AtomicUsize,
pub pending: AtomicUsize,
pub reindex_signal: AtomicBool,
}
impl SharedState {
pub fn new(store: Store) -> Self {
SharedState {
store: Mutex::new(store),
embedder: Mutex::new(None),
model_ready: AtomicBool::new(false),
indexed: AtomicUsize::new(0),
pending: AtomicUsize::new(0),
reindex_signal: AtomicBool::new(false),
}
}
}
pub struct Indexer {
state: Arc<SharedState>,
config: breadsearch_shared::Config,
state_dir: PathBuf,
}
impl Indexer {
pub fn new(state: Arc<SharedState>, config: breadsearch_shared::Config, state_dir: PathBuf) -> Self {
Indexer { state, config, state_dir }
}
pub fn run(self) {
self.initial_scan();
self.watch_loop();
}
/// True when embedding should be skipped right now: the user turned
/// indexing off entirely, or the machine is on battery and
/// `power.run_on_battery` is not set. Cheap sysfs reads — safe to call
/// per-file and on every watch_loop tick.
fn indexing_paused(&self) -> bool {
if !self.config.power.enabled {
return true;
}
!self.config.power.run_on_battery && !power::on_ac_power()
}
pub fn full_reindex(&self) {
eprintln!("breadmill: full reindex triggered");
{
let mut store = self.state.store.lock().unwrap();
// Clear all state
let _ = store.conn.execute_batch("DELETE FROM chunks; DELETE FROM files;");
let _ = store.index.reserve(4096);
}
self.initial_scan();
}
fn initial_scan(&self) {
eprintln!("breadmill: scanning roots...");
let roots: Vec<PathBuf> = self.config.index.roots
.iter()
.map(|r| expand_home(r))
.collect();
let excludes: Vec<PathBuf> = self.config.index.excludes
.iter()
.map(|r| expand_home(r))
.collect();
// Snapshot existing indexed files
let known: HashMap<String, (i64, String)> = {
let store = self.state.store.lock().unwrap();
store.all_files()
.unwrap_or_default()
.into_iter()
.map(|f| (f.path, (f.mtime, f.hash)))
.collect()
};
let mut seen: HashSet<String> = HashSet::new();
let max_bytes = (self.config.index.max_file_mb * 1024.0 * 1024.0) as u64;
for root in &roots {
if !root.exists() {
continue;
}
for entry in WalkBuilder::new(root)
.hidden(false)
.ignore(true)
.git_ignore(true)
.build()
.filter_map(|e| e.ok())
.filter(|e| e.path().is_file())
{
let path = entry.path();
if excludes.iter().any(|excl| path.starts_with(excl)) {
continue;
}
if !self.is_indexed_extension(path) {
continue;
}
let meta = match fs::metadata(path) {
Ok(m) => m,
Err(_) => continue,
};
if meta.len() > max_bytes {
continue;
}
let path_str = path.to_string_lossy().into_owned();
seen.insert(path_str.clone());
let mtime = mtime_secs(&meta);
if let Some((known_mtime, _)) = known.get(&path_str) {
if *known_mtime == mtime {
continue; // unchanged
}
}
self.state.pending.fetch_add(1, Ordering::Relaxed);
self.index_file(path, &path_str, mtime);
self.state.pending.fetch_sub(1, Ordering::Relaxed);
}
}
// Drop files that were deleted
let to_delete: Vec<String> = known
.keys()
.filter(|p| !seen.contains(*p))
.cloned()
.collect();
if !to_delete.is_empty() {
let mut store = self.state.store.lock().unwrap();
for path in to_delete {
eprintln!("breadmill: removing deleted file: {}", path);
let _ = store.delete_file(&path);
}
}
let count = {
let store = self.state.store.lock().unwrap();
let n = store.chunk_count();
let _ = store.save_index(&self.state_dir);
n
};
self.state.indexed.store(count, Ordering::Relaxed);
eprintln!("breadmill: initial scan done — {} chunks indexed", count);
}
fn watch_loop(self) {
let (tx, rx) = std::sync::mpsc::channel();
let mut watcher: RecommendedWatcher = match notify::recommended_watcher(tx) {
Ok(w) => w,
Err(e) => {
eprintln!("breadmill: watcher init failed: {}", e);
return;
}
};
for root in self.config.index.roots.iter().map(|r| expand_home(r)) {
if root.exists() {
let _ = watcher.watch(&root, RecursiveMode::Recursive);
}
}
let mut pending_paths: HashSet<PathBuf> = HashSet::new();
let mut last_event = Instant::now();
let quiet = Duration::from_secs(2);
let mut was_paused = self.indexing_paused();
let mut last_power_check = Instant::now();
eprintln!("breadmill: watching for changes");
loop {
// Drain the reindex signal
if self.state.reindex_signal.swap(false, Ordering::Relaxed) {
self.full_reindex();
}
// Re-check the power gate periodically (sysfs reads are cheap but
// no need to do it every 500ms tick). On a paused->active
// transition, re-run the incremental scan to catch up anything
// skipped while gated.
if last_power_check.elapsed() >= Duration::from_secs(30) {
last_power_check = Instant::now();
let now_paused = self.indexing_paused();
if was_paused && !now_paused {
eprintln!("breadmill: power gate opened — resuming indexing");
self.initial_scan();
}
was_paused = now_paused;
}
match rx.recv_timeout(Duration::from_millis(500)) {
Ok(Ok(event)) => {
match event.kind {
EventKind::Create(_) | EventKind::Modify(_) | EventKind::Remove(_) => {
for p in event.paths {
pending_paths.insert(p);
}
last_event = Instant::now();
}
_ => {}
}
}
Ok(Err(e)) => eprintln!("breadmill: watch error: {}", e),
Err(_) => {} // timeout — check quiet period
}
if !pending_paths.is_empty() && last_event.elapsed() >= quiet {
for path in pending_paths.drain() {
self.handle_fs_event(&path);
}
let count = {
let store = self.state.store.lock().unwrap();
let n = store.chunk_count();
let _ = store.save_index(&self.state_dir);
n
};
self.state.indexed.store(count, Ordering::Relaxed);
}
}
}
fn handle_fs_event(&self, path: &Path) {
let excludes: Vec<PathBuf> = self.config.index.excludes
.iter()
.map(|r| expand_home(r))
.collect();
if excludes.iter().any(|excl| path.starts_with(excl)) {
return;
}
if !path.is_file() {
let path_str = path.to_string_lossy().into_owned();
// File deleted — remove from index
let mut store = self.state.store.lock().unwrap();
let _ = store.delete_file(&path_str);
return;
}
if !self.is_indexed_extension(path) {
return;
}
let meta = match fs::metadata(path) {
Ok(m) => m,
Err(_) => return,
};
let max_bytes = (self.config.index.max_file_mb * 1024.0 * 1024.0) as u64;
if meta.len() > max_bytes {
return;
}
let path_str = path.to_string_lossy().into_owned();
let mtime = mtime_secs(&meta);
self.state.pending.fetch_add(1, Ordering::Relaxed);
self.index_file(path, &path_str, mtime);
self.state.pending.fetch_sub(1, Ordering::Relaxed);
}
fn index_file(&self, path: &Path, path_str: &str, mtime: i64) {
if self.indexing_paused() {
// Leave the file unrecorded so it's picked up again once indexing
// resumes (initial_scan/watch_loop treat it as not-yet-indexed).
return;
}
eprintln!("breadmill: extracting {}", path_str);
let text = match extract::extract(path) {
Ok(t) if !t.trim().is_empty() => t,
Ok(_) => return,
Err(e) => {
eprintln!("breadmill: extract {}: {}", path_str, e);
return;
}
};
let hash = sha256_str(text.as_bytes());
// Check if hash changed (catches content changes without mtime change)
{
let store = self.state.store.lock().unwrap();
if let Ok(files) = store.all_files() {
if files.iter().any(|f| f.path == path_str && f.hash == hash) {
return;
}
}
}
// 2000 char cap keeps even minified single-line files to ~500–2000 tokens,
// avoiding quadratic attention blowup while still splitting at word boundaries
// for natural-language files.
let chunks = chunk::chunk_text(&text, 400, 80, 2_000);
eprintln!("breadmill: embedding {} ({} chars, {} chunks)", path_str, text.len(), chunks.len());
let mut embedder_guard = self.state.embedder.lock().unwrap();
if !self.state.model_ready.load(Ordering::Relaxed) {
eprintln!("breadmill: model not ready, skipping embed for {}", path_str);
return;
}
let embedder = match embedder_guard.as_mut() {
Some(e) => e,
None => return,
};
{
let mut store = self.state.store.lock().unwrap();
let _ = store.delete_file(path_str); // remove old chunks/vectors first
}
let mut any_ok = false;
let mut chunks_added = 0usize;
for (i, chunk) in chunks.iter().enumerate() {
eprintln!("breadmill: embed chunk {}/{} ({} chars) for {}", i + 1, chunks.len(), chunk.text.len(), path_str);
match embedder.embed_document(&chunk.text) {
Ok(embedding) => {
let mut store = self.state.store.lock().unwrap();
// Ensure file row exists before inserting chunks (FK constraint)
let _ = store.upsert_file(path_str, mtime, &hash);
let _ = store.insert_chunk(
path_str,
&chunk.text,
chunk.start,
chunk.end,
&embedding,
);
any_ok = true;
chunks_added += 1;
}
Err(e) => eprintln!("breadmill: embed error for {}: {}", path_str, e),
}
}
if !any_ok {
eprintln!("breadmill: no chunks embedded for {}", path_str);
// Record the file so the mtime+hash check skips it on the next startup
// rather than re-entering the same embed-fail loop.
let store = self.state.store.lock().unwrap();
let _ = store.upsert_file(path_str, mtime, &hash);
} else {
// Increment live so `status` reflects progress before the full scan ends.
self.state.indexed.fetch_add(chunks_added, Ordering::Relaxed);
eprintln!("breadmill: done {} ({} chunks indexed)", path_str, chunks_added);
}
}
fn is_indexed_extension(&self, path: &Path) -> bool {
path.extension()
.and_then(|e| e.to_str())
.map(|ext| {
self.config
.index
.extensions
.iter()
.any(|e| e.eq_ignore_ascii_case(ext))
})
.unwrap_or(false)
}
}
fn mtime_secs(meta: &fs::Metadata) -> i64 {
meta.modified()
.ok()
.and_then(|t| t.duration_since(UNIX_EPOCH).ok())
.map(|d| d.as_secs() as i64)
.unwrap_or(0)
}
fn sha256_str(bytes: &[u8]) -> String {
let mut hasher = Sha256::new();
hasher.update(bytes);
hex::encode(hasher.finalize())
}
pub fn expand_home(path: &str) -> PathBuf {
if path.starts_with("~/") {
let home = std::env::var("HOME").unwrap_or_else(|_| "/tmp".into());
PathBuf::from(home).join(&path[2..])
} else {
PathBuf::from(path)
}
}

242
breadmill/src/main.rs Normal file
View file

@ -0,0 +1,242 @@
use std::{
io::Read,
path::{Path, PathBuf},
sync::{Arc, atomic::Ordering},
};
use breadsearch_shared::{Request, Response};
mod chunk;
mod embed;
mod extract;
mod indexer;
mod power;
mod serve;
mod store;
use embed::{Backend, OrtEmbedder};
use indexer::{Indexer, SharedState};
use store::Store;
const MODEL_URL: &str =
"https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/onnx/model.onnx";
const TOKENIZER_URL: &str =
"https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json";
fn main() {
let raw_args: Vec<String> = std::env::args().collect();
// Extract global flags before command dispatch.
let use_npu = raw_args.iter().any(|a| a == "--npu");
let use_rocm = raw_args.iter().any(|a| a == "--rocm");
// Build a view of argv without backend flags for command matching.
let args: Vec<&str> = raw_args
.iter()
.skip(1)
.filter(|a| a.as_str() != "--npu" && a.as_str() != "--rocm")
.map(|s| s.as_str())
.collect();
match args.first().copied() {
Some("--version") | Some("-V") => {
println!("breadmill {}", env!("CARGO_PKG_VERSION"));
}
Some("--fetch-model") | Some("fetch-model") => {
if let Err(e) = fetch_model() {
eprintln!("breadmill: {}", e);
std::process::exit(1);
}
}
Some("--reindex") | Some("reindex") => {
if let Err(e) = run_daemon(true, use_npu, use_rocm) {
eprintln!("breadmill: {}", e);
std::process::exit(1);
}
}
Some("query") => {
let q = args.get(1).copied().unwrap_or("");
if q.is_empty() {
eprintln!("usage: breadmill query <text>");
std::process::exit(1);
}
cli_query(q);
}
Some("status") => {
cli_status();
}
None | Some("serve") | Some("--serve") => {
if let Err(e) = run_daemon(false, use_npu, use_rocm) {
eprintln!("breadmill: {}", e);
std::process::exit(1);
}
}
Some(cmd) => {
eprintln!("breadmill: unknown command: {}", cmd);
eprintln!(
"usage: breadmill [serve|reindex|fetch-model|query <text>|status] [--npu|--rocm] [--version]"
);
std::process::exit(1);
}
}
}
// ---- Daemon -----------------------------------------------------------------
fn run_daemon(force_reindex: bool, use_npu: bool, use_rocm: bool) -> Result<(), String> {
let config = breadsearch_shared::Config::load();
let state_dir = breadsearch_shared::state_dir();
let cache_dir = breadsearch_shared::cache_dir();
let socket_path = breadsearch_shared::socket_path();
let dim = config.model.dim;
let snippet_len = config.search.snippet_len;
let search_limit = config.search.limit;
std::fs::create_dir_all(&state_dir).map_err(|e| e.to_string())?;
std::fs::create_dir_all(&cache_dir).map_err(|e| e.to_string())?;
let backend = if use_npu || config.model.backend == "npu" {
eprintln!("breadmill: NPU backend selected");
Backend::Npu { cache_dir: cache_dir.clone() }
} else if use_rocm || config.model.backend == "rocm" {
eprintln!("breadmill: ROCm backend selected");
Backend::Rocm
} else {
Backend::Cpu
};
let store = Store::open(&state_dir, dim)?;
let state = Arc::new(SharedState::new(store));
// Load embedder if model files present
let model_dir = model_dir(&cache_dir);
let model_path = model_dir.join("model.onnx");
let tokenizer_path = model_dir.join("tokenizer.json");
if model_path.exists() && tokenizer_path.exists() {
eprintln!("breadmill: loading model...");
match OrtEmbedder::load(&model_path, &tokenizer_path, dim, backend) {
Ok(embedder) => {
*state.embedder.lock().unwrap() = Some(embedder);
state.model_ready.store(true, Ordering::Relaxed);
eprintln!("breadmill: model loaded");
}
Err(e) => eprintln!("breadmill: model load failed: {} — run --fetch-model", e),
}
} else {
eprintln!(
"breadmill: model files not found in {} — run: breadmill --fetch-model",
model_dir.display()
);
}
// Indexer runs in a background thread
{
let state_clone = Arc::clone(&state);
let config_clone = config.clone();
let state_dir_clone = state_dir.clone();
std::thread::spawn(move || {
let indexer = Indexer::new(state_clone, config_clone, state_dir_clone);
if force_reindex {
indexer.full_reindex();
}
indexer.run();
});
}
// Server runs on the main thread (blocking)
serve::run(&socket_path, Arc::clone(&state), snippet_len, search_limit);
Ok(())
}
// ---- Model fetch ------------------------------------------------------------
fn fetch_model() -> Result<(), String> {
let cache_dir = breadsearch_shared::cache_dir();
let model_dir = model_dir(&cache_dir);
std::fs::create_dir_all(&model_dir).map_err(|e| e.to_string())?;
download_if_missing(MODEL_URL, &model_dir.join("model.onnx"))?;
download_if_missing(TOKENIZER_URL, &model_dir.join("tokenizer.json"))?;
eprintln!("breadmill: model files ready in {}", model_dir.display());
Ok(())
}
fn download_if_missing(url: &str, dest: &Path) -> Result<(), String> {
if dest.exists() {
eprintln!(" already present: {}", dest.display());
return Ok(());
}
eprintln!(" downloading {} ...", url);
let agent = ureq::AgentBuilder::new()
.timeout(std::time::Duration::from_secs(300))
.build();
let response = agent.get(url).call().map_err(|e| e.to_string())?;
let mut bytes = Vec::new();
response
.into_reader()
.read_to_end(&mut bytes)
.map_err(|e| e.to_string())?;
if bytes.is_empty() {
return Err(format!("empty download from {}", url));
}
// Write atomically via temp file
let tmp = dest.with_extension("tmp");
std::fs::write(&tmp, &bytes).map_err(|e| e.to_string())?;
std::fs::rename(&tmp, dest).map_err(|e| e.to_string())?;
eprintln!(" saved {} ({:.1} MB)", dest.display(), bytes.len() as f64 / 1_048_576.0);
Ok(())
}
fn model_dir(cache_dir: &Path) -> PathBuf {
cache_dir.join("models")
}
// ---- CLI helpers ------------------------------------------------------------
fn cli_query(query: &str) {
let req = Request::Query {
query: query.to_string(),
limit: 10,
};
match breadsearch_shared::send_request(&req) {
Ok(Response::Hits { hits }) => {
if hits.is_empty() {
println!("no results");
}
for (i, h) in hits.iter().enumerate() {
println!(
"{:2}. {} ({:.3})\n {}\n {}\n",
i + 1,
h.title,
h.score,
h.path,
h.snippet.lines().next().unwrap_or(""),
);
}
}
Ok(Response::Error { message }) => eprintln!("error: {}", message),
Ok(_) => eprintln!("unexpected response"),
Err(e) => eprintln!("could not reach breadmill: {}", e),
}
}
fn cli_status() {
match breadsearch_shared::send_request(&Request::Status) {
Ok(Response::StatusInfo(s)) => {
println!("indexed: {}", s.indexed);
println!("pending: {}", s.pending);
println!("model ready: {}", s.model_ready);
}
Ok(_) => eprintln!("unexpected response"),
Err(e) => eprintln!("could not reach breadmill: {}", e),
}
}

31
breadmill/src/power.rs Normal file
View file

@ -0,0 +1,31 @@
use std::fs;
use std::path::Path;
/// Best-effort check for whether the system is currently on AC/mains power.
/// Scans `/sys/class/power_supply` for a Mains or USB supply with `online=1`.
/// Systems with no such supply at all (desktops, no battery) are treated as
/// always on power, so this never blocks indexing on hardware without a
/// battery to protect.
pub fn on_ac_power() -> bool {
let dir = Path::new("/sys/class/power_supply");
let Ok(entries) = fs::read_dir(dir) else {
return true;
};
let mut found_mains = false;
for entry in entries.flatten() {
let path = entry.path();
let supply_type = fs::read_to_string(path.join("type")).unwrap_or_default();
let supply_type = supply_type.trim();
if supply_type != "Mains" && supply_type != "USB" {
continue;
}
found_mains = true;
let online = fs::read_to_string(path.join("online")).unwrap_or_default();
if online.trim() == "1" {
return true;
}
}
!found_mains
}

127
breadmill/src/serve.rs Normal file
View file

@ -0,0 +1,127 @@
use std::{
io::{BufRead, BufReader, Write},
os::unix::net::{UnixListener, UnixStream},
path::Path,
sync::Arc,
};
use breadsearch_shared::{Request, Response, StatusInfo};
use crate::indexer::SharedState;
pub fn run(socket_path: &Path, state: Arc<SharedState>, snippet_len: usize, search_limit: usize) {
let _ = std::fs::remove_file(socket_path);
let listener = match UnixListener::bind(socket_path) {
Ok(l) => l,
Err(e) => {
eprintln!("breadmill: bind {}: {}", socket_path.display(), e);
return;
}
};
eprintln!("breadmill: listening on {}", socket_path.display());
for stream in listener.incoming() {
match stream {
Ok(s) => {
let state = Arc::clone(&state);
std::thread::spawn(move || {
handle(s, state, snippet_len, search_limit);
});
}
Err(e) => eprintln!("breadmill: accept error: {}", e),
}
}
}
fn handle(stream: UnixStream, state: Arc<SharedState>, snippet_len: usize, search_limit: usize) {
let stream_write = match stream.try_clone() {
Ok(s) => s,
Err(_) => return,
};
let mut reader = BufReader::new(&stream);
let mut writer = std::io::BufWriter::new(stream_write);
let mut line = String::new();
if reader.read_line(&mut line).is_err() {
return;
}
let response = match serde_json::from_str::<Request>(line.trim()) {
Ok(req) => {
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
dispatch(req, &state, snippet_len, search_limit)
}));
match r {
Ok(resp) => resp,
Err(_) => Response::Error { message: "internal error".into() },
}
}
Err(e) => Response::Error { message: e.to_string() },
};
if let Ok(mut json) = serde_json::to_string(&response) {
json.push('\n');
let _ = writer.write_all(json.as_bytes());
let _ = writer.flush();
}
}
fn dispatch(
req: Request,
state: &SharedState,
snippet_len: usize,
search_limit: usize,
) -> Response {
use std::sync::atomic::Ordering;
match req {
Request::Query { query, limit } => {
if !state.model_ready.load(Ordering::Relaxed) {
return Response::Error {
message: "model not ready — run breadmill --fetch-model".into(),
};
}
let embedding = {
let mut embedder = state.embedder.lock().unwrap();
match embedder.as_mut() {
Some(e) => match e.embed_query(&query) {
Ok(v) => v,
Err(e) => return Response::Error { message: e },
},
None => {
return Response::Error {
message: "embedder unavailable".into(),
}
}
}
};
let limit = limit.min(search_limit).max(1);
let store = state.store.lock().unwrap();
match store.search(&embedding, limit, snippet_len) {
Ok(hits) => Response::Hits { hits },
Err(e) => Response::Error { message: e },
}
}
Request::Status => {
use std::sync::atomic::Ordering;
Response::StatusInfo(StatusInfo {
indexed: state.indexed.load(Ordering::Relaxed),
pending: state.pending.load(Ordering::Relaxed),
model_ready: state.model_ready.load(Ordering::Relaxed),
})
}
Request::Reindex => {
use std::sync::atomic::Ordering;
state.reindex_signal.store(true, Ordering::Relaxed);
Response::Ok
}
}
}

235
breadmill/src/store.rs Normal file
View file

@ -0,0 +1,235 @@
use std::path::Path;
use rusqlite::{Connection, params};
use usearch::{Index, IndexOptions, MetricKind, ScalarKind, new_index};
pub struct Store {
pub conn: Connection,
pub index: Index,
pub dim: usize,
}
// usearch::Index wraps a raw C++ pointer; access is serialized by the Mutex<Store>.
unsafe impl Send for Store {}
#[derive(Debug)]
pub struct FileMeta {
pub path: String,
pub mtime: i64,
pub hash: String,
}
impl Store {
pub fn open(state_dir: &Path, dim: usize) -> Result<Self, String> {
std::fs::create_dir_all(state_dir).map_err(|e| e.to_string())?;
let db_path = state_dir.join("meta.db");
let conn = Connection::open(&db_path).map_err(|e| e.to_string())?;
conn.execute_batch(
"PRAGMA journal_mode=WAL;
PRAGMA foreign_keys=ON;
CREATE TABLE IF NOT EXISTS files (
path TEXT PRIMARY KEY,
mtime INTEGER NOT NULL,
hash TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS chunks (
id INTEGER PRIMARY KEY,
path TEXT NOT NULL REFERENCES files(path) ON DELETE CASCADE,
chunk_text TEXT NOT NULL,
chunk_start INTEGER NOT NULL,
chunk_end INTEGER NOT NULL
);",
)
.map_err(|e| e.to_string())?;
let idx_path = state_dir.join("vectors.usearch");
let options = IndexOptions {
dimensions: dim,
metric: MetricKind::Cos,
quantization: ScalarKind::F32,
connectivity: 16,
expansion_add: 128,
expansion_search: 64,
multi: false,
};
let index = new_index(&options).map_err(|e| e.to_string())?;
if idx_path.exists() {
index
.load(idx_path.to_str().unwrap())
.map_err(|e| e.to_string())?;
} else {
index.reserve(4096).map_err(|e| e.to_string())?;
}
Ok(Self { conn, index, dim })
}
// ---- file state ---------------------------------------------------------
pub fn all_files(&self) -> Result<Vec<FileMeta>, String> {
let mut stmt = self
.conn
.prepare("SELECT path, mtime, hash FROM files")
.map_err(|e| e.to_string())?;
let rows = stmt
.query_map([], |row| {
Ok(FileMeta {
path: row.get(0)?,
mtime: row.get(1)?,
hash: row.get(2)?,
})
})
.map_err(|e| e.to_string())?
.filter_map(|r| r.ok())
.collect();
Ok(rows)
}
pub fn upsert_file(&self, path: &str, mtime: i64, hash: &str) -> Result<(), String> {
self.conn
.execute(
"INSERT OR REPLACE INTO files (path, mtime, hash) VALUES (?1, ?2, ?3)",
params![path, mtime, hash],
)
.map_err(|e| e.to_string())?;
Ok(())
}
/// Delete a file and all its chunks from SQLite; also remove chunk vectors.
pub fn delete_file(&mut self, path: &str) -> Result<(), String> {
// Collect chunk IDs before deletion for usearch removal
let ids = self.chunk_ids_for(path)?;
self.conn
.execute("DELETE FROM files WHERE path = ?1", params![path])
.map_err(|e| e.to_string())?;
for id in ids {
let _ = self.index.remove(id); // best-effort; stale entries are harmless
}
Ok(())
}
fn chunk_ids_for(&self, path: &str) -> Result<Vec<u64>, String> {
let mut stmt = self
.conn
.prepare("SELECT id FROM chunks WHERE path = ?1")
.map_err(|e| e.to_string())?;
let ids = stmt
.query_map(params![path], |row| row.get::<_, i64>(0))
.map_err(|e| e.to_string())?
.filter_map(|r| r.ok())
.map(|id| id as u64)
.collect();
Ok(ids)
}
// ---- chunk operations ---------------------------------------------------
pub fn insert_chunk(
&mut self,
path: &str,
text: &str,
start: usize,
end: usize,
embedding: &[f32],
) -> Result<u64, String> {
self.conn
.execute(
"INSERT INTO chunks (path, chunk_text, chunk_start, chunk_end)
VALUES (?1, ?2, ?3, ?4)",
params![path, text, start as i64, end as i64],
)
.map_err(|e| e.to_string())?;
let id = self.conn.last_insert_rowid() as u64;
// Grow index if needed
if self.index.size() + 1 > self.index.capacity() {
self.index
.reserve(self.index.capacity() + 4096)
.map_err(|e| e.to_string())?;
}
self.index.add(id, embedding).map_err(|e| e.to_string())?;
Ok(id)
}
pub fn chunk_count(&self) -> usize {
self.conn
.query_row("SELECT COUNT(*) FROM chunks", [], |row| {
row.get::<_, i64>(0)
})
.unwrap_or(0) as usize
}
// ---- query --------------------------------------------------------------
pub fn search(
&self,
embedding: &[f32],
limit: usize,
snippet_len: usize,
) -> Result<Vec<breadsearch_shared::Hit>, String> {
let results = self
.index
.search(embedding, limit)
.map_err(|e| e.to_string())?;
let mut hits = Vec::new();
for (key, distance) in results.keys.iter().zip(results.distances.iter()) {
// Convert cosine distance → similarity score (higher = better)
let score = 1.0 - distance;
let maybe_chunk = self
.conn
.query_row(
"SELECT chunk_text, path FROM chunks WHERE id = ?1",
params![*key as i64],
|row| Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)),
)
.ok();
if let Some((chunk_text, path)) = maybe_chunk {
let snippet = truncate_to_chars(&chunk_text, snippet_len);
let title = std::path::Path::new(&path)
.file_name()
.and_then(|n| n.to_str())
.unwrap_or(&path)
.to_string();
hits.push(breadsearch_shared::Hit {
title,
path,
snippet,
score,
});
}
}
Ok(hits)
}
// ---- persistence --------------------------------------------------------
pub fn save_index(&self, state_dir: &Path) -> Result<(), String> {
let idx_path = state_dir.join("vectors.usearch");
self.index
.save(idx_path.to_str().unwrap())
.map_err(|e| e.to_string())
}
}
fn truncate_to_chars(s: &str, max_chars: usize) -> String {
if s.chars().count() <= max_chars {
return s.to_string();
}
let truncated: String = s.chars().take(max_chars).collect();
format!("{}…", truncated.trim_end())
}

14
breadsearch-shared/Cargo.toml Normal file
View file

@ -0,0 +1,14 @@
[package]
name = "breadsearch-shared"
version = "0.1.0"
edition = "2021"
license = "MIT"
[lib]
name = "breadsearch_shared"
path = "src/lib.rs"
[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
toml = "0.8"

268
breadsearch-shared/src/lib.rs Normal file
View file

@ -0,0 +1,268 @@
use std::{
env, fs,
io::{BufRead, BufReader, Write},
os::unix::net::UnixStream,
path::PathBuf,
};
use serde::{Deserialize, Serialize};
// ---- XDG path helpers -------------------------------------------------------
pub fn home_dir() -> PathBuf {
PathBuf::from(env::var("HOME").unwrap_or_else(|_| "/tmp".into()))
}
pub fn config_dir() -> PathBuf {
env::var("XDG_CONFIG_HOME")
.map(PathBuf::from)
.unwrap_or_else(|_| home_dir().join(".config"))
.join("breadsearch")
}
pub fn state_dir() -> PathBuf {
env::var("XDG_STATE_HOME")
.map(PathBuf::from)
.unwrap_or_else(|_| home_dir().join(".local/state"))
.join("breadsearch")
}
pub fn cache_dir() -> PathBuf {
env::var("XDG_CACHE_HOME")
.map(PathBuf::from)
.unwrap_or_else(|_| home_dir().join(".cache"))
.join("breadsearch")
}
pub fn socket_path() -> PathBuf {
env::var("XDG_RUNTIME_DIR")
.map(PathBuf::from)
.unwrap_or_else(|_| PathBuf::from("/tmp"))
.join("breadmill.sock")
}
// ---- Config -----------------------------------------------------------------
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Config {
#[serde(default)]
pub index: IndexConfig,
#[serde(default)]
pub search: SearchConfig,
#[serde(default)]
pub model: ModelConfig,
#[serde(default)]
pub power: PowerConfig,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct IndexConfig {
#[serde(default = "default_roots")]
pub roots: Vec<String>,
#[serde(default)]
pub excludes: Vec<String>,
#[serde(default = "default_extensions")]
pub extensions: Vec<String>,
#[serde(default = "default_max_file_mb")]
pub max_file_mb: f64,
}
fn default_roots() -> Vec<String> {
let home = home_dir();
vec![
home.join("Documents").to_string_lossy().into_owned(),
home.join("Projects").to_string_lossy().into_owned(),
home.join(".config/breadpad").to_string_lossy().into_owned(),
]
}
fn default_extensions() -> Vec<String> {
["md", "txt", "org", "pdf", "odt", "docx"]
.iter()
.map(|s| s.to_string())
.collect()
}
fn default_max_file_mb() -> f64 {
10.0
}
impl Default for IndexConfig {
fn default() -> Self {
Self {
roots: default_roots(),
excludes: vec![],
extensions: default_extensions(),
max_file_mb: default_max_file_mb(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SearchConfig {
#[serde(default = "default_limit")]
pub limit: usize,
#[serde(default = "default_snippet_len")]
pub snippet_len: usize,
}
fn default_limit() -> usize {
10
}
fn default_snippet_len() -> usize {
200
}
impl Default for SearchConfig {
fn default() -> Self {
Self {
limit: default_limit(),
snippet_len: default_snippet_len(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ModelConfig {
#[serde(default = "default_model_name")]
pub name: String,
#[serde(default = "default_dim")]
pub dim: usize,
/// Compute backend: "cpu" or "npu" (VitisAI/XDNA).
#[serde(default = "default_backend")]
pub backend: String,
}
fn default_model_name() -> String {
"nomic-embed-text-v1.5".into()
}
fn default_dim() -> usize {
768
}
fn default_backend() -> String {
"cpu".into()
}
impl Default for ModelConfig {
fn default() -> Self {
Self {
name: default_model_name(),
dim: default_dim(),
backend: default_backend(),
}
}
}
impl Default for Config {
fn default() -> Self {
Self {
index: IndexConfig::default(),
search: SearchConfig::default(),
model: ModelConfig::default(),
power: PowerConfig::default(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PowerConfig {
/// Master on/off switch for indexing (embedding). When false, breadmill
/// still serves queries over the existing index but never embeds new files.
#[serde(default = "default_true")]
pub enabled: bool,
/// Whether to keep embedding while running on battery. Embedding is the
/// compute-heavy step (CPU/NPU/GPU forward pass), so this defaults to
/// false to avoid draining battery; indexing resumes automatically once
/// AC power is reconnected.
#[serde(default)]
pub run_on_battery: bool,
}
fn default_true() -> bool {
true
}
impl Default for PowerConfig {
fn default() -> Self {
Self {
enabled: true,
run_on_battery: false,
}
}
}
impl Config {
pub fn load() -> Self {
let path = config_dir().join("config.toml");
let content = match fs::read_to_string(&path) {
Ok(s) => s,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Self::default(),
Err(e) => {
eprintln!("breadsearch: could not read {}: {}", path.display(), e);
return Self::default();
}
};
match toml::from_str(&content) {
Ok(c) => c,
Err(e) => {
eprintln!("breadsearch: parse error in {}: {}", path.display(), e);
Self::default()
}
}
}
}
// ---- IPC types (newline-delimited JSON over Unix socket) --------------------
#[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Request {
Query { query: String, limit: usize },
Status,
Reindex,
}
#[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Response {
Hits { hits: Vec<Hit> },
StatusInfo(StatusInfo),
Ok,
Error { message: String },
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Hit {
pub title: String,
pub path: String,
pub snippet: String,
pub score: f32,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct StatusInfo {
pub indexed: usize,
pub pending: usize,
pub model_ready: bool,
}
// ---- Socket client ----------------------------------------------------------
pub fn send_request(req: &Request) -> std::io::Result<Response> {
let mut stream = UnixStream::connect(socket_path())?;
let mut line = serde_json::to_string(req)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
line.push('\n');
stream.write_all(line.as_bytes())?;
stream.flush()?;
let mut response = String::new();
BufReader::new(stream).read_line(&mut response)?;
serde_json::from_str(response.trim())
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))
}

16
breadsearch/Cargo.toml Normal file
View file

@ -0,0 +1,16 @@
[package]
name = "breadsearch"
version = "0.1.0"
edition = "2021"
license = "MIT"
[[bin]]
name = "breadsearch"
path = "src/main.rs"
[dependencies]
breadsearch-shared = { path = "../breadsearch-shared" }
bread-theme = { git = "https://github.com/Breadway/bread-ecosystem", tag = "v0.2.8", features = ["gtk"] }
gtk4 = { version = "0.11", features = ["v4_12"] }
gtk4-layer-shell = "0.8"
serde_json = "1"

429
breadsearch/src/main.rs Normal file
View file

@ -0,0 +1,429 @@
use bread_theme::{hex_to_rgba, ink_on, load_palette, Palette};
use breadsearch_shared::{Hit, Request, Response};
use std::{
cell::RefCell,
env, fs,
path::PathBuf,
process::Command,
rc::Rc,
sync::mpsc,
};
use gtk4::{
glib,
pango::EllipsizeMode,
prelude::*,
Application, ApplicationWindow, Box as GBox, CssProvider, EventControllerKey, Image, Label,
ListBox, Orientation, PolicyType, ScrolledWindow, SearchEntry, SelectionMode,
};
use gtk4_layer_shell::{Edge, KeyboardMode, Layer, LayerShell};
// ---- Theming ----------------------------------------------------------------
fn build_css(p: &Palette) -> String {
let bg_panel = hex_to_rgba(&p.background, 0.60);
format!(
"window {{ background-color: transparent; }}\
.launcher-bg {{ background-color: {bg_panel}; color: {on_bg}; border-radius: 8px;\
box-shadow: 0 8px 32px rgba(0,0,0,0.6); }}\
searchentry {{ background-color: {surface}; color: {on_surface}; caret-color: {accent};\
border: none; outline: none; box-shadow: none;\
padding: 12px 16px; border-radius: 6px 6px 0 0; }}\
listbox {{ background-color: transparent; padding: 4px; }}\
row {{ padding: 8px 12px; color: {on_bg}; background-color: transparent;\
border-radius: 6px; }}\
row:hover {{ background-color: {surface}; color: {on_surface}; }}\
row:selected {{ background-color: {surface}; color: {on_surface}; }}\
.hit-title {{ font-size: 14px; }}\
.hit-muted {{ opacity: 0.6; font-size: 12px; }}\
.hit-snippet {{ opacity: 0.75; font-size: 11px; font-style: italic; }}\
.hit-score {{ opacity: 0.5; font-size: 11px; }}\
image {{ margin-right: 8px; }}",
bg_panel = bg_panel,
surface = p.color0,
accent = p.color4,
on_bg = ink_on(&p.background),
on_surface = ink_on(&p.color0),
)
}
// ---- PID file toggle --------------------------------------------------------
fn pid_file() -> PathBuf {
env::var("XDG_RUNTIME_DIR")
.map(PathBuf::from)
.unwrap_or_else(|_| PathBuf::from("/tmp"))
.join("breadsearch.pid")
}
fn is_breadsearch_pid(pid: u32) -> bool {
fs::read_to_string(format!("/proc/{}/comm", pid))
.map(|s| s.trim() == "breadsearch")
.unwrap_or(false)
}
fn toggle_or_continue() -> bool {
let pf = pid_file();
if let Ok(content) = fs::read_to_string(&pf) {
if let Ok(pid) = content.trim().parse::<u32>() {
if is_breadsearch_pid(pid) {
let _ = Command::new("kill").arg(pid.to_string()).status();
return false;
}
}
}
let _ = fs::write(&pf, std::process::id().to_string());
true
}
fn cleanup_pid() {
let _ = fs::remove_file(pid_file());
}
// ---- Row builder ------------------------------------------------------------
fn make_hit_row(hit: &Hit) -> gtk4::ListBoxRow {
let row = gtk4::ListBoxRow::new();
let vbox = GBox::new(Orientation::Vertical, 2);
vbox.set_margin_start(6);
vbox.set_margin_end(6);
vbox.set_margin_top(4);
vbox.set_margin_bottom(4);
let hbox = GBox::new(Orientation::Horizontal, 0);
let (content_type, _) =
gtk4::gio::functions::content_type_guess(Some(&hit.path), None::<&[u8]>);
let gicon = gtk4::gio::content_type_get_icon(&content_type);
let img = Image::from_gicon(&gicon);
img.set_pixel_size(24);
hbox.append(&img);
let title = Label::new(Some(&hit.title));
title.add_css_class("hit-title");
title.set_xalign(0.0);
title.set_hexpand(true);
title.set_ellipsize(EllipsizeMode::End);
hbox.append(&title);
let score_lbl = Label::new(Some(&format!("{:.0}%", hit.score * 100.0)));
score_lbl.add_css_class("hit-score");
score_lbl.set_xalign(1.0);
hbox.append(&score_lbl);
vbox.append(&hbox);
let path_lbl = Label::new(Some(&hit.path));
path_lbl.add_css_class("hit-muted");
path_lbl.set_xalign(0.0);
path_lbl.set_ellipsize(EllipsizeMode::Start);
vbox.append(&path_lbl);
let snippet_text = hit.snippet.lines().next().unwrap_or("").trim().to_string();
if !snippet_text.is_empty() {
let snippet = Label::new(Some(&snippet_text));
snippet.add_css_class("hit-snippet");
snippet.set_xalign(0.0);
snippet.set_ellipsize(EllipsizeMode::End);
vbox.append(&snippet);
}
row.set_child(Some(&vbox));
unsafe { row.set_data("hit_path", hit.path.clone()) };
row
}
fn row_path(row: &gtk4::ListBoxRow) -> Option<String> {
unsafe { row.data::<String>("hit_path").map(|p| p.as_ref().clone()) }
}
// ---- List population --------------------------------------------------------
fn clear_list(list: &ListBox) {
while let Some(child) = list.first_child() {
list.remove(&child);
}
}
fn info_row(text: &str) -> gtk4::ListBoxRow {
let row = gtk4::ListBoxRow::new();
let lbl = Label::new(Some(text));
lbl.add_css_class("hit-muted");
lbl.set_margin_top(12);
lbl.set_margin_bottom(12);
row.set_child(Some(&lbl));
row.set_activatable(false);
row.set_selectable(false);
row
}
fn populate_list(list: &ListBox, result: std::io::Result<Response>) {
clear_list(list);
match result {
Ok(Response::Hits { hits }) => {
if hits.is_empty() {
list.append(&info_row("No results"));
} else {
for hit in &hits {
list.append(&make_hit_row(hit));
}
if let Some(first) = list.row_at_index(0) {
list.select_row(Some(&first));
}
}
}
Ok(Response::Error { message }) => {
list.append(&info_row(&format!("Error: {}", message)));
}
Err(e) => {
list.append(&info_row(&format!("breadmill not running: {}", e)));
}
_ => {}
}
}
// ---- Actions ----------------------------------------------------------------
fn open_file(path: &str) {
let _ = Command::new("xdg-open")
.arg(path)
.stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn();
}
fn open_folder(path: &str) {
let parent = std::path::Path::new(path)
.parent()
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|| path.to_string());
let _ = Command::new("xdg-open")
.arg(&parent)
.stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn();
}
// ---- UI ---------------------------------------------------------------------
fn run_ui() {
let app = Application::builder()
.application_id("com.breadway.breadsearch")
.build();
let debounce_id: Rc<RefCell<Option<glib::SourceId>>> = Rc::new(RefCell::new(None));
app.connect_activate(move |app| {
bread_theme::gtk::apply_shared();
bread_theme::gtk::apply_app_css(|| build_css(&load_palette()));
{
let user_css_path = breadsearch_shared::config_dir().join("style.css");
let user_cell: RefCell<Option<CssProvider>> = RefCell::new(None);
bread_theme::gtk::apply_user_css(&user_css_path, &user_cell);
}
let window = ApplicationWindow::builder().application(app).build();
window.init_layer_shell();
window.set_namespace(Some("breadsearch"));
window.set_layer(Layer::Overlay);
window.set_keyboard_mode(KeyboardMode::Exclusive);
for edge in [Edge::Top, Edge::Bottom, Edge::Left, Edge::Right] {
window.set_anchor(edge, true);
}
window.set_exclusive_zone(0);
let close_all: Rc<dyn Fn()> = Rc::new({
let w = window.clone();
move || {
cleanup_pid();
w.close();
}
});
let vbox = GBox::new(Orientation::Vertical, 0);
vbox.add_css_class("launcher-bg");
vbox.set_halign(gtk4::Align::Center);
vbox.set_valign(gtk4::Align::Start);
vbox.set_margin_top(120);
vbox.set_size_request(640, -1);
let search = SearchEntry::new();
search.set_placeholder_text(Some("breadsearch — find anything by meaning"));
vbox.append(&search);
let scroll = ScrolledWindow::new();
scroll.set_policy(PolicyType::Never, PolicyType::Automatic);
scroll.set_max_content_height(520);
scroll.set_propagate_natural_height(true);
let list = ListBox::new();
list.set_selection_mode(SelectionMode::Browse);
list.append(&info_row("Type to search across your documents…"));
scroll.set_child(Some(&list));
vbox.append(&scroll);
window.set_child(Some(&vbox));
// Search with 150ms debounce + off-UI-thread query
let list_s = list.clone();
let debounce = Rc::clone(&debounce_id);
search.connect_changed(move |entry| {
let query = entry.text().to_string();
if let Some(id) = debounce.borrow_mut().take() {
id.remove();
}
if query.is_empty() {
clear_list(&list_s);
list_s.append(&info_row("Type to search across your documents…"));
return;
}
let list_clone = list_s.clone();
let debounce_clone = Rc::clone(&debounce);
let id = glib::timeout_add_local(std::time::Duration::from_millis(150), move || {
debounce_clone.borrow_mut().take();
let q = query.clone();
let (tx, rx) = mpsc::sync_channel::<std::io::Result<Response>>(1);
std::thread::spawn(move || {
let req = Request::Query { query: q, limit: 10 };
let _ = tx.send(breadsearch_shared::send_request(&req));
});
// Poll via idle_add_local until the thread delivers its result.
// Unix socket round-trips are sub-millisecond so this fires once.
let rx = Rc::new(rx);
let list_t = list_clone.clone();
glib::idle_add_local(move || {
match rx.try_recv() {
Ok(result) => {
populate_list(&list_t, result);
glib::ControlFlow::Break
}
Err(mpsc::TryRecvError::Empty) => glib::ControlFlow::Continue,
Err(mpsc::TryRecvError::Disconnected) => glib::ControlFlow::Break,
}
});
glib::ControlFlow::Break
});
*debounce.borrow_mut() = Some(id);
});
// Keyboard handling
let key_ctrl = EventControllerKey::new();
key_ctrl.set_propagation_phase(gtk4::PropagationPhase::Capture);
let close_k = Rc::clone(&close_all);
let list_k = list.clone();
key_ctrl.connect_key_pressed(move |_, key, _, mods| {
use gtk4::gdk::Key;
match key {
Key::Escape => {
close_k();
glib::Propagation::Stop
}
Key::Return | Key::KP_Enter => {
if let Some(row) = list_k.selected_row() {
if let Some(path) = row_path(&row) {
if mods.contains(gtk4::gdk::ModifierType::CONTROL_MASK) {
open_folder(&path);
} else {
open_file(&path);
}
close_k();
}
}
glib::Propagation::Stop
}
Key::Down => {
let cur = list_k.selected_row().map(|r| r.index()).unwrap_or(-1);
let mut i = cur + 1;
loop {
match list_k.row_at_index(i) {
Some(r) if r.is_selectable() => {
list_k.select_row(Some(&r));
break;
}
Some(_) => i += 1,
None => break,
}
}
glib::Propagation::Stop
}
Key::Up => {
let cur = list_k.selected_row().map(|r| r.index()).unwrap_or(0);
let mut i = cur - 1;
loop {
if i < 0 {
break;
}
match list_k.row_at_index(i) {
Some(r) if r.is_selectable() => {
list_k.select_row(Some(&r));
break;
}
Some(_) => i -= 1,
None => break,
}
}
glib::Propagation::Stop
}
_ => glib::Propagation::Proceed,
}
});
window.add_controller(key_ctrl);
// Row click opens file
let close_a = Rc::clone(&close_all);
list.connect_row_activated(move |_, row| {
if let Some(path) = row_path(row) {
open_file(&path);
close_a();
}
});
// Click outside launcher panel → close
let close_outside = Rc::clone(&close_all);
let vbox_ref = vbox.clone();
let win_ref = window.clone();
let outside_click = gtk4::GestureClick::new();
outside_click.connect_pressed(move |_, _, x, y| {
if let Some(b) = vbox_ref.compute_bounds(&win_ref) {
if x < b.x() as f64
|| x > (b.x() + b.width()) as f64
|| y < b.y() as f64
|| y > (b.y() + b.height()) as f64
{
close_outside();
}
}
});
window.add_controller(outside_click);
window.connect_destroy(|_| cleanup_pid());
window.present();
search.grab_focus();
});
app.run();
}
// ---- Main -------------------------------------------------------------------
fn main() {
if !toggle_or_continue() {
return;
}
run_ui();
}

41
config.example.toml Normal file
View file

@ -0,0 +1,41 @@
# breadsearch configuration
# Copy to ~/.config/breadsearch/config.toml
[index]
# Directories to crawl for documents
roots = [
"~/Documents",
"~/Projects",
"~/.config/breadpad",
]
# File extensions to index
extensions = ["md", "txt", "org", "pdf", "odt", "docx"]
# Skip files larger than this (MB)
max_file_mb = 10.0
[search]
# Default number of results
limit = 10
# Max characters in snippet
snippet_len = 200
[model]
# Embedding model name (informational; breadmill uses the ONNX files it fetches)
name = "nomic-embed-text-v1.5"
# Embedding dimension
dim = 768
# Compute backend: "cpu" (default), "npu" (AMD XDNA via VitisAI EP), or
# "rocm" (AMD iGPU/dGPU). NPU requires breadmill compiled with --features npu
# and the AMD Ryzen AI SDK installed; ROCm requires --features rocm.
# Override at runtime with: breadmill --npu / --rocm
backend = "cpu"
[power]
# Master switch for indexing. When false, breadmill still answers queries
# against the existing index but never embeds new/changed files.
enabled = true
# Embedding is the compute-heavy step. By default breadmill pauses indexing
# while on battery and resumes automatically when AC power is reconnected.
# Set true to index on battery too. Machines with no battery (desktops) are
# always treated as on-power regardless of this setting.
run_on_battery = false

238
docs/error-reports/2026-06-24-oom-minified-json-chunking.md Normal file
View file

@ -0,0 +1,238 @@
# Error Report: breadmill OOM loop on minified JSON `.txt` file
**Date:** 2026-06-24
**Reporter:** breadway
**Component:** `breadmill` (semantic search indexer)
**Version:** 0.1.0 (`/home/breadway/Projects/breadsearch/breadmill`)
**Severity:** Critical — repeated kernel OOM kills, ~28 GB RAM consumption, system instability
---
## Summary
`breadmill` enters a crash loop when indexing a 705 KB minified JSON game save file exposed as `.txt`. Word-based chunking produces ~486–616 KB chunks that are passed unbounded to the ONNX embedding model. ONNX Runtime then attempts multi-terabyte allocations in attention `MatMul`, the process balloons to ~28 GB RSS, and the kernel OOM killer terminates it. `Restart=on-failure` causes an immediate restart and the cycle repeats.
On 2026-06-24 this produced **40 embed errors** and **76 OOM kills** in journal logs within ~45 minutes.
---
## Environment
| Item | Value |
|------|-------|
| Host | Lenovo Yoga Slim 7 14AKP10 (`83JY`) |
| OS | BOS (Arch-based), kernel `7.0.12-arch1-1` |
| RAM | 32 GB |
| Swap | 4 GB zram |
| Model | `nomic-embed-text-v1.5` (768-dim, ONNX via `ort` 2.0.0-rc.12) |
| Service | `breadmill.service` (user systemd unit) |
| Config | `~/.config/breadsearch/config.toml` |
### Service unit
```ini
[Service]
Type=simple
ExecStart=%h/.cargo/bin/breadmill
Restart=on-failure
RestartSec=5
# No MemoryMax set (unlimited)
# OOMScoreAdjust=200 (default for user services — preferential OOM victim)
```
### Relevant config
```toml
[index]
roots = ["~/Documents", "~/.config/breadpad"]
extensions = ["md", "txt", "org", "pdf", "odt", "docx"]
max_file_mb = 5.0
```
---
## Triggering file
**Path:**
`~/Documents/Gaming/Games/Spaceflight Simulator Game/Saving/Worlds/Bread/Persistent/Rockets.txt`
| Property | Value |
|----------|-------|
| Size | 721,601 bytes (705 KB) |
| Type | JSON array (11 rocket objects), stored as `.txt` |
| Lines | **1** (single-line minified JSON) |
| Whitespace-delimited "words" | 601 |
| Longest "word" | 68,068 characters |
The file is under `max_file_mb = 5.0` and has extension `.txt`, so it is indexed.
---
## Root cause
### 1. Word-based chunking breaks on minified JSON
`chunk::chunk_text()` splits on whitespace only (`chunk.rs`). For minified JSON, most structure is packed into very long tokens with few spaces.
Chunking with `words_per_chunk=400`, `overlap_words=80` (`indexer.rs:268`) yields:
| Chunk | Word range | Character length |
|-------|------------|------------------|
| 0 | 0–399 | **486,347** |
| 1 | 320–600 | **616,582** |
A "400-word chunk" is not ~400 natural-language words; it is hundreds of kilobytes of dense JSON.
### 2. No token/character limit before embedding
`embed::OrtEmbedder::embed_with_prefix()` tokenizes the full chunk and runs ONNX inference with no `max_length` truncation (`embed.rs`). The nomic model supports ~8192 tokens; these chunks are orders of magnitude larger when tokenized.
### 3. ONNX attention allocation explodes
Embedding fails inside ONNX Runtime:
```
Non-zero status code returned while running FusedMatMul node.
Name: '/encoder/layers.0/attn/MatMul/MatMulScaleFusion/'
Status Message: ... BFCArena::AllocateRawInternal ...
Failed to allocate memory for requested buffer of size 5525780084992
```
Requested buffer: **~5.1 TiB** (5,525,780,084,992 bytes).
Kernel also logs repeated allocation attempts before OOM:
```
__vm_enough_memory: pid: NNNN, comm: breadmill,
bytes: 8796093026304 not enough memory for the allocation (~8.0 TiB)
bytes: 7916483514368 not enough memory for the allocation (~7.2 TiB)
bytes: 7124834848768 not enough memory for the allocation (~6.5 TiB)
```
### 4. Restart loop amplifies damage
`Restart=on-failure` + `RestartSec=5` restarts breadmill immediately after each OOM kill. Each restart reloads the model, rescans `~/Documents`, hits the same file, and OOMs again. Peak memory per attempt: **~27–28 GB RSS** (`anon-rss:27867432kB`).
---
## Log excerpts
### Embed error (repeats on every restart)
```
breadmill: embed error for .../Rockets.txt: Non-zero status code returned while running FusedMatMul node.
Name:'/encoder/layers.0/attn/MatMul/MatMulScaleFusion/' Status Message:
... Failed to allocate memory for requested buffer of size 5525780084992
```
### OOM kill
```
oom-kill: ... task_memcg=.../breadmill.service, task=breadmill, pid=4515, uid=1000
Out of memory: Killed process 4515 (breadmill)
total-vm:64081848kB, anon-rss:27867432kB, ... oom_score_adj:200
systemd[1649]: breadmill.service: Failed with result 'oom-kill'.
systemd[1649]: breadmill.service: Consumed ... 27.9G memory peak, 2.1G memory swap peak.
systemd[1649]: breadmill.service: Scheduled restart job, restart counter is at 8.
```
### Secondary noise (non-fatal)
```
breadmill: extract .../Novacana Info.txt: stream did not contain valid UTF-8
```
---
## Reproduction
1. Place a single-line minified JSON file (>400 KB, `.txt` extension) under a configured index root.
2. Start `breadmill serve` (or `systemctl --user start breadmill.service`).
3. Wait for initial scan to reach the file.
4. Observe embed errors, climbing RSS, and OOM kills in `journalctl --user -u breadmill.service -f`.
**Minimal reproducer characteristics:**
- Extension in `extensions` list (e.g. `txt`)
- File size `< max_file_mb`
- Content is minified JSON or other whitespace-sparse text on one line
- Results in chunks >> model `max_seq_len` when tokenized
---
## Impact
- **breadmill** unusable while the file is present in index roots
- **System-wide** memory pressure: swap exhaustion, unrelated process kills, journal/cache pressure
- Can be mistaken for unrelated storage or suspend issues when swap write errors appear in kernel log
---
## Suggested fixes
### Required (correctness)
1. **Truncate before tokenization** — cap input to model `max_seq_len` (8192 tokens for nomic-embed-text-v1.5) in `embed.rs`, with explicit logging when truncation occurs.
2. **Character-based chunk limits** — add `max_chunk_chars` (e.g. 8_000–32_000) independent of word count; split oversized chunks before embedding.
3. **Sanity-check chunk size** — refuse to embed chunks above a byte/token threshold; log and skip rather than calling ONNX.
### Recommended (resilience)
4. **Per-chunk error isolation** — on embed failure for one chunk, skip that chunk but do not retry the entire file in a tight loop; mark file as `failed` in SQLite.
5. **Service memory limit** — set `MemoryMax=4G` (or similar) on `breadmill.service` so a runaway embed cannot take the whole machine.
6. **Lower `OOMScoreAdjust`** — use `0` or negative so breadmill is not preferentially killed while still allowing limits.
7. **Backoff on repeated OOM** — `RestartSec=exponential` or stop after N OOM kills per file.
### Optional (UX)
8. **Exclude patterns** — config option for glob excludes (e.g. `**/Saving/**`, `**/*.json` even if renamed `.txt`).
9. **Detect minified JSON** — if `serde_json::from_str` succeeds on `.txt`, skip or pretty-print/chunk differently.
10. **`--version` flag** — aids bug reports (currently `unknown command: --version`).
---
## Workaround (immediate)
Exclude the game save directory from index roots in `~/.config/breadsearch/config.toml`:
```toml
[index]
roots = [
"~/Documents",
"~/.config/breadpad",
]
# Then add an exclude mechanism when available, OR temporarily narrow roots:
# roots = ["~/Documents/Creative", "~/.config/breadpad"]
```
Or stop the service until a fix is deployed:
```bash
systemctl --user stop breadmill.service
```
---
## Files involved
| File | Role |
|------|------|
| `breadmill/src/chunk.rs` | Whitespace word chunking — no char/token cap |
| `breadmill/src/indexer.rs:268` | `chunk_text(&text, 400, 80)` — hardcoded params |
| `breadmill/src/embed.rs` | No truncation before `tokenizer.encode()` / ONNX run |
| `breadmill/src/extract.rs` | Treats `.txt` as raw UTF-8 (JSON passes through) |
| `~/.config/breadsearch/config.toml` | `max_file_mb` only; no chunk/token limits |
---
## Related
- ONNX node: `/encoder/layers.0/attn/MatMul/MatMulScaleFusion/`
- Model: [nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) (max sequence length 8192)

1
original-info-signature.txt Normal file
View file

@ -0,0 +1 @@
6c422027bd642c5e1f409e020b7b7a9b

1
original-model-signature.txt Normal file
View file

@ -0,0 +1 @@
e9339c75bca7719f67ac377f03e28b83

15
packaging/breadmill.service Normal file
View file

@ -0,0 +1,15 @@
[Unit]
Description=Breadmill semantic search indexer
Documentation=https://github.com/breadway/breadsearch
After=default.target
[Service]
Type=simple
ExecStart=%h/.cargo/bin/breadmill
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=default.target

144
quantize_nomic.py Normal file
View file

@ -0,0 +1,144 @@
#!/usr/bin/env python3
"""
Quantize nomic-embed-text-v1.5 with AMD Quark for NPU (XDNA) inference.
Targets int8 QDQ format with enable_npu_transformer=True so VAIML can map
attention MatMul ops to the NPU.
"""
import json
import numpy as np
import onnxruntime
from pathlib import Path
MODEL_PATH = Path.home() / ".cache/breadsearch/models/model.onnx"
TOKENIZER_PATH = Path.home() / ".cache/breadsearch/models/tokenizer.json"
OUTPUT_PATH = Path.home() / ".cache/breadsearch/models/model_quantized.onnx"
CALIB_SENTENCES = [
"search_document: The quick brown fox jumps over the lazy dog.",
"search_document: Machine learning is a branch of artificial intelligence.",
"search_document: The Eiffel Tower is located in Paris, France.",
"search_document: Python is a high-level programming language.",
"search_document: The speed of light is approximately 299,792,458 meters per second.",
"search_document: Climate change is one of the greatest challenges facing humanity.",
"search_document: Quantum computing leverages quantum mechanical phenomena.",
"search_document: The human genome contains approximately 3 billion base pairs.",
"search_document: Rust is a systems programming language focused on safety.",
"search_document: Neural networks are inspired by the structure of the brain.",
"search_document: The Milky Way galaxy contains over 100 billion stars.",
"search_document: Cryptography is the practice of secure communication.",
"search_document: The Linux kernel was first released in 1991 by Linus Torvalds.",
"search_document: Databases store and retrieve structured data efficiently.",
"search_document: The TCP/IP protocol suite is the backbone of the internet.",
"search_document: Embedded systems run on resource-constrained hardware.",
"search_document: The ONNX format provides a standard for machine learning models.",
"search_document: Transformer models use attention mechanisms for sequence tasks.",
"search_document: File systems organize and manage data storage on disks.",
"search_document: Async programming allows concurrent execution without threads.",
"search_query: What is machine learning?",
"search_query: How does attention work in transformers?",
"search_query: Where is the Eiffel Tower?",
"search_query: What programming language should I learn first?",
"search_query: How fast is the speed of light?",
]
MAX_SEQ_LEN = 512
def load_tokenizer(path: Path):
with open(path) as f:
return json.load(f)
def simple_tokenize(tok_data: dict, text: str, max_len: int) -> dict[str, np.ndarray]:
vocab = tok_data["model"]["vocab"]
unk_id = vocab.get("[UNK]", 100)
cls_id = vocab.get("[CLS]", 101)
sep_id = vocab.get("[SEP]", 102)
pad_id = vocab.get("[PAD]", 0)
words = text.lower().split()
token_ids = [cls_id]
for w in words:
token_ids.append(vocab.get(w, unk_id))
token_ids.append(sep_id)
if len(token_ids) > max_len:
token_ids = token_ids[:max_len - 1] + [sep_id]
seq_len = len(token_ids)
padded = token_ids + [pad_id] * (max_len - seq_len)
mask = [1] * seq_len + [0] * (max_len - seq_len)
type_ids = [0] * max_len
return {
"input_ids": np.array([padded], dtype=np.int64),
"attention_mask": np.array([mask], dtype=np.int64),
"token_type_ids": np.array([type_ids], dtype=np.int64),
}
class NomicCalibrationReader:
def __init__(self, sentences, tok_data, max_len=MAX_SEQ_LEN):
self.inputs = [simple_tokenize(tok_data, s, max_len) for s in sentences]
self.idx = 0
def get_next(self):
if self.idx >= len(self.inputs):
return None
sample = self.inputs[self.idx]
self.idx += 1
return sample
def rewind(self):
self.idx = 0
def main():
print(f"Loading tokenizer from {TOKENIZER_PATH}")
tok_data = load_tokenizer(TOKENIZER_PATH)
calib_reader = NomicCalibrationReader(CALIB_SENTENCES, tok_data)
# Verify model input names match what we provide
sess = onnxruntime.InferenceSession(str(MODEL_PATH), providers=["CPUExecutionProvider"])
input_names = [inp.name for inp in sess.get_inputs()]
print(f"Model inputs: {input_names}")
del sess
from quark.onnx import ModelQuantizer, QuantizationConfig
from quark.onnx.quantization.config.config import Config
from onnxruntime.quantization import CalibrationMethod, QuantFormat, QuantType
quant_config = QuantizationConfig(
calibrate_method=CalibrationMethod.MinMax,
quant_format=QuantFormat.QDQ,
activation_type=QuantType.QInt8,
weight_type=QuantType.QInt8,
per_channel=False,
reduce_range=False,
optimize_model=True,
enable_npu_transformer=True,
include_cle=True,
print_summary=True,
extra_options={
"ActivationSymmetric": True,
"WeightSymmetric": True,
},
)
config = Config(global_quant_config=quant_config)
quantizer = ModelQuantizer(config)
print(f"Quantizing {MODEL_PATH} → {OUTPUT_PATH}")
print("This may take several minutes...")
quantizer.quantize_model(
model_input=str(MODEL_PATH),
model_output=str(OUTPUT_PATH),
calibration_data_reader=calib_reader,
)
print(f"\nQuantized model saved to {OUTPUT_PATH}")
print(f"Size: {OUTPUT_PATH.stat().st_size / 1024 / 1024:.1f} MB")
if __name__ == "__main__":
main()

151
quantized_info.csv Normal file
View file

@ -0,0 +1,151 @@
/home/breadway/.cache/breadsearch/models/model.onnx quantization quantization info,2026-06-24 18:14:20
quantization stage,time consumed(s),sub stage,time consumed(s)
pre process,5.43
,,calibration: collect data (onnx inference + numpy statistics),16.60
,,calibration: compute data,0.01
calibration (collect data + compute data),19.75
static quantization,0.81
post process(including finetuning),0.00
e2e,40.621040006999465
Node Name,Op Type,Activation,Weights,Bias
/encoder/layers.0/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.0/attn/Reshape,Reshape,INT8,,
/encoder/layers.0/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.0/Add,Add,INT8,,
/encoder/layers.0/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.0/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.0/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.0/mlp/Mul,Mul,INT8,,
/encoder/layers.0/mlp/Mul_1,Mul,INT8,,
/encoder/layers.0/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.0/Add_1,Add,INT8,,
/encoder/layers.1/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.1/attn/Reshape,Reshape,INT8,,
/encoder/layers.1/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.1/Add,Add,INT8,,
/encoder/layers.1/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.1/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.1/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.1/mlp/Mul,Mul,INT8,,
/encoder/layers.1/mlp/Mul_1,Mul,INT8,,
/encoder/layers.1/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.1/Add_1,Add,INT8,,
/encoder/layers.2/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.2/attn/Reshape,Reshape,INT8,,
/encoder/layers.2/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.2/Add,Add,INT8,,
/encoder/layers.2/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.2/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.2/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.2/mlp/Mul,Mul,INT8,,
/encoder/layers.2/mlp/Mul_1,Mul,INT8,,
/encoder/layers.2/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.2/Add_1,Add,INT8,,
/encoder/layers.3/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.3/attn/Reshape,Reshape,INT8,,
/encoder/layers.3/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.3/Add,Add,INT8,,
/encoder/layers.3/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.3/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.3/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.3/mlp/Mul,Mul,INT8,,
/encoder/layers.3/mlp/Mul_1,Mul,INT8,,
/encoder/layers.3/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.3/Add_1,Add,INT8,,
/encoder/layers.4/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.4/attn/Reshape,Reshape,INT8,,
/encoder/layers.4/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.4/Add,Add,INT8,,
/encoder/layers.4/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.4/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.4/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.4/mlp/Mul,Mul,INT8,,
/encoder/layers.4/mlp/Mul_1,Mul,INT8,,
/encoder/layers.4/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.4/Add_1,Add,INT8,,
/encoder/layers.5/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.5/attn/Reshape,Reshape,INT8,,
/encoder/layers.5/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.5/Add,Add,INT8,,
/encoder/layers.5/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.5/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.5/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.5/mlp/Mul,Mul,INT8,,
/encoder/layers.5/mlp/Mul_1,Mul,INT8,,
/encoder/layers.5/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.5/Add_1,Add,INT8,,
/encoder/layers.6/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.6/attn/Reshape,Reshape,INT8,,
/encoder/layers.6/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.6/Add,Add,INT8,,
/encoder/layers.6/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.6/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.6/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.6/mlp/Mul,Mul,INT8,,
/encoder/layers.6/mlp/Mul_1,Mul,INT8,,
/encoder/layers.6/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.6/Add_1,Add,INT8,,
/encoder/layers.7/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.7/attn/Reshape,Reshape,INT8,,
/encoder/layers.7/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.7/Add,Add,INT8,,
/encoder/layers.7/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.7/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.7/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.7/mlp/Mul,Mul,INT8,,
/encoder/layers.7/mlp/Mul_1,Mul,INT8,,
/encoder/layers.7/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.7/Add_1,Add,INT8,,
/encoder/layers.8/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.8/attn/Reshape,Reshape,INT8,,
/encoder/layers.8/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.8/Add,Add,INT8,,
/encoder/layers.8/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.8/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.8/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.8/mlp/Mul,Mul,INT8,,
/encoder/layers.8/mlp/Mul_1,Mul,INT8,,
/encoder/layers.8/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.8/Add_1,Add,INT8,,
/encoder/layers.9/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.9/attn/Reshape,Reshape,INT8,,
/encoder/layers.9/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.9/Add,Add,INT8,,
/encoder/layers.9/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.9/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.9/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.9/mlp/Mul,Mul,INT8,,
/encoder/layers.9/mlp/Mul_1,Mul,INT8,,
/encoder/layers.9/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.9/Add_1,Add,INT8,,
/encoder/layers.10/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.10/attn/Reshape,Reshape,INT8,,
/encoder/layers.10/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.10/Add,Add,INT8,,
/encoder/layers.10/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.10/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.10/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.10/mlp/Mul,Mul,INT8,,
/encoder/layers.10/mlp/Mul_1,Mul,INT8,,
/encoder/layers.10/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.10/Add_1,Add,INT8,,
/encoder/layers.11/attn/Wqkv/MatMul,MatMul,INT8,INT8,
/encoder/layers.11/attn/Reshape,Reshape,INT8,,
/encoder/layers.11/attn/out_proj/MatMul,MatMul,INT8,INT8,
/encoder/layers.11/Add,Add,INT8,,
/encoder/layers.11/mlp/fc12/MatMul,MatMul,INT8,INT8,
/encoder/layers.11/mlp/fc11/MatMul,MatMul,INT8,INT8,
/encoder/layers.11/mlp/Sigmoid,Sigmoid,INT8,,
/encoder/layers.11/mlp/Mul,Mul,INT8,,
/encoder/layers.11/mlp/Mul_1,Mul,INT8,,
/encoder/layers.11/mlp/fc2/MatMul,MatMul,INT8,INT8,
/encoder/layers.11/Add_1,Add,INT8,,
Op Type,Activation,Weights,Bias
MatMul,INT8(60),INT8(60),
Reshape,INT8(12),,
Add,INT8(24),,
Sigmoid,INT8(12),,
Mul,INT8(24),,
1 /home/breadway/.cache/breadsearch/models/model.onnx quantization quantization info,2026-06-24 18:14:20
2 quantization stage,time consumed(s),sub stage,time consumed(s)
3 pre process,5.43
4 ,,calibration: collect data (onnx inference + numpy statistics),16.60
5 ,,calibration: compute data,0.01
6 calibration (collect data + compute data),19.75
7 static quantization,0.81
8 post process(including finetuning),0.00
9 e2e,40.621040006999465
10 Node Name,Op Type,Activation,Weights,Bias
11 /encoder/layers.0/attn/Wqkv/MatMul,MatMul,INT8,INT8,
12 /encoder/layers.0/attn/Reshape,Reshape,INT8,,
13 /encoder/layers.0/attn/out_proj/MatMul,MatMul,INT8,INT8,
14 /encoder/layers.0/Add,Add,INT8,,
15 /encoder/layers.0/mlp/fc12/MatMul,MatMul,INT8,INT8,
16 /encoder/layers.0/mlp/fc11/MatMul,MatMul,INT8,INT8,
17 /encoder/layers.0/mlp/Sigmoid,Sigmoid,INT8,,
18 /encoder/layers.0/mlp/Mul,Mul,INT8,,
19 /encoder/layers.0/mlp/Mul_1,Mul,INT8,,
20 /encoder/layers.0/mlp/fc2/MatMul,MatMul,INT8,INT8,
21 /encoder/layers.0/Add_1,Add,INT8,,
22 /encoder/layers.1/attn/Wqkv/MatMul,MatMul,INT8,INT8,
23 /encoder/layers.1/attn/Reshape,Reshape,INT8,,
24 /encoder/layers.1/attn/out_proj/MatMul,MatMul,INT8,INT8,
25 /encoder/layers.1/Add,Add,INT8,,
26 /encoder/layers.1/mlp/fc12/MatMul,MatMul,INT8,INT8,
27 /encoder/layers.1/mlp/fc11/MatMul,MatMul,INT8,INT8,
28 /encoder/layers.1/mlp/Sigmoid,Sigmoid,INT8,,
29 /encoder/layers.1/mlp/Mul,Mul,INT8,,
30 /encoder/layers.1/mlp/Mul_1,Mul,INT8,,
31 /encoder/layers.1/mlp/fc2/MatMul,MatMul,INT8,INT8,
32 /encoder/layers.1/Add_1,Add,INT8,,
33 /encoder/layers.2/attn/Wqkv/MatMul,MatMul,INT8,INT8,
34 /encoder/layers.2/attn/Reshape,Reshape,INT8,,
35 /encoder/layers.2/attn/out_proj/MatMul,MatMul,INT8,INT8,
36 /encoder/layers.2/Add,Add,INT8,,
37 /encoder/layers.2/mlp/fc12/MatMul,MatMul,INT8,INT8,
38 /encoder/layers.2/mlp/fc11/MatMul,MatMul,INT8,INT8,
39 /encoder/layers.2/mlp/Sigmoid,Sigmoid,INT8,,
40 /encoder/layers.2/mlp/Mul,Mul,INT8,,
41 /encoder/layers.2/mlp/Mul_1,Mul,INT8,,
42 /encoder/layers.2/mlp/fc2/MatMul,MatMul,INT8,INT8,
43 /encoder/layers.2/Add_1,Add,INT8,,
44 /encoder/layers.3/attn/Wqkv/MatMul,MatMul,INT8,INT8,
45 /encoder/layers.3/attn/Reshape,Reshape,INT8,,
46 /encoder/layers.3/attn/out_proj/MatMul,MatMul,INT8,INT8,
47 /encoder/layers.3/Add,Add,INT8,,
48 /encoder/layers.3/mlp/fc12/MatMul,MatMul,INT8,INT8,
49 /encoder/layers.3/mlp/fc11/MatMul,MatMul,INT8,INT8,
50 /encoder/layers.3/mlp/Sigmoid,Sigmoid,INT8,,
51 /encoder/layers.3/mlp/Mul,Mul,INT8,,
52 /encoder/layers.3/mlp/Mul_1,Mul,INT8,,
53 /encoder/layers.3/mlp/fc2/MatMul,MatMul,INT8,INT8,
54 /encoder/layers.3/Add_1,Add,INT8,,
55 /encoder/layers.4/attn/Wqkv/MatMul,MatMul,INT8,INT8,
56 /encoder/layers.4/attn/Reshape,Reshape,INT8,,
57 /encoder/layers.4/attn/out_proj/MatMul,MatMul,INT8,INT8,
58 /encoder/layers.4/Add,Add,INT8,,
59 /encoder/layers.4/mlp/fc12/MatMul,MatMul,INT8,INT8,
60 /encoder/layers.4/mlp/fc11/MatMul,MatMul,INT8,INT8,
61 /encoder/layers.4/mlp/Sigmoid,Sigmoid,INT8,,
62 /encoder/layers.4/mlp/Mul,Mul,INT8,,
63 /encoder/layers.4/mlp/Mul_1,Mul,INT8,,
64 /encoder/layers.4/mlp/fc2/MatMul,MatMul,INT8,INT8,
65 /encoder/layers.4/Add_1,Add,INT8,,
66 /encoder/layers.5/attn/Wqkv/MatMul,MatMul,INT8,INT8,
67 /encoder/layers.5/attn/Reshape,Reshape,INT8,,
68 /encoder/layers.5/attn/out_proj/MatMul,MatMul,INT8,INT8,
69 /encoder/layers.5/Add,Add,INT8,,
70 /encoder/layers.5/mlp/fc12/MatMul,MatMul,INT8,INT8,
71 /encoder/layers.5/mlp/fc11/MatMul,MatMul,INT8,INT8,
72 /encoder/layers.5/mlp/Sigmoid,Sigmoid,INT8,,
73 /encoder/layers.5/mlp/Mul,Mul,INT8,,
74 /encoder/layers.5/mlp/Mul_1,Mul,INT8,,
75 /encoder/layers.5/mlp/fc2/MatMul,MatMul,INT8,INT8,
76 /encoder/layers.5/Add_1,Add,INT8,,
77 /encoder/layers.6/attn/Wqkv/MatMul,MatMul,INT8,INT8,
78 /encoder/layers.6/attn/Reshape,Reshape,INT8,,
79 /encoder/layers.6/attn/out_proj/MatMul,MatMul,INT8,INT8,
80 /encoder/layers.6/Add,Add,INT8,,
81 /encoder/layers.6/mlp/fc12/MatMul,MatMul,INT8,INT8,
82 /encoder/layers.6/mlp/fc11/MatMul,MatMul,INT8,INT8,
83 /encoder/layers.6/mlp/Sigmoid,Sigmoid,INT8,,
84 /encoder/layers.6/mlp/Mul,Mul,INT8,,
85 /encoder/layers.6/mlp/Mul_1,Mul,INT8,,
86 /encoder/layers.6/mlp/fc2/MatMul,MatMul,INT8,INT8,
87 /encoder/layers.6/Add_1,Add,INT8,,
88 /encoder/layers.7/attn/Wqkv/MatMul,MatMul,INT8,INT8,
89 /encoder/layers.7/attn/Reshape,Reshape,INT8,,
90 /encoder/layers.7/attn/out_proj/MatMul,MatMul,INT8,INT8,
91 /encoder/layers.7/Add,Add,INT8,,
92 /encoder/layers.7/mlp/fc12/MatMul,MatMul,INT8,INT8,
93 /encoder/layers.7/mlp/fc11/MatMul,MatMul,INT8,INT8,
94 /encoder/layers.7/mlp/Sigmoid,Sigmoid,INT8,,
95 /encoder/layers.7/mlp/Mul,Mul,INT8,,
96 /encoder/layers.7/mlp/Mul_1,Mul,INT8,,
97 /encoder/layers.7/mlp/fc2/MatMul,MatMul,INT8,INT8,
98 /encoder/layers.7/Add_1,Add,INT8,,
99 /encoder/layers.8/attn/Wqkv/MatMul,MatMul,INT8,INT8,
100 /encoder/layers.8/attn/Reshape,Reshape,INT8,,
101 /encoder/layers.8/attn/out_proj/MatMul,MatMul,INT8,INT8,
102 /encoder/layers.8/Add,Add,INT8,,
103 /encoder/layers.8/mlp/fc12/MatMul,MatMul,INT8,INT8,
104 /encoder/layers.8/mlp/fc11/MatMul,MatMul,INT8,INT8,
105 /encoder/layers.8/mlp/Sigmoid,Sigmoid,INT8,,
106 /encoder/layers.8/mlp/Mul,Mul,INT8,,
107 /encoder/layers.8/mlp/Mul_1,Mul,INT8,,
108 /encoder/layers.8/mlp/fc2/MatMul,MatMul,INT8,INT8,
109 /encoder/layers.8/Add_1,Add,INT8,,
110 /encoder/layers.9/attn/Wqkv/MatMul,MatMul,INT8,INT8,
111 /encoder/layers.9/attn/Reshape,Reshape,INT8,,
112 /encoder/layers.9/attn/out_proj/MatMul,MatMul,INT8,INT8,
113 /encoder/layers.9/Add,Add,INT8,,
114 /encoder/layers.9/mlp/fc12/MatMul,MatMul,INT8,INT8,
115 /encoder/layers.9/mlp/fc11/MatMul,MatMul,INT8,INT8,
116 /encoder/layers.9/mlp/Sigmoid,Sigmoid,INT8,,
117 /encoder/layers.9/mlp/Mul,Mul,INT8,,
118 /encoder/layers.9/mlp/Mul_1,Mul,INT8,,
119 /encoder/layers.9/mlp/fc2/MatMul,MatMul,INT8,INT8,
120 /encoder/layers.9/Add_1,Add,INT8,,
121 /encoder/layers.10/attn/Wqkv/MatMul,MatMul,INT8,INT8,
122 /encoder/layers.10/attn/Reshape,Reshape,INT8,,
123 /encoder/layers.10/attn/out_proj/MatMul,MatMul,INT8,INT8,
124 /encoder/layers.10/Add,Add,INT8,,
125 /encoder/layers.10/mlp/fc12/MatMul,MatMul,INT8,INT8,
126 /encoder/layers.10/mlp/fc11/MatMul,MatMul,INT8,INT8,
127 /encoder/layers.10/mlp/Sigmoid,Sigmoid,INT8,,
128 /encoder/layers.10/mlp/Mul,Mul,INT8,,
129 /encoder/layers.10/mlp/Mul_1,Mul,INT8,,
130 /encoder/layers.10/mlp/fc2/MatMul,MatMul,INT8,INT8,
131 /encoder/layers.10/Add_1,Add,INT8,,
132 /encoder/layers.11/attn/Wqkv/MatMul,MatMul,INT8,INT8,
133 /encoder/layers.11/attn/Reshape,Reshape,INT8,,
134 /encoder/layers.11/attn/out_proj/MatMul,MatMul,INT8,INT8,
135 /encoder/layers.11/Add,Add,INT8,,
136 /encoder/layers.11/mlp/fc12/MatMul,MatMul,INT8,INT8,
137 /encoder/layers.11/mlp/fc11/MatMul,MatMul,INT8,INT8,
138 /encoder/layers.11/mlp/Sigmoid,Sigmoid,INT8,,
139 /encoder/layers.11/mlp/Mul,Mul,INT8,,
140 /encoder/layers.11/mlp/Mul_1,Mul,INT8,,
141 /encoder/layers.11/mlp/fc2/MatMul,MatMul,INT8,INT8,
142 /encoder/layers.11/Add_1,Add,INT8,,
143 Op Type,Activation,Weights,Bias
144 MatMul,INT8(60),INT8(60),
145 Reshape,INT8(12),,
146 Add,INT8(24),,
147 Sigmoid,INT8(12),,
148 Mul,INT8(24),,

97
test_npu_quantized.py Normal file
View file

@ -0,0 +1,97 @@
#!/usr/bin/env python3
"""
Test VitisAI EP with the quantized nomic-embed-text-v1.5 model.
Checks whether the NPU VAIML pass achieves meaningful GOPs coverage.
"""
import os
import json
import time
import numpy as np
from pathlib import Path
MODEL_PATH = Path.home() / ".cache/breadsearch/models/model_quantized_static.onnx"
CACHE_DIR = Path.home() / ".cache/breadsearch/npu/nomic-quantized-static"
VAIP_CONFIG = Path.home() / ".config/breadsearch/vaip_config.json"
RYZEN_AI_LIB = Path.home() / ".local/share/ryzen-ai-1.7.1/lib"
# libvaiml.so must be discoverable
os.environ["RYZEN_AI_INSTALLATION_PATH"] = str(RYZEN_AI_LIB)
os.environ["LD_LIBRARY_PATH"] = str(RYZEN_AI_LIB) + ":" + os.environ.get("LD_LIBRARY_PATH", "")
CACHE_DIR.mkdir(parents=True, exist_ok=True)
if not VAIP_CONFIG.exists():
print(f"ERROR: vaip_config.json not found at {VAIP_CONFIG}")
print("Check breadmill embed.rs for the find_vaip_config() paths")
exit(1)
print(f"Testing quantized model: {MODEL_PATH}")
print(f"Cache dir: {CACHE_DIR}")
print(f"VAIP config: {VAIP_CONFIG}")
print(f"RYZEN_AI_INSTALLATION_PATH: {os.environ['RYZEN_AI_INSTALLATION_PATH']}")
import onnxruntime as ort
providers = [
("VitisAIExecutionProvider", {
"config_file": str(VAIP_CONFIG),
"cacheDir": str(CACHE_DIR),
"cacheKey": "nomic-quantized-static",
}),
"CPUExecutionProvider",
]
print("\nCreating InferenceSession with VitisAI EP...")
t0 = time.time()
try:
sess = ort.InferenceSession(str(MODEL_PATH), providers=providers)
t1 = time.time()
print(f"Session created in {t1-t0:.1f}s")
print(f"Active providers: {sess.get_providers()}")
except Exception as e:
print(f"ERROR creating session: {e}")
exit(1)
# Check for the VAIML pass summary
summary_path = CACHE_DIR / "nomic-quantized" / "preliminary-vaiml-pass-summary.txt"
if not summary_path.exists():
# Try variations
for p in CACHE_DIR.rglob("preliminary-vaiml-pass-summary.txt"):
summary_path = p
break
if summary_path.exists():
print(f"\n--- VAIML Pass Summary ---")
print(summary_path.read_text())
else:
print(f"\nNo VAIML summary found at {summary_path}")
print("Files in cache dir:")
for f in CACHE_DIR.rglob("*"):
if f.is_file():
print(f" {f}")
# Run a quick inference test
print("\nRunning inference test...")
seq_len = 128
dummy_ids = np.ones((1, seq_len), dtype=np.int64)
dummy_mask = np.ones((1, seq_len), dtype=np.int64)
dummy_types = np.zeros((1, seq_len), dtype=np.int64)
input_names = [inp.name for inp in sess.get_inputs()]
print(f"Input names: {input_names}")
feed = {}
for name in input_names:
if "type" in name:
feed[name] = dummy_types
elif "mask" in name:
feed[name] = dummy_mask
else:
feed[name] = dummy_ids
t0 = time.time()
outputs = sess.run(None, feed)
t1 = time.time()
print(f"Inference completed in {(t1-t0)*1000:.1f}ms")
print(f"Output shape: {outputs[0].shape}")