Indago

A full-text search engine, built from scratch in Rust — inverted index, BM25 ranking, a boolean query language, segmented on-disk storage, a CLI, a live web dashboard, and benchmarks. No existing search library is used; the engine, the binary segment format, and the query parser are all original.

$ indago generate --count 10000        # synthetic corpus
$ indago index corpus/docs             # ~10k docs in <1s
$ indago query 'quantum entanglement'  # ranked results with snippets
$ indago serve                         # live dashboard at :8080

Highlights

• Inverted index with positional postings (title + body fields), built for ~44k docs/sec and single-term queries in ~10 ms at 100k documents.
• BM25 ranking with tunable k1 / b and field weighting (title matches weighted 3× body matches).
• Query language: implicit-OR terms, AND / OR / NOT, exact phrases, title: / body: fielded search, term~ fuzzy (edit distance), +term / -term required/excluded, term^2 boost, parentheses — plus automatic relaxation to OR when a conjunction matches nothing.
• Segmented storage: immutable segments in a custom little-endian binary format (INDGSEG1), delta-encoded doc ids, zero-padded segment files, and an online merge.
• Web dashboard served by an axum backend: live search with <mark> highlighting, prefix suggestions, top-term and latency charts drawn on <canvas> (no frontend framework, no chart library), and a live query log streamed over WebSocket.
• Testing: 47 unit/integration tests plus property-based tests (proptest) asserting tokenizer and index invariants; clippy -D warnings clean; rustfmt enforced.
• Benchmarks with Criterion (cargo bench).

Layout

crates/indago-core   the engine: tokenizer, index, scoring, query, search, segment, storage
crates/indago-cli    the `indago` binary: generate / index / query / stats / ingest / merge / serve
crates/indago-server axum HTTP + WebSocket API, serves web/
web/                 dashboard (plain HTML/CSS/JS + canvas charts)
docs/design.md       architecture and design decisions
docs/query-language.md  full query syntax reference

Getting started

Indago builds on any machine with a stable Rust toolchain (Linux/macOS/ Windows; the server uses no system C libraries — pure Rust TLS). The corpus and index are generated locally, so a fresh clone is all you need.

1. Install Rust (one-time, if you don't have it)

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"     # or restart your shell
rustc --version               # check it works

On Windows, install via https://rustup.rs. Everything else in this project is built by Cargo — there is no system package to install.

2. Build, generate data, search

# from the repository root
make build                     # cargo build --release (first build ~1 min)
make corpus                    # generate the seeded 10k-document corpus
make index                     # index corpus/docs into index/
make query q='quantum entanglement'   # first results appear in <1 ms

Without make, the equivalent commands are:

cargo build --release
cargo run --release -p indago-cli -- generate --count 10000
cargo run --release -p indago-cli -- index corpus/docs
cargo run --release -p indago-cli -- query 'quantum entanglement'

3. Explore

indago query                      # interactive REPL (try /help)
indago query '"dark energy" -physics'
make serve                        # web dashboard → http://127.0.0.1:8080

make serve runs the server binary from the workspace root, which is required so the dashboard files under web/ resolve. To run it manually:

cargo run --release -p indago-cli -- serve --port 8080

4. Verify it works (optional)

See Development below for the test, lint and benchmark commands.

Other useful invocations:

indago stats                 # index statistics
indago query 'quntm~'        # fuzzy search
indago query 'title:rust'    # fielded search
indago ingest https://example.com/             # index a web page
indago ingest https://blog.example.com/ --pages 50 --delay 300   # crawl
indago merge                 # merge all segments into one

Development

make test       # cargo test --workspace
make bench      # cargo bench
make check      # cargo fmt --check && cargo clippy -- -D warnings

The corpus generator is seeded and deterministic: indago generate --seed 42 always produces the same 10,000 documents, so benchmarks and demos are reproducible.

Design notes

The interesting decisions — why segments are immutable, how positions are field-tagged, why BM25 over tf-idf, the OR-fallback heuristic, and the binary format layout — are explained in docs/design.md.

License

MIT. See LICENSE.