filter

[shaleenji]

No description provided.

[shaleenji]

1. Numeric bucket: fix the duplicate-heavy cliff bug.

When >65,536 ids shared the same numeric filter value, Bucket::serialize
truncated the on-disk count to a uint16_t and corrupted the bucket --
recall collapsed to zero past that boundary (reproducible with
tests/repo_filter.py).

The fix has four parts in src/filter/numeric_index.hpp:

• Bucket::add now caps deltas/ids at MAX_SIZE for delta_32==0
  duplicates and routes the excess id into summary_bitmap only.
  Cardinality is preserved, on-disk arrays no longer grow without
  bound.

• The on-disk count field is removed from Bucket::serialize entirely.
  Bucket::deserialize derives nr_array_entries from the residual
  bytes after the bitmap, so there is no count to overflow.

• Bucket::deserialize stays backward compatible with the old on-disk
  format via a modulus check on the residual: existing DBs written
  by the old code keep working with no migration.

• range() gains two correctness branches: a legacy-salvage branch
  for cliff-corrupted buckets (ids.empty() but bitmap non-empty),
  and a bitmap-only-inclusion branch in the partial-overlap path
  for buckets with cardinality > ids.size().

• range() also gains a coarse full-coverage fast path: when a
  bucket's [base, base+MAX_DELTA] extent is wholly inside the query
  range, skip the deltas/ids deserialize and union just the bitmap
  via Bucket::read_summary_bitmap.

---

2. Build: pin -falign-functions=64 in the release flags (CMakeLists.txt).

Editing any header transitively included by ndd.hpp (filter.hpp,
numeric_index.hpp, vector_storage.hpp) was producing 10-30% QPS
swings on the int-filter HTTP bench with no algorithmic change,
because the HNSW search loop is sensitive to function placement
relative to cache lines. Microbenches of range() and bitmap.contains()
were byte-identical between affected builds; the cost lived in
i-cache effects on the surrounding HNSW inner loop. Forcing 64-byte
function alignment removes the variance so future header touches
don't masquerade as perf regressions.

[shaleenji]

Adjacent issue not addressed here: the slide-split LEFT-bucket rebuild
in add_to_buckets() rebuilds summary_bitmap from ids only, which
silently drops bitmap-only entries. Saturation handling is therefore
not durable across splits; tracked separately.

[shaleenji]

THIS IS A Breaking change. Need to reindex

[shaleenji]

1. Now doing metadata updates when filters are updated
2. Stale entries are removed when updating filters

[shaleenji]

• add explicit Roaring bitmap payload validation for category filter reads
• replace unsafe filter bitmap reads with bounded readSafe + exact byte-count checks
• validate deserialized Roaring internals before using stored bitmaps
• apply the same hardening to numeric bucket bitmap payloads, including the range fast path
• return/propagate corruption as OperationResult code 200 instead of trusting malformed payloads
• add regression coverage for valid, truncated, trailing-byte, and garbage bitmap payloads

[shaleenji]

solves #237 , #238 , #239 , #240 , #241 , #242

partially #244 and #25

[shaleenji]

Requires reindexing

[shaleenji]

Server:
8 CPUs
32GB Memory
100GB NVME SSD

Label Filter

Int Filter