You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
experimental: TestSemanticKnnBudget ceiling test fails intermittently on Postgres #1401
Automated groom (an AI agent working for the maintainer). Read on dev at 8ca2b068. I read CI logs; I ran nothing against Postgres. Written so someone new to Pyrite can start from this issue alone.
Short version. The test is probably not flaky. It probably catches a real, intermittent bug in the Postgres backend: a filtered semantic search can come back empty, without saying so, when a matching row exists. Whether it fails depends on which query plan Postgres chooses on a given run. The first step is to confirm that with EXPLAIN (below).
Principle at stake: 9 (a small supported surface; the Postgres backend is experimental, which is why the job is named experimental and is not required). The property the test pins is #668's: an empty semantic result must not read as "nothing matches" when rows were never considered. Either the row comes back, or the result is marked capped (SemanticRows).
What a user expects. "A semantic search filtered to entry_type=rare returns the rare entry if it is within the similarity floor." What the code claims. The docstring of PostgresBackend.search_semantic (pyrite/storage/backends/postgres_backend.py) says every predicate goes into one WHERE with the distance ordering, "so LIMIT applies after filtering and cannot under-return". That holds for an exact plan. With an approximate (HNSW) index it does not hold (see below).
Known (from CI logs)
The same failure, every time:
tests/backends/test_backend_conformance.py:1467
assert [r["id"] for r in rows] == ["far"] or getattr(rows, "capped", False)
AssertionError: assert ([] == ['far'] ... or False)
Rate: in the last 60 CI runs (2026-10-31 to 2026-11-10), the experimental job completed 47 times. It failed 4 times, and 3 of those were this test. The fourth was an unrelated MCP session failure. So about 3 in 47, roughly 6%.
The heads that failed do not touch the backend. The test calls the backend directly.
What the code does
ensure_schema creates idx_entry_embedding ON entry USING hnsw(embedding vector_cosine_ops) (postgres_backend.py, near the FTS index).
search_semantic builds a single SELECT ... WHERE <filters> AND distance <= floor ORDER BY embedding <=> :vec LIMIT :limit.
The test fixture (big_embedded_backend in TestSemanticKnnBudget, tests/backends/test_backend_conformance.py) inserts 4,097 near-identical note vectors one row at a time. The test then adds one rare row that is farther from the query than all of them, and searches with entry_type="rare", limit=1. Every test gets freshly created tables (_teardown_postgres in tests/backends/conftest.py drops them), so table statistics depend on whether autovacuum's analyze has run by the time of the query.
Hypothesis (assumed, not verified)
Postgres can run that query in two ways:
An exact plan. Scan the rows that match the filter, sort them by distance, take 1. This returns far.
An HNSW index scan. Walk the index in distance order, and apply the WHERE to the candidates it yields. pgvector's documentation says that with approximate indexes, filtering happens after the index scan. With the default hnsw.ef_search = 40, only about the 40 nearest candidates are considered. far is the farthest row, so it is never among them, and the result is []. Nothing marks it capped.
Which plan the planner picks depends on its cost estimates, and those depend on whether the fresh table has been analyzed yet. That would explain why it is intermittent and unrelated to the PR under test. pgvector 0.8 added hnsw.iterative_scan, which keeps scanning until enough rows pass the filter. Which pgvector version the pg16 image ships on a given day is not pinned.
Options
A (recommended). Make the filtered query exact, or say when it is not. Inside the query's transaction, SET LOCAL hnsw.iterative_scan = strict_order (if the server's pgvector is 0.8 or later), with a hnsw.max_scan_tuples bound. When the bound is reached and fewer than limit rows came back, mark the result capped, as the SQLite backend does. On an older pgvector, either force the exact plan for filtered queries, or mark every filtered result that is shorter than limit as capped.
B. Force the exact plan whenever a filter is present (for example SET LOCAL enable_indexscan = off for that one statement). This is simple and correct, but each filtered search becomes a full scan. That is acceptable at experimental scale; say so in the docstring.
C. Record it as a known failure (scripts/known-cloud-failures.json) under this cause. That is only right if the maintainer accepts the under-return as a limit of the experimental backend. It hides a real wrong answer, so I recommend against it.
Acceptance: 1. Confirm the cause first. Against a pgvector container, run the test's query under EXPLAIN
(ANALYZE) twice: once right after the fixture, and once after an explicit `ANALYZE entry`
(or with `SET enable_seqscan = off`). Post both plans on this issue. If the HNSW plan
returns [] and the exact plan returns `far`, the hypothesis holds.
2. A deterministic test: the same fixture, with the index plan forced (for example
`SET enable_seqscan = off` in the test). It fails on dev every time and passes after the
fix. The existing test then stops being intermittent.
3. The docstring of search_semantic no longer claims the query "cannot under-return"
unless that is true.
Regimes: pgvector < 0.8 vs >= 0.8 (detect it with `SELECT extversion FROM pg_extension WHERE
extname = 'vector'`); no filter (the control test
test_a_result_the_floor_cut_short_is_not_capped must stay not-capped); a tags filter
(subquery) as well as an entry_type filter
Shapes: the filters search_semantic applies (kb_name, kb_names, entry_type, entry_types,
exclude_types, dates, tags, fips, state, status, non_empty); parametrise the forced-plan
test over at least entry_type and tags
Touches: existing: pyrite/storage/backends/postgres_backend.py (search_semantic),
tests/backends/test_backend_conformance.py new: changelog. About 40-80 lines.
Sequence: independent. Open PR #1021 edits postgres_backend.py elsewhere (the alembic chain).
Model: needs Postgres and pgvector knowledge
heavy: no, but it needs a local Postgres with pgvector
(`docker run -e POSTGRES_PASSWORD=postgres -p 5432:5432 pgvector/pgvector:pg16`, then
PYRITE_TEST_PG_URL; see tests/backends/conftest.py)
Cold read: yes (storage behaviour, and it changes what a search returns)
Out of scope: the SQLite leg (it already caps and marks); changing the HNSW index parameters
globally; pinning the CI image (worth a separate note if versions turn out to matter)
Decision: A or B. Recommendation: A, falling back to B on an older pgvector.
Spike first, briefly. Step 1 of the acceptance is the spike: an hour with a container and EXPLAIN. The fix is small once the plan is known.
Checked versus assumed. Checked: the four failing runs and their assertion text; the rate across 60 runs; the index DDL; the query shape; the fixture's teardown. Assumed: the two-plan mechanism, which comes from pgvector's documented post-filtering behaviour and has not been reproduced here; and the pgvector version in the image. This groom is wrong ifEXPLAIN shows the same plan on a failing run and a passing one. In that case, look at the HNSW graph build next: with 4,097 near-identical vectors, recall can vary between builds.
Sighting, 2026-11-17: the non-blocking experimental job failed on #1570 at 0580fac with only tests/backends/test_backend_conformance.py::TestSemanticKnnBudget::test_rows_the_ceiling_hides_are_returned_or_the_result_says_capped[postgres] as a new failure (run 44033508908, attempt 1). That push changed a changelog fragment and a test docstring, and the same job passed on the previous head d508760, so this is the intermittent failure described here, not the change. Re-run once.
Posted by the maintainer's conductor agent (Claude Code).
Automated: the maintainer's conductor agent (2026-11-17). Another sighting: the experimental job on the dev push at e0434281 (run 44125437779) failed on this test with assert ([] == ['far'] .... The same job passed on the pushes before and after (061388c7, 38d98071), and that commit does not touch the backend. The ratchet filed it as #1641, closed as a duplicate of this issue; its groom comment has the run table and repeats step 1 here (the two EXPLAIN (ANALYZE) plans) as the next step.
Activity
markramm commentedon Nov 10, 2026
Groom 2026-11-09
Automated groom (an AI agent working for the maintainer). Read on
devat8ca2b068. I read CI logs; I ran nothing against Postgres. Written so someone new to Pyrite can start from this issue alone.Short version. The test is probably not flaky. It probably catches a real, intermittent bug in the Postgres backend: a filtered semantic search can come back empty, without saying so, when a matching row exists. Whether it fails depends on which query plan Postgres chooses on a given run. The first step is to confirm that with
EXPLAIN(below).Principle at stake: 9 (a small supported surface; the Postgres backend is experimental, which is why the job is named
experimentaland is not required). The property the test pins is #668's: an empty semantic result must not read as "nothing matches" when rows were never considered. Either the row comes back, or the result is markedcapped(SemanticRows).What a user expects. "A semantic search filtered to
entry_type=rarereturns therareentry if it is within the similarity floor."What the code claims. The docstring of
PostgresBackend.search_semantic(pyrite/storage/backends/postgres_backend.py) says every predicate goes into oneWHEREwith the distance ordering, "soLIMITapplies after filtering and cannot under-return". That holds for an exact plan. With an approximate (HNSW) index it does not hold (see below).Known (from CI logs)
experimental, imagepgvector/pgvector:pg16, Python packagepgvector==0.5.1):dfb493b7, 2026-10-27: run 40477500618acba9c39, 2026-11-01: run 4131981473506158e48, 2026-11-02: run 41516924703a79beab1, 2026-11-03: run 41649062678experimentaljob completed 47 times. It failed 4 times, and 3 of those were this test. The fourth was an unrelated MCP session failure. So about 3 in 47, roughly 6%.What the code does
ensure_schemacreatesidx_entry_embedding ON entry USING hnsw(embedding vector_cosine_ops)(postgres_backend.py, near the FTS index).search_semanticbuilds a singleSELECT ... WHERE <filters> AND distance <= floor ORDER BY embedding <=> :vec LIMIT :limit.big_embedded_backendinTestSemanticKnnBudget,tests/backends/test_backend_conformance.py) inserts 4,097 near-identicalnotevectors one row at a time. The test then adds onerarerow that is farther from the query than all of them, and searches withentry_type="rare", limit=1. Every test gets freshly created tables (_teardown_postgresintests/backends/conftest.pydrops them), so table statistics depend on whether autovacuum's analyze has run by the time of the query.Hypothesis (assumed, not verified)
Postgres can run that query in two ways:
far.WHEREto the candidates it yields. pgvector's documentation says that with approximate indexes, filtering happens after the index scan. With the defaulthnsw.ef_search = 40, only about the 40 nearest candidates are considered.faris the farthest row, so it is never among them, and the result is[]. Nothing marks it capped.Which plan the planner picks depends on its cost estimates, and those depend on whether the fresh table has been analyzed yet. That would explain why it is intermittent and unrelated to the PR under test. pgvector 0.8 added
hnsw.iterative_scan, which keeps scanning until enough rows pass the filter. Which pgvector version thepg16image ships on a given day is not pinned.Options
SET LOCAL hnsw.iterative_scan = strict_order(if the server's pgvector is 0.8 or later), with ahnsw.max_scan_tuplesbound. When the bound is reached and fewer thanlimitrows came back, mark the resultcapped, as the SQLite backend does. On an older pgvector, either force the exact plan for filtered queries, or mark every filtered result that is shorter thanlimitas capped.SET LOCAL enable_indexscan = offfor that one statement). This is simple and correct, but each filtered search becomes a full scan. That is acceptable at experimental scale; say so in the docstring.scripts/known-cloud-failures.json) under this cause. That is only right if the maintainer accepts the under-return as a limit of the experimental backend. It hides a real wrong answer, so I recommend against it.Spike first, briefly. Step 1 of the acceptance is the spike: an hour with a container and
EXPLAIN. The fix is small once the plan is known.Checked versus assumed. Checked: the four failing runs and their assertion text; the rate across 60 runs; the index DDL; the query shape; the fixture's teardown. Assumed: the two-plan mechanism, which comes from pgvector's documented post-filtering behaviour and has not been reproduced here; and the pgvector version in the image. This groom is wrong if
EXPLAINshows the same plan on a failing run and a passing one. In that case, look at the HNSW graph build next: with 4,097 near-identical vectors, recall can vary between builds.markramm commentedon Nov 17, 2026
Sighting, 2026-11-17: the non-blocking
experimentaljob failed on #1570 at 0580fac with onlytests/backends/test_backend_conformance.py::TestSemanticKnnBudget::test_rows_the_ceiling_hides_are_returned_or_the_result_says_capped[postgres]as a new failure (run 44033508908, attempt 1). That push changed a changelog fragment and a test docstring, and the same job passed on the previous head d508760, so this is the intermittent failure described here, not the change. Re-run once.Posted by the maintainer's conductor agent (Claude Code).
markramm commentedon Nov 17, 2026
Automated: the maintainer's conductor agent (2026-11-17). Another sighting: the
experimentaljob on thedevpush ate0434281(run 44125437779) failed on this test withassert ([] == ['far'] .... The same job passed on the pushes before and after (061388c7,38d98071), and that commit does not touch the backend. The ratchet filed it as #1641, closed as a duplicate of this issue; its groom comment has the run table and repeats step 1 here (the twoEXPLAIN (ANALYZE)plans) as the next step.