Skip to content

experimental: TestSemanticKnnBudget ceiling test fails intermittently on Postgres #1401

Open

Description

@markramm

experimental intermittently fails on one test, on heads that don't touch the backend:

tests/backends/test_backend_conformance.py::TestSemanticKnnBudget::test_rows_the_ceiling_hides_are_returned_or_the_result_says_capped[postgres]

Seen failing:

It passed on #1381's earlier head 9785010b, and #1381's diff doesn't reach the backend: the test calls it directly and never touches KBService.

experimental isn't required, so nothing is blocked, but contributors see a red check that isn't theirs. It's not on the known-failures list.

Done means one of:

  • the cause is found and fixed, for example a row-ordering or budget race on the Postgres leg, or
  • the test is recorded as a known failure, with the cause it was recorded under.

The cause is unknown; nobody has investigated yet.

🤖 Generated with Claude Code

Activity

  1. markramm commented on Nov 10, 2026

    @markramm
    CollaboratorAuthor

    Groom 2026-11-09

    Automated groom (an AI agent working for the maintainer). Read on dev at 8ca2b068. I read CI logs; I ran nothing against Postgres. Written so someone new to Pyrite can start from this issue alone.

    Short version. The test is probably not flaky. It probably catches a real, intermittent bug in the Postgres backend: a filtered semantic search can come back empty, without saying so, when a matching row exists. Whether it fails depends on which query plan Postgres chooses on a given run. The first step is to confirm that with EXPLAIN (below).

    Principle at stake: 9 (a small supported surface; the Postgres backend is experimental, which is why the job is named experimental and is not required). The property the test pins is #668's: an empty semantic result must not read as "nothing matches" when rows were never considered. Either the row comes back, or the result is marked capped (SemanticRows).

    What a user expects. "A semantic search filtered to entry_type=rare returns the rare entry if it is within the similarity floor."
    What the code claims. The docstring of PostgresBackend.search_semantic (pyrite/storage/backends/postgres_backend.py) says every predicate goes into one WHERE with the distance ordering, "so LIMIT applies after filtering and cannot under-return". That holds for an exact plan. With an approximate (HNSW) index it does not hold (see below).

    Known (from CI logs)

    • The same failure, every time:
      tests/backends/test_backend_conformance.py:1467
      assert [r["id"] for r in rows] == ["far"] or getattr(rows, "capped", False)
      AssertionError: assert ([] == ['far'] ... or False)
      
      The result is empty and not marked capped.
    • Failing runs (job experimental, image pgvector/pgvector:pg16, Python package pgvector==0.5.1):
    • Rate: in the last 60 CI runs (2026-10-31 to 2026-11-10), the experimental job completed 47 times. It failed 4 times, and 3 of those were this test. The fourth was an unrelated MCP session failure. So about 3 in 47, roughly 6%.
    • The heads that failed do not touch the backend. The test calls the backend directly.

    What the code does

    • ensure_schema creates idx_entry_embedding ON entry USING hnsw(embedding vector_cosine_ops) (postgres_backend.py, near the FTS index).
    • search_semantic builds a single SELECT ... WHERE <filters> AND distance <= floor ORDER BY embedding <=> :vec LIMIT :limit.
    • The test fixture (big_embedded_backend in TestSemanticKnnBudget, tests/backends/test_backend_conformance.py) inserts 4,097 near-identical note vectors one row at a time. The test then adds one rare row that is farther from the query than all of them, and searches with entry_type="rare", limit=1. Every test gets freshly created tables (_teardown_postgres in tests/backends/conftest.py drops them), so table statistics depend on whether autovacuum's analyze has run by the time of the query.

    Hypothesis (assumed, not verified)

    Postgres can run that query in two ways:

    1. An exact plan. Scan the rows that match the filter, sort them by distance, take 1. This returns far.
    2. An HNSW index scan. Walk the index in distance order, and apply the WHERE to the candidates it yields. pgvector's documentation says that with approximate indexes, filtering happens after the index scan. With the default hnsw.ef_search = 40, only about the 40 nearest candidates are considered. far is the farthest row, so it is never among them, and the result is []. Nothing marks it capped.

    Which plan the planner picks depends on its cost estimates, and those depend on whether the fresh table has been analyzed yet. That would explain why it is intermittent and unrelated to the PR under test. pgvector 0.8 added hnsw.iterative_scan, which keeps scanning until enough rows pass the filter. Which pgvector version the pg16 image ships on a given day is not pinned.

    Options

    • A (recommended). Make the filtered query exact, or say when it is not. Inside the query's transaction, SET LOCAL hnsw.iterative_scan = strict_order (if the server's pgvector is 0.8 or later), with a hnsw.max_scan_tuples bound. When the bound is reached and fewer than limit rows came back, mark the result capped, as the SQLite backend does. On an older pgvector, either force the exact plan for filtered queries, or mark every filtered result that is shorter than limit as capped.
    • B. Force the exact plan whenever a filter is present (for example SET LOCAL enable_indexscan = off for that one statement). This is simple and correct, but each filtered search becomes a full scan. That is acceptable at experimental scale; say so in the docstring.
    • C. Record it as a known failure (scripts/known-cloud-failures.json) under this cause. That is only right if the maintainer accepts the under-return as a limit of the experimental backend. It hides a real wrong answer, so I recommend against it.
    Acceptance:   1. Confirm the cause first. Against a pgvector container, run the test's query under EXPLAIN
                     (ANALYZE) twice: once right after the fixture, and once after an explicit `ANALYZE entry`
                     (or with `SET enable_seqscan = off`). Post both plans on this issue. If the HNSW plan
                     returns [] and the exact plan returns `far`, the hypothesis holds.
                  2. A deterministic test: the same fixture, with the index plan forced (for example
                     `SET enable_seqscan = off` in the test). It fails on dev every time and passes after the
                     fix. The existing test then stops being intermittent.
                  3. The docstring of search_semantic no longer claims the query "cannot under-return"
                     unless that is true.
    Regimes:      pgvector < 0.8 vs >= 0.8 (detect it with `SELECT extversion FROM pg_extension WHERE
                  extname = 'vector'`); no filter (the control test
                  test_a_result_the_floor_cut_short_is_not_capped must stay not-capped); a tags filter
                  (subquery) as well as an entry_type filter
    Shapes:       the filters search_semantic applies (kb_name, kb_names, entry_type, entry_types,
                  exclude_types, dates, tags, fips, state, status, non_empty); parametrise the forced-plan
                  test over at least entry_type and tags
    Touches:      existing: pyrite/storage/backends/postgres_backend.py (search_semantic),
                  tests/backends/test_backend_conformance.py   new: changelog. About 40-80 lines.
    Sequence:     independent. Open PR #1021 edits postgres_backend.py elsewhere (the alembic chain).
    Model:        needs Postgres and pgvector knowledge
    heavy:        no, but it needs a local Postgres with pgvector
                  (`docker run -e POSTGRES_PASSWORD=postgres -p 5432:5432 pgvector/pgvector:pg16`, then
                  PYRITE_TEST_PG_URL; see tests/backends/conftest.py)
    Cold read:    yes (storage behaviour, and it changes what a search returns)
    Out of scope: the SQLite leg (it already caps and marks); changing the HNSW index parameters
                  globally; pinning the CI image (worth a separate note if versions turn out to matter)
    Decision:     A or B. Recommendation: A, falling back to B on an older pgvector.
    

    Spike first, briefly. Step 1 of the acceptance is the spike: an hour with a container and EXPLAIN. The fix is small once the plan is known.

    Checked versus assumed. Checked: the four failing runs and their assertion text; the rate across 60 runs; the index DDL; the query shape; the fixture's teardown. Assumed: the two-plan mechanism, which comes from pgvector's documented post-filtering behaviour and has not been reproduced here; and the pgvector version in the image. This groom is wrong if EXPLAIN shows the same plan on a failing run and a passing one. In that case, look at the HNSW graph build next: with 4,097 near-identical vectors, recall can vary between builds.

  2. added
    help wantedExtra attention is needed
    bugSomething isn't working
    and removed
    bugSomething isn't working
    on Nov 10, 2026
  3. markramm commented on Nov 17, 2026

    @markramm
    CollaboratorAuthor

    Sighting, 2026-11-17: the non-blocking experimental job failed on #1570 at 0580fac with only tests/backends/test_backend_conformance.py::TestSemanticKnnBudget::test_rows_the_ceiling_hides_are_returned_or_the_result_says_capped[postgres] as a new failure (run 44033508908, attempt 1). That push changed a changelog fragment and a test docstring, and the same job passed on the previous head d508760, so this is the intermittent failure described here, not the change. Re-run once.

    Posted by the maintainer's conductor agent (Claude Code).

  4. markramm commented on Nov 17, 2026

    @markramm
    CollaboratorAuthor

    Automated: the maintainer's conductor agent (2026-11-17). Another sighting: the experimental job on the dev push at e0434281 (run 44125437779) failed on this test with assert ([] == ['far'] .... The same job passed on the pushes before and after (061388c7, 38d98071), and that commit does not touch the backend. The ratchet filed it as #1641, closed as a duplicate of this issue; its groom comment has the run table and repeats step 1 here (the two EXPLAIN (ANALYZE) plans) as the next step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions