Skip to content

experimental-broken: tests/backends/test_backend_conformance.py #1641

Closed as not planned

Description

@github-actions

Experimental tests in tests/backends/test_backend_conformance.py newly failing on dev (not in tests/experimental_known_failures.txt):

  • tests/backends/test_backend_conformance.py::TestSemanticKnnBudget::test_rows_the_ceiling_hides_are_returned_or_the_result_says_capped[postgres]

    AssertionError: assert ([] == ['far']

Commit: e043428
Run: https://github.com/pyrite-wiki/pyrite/actions/runs/44125437779

Experimental tests do not block a merge, but are never silent (#657). Filed by scripts/experimental_ratchet.py.

Activity

  1. markramm commented on Nov 17, 2026

    @markramm
    Collaborator

    Automated: written by the maintainer's conductor agent (groom lane); checked against dev f1ea727.

    Groom 2026-11-17

    Verdict: not a regression; the intermittent failure already tracked in #1401. This issue is a duplicate of it (the same test, the same assertion, filed again by the ratchet). I read CI logs only and ran nothing.

    Evidence (pass, fail, pass). The experimental job in run 44125437779 (dev push e043428, #1613, a board.yaml gate change) failed on this one test, AssertionError: assert ([] == ['far'] ... or False), with 1 failed, 2241 passed, 14 skipped. The same test, [postgres], was executed and passed in the experimental job of the dev pushes before and after:

    dev push result of this test
    a5f942b (run 44113549682) passed
    d4fcea8 (run 44117748070) passed
    061388c (run 44122529977) passed
    e043428 (run 44125437779) failed
    38d9807 (run 44125933819) passed
    f1ea727 (run 44127867798) passed

    Each of those logs shows the test running (not skipped) with the same Docker image digest for pgvector/pgvector:pg16 (sha256:484e8380...) and the same Python package pgvector==0.5.1. So the environment did not change between the runs. The last five commits touching tests/backends, pyrite/storage or tests/experimental_known_failures.txt (git log -5 origin/dev -- ...) are 061388c (#1552, writes to the path the walk lists), b0a6128, 43d3682, 151e816 and 250332d. None changes pyrite/storage/backends/postgres_backend.py or tests/backends/test_backend_conformance.py, and the test calls the backend directly, so #1613 cannot reach it. The test file and the known-failures list did not change.

    What the test asserts. TestSemanticKnnBudget in tests/backends/test_backend_conformance.py inserts 4,097 near-identical note embeddings, then one rare entry that is farther from the query than all of them (cosine about 0.6, above the similarity floor). It searches with entry_type="rare", limit=1. The property is #668's: an empty semantic result must not read as "nothing matches" when rows were never considered. Either the row far comes back, or the result is marked capped. Postgres is expected to return it, because every filter is in the same WHERE as the distance ordering (PostgresBackend.search_semantic). It returned [], not capped.

    Why it comes and goes. #1401 has the groom and the history (failures on 10-27, 11-01, 11-02, 11-03, and a sighting on 11-17 on #1570, about 3 in 47 runs). The hypothesis there, still not confirmed: the table has an HNSW index (idx_entry_embedding), and Postgres sometimes plans an index scan that takes about the 40 nearest candidates (hnsw.ef_search) and applies the filter afterwards, so the farthest row is never seen. Which plan runs depends on table statistics on a freshly built table. That would make it a real wrong answer of the experimental backend, not a flake in the test. This failure is consistent with that and adds nothing against it.

    Smallest next step. Close #1641 as a duplicate of #1401 and add this run to #1401's list. Then do #1401's step 1: with a pgvector/pgvector:pg16 container (tests/backends/conftest.py shows the URL variable), run the test's query under EXPLAIN (ANALYZE) right after the fixture and again after ANALYZE entry, and post both plans. Anyone with Docker can do that; no maintainer rights are needed.

    What would settle it from CI alone. A run of the experimental job with a step that prints EXPLAIN for the failing query when the test fails, or a re-run of run 44125437779 (the job is experimental, id 134196476410) to see pass-fail-pass on the same commit. I did not re-run it.

    Checked versus assumed. Checked: the failing job log, the experimental job's result and this test's line in the logs of five other dev runs, the image digest in each, and the git log above. Not checked: the merge-group runs (the two I could list had no finished experimental result), run 44097460262 (the jobs API returned an error), the plan Postgres chose (no log has it). This groom is wrong if a log shows e043428's job running a different image digest or different test code than its neighbours; I compared the digest and the test line, not the whole install log.

    Dispatch (for #1401, since #1641 is a duplicate)

    • Model: opus (storage behaviour, Postgres planner knowledge)
    • Size: M (spike S, fix about 40-80 lines)
    • Depends on: none
    • Good first issue: no
    • Proposed milestone: the next patch release (it is already labelled help wanted)
    • For experimental-broken: tests/backends/test_backend_conformance.py #1641 itself: close as duplicate; no milestone, no label change beyond what the ratchet applies.
  2. markramm commented on Nov 17, 2026

    @markramm
    Collaborator

    Automated: the maintainer's conductor agent (2026-11-17). Closing as a duplicate of #1401: the same test and the same assertion, an intermittent failure that is already tracked there. Checked: the experimental job passed on the dev pushes before and after this one (061388c7, 38d98071) and failed only at e0434281, whose change does not reach the backend. The sighting is recorded on #1401. This is not a regression in dev.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    experimental-brokenAn experimental test newly failing on dev (#657)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions