Skip to content

fix: rows under a folder the process could not list are kept and named (#1546) - #1647

Draft
markramm wants to merge 9 commits into
devfrom
claude/index-sync-row-retirement-vmdcwe
Draft

markramm wants to merge 9 commits into
devfrom
claude/index-sync-row-retirement-vmdcwe

Conversation

@markramm

Copy link
Copy Markdown
Collaborator

Fixes #1546

Branch: this cloud session was given its own branch name, claude/index-sync-row-retirement-vmdcwe, which is used instead of fix/1546-unreachable-folder-keeps-rows.

Theme (from the groom of 2026-11-16)

Acceptance:   With a KB folder at mode 000 and at 600 (the root readable), and with one
              unreadable file whose mtime moved, every reconcile path keeps the rows of the
              paths it could not reach (the row set and each content_hash unchanged). It names
              the folder (000) or the file (600, and the unreadable file). CLI sync and build
              exit 3, or 1 when nothing synced. Health is unhealthy and names them. After the
              folder is made readable, the next sync reports added=updated=removed=0 for those
              rows. A file that reads and does not parse still loses its row.
Regimes:      An unlisted folder under a hidden folder or _templates: not reported, exit code
              unchanged. Nested unlisted folders: the outermost only. Empty KB. Unreadable KB
              root: unchanged (KB_UNREADABLE). A user whom mode 000 does not stop (root): skip,
              as _lock does. A network caller gets the fixed text, never the OS's (N3).
Shapes:       The 9 in section 3. The test is parametrised over shapes 1, 2, 3, 5 and 6 times
              the four reconcile entry points (index_kb, sync_incremental, sync_kb,
              index_with_attribution), plus CLI index sync, MCP kb_index_sync and REST
              POST /api/index/sync?wait=true for shape 1.

Maintainer's added acceptance (2026-11-17): an in-KB link (alias) into a folder that cannot be searched is reported as unreadable, not as a dangling link to a missing file, and after a sync pyrite index health is unhealthy (exit 1).

Design (groom's recommendations, as briefed)

  • Option A: the one walk records each folder it could not list (after pruning hidden and _templates folders by the walk's own rule); plan_reconcile keeps the row of every known path it could not reach by a claim built from the row (the verify branch's shape). One rule, inside the one reconcile.
  • Result key unreadable_folders. Exit code counts unreadable KBs, folders and files as failed (exit_unless_whole): a KB with one shut folder exits 3.
  • A row kept on faith never beats a file read now.
  • A file the sync cannot read stays under malformed on the CLI and counts as failed.
  • Network callers: path rows on the local CLI; over REST and MCP a count and Pyrite's fixed text (following the maintainer's create writes an entry file the index walk skips (a note titled README): the entry disappears at the next sync #1552 decision; put to the maintainer).

Plan: in the first comment below. Report: appended here when done.


Generated by Claude Code

…ept and named

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs

Copy link
Copy Markdown
Collaborator Author

Plan (#1546), read on dev f95e9b3

0. The riskiest assumption first

  • The container runs as root (uid 0), and mode 000 does not stop root. Every probe below runs as nobody (setpriv --reuid=65534 --regid=65534 --clear-groups), with throwaway HOME, PYRITE_CONFIG_DIR and PYRITE_DATA_DIR under a temp dir. (On this checkout PYRITE_DATA_DIR wins over PYRITE_CONFIG_DIR for finding config.yaml, config.py:1776, so both point at one dir.) The new tests skip for a user mode 000 does not stop, as _lock does; locally I run them as nobody, and CI runs them as a normal user.
  • os.walk's onerror fires for a folder at mode 000 on Linux (Python 3.13.18, as nobody): onerror got PermissionError(13) for k/sub, and Path.rglob("*.md") skipped it without a sign. At mode 600 the folder is listed (names come from readdir, with no stat), its files are yielded, and each stat fails. A folder inside a 600 folder fires onerror. So the groom's "wrong if" does not hold, and the design stands.

1. Section 2 on today's dev (Linux, as nobody)

Setup as in the groom: KB k holding top.md and sub/inner.md, indexed, then the mode set.

Case Output Exit Rows after Health after vs groom
sub at 000 Removed: 1 0 top healthy, exit 0 same
sub at 600 Removed: 1, Malformed: 1 • sub/inner.md 0 top unhealthy, exit 1; unreadable_files names the absolute path, OS text same, except the path is absolute, not sub/inner.md
sub/inner.md at 000, mtime moved Removed: 1, Malformed: 1 0 top unhealthy (unreadable_files) same (health column filled in)
sub/inner.md at 000, index build Index build complete. 0 top unhealthy (unreadable_files) same (health column filled in)
sub at 000, kb reindex -k k --format json "removed": 1 0 top healthy same
sub at 000 holding c/__collection.yaml Removed: 2 0 top healthy same
sub/inner.md at 000, stat unchanged Removed: 0 0 kept unhealthy same
sub/inner.md at 000, index sync --verify Could not hash ... keeping the existing indexed entry (under Malformed) 0 kept unhealthy same
(groom addition) a.md -> locked/b.md, locked at 000 Removed: 1, Dangling links: 1 (in-KB links to a missing file ...) 0 top warning, exit 0, dangling_links names a.md same as the 2026-11-17 addition, plus: b's row is also retired

2. The rule and where it lives (option A)

  • The walk records. One walk of the folders on os.walk(onerror=...) behind both listings the plan uses: KBRepository.walk_files (today rglob("*.md")) and _named (today os.walk without onerror, the collections and the by-name lookups). Both take an optional unreadable list; onerror adds the KB-relative path of a folder it could not list, only when the walk's own folder rule (entry_paths.reason_for_folders) would list things in it: never under a hidden part or a _templates part (shape 6), never the root itself (fix: an unreadable KB is reported, never read as empty; an id that cannot exist is not found (#1542) #1551's gate owns it), and only the outermost (shape 7: os.walk cannot descend further). The md listing keeps rglob's matches: names ending .md among files and folders (a folder x.md is still "not a regular file", index health: a file whose stat fails during the walk (symlink loop, dangling new symlink, non-regular file) is dropped silently, and pyrite-admin's verdict ignores duplicates/content_changed/off_list_values #1119), links listed by name, folder links not entered.
  • The plan keeps. plan_reconcile gets plan.unreadable_folders (the walk's list). It adds a claim built from the row, the verify branch's shape (_Claim(rel, path, row_id, stat=None, unreached=True)), for every known row whose path (a) lies under a recorded folder, or (b) was listed and could not be stat-ed (an OSError that is not a dangling link), or (c) was listed and could not be read (OSError from the load), or (d) could not be hashed under --verify (already kept today). A UnicodeError is "read, and does not decode": it stays as today (row retired, health's unreadable_files), like any file that reads and does not parse.
  • A row kept on faith never beats a file read now. When a read claim and an unreached claim share an id, the read claim wins whatever the path order, and the duplicate row names both paths.
  • The removal loop is untouched: an unreached claim is a winner with no entry, so _write_claim is not called and the row is neither rewritten nor removed. One rule, inside the one reconcile; remove_entry still has one caller.
  • The dangling-link probe. dangling_target asks KBFiles.present (three-way) first: "could not tell" is not dangling, so the link falls to the unstattable branch (reported unreadable, kept if it had a row). lexists stays as the second probe so a link loop (present: ELOOP = absent) is still "there" as today.

3. Enumeration (every place the property must hold)

Place Class Evidence
reconcile_kb's removal loop (the one remove_entry caller) fixed here the plan's unreached claims
index_kb, sync_incremental, sync_kb, index_with_attribution guarded test_an_unlisted_folder_keeps_its_rows.py, parametrised over RECONCILES = test_one_reconcile.PATHS + index_with_attribution, and a structural test that the set equals every IndexManager method whose body calls self.reconcile_kb (AST), so a fifth cannot be added untested
CLI index sync fixed + guarded names folders, exit 3/1
CLI index build (and --with-attribution) fixed + guarded index_kb / index_with_attribution gain an optional report dict the CLI passes; exit 3/1
CLI kb reindex fixed + guarded names, exit 3 (1 KB done)
MCP kb_index_sync, REST POST /api/index/sync?wait=true fixed + guarded unreadable_folders: <count>
MCP kb_registry_reindex, REST admin reindex fixed + guarded MCP spreads the whole result (**result) today, so it goes through the same network shaping; REST KBReindexResponse gains the count
check_health (CLI index health, kb health, pyrite-admin, QA's gate via health_verdict) fixed + guarded unreadable_folders rows; health_verdict reads it as unhealthy
REST/MCP kb_registry_health needs a file outside Touches, see §7
dangling_target (plan, document_manager) fixed + guarded the alias cell
alias_target's _is_regular_file safe by reading a target it cannot stat is "not an alias", so the link goes to the plan's stat, which fails and lands in (b): reported unreadable, never dangling
pyrite init, pyrite-admin index, KBService.sync_index/index_kb, rename's sync, repo_service subscribe/sync, the background jobs rows guarded (they reach the reconcile); naming out of scope the background job is #1548; the rest go in the follow-up issue
Report-only walkers of groom §3, the read path's NOT_FOUND out of scope follow-up issue (one)

4. What each key carries, on today's dev and with this change

Key CLI REST MCP
unreadable_files (health) rows {kb, path (absolute), error (OS text)} not carried (healthy: false only) not carried (healthy: false only)
unreadable_kbs (sync, health) sync: Not synced: kb: <fixed> lines; health: rows {kb, error (fixed)} sync: rows {kb, error (fixed), error_code}; health: not carried same as REST
unlisted_entry_files (health) rows {kb, path, reason} count (KBHealthResponse) count (health_kb)
reindex malformed (today) rows not carried (KBReindexResponse) rows with absolute path and OS text (**result; existing, not this theme's)
unreadable_folders (new) rows {kb, path (KB-relative), error (fixed KB_UNREADABLE text)} on sync, build, reindex, health count on sync, reindex (and health, §7) count on sync, reindex (and health, §7)
unreadable_files on a reconcile result (new) rows {kb, path, error}; files also stay under malformed and count for the exit not carried not carried (dropped from the spread)

All network shaping is one function, network_shape(result) in pyrite/services/index_health.py: rows to a count. Changing to rows on every surface is a few lines there.

5. The maintainer's acceptance line

The alias cell: KB with locked/b.md and a.md -> locked/b.md, synced; locked at 000; sync. Expected: no "Dangling links", a.md under Malformed, locked named under unreadable folders, b's row kept, exit 3; index health unhealthy, exit 1, unreadable_folders names locked, unreadable_files names a.md, dangling_links empty.

6. Files expected, against the groom's Touches

In Touches: pyrite/storage/repository.py, pyrite/storage/index.py, pyrite/cli/index_commands.py, pyrite/cli/kb_commands.py, pyrite/server/mcp_server.py, pyrite/server/schemas.py (sync, reindex, health classes only), pyrite/server/endpoints/admin.py, pyrite/services/index_health.py, docs/json-contracts.md, kb/maps/index-database-and-derived-state.md, tests/test_kb_reads_are_one_function.py, goldens if they move, kb/components/{kb-repository,index-manager}.md, new tests/test_an_unlisted_folder_keeps_its_rows.py, changelog.d/1546-*.fixed.md. Not touching: the lines above plan_reconcile (#1628), ErrorResponse and the request classes (#1633), --ttl (#1644).

7. Stop item: a file outside Touches

REST and MCP kb health are shaped in one place since #1576: KBRegistryService.health_kb in pyrite/services/kb_registry_service.py (not in the groom's Touches, which predates #1576). A count of unreadable_folders on network health needs one line there (drift["unreadable_folders"] = ..., through network_shape). Until the maintainer says yes, I leave that line out: network health still turns healthy: false for a shut folder (through health_verdict, as it does for unreadable_files and unreadable_kbs, which network health doesn't carry either), and KBHealthResponse is unchanged. Everything else goes ahead.

8. Where the groom is out of date or wrong on today's dev

9. Timings (#135), baseline on this dev

2,000-file throwaway KB, median of 15: by-id hit by filename 3.0 ms; miss (walk + prefilter) 142.4 ms; list_files 15.5 ms. Re-measured after the change, in the report.


Generated by Claude Code

#1546)

A folder the walk could not list, and a file it could not stat or read,
said nothing about the entries in them, yet the reconcile read them as
"no file" and retired their rows, exit 0, and health then said healthy.

The one walk (KBRepository._folders, os.walk with onerror, behind both
walk_files and _named) records each folder it could not list, unless the
walk's own rule would never list anything in it (hidden, _templates) or it
is the root (#1551's gate). plan_reconcile keeps the row of every known
path it could not reach as a claim built from the row (the verify branch's
shape), so the one removal loop neither rewrites nor removes it; a row kept
on faith never beats a file read now. A file that reads and does not parse
still loses its row. dangling_target asks KBFiles.present first, so a link
into a folder that cannot be searched is unreadable, never dangling.

The reconcile result gains unreadable_folders ({kb, path, error}, the fixed
KB_UNREADABLE text) and unreadable_files; health gains unreadable_folders
(unhealthy). CLI sync, build and kb reindex name them and count them as
failed for exit_unless_whole (3, or 1 when nothing synced). Over REST and
MCP the folders are a count, decided in one function (network_shape).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
…t file (#1546)

Moving walk_files from rglob("*.md") to the one os.walk made list_files
about 5 ms slower on a 2,000-file KB: each name became a Path that every
probe parsed again. The name is now joined and checked as text, and the
alias probe takes it as text; best of 200 runs, list_files 11.8 ms on dev
and 11.7 ms here, a lookup miss 101.0 and 103.2 ms (#135).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
@markramm

Copy link
Copy Markdown
Collaborator Author

Conductor answer to your plan on #1647 (#1546), stop item §7: YES, add the one line in pyrite/services/kb_registry_service.py.

  • Scope: in KBRegistryService.health_kb, carry unreadable_folders through network_shape as a count, exactly as fix: a write never places an entry where the index walk does not list it (#1552) #1576 does for unlisted_entry_files. Add the matching optional count field on KBHealthResponse in pyrite/server/schemas.py (the health class, which is already in your Touches). Nothing else in that file.
  • Why this is footprint, not a design choice: the network shape was already set as your default in the brief (rows on the local CLI, a count over REST and MCP, the create writes an entry file the index walk skips (a note titled README): the entry disappears at the next sync #1552 rule). The line makes network health agree with network sync and reindex. No open PR touches kb_registry_service.py (checked against every open PR's file list at 19:11Z).
  • The rows-vs-count choice is still on the maintainer's desk. Keep it in one function (network_shape) as your plan says, so a "rows everywhere" answer is a few lines.
  • Your enumeration table (§3): move the kb_registry_health row from "needs a file outside Touches" to fixed + guarded, with a cell for the count over REST and MCP.
  • The rest of the plan stands as posted, including §8's corrections to the groom. Remember: each commit passes the commit hooks (a fix: commit touches tests/), never --no-verify on commit; no force-push.

Carry on to the report ("Report posted at " as a PR comment when gate and verify-red finish on that head).

…or (#1546)

os.walk's onerror hands over the error, and its .filename is text derived
from an exception (tests/test_response_text_is_pyrites.py, ADR-0052 N3),
which the folder rows would carry into a result. _folders now writes out
os.walk's top-down walk over os.scandir and records the folder it asked to
list, never text from the error; the ratchet classifies its scandir and the
DirEntry's own is_dir / is_symlink answers. _print_unreadable_folders is
judged in the exit-code ratchet: its callers exit 3 or 1.

Same timings on a 2,000-file KB, best of 200: a by-id hit 2.5 ms on dev and
2.2 ms here, a miss 103.1 and 105.1 ms, list_files 13.1 and 11.9 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
…#1546)

Network health is shaped in one place since #1576, KBRegistryService.
health_kb. It now carries unreadable_folders through network_shape, the
count sync and reindex give, beside #1552's unlisted_entry_files; healthy
is already false through health_verdict. KBHealthResponse gains the field.
The conductor approved the file on #1647 (outside the groom's Touches,
which predates #1576).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
Two keys, both this theme's, nothing else: MCP kb_index_sync gains
unreadable_folders: 0 (3 cells in each of the read, write and admin key
files), and REST GET /api/kbs/{kb}/health gains unreadable_folders: 0
(16 cells; the unlisted_entry_files lines only gain a trailing comma).
Regenerated serially with PYRITE_CHARACTERIZATION_REGENERATE=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
#1546)

docs/json-contracts.md gains "A folder or file Pyrite cannot reach" (the
rows kept, the result keys, the network count, health, the alias, the
limits), the health table row unreadable_folders and the exit-code line for
index sync, build and kb reindex. Invariant I2 in the index map now holds
the rows of paths the process could not reach; kb-repository and
index-manager say where the walk records and the plan keeps. The changelog
fragment is the upgrade note: exit 3 where it was 0, health unhealthy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

index sync retires the rows of a subfolder it cannot list, exits 0, and health says healthy

2 participants