Repository navigation
Conversation
…ept and named Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
Plan (#1546), read on
|
| Case | Output | Exit | Rows after | Health after | vs groom |
|---|---|---|---|---|---|
sub at 000 |
Removed: 1 |
0 | top |
healthy, exit 0 |
same |
sub at 600 |
Removed: 1, Malformed: 1 • sub/inner.md |
0 | top |
unhealthy, exit 1; unreadable_files names the absolute path, OS text |
same, except the path is absolute, not sub/inner.md |
sub/inner.md at 000, mtime moved |
Removed: 1, Malformed: 1 |
0 | top |
unhealthy (unreadable_files) |
same (health column filled in) |
sub/inner.md at 000, index build |
Index build complete. |
0 | top |
unhealthy (unreadable_files) |
same (health column filled in) |
sub at 000, kb reindex -k k --format json |
"removed": 1 |
0 | top |
healthy | same |
sub at 000 holding c/__collection.yaml |
Removed: 2 |
0 | top |
healthy | same |
sub/inner.md at 000, stat unchanged |
Removed: 0 |
0 | kept | unhealthy | same |
sub/inner.md at 000, index sync --verify |
Could not hash ... keeping the existing indexed entry (under Malformed) |
0 | kept | unhealthy | same |
(groom addition) a.md -> locked/b.md, locked at 000 |
Removed: 1, Dangling links: 1 (in-KB links to a missing file ...) |
0 | top |
warning, exit 0, dangling_links names a.md |
same as the 2026-11-17 addition, plus: b's row is also retired |
2. The rule and where it lives (option A)
- The walk records. One walk of the folders on
os.walk(onerror=...)behind both listings the plan uses:KBRepository.walk_files(todayrglob("*.md")) and_named(todayos.walkwithoutonerror, the collections and the by-name lookups). Both take an optionalunreadablelist;onerroradds the KB-relative path of a folder it could not list, only when the walk's own folder rule (entry_paths.reason_for_folders) would list things in it: never under a hidden part or a_templatespart (shape 6), never the root itself (fix: an unreadable KB is reported, never read as empty; an id that cannot exist is not found (#1542) #1551's gate owns it), and only the outermost (shape 7:os.walkcannot descend further). The md listing keepsrglob's matches: names ending.mdamong files and folders (a folderx.mdis still "not a regular file", index health: a file whose stat fails during the walk (symlink loop, dangling new symlink, non-regular file) is dropped silently, and pyrite-admin's verdict ignores duplicates/content_changed/off_list_values #1119), links listed by name, folder links not entered. - The plan keeps.
plan_reconcilegetsplan.unreadable_folders(the walk's list). It adds a claim built from the row, the verify branch's shape (_Claim(rel, path, row_id, stat=None, unreached=True)), for every known row whose path (a) lies under a recorded folder, or (b) was listed and could not be stat-ed (anOSErrorthat is not a dangling link), or (c) was listed and could not be read (OSErrorfrom the load), or (d) could not be hashed under--verify(already kept today). AUnicodeErroris "read, and does not decode": it stays as today (row retired, health'sunreadable_files), like any file that reads and does not parse. - A row kept on faith never beats a file read now. When a read claim and an unreached claim share an id, the read claim wins whatever the path order, and the duplicate row names both paths.
- The removal loop is untouched: an unreached claim is a winner with no entry, so
_write_claimis not called and the row is neither rewritten nor removed. One rule, inside the one reconcile;remove_entrystill has one caller. - The dangling-link probe.
dangling_targetasksKBFiles.present(three-way) first: "could not tell" is not dangling, so the link falls to the unstattable branch (reported unreadable, kept if it had a row).lexistsstays as the second probe so a link loop (present: ELOOP = absent) is still "there" as today.
3. Enumeration (every place the property must hold)
| Place | Class | Evidence |
|---|---|---|
reconcile_kb's removal loop (the one remove_entry caller) |
fixed here | the plan's unreached claims |
index_kb, sync_incremental, sync_kb, index_with_attribution |
guarded | test_an_unlisted_folder_keeps_its_rows.py, parametrised over RECONCILES = test_one_reconcile.PATHS + index_with_attribution, and a structural test that the set equals every IndexManager method whose body calls self.reconcile_kb (AST), so a fifth cannot be added untested |
CLI index sync |
fixed + guarded | names folders, exit 3/1 |
CLI index build (and --with-attribution) |
fixed + guarded | index_kb / index_with_attribution gain an optional report dict the CLI passes; exit 3/1 |
CLI kb reindex |
fixed + guarded | names, exit 3 (1 KB done) |
MCP kb_index_sync, REST POST /api/index/sync?wait=true |
fixed + guarded | unreadable_folders: <count> |
MCP kb_registry_reindex, REST admin reindex |
fixed + guarded | MCP spreads the whole result (**result) today, so it goes through the same network shaping; REST KBReindexResponse gains the count |
check_health (CLI index health, kb health, pyrite-admin, QA's gate via health_verdict) |
fixed + guarded | unreadable_folders rows; health_verdict reads it as unhealthy |
REST/MCP kb_registry_health |
needs a file outside Touches, see §7 | |
dangling_target (plan, document_manager) |
fixed + guarded | the alias cell |
alias_target's _is_regular_file |
safe by reading | a target it cannot stat is "not an alias", so the link goes to the plan's stat, which fails and lands in (b): reported unreadable, never dangling |
pyrite init, pyrite-admin index, KBService.sync_index/index_kb, rename's sync, repo_service subscribe/sync, the background jobs |
rows guarded (they reach the reconcile); naming out of scope | the background job is #1548; the rest go in the follow-up issue |
Report-only walkers of groom §3, the read path's NOT_FOUND |
out of scope | follow-up issue (one) |
4. What each key carries, on today's dev and with this change
| Key | CLI | REST | MCP |
|---|---|---|---|
unreadable_files (health) |
rows {kb, path (absolute), error (OS text)} |
not carried (healthy: false only) |
not carried (healthy: false only) |
unreadable_kbs (sync, health) |
sync: Not synced: kb: <fixed> lines; health: rows {kb, error (fixed)} |
sync: rows {kb, error (fixed), error_code}; health: not carried |
same as REST |
unlisted_entry_files (health) |
rows {kb, path, reason} |
count (KBHealthResponse) |
count (health_kb) |
reindex malformed (today) |
rows | not carried (KBReindexResponse) |
rows with absolute path and OS text (**result; existing, not this theme's) |
unreadable_folders (new) |
rows {kb, path (KB-relative), error (fixed KB_UNREADABLE text)} on sync, build, reindex, health |
count on sync, reindex (and health, §7) | count on sync, reindex (and health, §7) |
unreadable_files on a reconcile result (new) |
rows {kb, path, error}; files also stay under malformed and count for the exit |
not carried | not carried (dropped from the spread) |
All network shaping is one function, network_shape(result) in pyrite/services/index_health.py: rows to a count. Changing to rows on every surface is a few lines there.
5. The maintainer's acceptance line
The alias cell: KB with locked/b.md and a.md -> locked/b.md, synced; locked at 000; sync. Expected: no "Dangling links", a.md under Malformed, locked named under unreadable folders, b's row kept, exit 3; index health unhealthy, exit 1, unreadable_folders names locked, unreadable_files names a.md, dangling_links empty.
6. Files expected, against the groom's Touches
In Touches: pyrite/storage/repository.py, pyrite/storage/index.py, pyrite/cli/index_commands.py, pyrite/cli/kb_commands.py, pyrite/server/mcp_server.py, pyrite/server/schemas.py (sync, reindex, health classes only), pyrite/server/endpoints/admin.py, pyrite/services/index_health.py, docs/json-contracts.md, kb/maps/index-database-and-derived-state.md, tests/test_kb_reads_are_one_function.py, goldens if they move, kb/components/{kb-repository,index-manager}.md, new tests/test_an_unlisted_folder_keeps_its_rows.py, changelog.d/1546-*.fixed.md. Not touching: the lines above plan_reconcile (#1628), ErrorResponse and the request classes (#1633), --ttl (#1644).
7. Stop item: a file outside Touches
REST and MCP kb health are shaped in one place since #1576: KBRegistryService.health_kb in pyrite/services/kb_registry_service.py (not in the groom's Touches, which predates #1576). A count of unreadable_folders on network health needs one line there (drift["unreadable_folders"] = ..., through network_shape). Until the maintainer says yes, I leave that line out: network health still turns healthy: false for a shut folder (through health_verdict, as it does for unreadable_files and unreadable_kbs, which network health doesn't carry either), and KBHealthResponse is unchanged. Everything else goes ahead.
8. Where the groom is out of date or wrong on today's dev
list_filesno longer walks itself:walk_files(from fix(storage): an in-KB link to an entry file is an alias, indexed once (#1262) #1483) doesrglob("*.md")andlist_filesfilters it. The recording goes inwalk_files, as the groom expected after fix(storage): an in-KB link to an entry file is an alias, indexed once (#1262) #1483.- Health's
unreadable_filespath is absolute, not KB-relative. - MCP
kb_registry_reindexspreads the whole reconcile result, so any new reconcile key reaches MCP as it is unless shaped (groom §3 lists the tool but not this). - Network health carries neither
unreadable_filesnorunreadable_kbs(groom §5 assumed rows on health everywhere). - In the alias shape the target's row is also retired (the 2026-11-17 addition reported only the "dangling" line).
9. Timings (#135), baseline on this dev
2,000-file throwaway KB, median of 15: by-id hit by filename 3.0 ms; miss (walk + prefilter) 142.4 ms; list_files 15.5 ms. Re-measured after the change, in the report.
Generated by Claude Code
#1546) A folder the walk could not list, and a file it could not stat or read, said nothing about the entries in them, yet the reconcile read them as "no file" and retired their rows, exit 0, and health then said healthy. The one walk (KBRepository._folders, os.walk with onerror, behind both walk_files and _named) records each folder it could not list, unless the walk's own rule would never list anything in it (hidden, _templates) or it is the root (#1551's gate). plan_reconcile keeps the row of every known path it could not reach as a claim built from the row (the verify branch's shape), so the one removal loop neither rewrites nor removes it; a row kept on faith never beats a file read now. A file that reads and does not parse still loses its row. dangling_target asks KBFiles.present first, so a link into a folder that cannot be searched is unreadable, never dangling. The reconcile result gains unreadable_folders ({kb, path, error}, the fixed KB_UNREADABLE text) and unreadable_files; health gains unreadable_folders (unhealthy). CLI sync, build and kb reindex name them and count them as failed for exit_unless_whole (3, or 1 when nothing synced). Over REST and MCP the folders are a count, decided in one function (network_shape). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
…t file (#1546) Moving walk_files from rglob("*.md") to the one os.walk made list_files about 5 ms slower on a 2,000-file KB: each name became a Path that every probe parsed again. The name is now joined and checked as text, and the alias probe takes it as text; best of 200 runs, list_files 11.8 ms on dev and 11.7 ms here, a lookup miss 101.0 and 103.2 ms (#135). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
|
Conductor answer to your plan on #1647 (#1546), stop item §7: YES, add the one line in
Carry on to the report ("Report posted at " as a PR comment when gate and verify-red finish on that head). |
…or (#1546) os.walk's onerror hands over the error, and its .filename is text derived from an exception (tests/test_response_text_is_pyrites.py, ADR-0052 N3), which the folder rows would carry into a result. _folders now writes out os.walk's top-down walk over os.scandir and records the folder it asked to list, never text from the error; the ratchet classifies its scandir and the DirEntry's own is_dir / is_symlink answers. _print_unreadable_folders is judged in the exit-code ratchet: its callers exit 3 or 1. Same timings on a 2,000-file KB, best of 200: a by-id hit 2.5 ms on dev and 2.2 ms here, a miss 103.1 and 105.1 ms, list_files 13.1 and 11.9 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
…#1546) Network health is shaped in one place since #1576, KBRegistryService. health_kb. It now carries unreadable_folders through network_shape, the count sync and reindex give, beside #1552's unlisted_entry_files; healthy is already false through health_verdict. KBHealthResponse gains the field. The conductor approved the file on #1647 (outside the groom's Touches, which predates #1576). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
Two keys, both this theme's, nothing else: MCP kb_index_sync gains
unreadable_folders: 0 (3 cells in each of the read, write and admin key
files), and REST GET /api/kbs/{kb}/health gains unreadable_folders: 0
(16 cells; the unlisted_entry_files lines only gain a trailing comma).
Regenerated serially with PYRITE_CHARACTERIZATION_REGENERATE=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
#1546) docs/json-contracts.md gains "A folder or file Pyrite cannot reach" (the rows kept, the result keys, the network count, health, the alias, the limits), the health table row unreadable_folders and the exit-code line for index sync, build and kb reindex. Invariant I2 in the index map now holds the rows of paths the process could not reach; kb-repository and index-manager say where the walk records and the plan keeps. The changelog fragment is the upgrade note: exit 3 where it was 0, health unhealthy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
…retirement-vmdcwe
…pt and named Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013BnEiXeX1jTydWtXbEVSKs
Fixes #1546
Branch: this cloud session was given its own branch name,
claude/index-sync-row-retirement-vmdcwe, which is used instead offix/1546-unreachable-folder-keeps-rows.Theme (from the groom of 2026-11-16)
Maintainer's added acceptance (2026-11-17): an in-KB link (alias) into a folder that cannot be searched is reported as unreadable, not as a dangling link to a missing file, and after a sync
pyrite index healthis unhealthy (exit 1).Design (groom's recommendations, as briefed)
_templatesfolders by the walk's own rule);plan_reconcilekeeps the row of every known path it could not reach by a claim built from the row (the verify branch's shape). One rule, inside the one reconcile.unreadable_folders. Exit code counts unreadable KBs, folders and files as failed (exit_unless_whole): a KB with one shut folder exits 3.malformedon the CLI and counts as failed.Plan: in the first comment below. Report: appended here when done.
Generated by Claude Code