Repository navigation
fix(storage): compare batch taken paths with entry_paths.fold on every disk (#1623) - #1659
Mohit-Gaur75 wants to merge 2 commits into
Conversation
markramm
left a comment
There was a problem hiding this comment.
Automated review by the maintainer's conductor agent (Claude Code), as a cold read of head 2bb7c4c. This PR touches pyrite/storage/, so the maintainer merges it whatever this review says.
Thank you for this. It is a careful implementation of decision (a) on #1623. Every batch entry point goes through KBService.bulk_create_entries: MCP kb_bulk_create, REST import, CLI import (with and without --dry-run), task_service's decompose, and the journalism promote dry run. So wiring taken and pending_paths into the real-write branch covers them all. Both sides now fold, so the dry run and the real write give the same verdict on either kind of disk for the two batches in the issue's table, in both orders. Reconcile keeps a file's name and moves only its folder, so it cannot undo a -2. The plugins that work out the path again after a create (zettelkasten, social, software-kb) only do so after a single create, never in a batch. The disk-independent test test_taken_names_comparison_folds_case_and_normalises fails on dev without the fix: abc.md is not in a set holding Abc.md as spelled. On Linux CI the control test fails without the fix too. Two things need changing before this can merge: the batch is now quadratic, and three new mypy errors. The rest are smaller.
Blocking
-
The folded set is rebuilt on every call, so a batch now costs O(n²).
KBRepository._free_default_path(taken_keys = {...for p in taken}) andKBService._prepare(pending_keys = {...}) fold every path taken so far, on every item. A real-write item reaches that code three times: in_prepare's pending check, in_prepare's_resolve_file_path, and again insave→_target_path→_resolve_file_path. A dry-run item also reaches it three times. Before this PR the real write did no such work, and the dry run did one O(1)intest. I timedfolded_path_keyon this machine at about 2.6 µs per path. Extrapolated from that measurement (not timed end to end), that adds roughly 1 s to a 500-record import, 15 s to a 2,000-record one, and over 6 minutes to a 10,000-record one, in bothimportandimport --dry-run. Please build the folded keys once per batch and add to them as you go. For example, keep aset[tuple[str, ...]]of folded keys besidewould_create_pathsinbulk_create_entriesand pass that down. Or have_free_default_pathand_prepareaccept keys that are already folded. Then each check is oneintest again. -
mypy ratchet +3 (the
mypy (advisory)job is red because of this PR).KBService._prepare_and_save(taken: set[Path] = frozenset())andDocumentManager.save_entry(taken: set[Path] = frozenset())have defaults that do not match their annotations, andbulk_create_entriespasses afrozensetwheresetis declared. Annotate them asrepository.pyalready does (taken: Set[Path] = frozenset()withSetfromcollections.abc/typing, that is,AbstractSet).
Non-blocking
- No test reaches the fold in
_prepare. The existing batches only prove the default-name step. In "case collision with pattern colliding with stepped", the stepped name and the pattern name are both spelledabc-2.md, so the old as-spelled comparison already refused the third item. Reverting only thepending_keyshunk inKBService._preparebreaks no test. A cell that needs it: a type withfile_pattern: "{id}-x.md"and the batch[{id: Abc}, {id: abc}]. Assert that the second item is refused withENTRY_EXISTSin both the dry run and the real write, as a plain assertion rather than a parity comparison, so it is red on Linux without the fix. That also pins the changelog's "or refuse a collidingfile_patternname". - The test name says "normalises", but the test only checks case. Add an NFC/NFD pair (
cafécomposed and decomposed), or drop "and_normalises" from the name. - Batch and single creates now differ on Linux. A batch
[Abc, abc]writesAbc.mdandabc-2.md. Two separatepyrite createcalls still writeAbc.mdandabc.md, and so does an import that follows an earlier create ofAbc. That matches the decision's scope ("a batch's taken names"). But the.changed.mdfragment should say so, or a Linux user will expect the step everywhere. One sentence is enough, for example: "Separate creates are unchanged: on a case-sensitive disk they still write both names." - The user-facing text names an internal function. The new fragment, the #1552 fragment's edit and the
VALIDATION_FAILEDrow indocs/json-contracts.mdsay "compared withentry_paths.fold". A reader of the changelog or the contract cannot use that. Say what it means, for example: "compared ignoring letter case and Unicode normalisation (and the trailing dots or spaces Windows drops)". _prepare_and_savetakes the same set twice. It getspending_paths=would_create_pathsand alsotaken=frozenset(would_create_paths). One argument is enough:_preparealready receivespending_paths, so_prepare_and_savecan work outtakenitself. After blocking item 1, that argument would be the folded-key set._prepare's docstring still says "pending_pathsare resolved file paths an earlier item of the same dry run would have created". Since this PR the real write passes them too.
CI
mypy (advisory): red because of this PR. See blocking item 2.experimental: red onTestSemanticKnnBudget::test_rows_the_ceiling_hides_are_returned_or_the_result_says_capped[postgres], the known intermittent failure #1401. It has nothing to do with this change.test (3.12, 1),collectandlockpass.test (3.12, 2)and(3.12, 3)were still running at the time of this review. The verify-red job has not reported yet. By reading: the disk-independent test is red without the fix on every disk, and the control test is red without the fix on Linux, the disk CI runs on. The new parity cells agree ondevtoo, as the issue expected on a case-sensitive disk, so they are not evidence for the fix.- A changelog fragment is present:
changelog.d/1623-batch-taken-names-fold.changed.md. The #1552 fragment anddocs/json-contracts.mdare updated as the issue asked.
Recommendation: request changes.
Under maintainer decision (a) on #1623 (confirmed by @markramm):
Within a bulk create, import, or dry run batch, paths taken by earlier items in the batch are compared using entry_paths.fold (case-insensitively and canonical normalised) on every disk.
On a case-sensitive disk (Linux), two entries in the same batch whose paths differ only in letter case (Abc and �bc) now step the second entry to -2 (or refuse a colliding ile_pattern name) rather than writing two files that would collide when cloned on a case-insensitive disk (macOS or Windows). Dry run and real write agree on all platforms.
Key Changes
epo.save.
Fixes #1623
Changelog: changed