Skip to content

fix(storage): compare batch taken paths with entry_paths.fold on every disk (#1623) - #1659

Open
Mohit-Gaur75 wants to merge 2 commits into
pyrite-wiki:devfrom
Mohit-Gaur75:fix/1623-batch-taken-names-fold
Open

Mohit-Gaur75 wants to merge 2 commits into
pyrite-wiki:devfrom
Mohit-Gaur75:fix/1623-batch-taken-names-fold

Conversation

@Mohit-Gaur75

Copy link
Copy Markdown
Contributor

Under maintainer decision (a) on #1623 (confirmed by @markramm):
Within a bulk create, import, or dry run batch, paths taken by earlier items in the batch are compared using entry_paths.fold (case-insensitively and canonical normalised) on every disk.

On a case-sensitive disk (Linux), two entries in the same batch whose paths differ only in letter case (Abc and �bc) now step the second entry to -2 (or refuse a colliding ile_pattern name) rather than writing two files that would collide when cloned on a case-insensitive disk (macOS or Windows). Dry run and real write agree on all platforms.

Key Changes

  • pyrite/storage/entry_paths.py: Added old_parts and olded_path_key helpers.
  • pyrite/storage/repository.py:
    • _free_default_path: Compares candidate paths against aken using entry_paths.folded_path_key.
    • _resolve_file_path: Checks entry_paths.folded_path_key(self.path, candidate) in taken_keys.
    • save and _target_path: Forward aken: set[Path] = frozenset() to _resolve_file_path.
  • pyrite/storage/document_manager.py: Forwards aken to
    epo.save.
  • pyrite/services/kb_service.py:
    • _prepare: Uses entry_paths.folded_path_key to check against pending_paths.
    • _prepare_and_save: Accepts and passes pending_paths and aken to ODM save_entry.
    • �ulk_create_entries: Tracks would_create_paths and written_paths, folding both in dry run and real write modes.
  • ests/test_writes_land_where_the_walk_lists.py:
  • Added 4 case-collision batch variants to PARITY_BATCHES.
  • Added est_taken_names_comparison_folds_case_and_normalises (disk-independent).
  • Added est_case_sensitive_disk_steps_second_case_variant_id (verifying second case variant steps to -2).
  • docs/json-contracts.md and changelog.d/1552-writes-land-where-the-walk-lists.fixed.md: Updated documentation regarding disk comparison.
  • Added changelog fragment changelog.d/1623-batch-taken-names-fold.changed.md.

Fixes #1623

Changelog: changed

@markramm markramm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by the maintainer's conductor agent (Claude Code), as a cold read of head 2bb7c4c. This PR touches pyrite/storage/, so the maintainer merges it whatever this review says.

Thank you for this. It is a careful implementation of decision (a) on #1623. Every batch entry point goes through KBService.bulk_create_entries: MCP kb_bulk_create, REST import, CLI import (with and without --dry-run), task_service's decompose, and the journalism promote dry run. So wiring taken and pending_paths into the real-write branch covers them all. Both sides now fold, so the dry run and the real write give the same verdict on either kind of disk for the two batches in the issue's table, in both orders. Reconcile keeps a file's name and moves only its folder, so it cannot undo a -2. The plugins that work out the path again after a create (zettelkasten, social, software-kb) only do so after a single create, never in a batch. The disk-independent test test_taken_names_comparison_folds_case_and_normalises fails on dev without the fix: abc.md is not in a set holding Abc.md as spelled. On Linux CI the control test fails without the fix too. Two things need changing before this can merge: the batch is now quadratic, and three new mypy errors. The rest are smaller.

Blocking

  1. The folded set is rebuilt on every call, so a batch now costs O(n²). KBRepository._free_default_path (taken_keys = {...for p in taken}) and KBService._prepare (pending_keys = {...}) fold every path taken so far, on every item. A real-write item reaches that code three times: in _prepare's pending check, in _prepare's _resolve_file_path, and again in save → _target_path → _resolve_file_path. A dry-run item also reaches it three times. Before this PR the real write did no such work, and the dry run did one O(1) in test. I timed folded_path_key on this machine at about 2.6 µs per path. Extrapolated from that measurement (not timed end to end), that adds roughly 1 s to a 500-record import, 15 s to a 2,000-record one, and over 6 minutes to a 10,000-record one, in both import and import --dry-run. Please build the folded keys once per batch and add to them as you go. For example, keep a set[tuple[str, ...]] of folded keys beside would_create_paths in bulk_create_entries and pass that down. Or have _free_default_path and _prepare accept keys that are already folded. Then each check is one in test again.

  2. mypy ratchet +3 (the mypy (advisory) job is red because of this PR). KBService._prepare_and_save(taken: set[Path] = frozenset()) and DocumentManager.save_entry(taken: set[Path] = frozenset()) have defaults that do not match their annotations, and bulk_create_entries passes a frozenset where set is declared. Annotate them as repository.py already does (taken: Set[Path] = frozenset() with Set from collections.abc/typing, that is, AbstractSet).

Non-blocking

  • No test reaches the fold in _prepare. The existing batches only prove the default-name step. In "case collision with pattern colliding with stepped", the stepped name and the pattern name are both spelled abc-2.md, so the old as-spelled comparison already refused the third item. Reverting only the pending_keys hunk in KBService._prepare breaks no test. A cell that needs it: a type with file_pattern: "{id}-x.md" and the batch [{id: Abc}, {id: abc}]. Assert that the second item is refused with ENTRY_EXISTS in both the dry run and the real write, as a plain assertion rather than a parity comparison, so it is red on Linux without the fix. That also pins the changelog's "or refuse a colliding file_pattern name".
  • The test name says "normalises", but the test only checks case. Add an NFC/NFD pair (café composed and decomposed), or drop "and_normalises" from the name.
  • Batch and single creates now differ on Linux. A batch [Abc, abc] writes Abc.md and abc-2.md. Two separate pyrite create calls still write Abc.md and abc.md, and so does an import that follows an earlier create of Abc. That matches the decision's scope ("a batch's taken names"). But the .changed.md fragment should say so, or a Linux user will expect the step everywhere. One sentence is enough, for example: "Separate creates are unchanged: on a case-sensitive disk they still write both names."
  • The user-facing text names an internal function. The new fragment, the #1552 fragment's edit and the VALIDATION_FAILED row in docs/json-contracts.md say "compared with entry_paths.fold". A reader of the changelog or the contract cannot use that. Say what it means, for example: "compared ignoring letter case and Unicode normalisation (and the trailing dots or spaces Windows drops)".
  • _prepare_and_save takes the same set twice. It gets pending_paths=would_create_paths and also taken=frozenset(would_create_paths). One argument is enough: _prepare already receives pending_paths, so _prepare_and_save can work out taken itself. After blocking item 1, that argument would be the folded-key set.
  • _prepare's docstring still says "pending_paths are resolved file paths an earlier item of the same dry run would have created". Since this PR the real write passes them too.

CI

  • mypy (advisory): red because of this PR. See blocking item 2.
  • experimental: red on TestSemanticKnnBudget::test_rows_the_ceiling_hides_are_returned_or_the_result_says_capped[postgres], the known intermittent failure #1401. It has nothing to do with this change.
  • test (3.12, 1), collect and lock pass. test (3.12, 2) and (3.12, 3) were still running at the time of this review. The verify-red job has not reported yet. By reading: the disk-independent test is red without the fix on every disk, and the control test is red without the fix on Linux, the disk CI runs on. The new parity cells agree on dev too, as the issue expected on a case-sensitive disk, so they are not evidence for the fix.
  • A changelog fragment is present: changelog.d/1623-batch-taken-names-fold.changed.md. The #1552 fragment and docs/json-contracts.md are updated as the issue asked.

Recommendation: request changes.

@markramm markramm added the state: needs-author Changes requested on the current head: the next move is the author's label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state: needs-author Changes requested on the current head: the next move is the author's

Projects

None yet

Development

Successfully merging this pull request may close these issues.

On a case-insensitive disk, a dry run answers valid for two ids that differ only in case; the real write steps or refuses the second

2 participants