Skip to content

fix: a field has one JSON type on every reader; list and search rows are shaped by one function (#1503) - #1628

Draft
markramm wants to merge 15 commits into
devfrom
claude/mcp-kb-get-json-types-jrgmmo
Draft

markramm wants to merge 15 commits into
devfrom
claude/mcp-kb-get-json-types-jrgmmo

Conversation

@markramm

@markramm markramm commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1503

Branch: this cloud session was given its own branch name, claude/mcp-kb-get-json-types-jrgmmo, which is used in place of fix/1503-index-rows-one-shape.

From the groom (#1503, ## Groom 2026-11-16):

Acceptance:   (A) for one task and one note with metadata, every tool in the "raw" and "ORM" classes returns metadata as an object and aliases as a list;
              a test parametrised over those tools fails today on kb_find_by_status and kb_search.
Regimes:      NULL metadata; an empty KB; Postgres (pg job).
Shapes:       the metadata/aliases/bag-field shapes above × each reader.

Decision (maintainer, 2026-11-16): option A1, one row-shaping function in pyrite/storage/ that every reader of entry rows returns through; and B option (i): list and search rows cast a TEXT column value by the field's declared type (KBSchema.type_fields) in that shaper, shipped with #730 P5's single read so kb_get and the list readers agree and a read written back never changes a file's type. Option (ii) is not built. A quoted "7" in a file reads as 7: a stated limit.

The plan is a comment on this PR; the report follows.

Report

Branch: claude/mcp-kb-get-json-types-jrgmmo. The session was given this branch name, so it replaces fix/1503-index-rows-one-shape. The work was done in the main cloud checkout, with no other session sharing it.
Pushed: 1bd955063051085e68035cadcbd1907b92b65327 == git ls-remote origin claude/mcp-kb-get-json-types-jrgmmo (checked).

Commits (no rebase, amend or force-push; dev was merged in as a merge commit):

  • 3cdb6af kb: the claim (backlog item, made with the CLI)
  • df1299a test: the row-shape cells, committed before any production code
  • 898d352 fix: every storage reader returns entry rows through one shaper
  • 9203ed2 docs: the "Entry row shape" section and the fragment
  • 150b4cc fix: the shaper adds no key; the lift stays with the ORM row
  • 1e2c350 Merge origin/dev (fix: a write never places an entry where the index walk does not list it (#1552) #1576, 061388c7), before touching index.py
  • e1bd018 refactor: the single read casts a column by its declared type
  • 3270b67 test: mark the 41 negative controls
  • 1bd9550 refactor: type the shaper's calls so mypy adds no error (the advisory ratchet read +6 on 3270b67; now +0)

The backlog item stays status: in_progress, not done. The conductor's instruction was to stop at the report while the stop item is open; it is set to done once the maintainer answers and the task-priority cells are settled.

Unsure

  1. STOP ITEM, open, on the maintainer's desk: the task's priority is not cast. KBSchema().type_fields("task") is {} (confirmed by the conductor on a5f942bd). task is not in CORE_TYPES, and no plugin implements get_field_schemas. So under "the declared type comes from KBSchema.type_fields only", the issue's own case keeps reading "7". It does so on every index reader and on kb_get alike: one type, but not the integer task_* returns. The kb_get → kb_update round trip on a task still rewrites 7 as "7". My recommendation is (a): a task layer in type_fields, read from a fields declaration beside TASK_TYPE_METADATA, the way resolve_type_metadata does at "Layer 1b". That needs pyrite/schema/kb_schema.py, which I did not touch (it belongs to fix: a tagged or out-of-range value in kb_authored is returned or withheld with a true reason (#1568) #1611).
    • How the cells are committed. The issue's integer cases are @pytest.mark.xfail(strict=True, reason=UNDECLARED_TASK_PRIORITY): test_a_task_priority_reads_as_the_integer_in_the_file_on_every_mcp_row_reader (9 readers) and test_a_task_priority_read_and_written_back_leaves_the_file_byte_identical (MCP, CLI, REST). They show as XFAIL on every run. The day a declaration makes the cast reach the task, they XPASS, and strict=True turns that into a failure, so the marker must come off.
    • Why the repo's rules allow it. A strict xfail per known violation is the established pattern (tests/test_storage_invariants.py, tests/test_doc_write_as_patch.py's KNOWN_DIVERGENCES; kb/maps/write-read-path-and-identity.md, "Tests that pin it"). It neither passes silently nor deletes or loosens a test. verify-red counts these 12 as n/a, not as passes.
    • What holds today. The part of the property that holds is pinned as a passing control: test_a_task_priority_reads_as_kb_get_reads_it_on_every_mcp_row_reader, one type across readers.
    • Docs and fragment say that a task's priority reads as text on every index reader and kb_get, so both are true under (b).
  2. A1's "declared fields lifted the same way" against the invariant "a row gains no key". My first version lifted declared bag fields onto raw rows too. That added keys: a search hit gained priority, a * search row gained actors, and 66 existing cells failed (test_reader_reads_where_the_writer_writes.py 62, test_search_contract.py::TestLeanSearchRows 4). I kept the invariant. The shaper decodes metadata and aliases and casts; the lift stays where it was, in _entry_to_dict (with its own get_entry_class call, untouched) and in read_fields. So a backlog item's rank is at its own name on the ORM rows and the single read, and only inside metadata on search and find rows. The maintainer might want raw rows lifted as well, but that is a projection change, which the brief puts out of scope.
  3. The declared types come from the index's kb table. Storage reads the KB's path (SELECT path FROM kb WHERE name = :n), then KBConfig(name, path).kb_schema (kept per name and path, so it is stamp-cached as A kb.yaml that is not UTF-8 or not valid YAML stops every command on every KB #1547 made it), then .type_fields(entry_type). There is no service import, no kb_type argument and no new get_entry_class call. A kb.yaml that does not load reads as the core and plugin layers for a list. Someone might instead prefer the services to hand the schema down.
  4. The lookup is lazy. test_list_entries_issues_two_statements_not_n_plus_one (Index rebuild writes one entry per transaction (2.8 ms/entry); list_entries is N+1 on tags #1044) caught one extra statement per listing. The test is unchanged. The lookup now runs only when a TEXT column holds text a cast could change (a number or a bool literal), once per KB and type per batch, never per row. A listing of task rows with numeric priorities therefore costs one more statement than before.
  5. What a raw reader's row carries, by its SQL. get_timeline selects location without entry_type, so no declared type can apply there. A kb.yaml that declared location: number would read as text on kb_timeline and as a number elsewhere. That is classed as safe by reading: no type declares it, and the subset carries no metadata.

Done, with evidence

  • The red list, before the fix. tests/test_index_rows_have_one_shape.py was run against untouched dev code (a detached worktree at origin/dev with only the test copied in): 42 failed, 40 passed, 1 skipped, 12 xfailed. The red cells:
    • MCP kb_get, kb_list_entries, kb_recent, kb_batch_read, kb_find_by_status, kb_find_by_assignee, kb_find_overdue, kb_find_by_location and kb_search. The groom named kb_find_by_status and kb_search; both are red: aliases '["Reg"]', NULL metadata, priority '7'.
    • CLI get, list-entries, recent, search --fields '*' and batch-read.
    • REST GET /api/entries, GET /api/entries/{id}, POST /api/entries/batch and GET /api/search?fields=metadata,importance.
    • Backend get_entry, get_entries, list_entries, search, search_by_tag, search_by_tag_prefix, search_by_date_range and list_entries_in_folder, on SQLite and Postgres, plus SQLite search_semantic.
    • PyriteDB.find_*.
    • The declared-number round trip on MCP, CLI and REST (the file's bytes changed).
    • "The single read and the list rows agree", "a quoted number reads as a number", and the structural scan.
  • Green after. On the merged head the file shows 836 passed, 1 skipped, 12 xfailed, together with tests/test_test_rules.py. The pre-push affected suite passed on e1bd018 before the push went through.
  • verify-red: 42 red · 0 import-only · 0 unexpected pass · 13 n/a · 41 control. The 13 n/a are the 12 strict xfails and the Postgres search_semantic skip (no vector support in that cell). Each control carries its reason.
  • Postgres. It was running in this container (scripts/cloud-postgres.sh, PYRITE_TEST_PG_URL). The pg cells and tests/backends/test_backend_conformance.py ran here: red before, green after (test_list_entries_issues_two_statements_not_n_plus_one[postgres] included).
  • The read = row pin. tests/test_get_reads_the_file.py::test_on_a_fresh_index_the_file_read_equals_the_row is unchanged and green: both sides cast with the same function and resolver.
  • Characterization goldens: none changed. The suite passed without regeneration. The goldens' entry rows are list and single reads of note-type entries with no declared numbers, and REST /api/search's SearchResult drops metadata.
  • Lint. ruff check and ruff format --check are clean on the changed files.
  • The draft PR's CI on the pushed head (1bd9550, run 44140292857): gate success, verify-red success, test (3.12, 1/2/3) success, mypy (advisory) success (ratchet +0), experimental success, kb success. The red gate on e1bd018 was a run cancelled by the next push, not a test failure.

The readers, and how each is classed (Covers)

Reader Class Where it is guarded
queries.find_by_status/assignee/overdue/location (SQLite, PyriteDB) fixed (raw, now shaped) test_the_find_readers_return_each_field_with_its_files_type; MCP kb_find_* cells
sqlite_backend / postgres_backend search, search_by_tag, search_by_tag_prefix, search_by_date_range, search_semantic fixed (raw) test_a_backend_row_reader_returns_each_field_with_its_files_type[sqlite-*, postgres-*], parametrised from SearchBackend
base_backend.list_entries_in_folder (REST folder collections; aliases was raw text there) fixed (raw) same, [*-list_entries_in_folder]
base_backend.get_entry/get_entries/list_entries (_entry_to_dict) fixed (ORM, now cast) same backend cells; MCP kb_list_entries/kb_recent/kb_batch_read; CLI and REST lists
IndexManager.read_fields (single read) fixed (casts, same resolver) MCP kb_get, CLI get, REST /api/entries/{id}; the round trip; the agree cell; the pin
other SearchBackend methods (31: counts, edges, links, tags, timeline, embedding stats ...) safe by reading: no metadata/aliases/entry_type test_every_backend_method_is_classified (a ratchet with a reason each)
overlay_backend safe: delegates to the main and diff backends and merges backend cells below it
MCP tool table (106 tools at admin tier) 9 row readers, 39 own-dict, 36 no entry rows, 22 investigation_* out of scope test_every_mcp_tool_is_classified; row cells; test_an_own_dict_tool_returns_no_metadata_or_aliases_as_text walks 25 own-dict tools' results for a string metadata/aliases (14 need state these files lack and are listed with None)
CLI --format json: get, list-entries, recent, search --fields '*', batch-read fixed through storage test_a_cli_json_row_reader_...
REST: GET /api/entries, /api/entries/{id}, POST /api/entries/batch, /api/search?fields=... fixed through storage test_a_rest_row_reader_...; /api/search without fields drops metadata (SearchResult): safe by its model
Extensions' readers (sw_*, wiki_*, social_*, zettel_*) own dicts over their own SQL (db._raw_conn) or ORM reads own-dict walk where callable; the structural scan covers pyrite/storage only
Internal: IndexManager._check_off_list_values (SELECT * via execute_sql) returns findings, never rows STRUCTURAL_ALLOWLIST, with its reason
Structural test_every_storage_reader_of_entry_rows_returns_through_the_shaper: any function in pyrite/storage with entry-row SQL must call shape_entry_row(s)
A raw row gains no key test_a_raw_readers_row_gains_no_key_its_sql_did_not_select (red against the lifting version)

Groom shapes covered:

  • metadata: {}, a nested object, and NULL (set on the row).
  • aliases: NULL, [] and a list.
  • A bag-held declared field (rank).
  • A declared number in a TEXT column; a quoted number; a non-number in a number field (stays text).
  • SQLite and Postgres.
  • Not covered: an empty KB.

Footprint against the groom's "Touches"

  • Inside it: storage/queries.py, storage/backends/sqlite_backend.py, postgres_backend.py, base_backend.py, services/task_service.py, services/rubric_checkers.py, server/endpoints/collections.py, docs/json-contracts.md (a new "Entry row shape" section only), the new test and the fragment.
  • Plus: the single read's one function (storage/index.py read_fields, after merging fix: a write never places an entry where the index walk does not list it (#1552) #1576) and the backlog item.
  • Not touched: kb_service.py (_normalize_metadata_rows is now a no-op and stays), kb_schema.py, core_types.py.
  • 12 files in all; the groom predicted about 10.

Left

  • The task-priority declaration (stop item 1).
  • An empty-KB regime cell.
  • The 14 own-dict tools listed with None are classified but not called: they need state (epics, components, an encyclopedia or social KB) these files lack.
  • kb_service._normalize_metadata_rows could be deleted. It is fix: a write never places an entry where the index walk does not list it (#1552) #1576's file, so I left it.
  • Found and not fixed: a plugin's MCP tool context is process-wide and names the database of the last PyriteMCPServer built. In a process with two servers, sw_* answered INTERNAL ("Cannot operate on a closed database") once the other server's world closed. Captured in: the comment on _mcp in the test. Worth an issue if the maintainer agrees.

Learned

Learned Captured in
KBSchema.type_fields("task") is {}; no plugin declares field schemas, so type_fields alone cannot type a built-in column field UNDECLARED_TASK_PRIORITY and the strict xfails; the docs row-shape paragraph; the plan comment
The groom's four _parse_metadata copies were already wrappers over utils.metadata.parse_metadata; it is now the only decoder (decode_metadata in storage is that function) comment at the _INTEGER_TEXT block in base_backend.py; the rubric_checkers comment
MCP kb_search fields takes a list: a string such as "*" is accepted and silently projects to id/kb_name; REST and MCP treat * as an unknown name, and only the CLI reads it as "all" the test's _rows_check now requires metadata on every checked row, so a vacuous cell fails
Lifting declared fields in a shared shaper changes projections; the lift belongs to the ORM row and the single read shape_entry_row docstring; test_a_raw_readers_row_gains_no_key_...; the fix commit message
The N+1 pin counts statements, so a per-batch schema lookup must be lazy the comment at the _castable gate in shape_entry_row
A note's status lives in its bag, not the column (no status field on a note), so the finds do not return it comment in test_the_find_readers_...
The keyword search row's rank is bm25 and shadows a backlog item's declared rank the docs row-shape table; _skips_for
A plugin's MCP context is process-wide (above) comment on _mcp in the test

Tokens

Not reported by the harness.

Report (round 1)

The property. An index reader returns a field's value with the JSON type the file gives it under the KB's declared schema. A read followed by a write-back of the value read never changes the file, with exactly one exception: a quoted plain number in a field declared a number. priority: "7" reads 7, and a write-back of that 7 stores priority: 7 without the quotes. qa validate reports such a value (number_as_text).

Branch: claude/mcp-kb-get-json-types-jrgmmo.
Pushed: 1cfb97d5e33d3f01762672ec88bd4eec1d27c06e == git ls-remote origin claude/mcp-kb-get-json-types-jrgmmo (checked).

Commits this round (no rebase, amend or force-push):

  • 01ccd55 Merge origin/dev (brings in fix: a tagged or out-of-range value in kb_authored is returned or withheld with a true reason (#1568) #1611, so kb_schema.py is free)
  • 664da3f fix: cast only plain ASCII numbers; a task declares its priority a number
  • 20cf15c fix: qa validate names text in a field declared a number
  • c5b7fe2 test: backend cells seeded as the indexer binds; the structural guard reads the AST
  • dd4b74d docs: the row shape says what the strict cast and a write-back do
  • 1cfb97d test: a REST GET put back whole leaves the file byte-identical

The decisions as built

  • Finding 1 = (a). read_declared_number (pyrite/storage/backends/base_backend.py) casts only 0|[1-9][0-9]* (ASCII), and decimals (0|[1-9][0-9]*)\.[0-9]+ whose float spells them back exactly (repr(float(t)) == t). +7, 007, 7, 7., 1e3, ٧, 7, -7 and 7.50 stay the file's text.
  • The stated exception is pinned. test_a_declared_number_read_and_written_back_leaves_what_the_rule_says runs 12 spellings × MCP kb_update, CLI -f and REST PATCH, comparing bytes. Every spelling that stays text leaves the file byte-identical. "7" leaves priority: 7.
  • Docs and changelog state the exception exactly and no longer say the file keeps its quotes.
  • qa validate. It did not report this before: _validate_field_value runs float(value), so "7" and "007" passed (checked on this branch before the change). A new number_as_text warning sits in the pass that reads each file's own values (QAService._check_number_as_text). It names numeric-looking text in any field the type declares a number, and says what Pyrite reads it as. Writes are unchanged. Guarded by test_qa_validate_reports_text_in_a_field_declared_a_number.
  • MCP list and search tools return priority and metadata with different JSON types than kb_get #1503 = (a). task declares one field: priority, type number ("int" in TASK_TYPE_METADATA["fields"], pyrite/models/task.py). KBSchema.type_fields reads it beside CORE_TYPES, the way resolve_type_metadata reads a task's settled statuses. No default is declared. MCP kb_create, task_create and CLI create produce byte-identical task files on dev and on this head (compared; pinned by test_declaring_a_task_priority_adds_nothing_to_a_created_tasks_file).
    • I declared only priority: it is the one TEXT-column field of a task with a non-text type. status, assignee and due_date read as text either way. Declaring the list and dict fields would feed KBService._with_declared_fields (the create path) for no gain.
    • The 12 task-priority cells that were strict xfails now pass as ordinary cells.

Enumeration: every place that casts or shapes a row, and every write-back surface

Place Class Guard
shape_entry_row(s) (decode metadata and aliases; cast by declared type) the one rule everything below
IndexManager.read_fields (single read: kb_get, CLI get, REST /api/entries/{id}) fixed: casts with the same function and resolver MCP, CLI and REST row cells; test_a_declared_numbers_file_spelling_reads_by_the_strict_rule (12 spellings); the read = row pin (unchanged)
_entry_to_dict (ORM: get_entry, get_entries, list_entries) fixed: shaped, then lifted as before backend cells (SQLite and Postgres); MCP kb_list_entries/kb_recent/kb_batch_read
queries.find_by_status/assignee/overdue/location fixed test_the_find_readers_...; MCP kb_find_*
SQLite and Postgres search, search_by_tag, search_by_tag_prefix, search_by_date_range, search_semantic fixed backend cells, parametrised from SearchBackend
base_backend.list_entries_in_folder fixed backend cells; the structural mutation check
Values the indexer binds (crud.bindable: int, float, bool) fixed test_typed_values_the_indexer_binds_read_the_same_on_every_backend (SQLite and Postgres, run here)
get_timeline (kb_timeline) safe by reading: it selects no entry_type, so no declaration applies; date and location return as text (now in the docs) STRUCTURAL_ALLOWLIST, with its reason
check_health, _check_off_list_values (index health) safe: they return findings, not rows allowlist
kb_ops.get_kb_stats safe: k.* is the kb table allowlist
Other SearchBackend methods safe: no shaped column test_every_backend_method_is_classified
MCP tool table (106 tools) 9 row readers, 39 own-dict, 36 no rows, 22 investigation_* out of scope test_every_mcp_tool_is_classified; own-dict walk
Write-back: MCP kb_update guarded the spelling write-back cells; the task and declared-number round trips; the other-field update
Write-back: CLI update -f guarded same, CLI column
Write-back: REST PATCH guarded same, REST column
Write-back: REST PUT safe by reading: it accepts title, body, importance, tags and metadata, never a column field; guarded anyway test_a_rest_get_put_back_whole_leaves_the_file_byte_identical (a nested bag; a task with a quoted priority)
Structural rule test_every_storage_reader_of_entry_rows_returns_through_the_shaper now requires a call in the AST. The SQL pattern is widened to any SELECT ... FROM/JOIN entry returning * or a shaped column; an allowlist with reasons; it asserts the known readers are seen. Removing the call from list_entries_in_folder (with the name left in a comment) makes it fail. The old text check passed that.

Evidence

  • scripts/test-affected --run on the tree of dd4b74d (the last commit after it adds one control cell): 26711 passed, 257 skipped, 7 xfailed, 0 failed.
  • verify-red: 72 red · 0 import-only · 0 unexpected pass · 1 n/a · 71 control. The n/a is the Postgres search_semantic skip. Every control carries its reason; the spelling cells that keep text are controls, red against the loose cast, not against dev.
  • Mutation of the checkbox. With _CHECKBOX_TEXT cut to "1"/"0", the three Postgres cells of test_typed_values_the_indexer_binds_... fail and the SQLite ones pass (run here).
  • Mutation of the cast. With the first version's loose rule restored, the read cells for "+7", " 7 ", "007", "٧", "7" and "-7" fail. "1e3", "7." and "7.50" are held by the decimal and round-trip checks.
  • Postgres ran here (scripts/cloud-postgres.sh), including the new crud.bindable cells.
  • Characterization goldens: none changed.
  • CI on the pushed head: started when this report was posted; its result follows as a comment on this PR.

Unsure

  1. Checkbox reads "1", "0", "true" and "false", not "1"/"0" only. The round brief restated my 19:12Z proposal of "1"/"0", but that proposal was wrong for Postgres: its assignment cast stores a Python True in a TEXT column as "true" (SQLite binds it as 1). test_typed_values_the_indexer_binds_read_the_same_on_every_backend[postgres-*] fails with "1"/"0" only. So the cast accepts exactly those four spellings, no folding and no whitespace. The cost: a quoted "true" in a checkbox field reads as a bool on either backend (the same class of limit as "7"). If the maintainer wants "1"/"0" only, Postgres checkboxes read as text.
  2. A decimal must round-trip. "7.50" stays text because its float writes back as 7.5. That is stricter than "plain decimals", in the direction of no respelling.
  3. A whole-valued float differs by backend. SQLite keeps 7.0; Postgres prints 7. Both are JSON numbers, but a write-back on Postgres could respell 7.0 as 7. This is stated in the docs and not pinned.
  4. Files outside the groom's Touches, by decision. pyrite/schema/kb_schema.py and pyrite/models/task.py (decision (a)); pyrite/services/qa_service.py (the decision's qa validate report).

Notes (from the cold read)

  • Falsy values. A cast 0 or False is falsy where the text "0" was truthy. software-kb's row["priority"] or meta.get(...) reads rows from its own SQL (db._raw_conn), which this change does not reach. A backlog item's priority is a select (text). Any new consumer of shaped rows must test is None, not truthiness.
  • The overlay. Declared types come from the index's kb.path, so an overlay's diff rows are cast by the worktree's kb.yaml and main rows by main's. A change set that edits a declaration reads its own rows by its own schema.
  • A broken kb.yaml. One that fails to load is read, for a listing, as the core and plugin layers. KBConfig.kb_schema logs the failure once per file; a single read of that KB refuses it (A kb.yaml that is not UTF-8 or not valid YAML stops every command on every KB #1547).
  • kb_search and the finds now return metadata as an object (in the changelog).

Learned

Learned Captured in
qa validate's number check is float(value), so numeric text in a declared number was never reported QAService._check_number_as_text docstring; its test
Postgres's TEXT spelling of a bool is true, SQLite's is 1; a whole float prints 7 on Postgres, 7.0 on SQLite _CHECKBOX_TEXT comment; test_typed_values_the_indexer_binds_... docstring; the docs
The single read returns ruamel's own string types (DoubleQuotedScalarString) the comment in test_a_declared_numbers_file_spelling_reads_by_the_strict_rule
A text-search structural guard passes a reader that only names the shaper in a comment _calls in the structural test; the mutation check above

Left

  • An empty-KB regime cell (as before).
  • The 14 own-dict tools listed with None are classified but not called.
  • The plugin MCP context hazard (process-wide, the last server's database), as before.

🤖 Generated with Claude Code

https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf

Copy link
Copy Markdown
Collaborator Author

Plan (#1503)

Checked against dev at aa7deeb. Reproduced the groom's table there over the real MCP dispatcher: kb_find_by_status and kb_find_by_assignee return metadata as '{"colour": "blue"}', aliases as '["Reg"]' and priority as '7'. kb_search returns metadata as a string. kb_list_entries and kb_get return '7' with the object. task_list returns 7. Sending kb_get's priority back through kb_update rewrote priority: 7 as priority: "7".

First, the riskiest assumptions

  1. Does any consumer depend on metadata being a string? No. I grepped pyrite/services, pyrite/server, pyrite/cli and extensions/*/src and found no string matching on a returned value. Every json_extract/LIKE on metadata runs in SQL against the stored column. Every decoder on a row the storage readers return is tolerant (utils.metadata.parse_metadata, or an isinstance(..., str) guard): mcp_server.py _task_status, cli/task_commands.py _task_get_impl, cli/collection_commands.py, qa_service.py, endpoints/entries.py (aliases). The extensions' many unguarded json.loads(row["metadata"]) calls (software-kb, encyclopedia, social) all read a sqlite3.Row from their own SQL through db._raw_conn, so no storage change reaches them. The one consumer that needs a dict is EntryResponse.metadata: dict, which is why KBService._normalize_metadata_rows exists.

  2. STOP ITEM: KBSchema.type_fields("task") returns {} on today's dev. task is not in CORE_TYPES; it is declared in models/task.py, which has no fields. No plugin implements get_field_schemas (get_all_field_schemas() is {}). So with "the declared type comes from KBSchema.type_fields and nowhere else", the headline case (a task's priority: 7) has no declared type, and option (i) would cast nothing for it. Only a KB whose kb.yaml declares the field casts (verified: types: {chore: {fields: {priority: {type: number}}}} gives type_fields("chore") == {"priority": "number"}, and the row's TEXT column holds '7'). Declaring task.priority as a number where type_fields reads it needs pyrite/schema/kb_schema.py (owned by fix: a tagged or out-of-range value in kb_authored is returned or withheld with a true reason (#1568) #1611 and A scalar written to a list[str] field is stored as a string, and qa validate does not flag it #1502) or pyrite/schema/core_types.py. Both are outside the groom's "Touches". Needs a decision:

    I'm going ahead with everything that does not depend on this (below). Until it is answered, the task-priority cells of the new test stay red and are not committed as passing.

The readers, and how each is classed

raw means fixed here by returning through the shaper. ORM means fixed here because _entry_to_dict calls the shaper. own means the reader builds its own dict. safe means safe by reading. Cells in tests/test_index_rows_have_one_shape.py.

Reader Today Class / action
storage queries.find_by_status/assignee/overdue/location raw dict(zip) raw: shape
sqlite_backend and postgres_backend search, search_by_tag, search_by_date_range, search_by_tag_prefix, search_semantic raw raw: shape
base_backend.list_entries_in_folder raw raw: shape (REST folder collections; aliases is a raw string there today)
base_backend.get_entry/get_entries/list_entries (_entry_to_dict) ORM ORM: _entry_to_dict returns through the shaper
IndexManager.read_fields (the single read) READ casts through the same cast function (below)
get_timeline, get_backlinks/outlinks, get_orphans, get_most_linked, edges, refs, tags subsets / link rows: no metadata, aliases or entry_type safe: the cell asserts they carry no metadata/aliases key
overlay_backend delegates to main/diff backends and merges safe: shaped below it
MCP tool table (106 tools at admin tier) every tool gets a cell, parametrised from server.tools
kb_get, kb_list_entries, kb_recent, kb_batch_read, kb_find_* x4, kb_search (keyword; semantic where embeddings are available) row readers: compared against the file
task_list, task_status, task_subtree, task_ancestors, task_blocked_by, task_critical_path, sw_*, wiki_*, social_*, zettel_*, kb_orient, kb_discover_neighbors, kb_batch_suggest, kb_timeline, kb_backlinks own dicts / subsets own: listed with what they return; checked that no metadata they return is a string
writes, admin, QA, registry tools no entry rows listed
investigation_* out of scope (experimental)
CLI --format json: search --fields '*', get, list-entries, recent, batch-read, collections query as above row cells
REST GET /api/entries, POST /api/entries/batch, /api/entries/{id}, /api/collections/{id}/entries (folder), /api/search?fields=* as above row cells (/api/search without fields drops metadata in SearchResult: safe)

A structural test scans pyrite/storage for SQL that selects entry rows (SELECT *, e.*, e.metadata from entry) in a function that does not return through the shaper. A small allowlist with reasons covers internal scans that are never returned to a caller (IndexManager's health scan).

The shaper

  • shape_entry_row(row, field_types) and shape_entry_rows(rows, session) go in pyrite/storage/backends/base_backend.py, beside _entry_to_dict (groom A1). For each key the row already has: metadata is decoded to an object (NULL or bad JSON reads {}), aliases to a list, declared fields are lifted with the existing lift_declared_fields, and a TEXT column value is cast by its declared type.
  • The cast: number turns an integer literal into int and a finite float literal into float, and leaves any other text as it is. checkbox turns "1"/"0"/"true"/"false" into a bool. Everything else is unchanged.
  • No key is added or removed, apart from the declared-field lift, which A1 decides and which ORM rows already do. base_backend._parse_metadata and _parse_aliases become the shaper's own decoders. The consumer wrappers in collections.py, task_service.py and rubric_checkers.py already delegate to utils.metadata.parse_metadata (the groom saw four copies; today they are three wrappers over one function). They become tolerant no-ops with a comment saying why they stay: an index row is already a dict, and a hand-built dict in a test or plugin may not be.
  • lift_declared_fields needs the type's class. The shaper calls get_entry_class in the one place _entry_to_dict calls it today: the call moves, and no call is added (read_fields passes type(entry)).

How the declared type reaches it, inside the layer rule

Storage resolves it itself, from what it already imports (pyrite.config, pyrite.schema; no services):

  • the KB's path comes from the index's own kb table (SELECT path FROM kb WHERE name = :n, the same on both backends);
  • the schema comes from a KBConfig(name, path).kb_schema kept per (name, path), so it is cached and stamp-checked exactly as the config loader's own is;
  • the types come from .type_fields(entry_type), memoised per (kb, type) for one batch.

A kb.yaml that does not load falls back to KBSchema().type_fields (the core and plugin layers) for a list, because a listing must not fail on it. The single read refuses that KB anyway (#1547). There is no new kb_type argument and no new get_entry_class call.

The single read

IndexManager.read_fields keeps building its result as today (_as_column). Then, after read_metadata, it runs the same cast function with the same resolver, inside that one function. For a declared number, _as_column(7) == "7" casts back to 7, so kb_get equals the list rows, and the value written back is the file's value: the round trip leaves the bytes alone. The pin test_on_a_fresh_index_the_file_read_equals_the_row should hold unchanged, since both sides cast. If it needs to change, it changes in its own commit, with the reason. A quoted "7" in the file reads as 7: the stated limit, in one sentence each in docs/json-contracts.md (row shape) and in the fragment.

Pinned tests expected to change

None loosened. The characterization goldens get regenerated only if a key changes type; each such key will be listed in the report. tests/backends/test_backend_conformance.py::test_entry_metadata already accepts both forms, so it is left alone.

Files, against the groom's "Touches"

What the groom got wrong on today's dev

  • The consumer _parse_metadata copies are wrappers over utils.metadata.parse_metadata, not four decoders.
  • KBService._normalize_metadata_rows is an existing per-call fix for the folder collection route. It decodes metadata but leaves aliases a raw string.
  • type_fields declares nothing for task (the stop item above).
  • The keyword search row's rank is bm25. A backlog item's declared rank therefore cannot be lifted onto a kb_search row (lift_declared_fields never overwrites a key), and renaming the key is a projection change, which is out of scope. The rank cell for kb_search is recorded as that known collision, not as a pass.
  • The line numbers have moved (queries.py find_* at 367/403/446/484).

Postgres is running in this container (scripts/cloud-postgres.sh), so the pg cells and tests/backends/test_backend_conformance.py will run here.


Generated by Claude Code

@markramm

Copy link
Copy Markdown
Collaborator Author

Conductor, answering the STOP ITEM in your plan on PR #1628 (2026-11-17). Only the stop item is answered here; the rest of the plan is yours to carry out and will be read cold when you report.

Your finding is confirmed on dev at a5f942b: KBSchema().type_fields("task") is {}. Whether task declares its fields where type_fields reads them (your a) or this ships without it (your b) is a choice between designs, and the maintainer keeps those. It is on his desk with your recommendation (a). Until an answer is posted on this PR:

  1. Carry on with everything that does not depend on it, as you planned.
  2. Do not change pyrite/schema/kb_schema.py or pyrite/schema/core_types.py. kb_schema.py belongs to PR fix: a tagged or out-of-range value in kb_authored is returned or withheld with a true reason (#1568) #1611 until that lands.
  3. The task-priority cells: commit them in a form that is visible and cannot pass silently (they fail loudly the day the cast arrives), not as passing and not left out. Say in the report which form you used and why the repo's rules allow it.
  4. The fragment and docs must not say a task's priority is a number on every reader while those cells are open. Write the sentence so it is true under (b); it is tightened if (a) is approved.
  5. If you finish everything else before the answer, post the report with the stop item at the top under "Unsure" and stop there.

Cells parametrised from the MCP tool table, the SearchBackend protocol
(SQLite and Postgres), the CLI --format json and REST row readers, plus the
kb_get -> kb_update round trip with the file's bytes compared and a
structural scan of pyrite/storage for entry-row SQL that bypasses the
shaper. The task-priority cells are strict xfails: KBSchema.type_fields
declares nothing for task today (stop item on #1628).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf
shape_entry_row (storage/backends/base_backend.py, beside _entry_to_dict)
decodes metadata to an object and aliases to a list, lifts declared fields,
and casts a TEXT column's value by its declared type from
KBSchema.type_fields. The ORM rows (_entry_to_dict), the raw readers
(queries find_*, search, search_by_tag/_prefix/_date_range, semantic,
list_entries_in_folder) on SQLite and Postgres all return through it.

The declared types reach storage without a service: the KB's path from the
index's own kb table, its schema from KBConfig.kb_schema (stamp-cached),
looked up once per (KB, type) per batch, and only when a column holds text
a cast could change, so a listing stays two statements (#1044).

The consumer _parse_metadata wrappers go: utils.metadata.parse_metadata is
the one decoder, and the shaper's.

The test's MCP helper builds a server per call: a plugin's tool context is
process-wide and named the closed database of another world's server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf
docs/json-contracts.md gains an "Entry row shape" section: metadata an
object, aliases a list, declared bag fields lifted, a column field read by
its declared type; the quoted-number limit, and what a task's priority
reads today (not declared in KBSchema.type_fields: text on every index
reader and kb_get alike).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf

Copy link
Copy Markdown
Collaborator Author

In progress. This head is not a reviewable state: every storage reader (ORM and raw, SQLite and Postgres) now returns rows through shape_entry_row, but the single read (IndexManager.read_fields in storage/index.py) does not cast yet. On this head kb_get and the list readers can still disagree on a kb.yaml-declared number. The brief says to merge dev once #1576 lands and before touching index.py. #1576 is still open, so that cast is the next commit. The 8 cells red on this head are exactly the single-read ones (kb_get, CLI get, REST /api/entries/{id}, and the round trip). The 12 task-priority cells are strict xfails, as the conductor asked.


Generated by Claude Code

Lifting declared bag fields onto raw rows added keys a reader never
selected (a search hit gained `priority`, a `*` search row `actors`):
tests/test_reader_reads_where_the_writer_writes.py (62 cells) and
tests/test_search_contract.py::TestLeanSearchRows (4) failed on it. The
shaper now only decodes metadata/aliases and casts by declared type; the
lift stays in `_entry_to_dict` (and `read_fields`), with its own
get_entry_class call, as test_the_row_shaping_and_the_file_read_share_the_one_lift_rule
pins. A new cell pins that a raw reader's row gains no key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf
IndexManager.read_fields runs cast_declared_columns with the same
DeclaredFieldTypes the row readers use, after read_metadata. kb_get, CLI
get and REST /api/entries/{id} now agree with every list and search row,
and a kb.yaml-declared number read and written back leaves the file's
bytes alone. The cells in tests/test_index_rows_have_one_shape.py
(committed in df1299a) go green with this commit; the read = row pin in
tests/test_get_reads_the_file.py holds unchanged, both sides cast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf
verify-red listed 41 cells that pass without the fix: the tool-table and
backend ratchets, the own-dict walks, the task-priority agreement, the
other-field write, the no-lossy-cast and no-new-key pins. Each is a
deliberate control and now says why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf

Copy link
Copy Markdown
Collaborator Author

The red gate on e1bd018 is CI run 44129513093. That run was superseded by the push of 3270b67, every job in it was cancelled, and gate fails on cancelled jobs. No test failed there. The run for the current head (3270b67, run 44130179321) is the one that counts.


Generated by Claude Code

The advisory mypy ratchet read +6 against dev: KBConfig's path as a str,
the ORM row dict typed object, and four new `self.session` reads on
QueryMixin. Each find now binds the session once; the ratchet is +0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf

Copy link
Copy Markdown
Collaborator Author

Report posted at 1bd9550


Generated by Claude Code

@markramm

Copy link
Copy Markdown
Collaborator Author

Cold read of 1bd9550 (Opus, read-only; no tests run): the property holds with limits; not landable yet. Every storage reader that returns whole entry rows goes through the one row shaper, and the list readers agree with the single read. Two claims go further than the code, and one limit is worse than dev.

1. Worse than dev, silent (blocks landing under the limits rule). The cast (base_backend._as_number) strips whitespace and accepts any Unicode digit: "007", "+7", " 7 ", "7.", "1e3", "٧" and "7" all read as numbers. A single-field write-back (MCP kb_update {priority: 7}, CLI -f, REST PATCH) then writes the value as sent, so a file holding priority: "7" or "007" becomes priority: 7 after a read and a write of that field. On dev the read gave "7" and the file kept its spelling. The new test checks only that the read leaves the file alone; it never writes back. The changelog and docs/json-contracts.md say "the file keeps its quotes", which is not true of this case.

2. No worse than dev. get_timeline returns date and location uncast and is not shaped (it does not select entry_type); "on every index reader" claims more than the code does.

3. Postgres, by reading only. A bool and a float stored through crud.bindable are spelled differently on Postgres and SQLite, so a float field may read as an int on one and a float on the other. The pg cell seeds the string "7" and cannot catch it.

Tests. test_a_task_priority_reads_as_kb_get_reads_it_on_every_mcp_row_reader passes on an empty loop (it never asserts a value was found). The structural test only checks that the text shape_entry_row appears in the function, and only sees readers that select *, e.* or metadata.

Notes. The declared types are looked up from the index's kb.path, so an overlay's diff rows are cast by the worktree's kb.yaml and main rows by main's; a kb.yaml that fails to load is read as the default schema with only a log line. A cast 0 or False is now falsy where "0" was truthy (row["priority"] or meta.get(...) in software-kb). kb_search and the finds now return metadata as an object (in the changelog).

Still open with the maintainer: #1503 (a) or (b) for the task type's undeclared fields. A fix round follows when that and finding 1's direction are settled.

🤖 Posted by the maintainer's conductor agent (Claude Code)

Copy link
Copy Markdown
Collaborator Author

Thanks for the cold read. I'm holding the fix round until finding 1's direction and the task (a)/(b) question are settled, as you said. Here is what I'd do, so the round can start as soon as they are.

Finding 1: proposed direction. Cast only text that round-trips. A TEXT value becomes a number only when the number's canonical text is the column's text exactly: str(int(t)) == t for an int (ASCII -?(0|[1-9][0-9]*), no +, no leading zeros, no whitespace), and repr(float(t)) == t for a finite float.

  • "007", "+7", " 7 ", "7.", "1e3", "٧" and "7" stay text, exactly as on dev. None of them can then be rewritten by a write-back.
  • Checkbox narrows the same way: "1"/"0" only, which is what the index stores for a YAML bool.
  • The one case left is the decided limit itself: a file with priority: "7" reads 7, and a single-field write-back of that 7 unquotes it. The docs and the fragment will say exactly that, instead of "the file keeps its quotes" (that is true of a read, not of a write-back).
  • Tests: the round trip gets a write-back cell for each of those spellings (bytes compared). For "007", "+7", " 7 " and the Unicode digits it asserts the file is byte-identical after read then write; for "7" it asserts the stated unquoting.

The rest:

  • Finding 2. I'll reword "every index reader" to name the readers that carry entry rows. The docs will say that kb_timeline returns date/location uncast (it selects no entry_type).
  • Finding 3. I'll seed the pg cell through the real indexer path (crud.bindable) with a float and a bool field, so a SQLite/Postgres spelling difference shows up as a red cell rather than only by reading.
  • The empty-loop control. It will assert the task's priority was found on each reader before it compares types.
  • The structural test. I'll match a call to shape_entry_row(s) in the AST (not the text), and widen the SQL pattern to any SELECT ... FROM entry that selects metadata, aliases or a TEXT field column. A narrower projection will need an allowlist entry with its reason.
  • Notes. I'll say in the report that a cast 0/False is falsy where "0" was truthy, and that software-kb's row["priority"] or meta.get(...) needs checking. That code reads its own SQL rows, so this change doesn't reach it today. I'll also note the overlay's per-worktree kb.yaml, and the broken-kb.yaml fallback.

Generated by Claude Code

@markramm

Copy link
Copy Markdown
Collaborator Author

Decisions (maintainer, 2026-11-17). Finding 1: (a) narrow the cast and accept the rest. Only plain ASCII whole numbers and decimals are cast (no surrounding whitespace, no sign prefix, no leading zeros, no exponent, no digits from other scripts), the same strictness as the importance rule in #1588. A quoted "7" in a field declared a number still reads as 7 and a write-back may store 7: the docs and the changelog say exactly that (they must not say the file keeps its quotes), and qa validate reports text in a declared number field. A test writes the read value back and pins what the file then holds. #1503: (a) declare the task type's field types in this PR, so the cast covers task priority. A fix round follows with both.

🤖 Recorded by the maintainer's conductor agent (Claude Code)

…mber (#1503)

The maintainer's decisions on #1628 (2026-11-17):
- Finding 1 of the cold read: only plain ASCII whole numbers and decimals
  are cast (no sign, whitespace, leading zero, exponent or other scripts;
  a decimal only when its float spells it back). "007", "+7", " 7 ", "1e3",
  "٧", "7", "7.50", "-7" stay the file's text, so a read written back
  cannot respell them. A checkbox reads only the backends' own spellings
  ("1"/"0" on SQLite, "true"/"false" on Postgres).
- #1503 (a): task declares `priority: int` in TASK_TYPE_METADATA["fields"],
  read by KBSchema.type_fields beside CORE_TYPES, with no default (a created
  task's file is unchanged: pinned).

The task-priority cells stop being strict xfails and pass. New cells pin
each spelling's read and what a write-back leaves in the file: identical,
except a quoted "7", which reads 7 and is stored 7 (the stated limit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf
The decision on #1628 accepts that a quoted "7" in a declared number reads
as 7 and a write-back stores 7, and asks `qa validate` to report the text.
A new `number_as_text` warning, in the pass that reads each file's own
values, names every numeric-looking text value in a field the type
declares a number (a kb.yaml number, a task's priority) and says what
Pyrite reads it as. Text that is no number stays the schema pass's
`field_number` and the task-priority check's. Writes are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf
… reads the AST (#1503)

Cold-read finding 3: the pg cell seeded the string "7" and could not see
the backends' spellings. New cells seed an int, a float and a bool through
crud.bindable, as the indexer does, and check a declared number and a
declared checkbox read the same on SQLite and Postgres.

The structural test now requires a call to shape_entry_row(s) in the
function's AST (a comment naming it no longer passes: checked by removing
the call from list_entries_in_folder), matches any SELECT that returns `*`
or a shaped column from or joining `entry`, and asserts it sees the known
readers. Four allowlisted functions say why they are not shaped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf
)

docs/json-contracts.md and the fragment no longer claim "the file keeps
its quotes": a quoted "7" in a declared number reads as 7 and a write of
that 7 stores it unquoted, and qa validate reports it (number_as_text).
They state the plain-ASCII rule, that a task's priority is declared a
number, the 7.0 spelling the backends differ on, and that kb_timeline
returns date and location as text (cold-read finding 2) instead of
claiming every index reader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf
@markramm

Copy link
Copy Markdown
Collaborator Author

Fix round 1 of 3 for #1628 (#1503), from the conductor. The maintainer decided both open questions at 19:28Z (comment above). This is the round. Your 19:12Z plan already covers most of it.

Property (state it first in your report, then enumerate): an index reader returns a field's value with the JSON type the file gives it under the KB's declared schema. A read followed by a write-back of the value read never changes the file, with exactly one exception, which must be stated: a quoted plain number in a field declared a number (priority: "7" reads 7, and a write-back may store 7). List every place that casts or shapes a row and every write-back surface (MCP kb_update, CLI -f, REST PATCH and PUT). Classify each: guarded by a test, safe by reading (say why), or fixed here.

Decision 1, finding 1 = (a): only plain ASCII whole numbers and decimals are cast. That means no surrounding whitespace, no sign prefix, no leading zeros, no exponent and no digits from other scripts, the same strictness as the importance rule in #1588. Your round-trip rule (str(int(t)) == t, repr(float(t)) == t for a finite float) fits this, as long as +7, 007, 7, 7., 1e3, ٧ and 7 all stay text. Checkbox casts "1"/"0" only.

  • The docs and the changelog must NOT say the file keeps its quotes. They state the one exception exactly.
  • qa validate reports text in a field declared a number. Check whether it does today; if not, add it, with a test.
  • A test writes the read value back and pins the file's bytes. The file must be byte-identical for each spelling that stays text, and hold the stated unquoting for "7".

Decision 2, #1503 = (a): declare the task type's field types in this PR, so the cast covers task priority. kb_schema.py is free (#1611 has merged; merge dev first by a new merge commit if you are behind). Name which task fields you declare, and with what types, in the report.

Also from your 19:12Z plan, all in: reword "every index reader" to name the readers that carry entry rows, and say kb_timeline returns date/location uncast. Seed the pg cell through crud.bindable with a float and a bool field. Make the empty-loop control assert that the priority was found before it compares. Make the structural test match the shape_entry_row(s) call in the AST, and widen the SQL pattern as you proposed, with an allowlist and reasons. Put the notes (falsy 0/False, the per-worktree kb.yaml, the broken-kb.yaml fallback) in the report.

Rules: every commit passes the commit hooks (never --no-verify on a commit). A fix: commit touches tests/; otherwise use test:, docs: or refactor:. No force-push. Run the suite only as scripts/test-affected --run (it holds the machine-wide lock), -n 4 at most. When done, push, append ## Report (round 1) to the PR body with the pushed SHA, the enumeration table, Learned, and Unsure. A delta cold read follows.

REST PUT is the fourth write-back surface the round names. It accepts
title, body, importance, tags and metadata, never a column field, so the
cell sends back what a GET returned for those and compares bytes, for a
nested bag and a task with a quoted priority. A control: it held before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BnEiWHHJjCS2XCy3CjZqf

Copy link
Copy Markdown
Collaborator Author

Round 1 report posted at 1cfb97d (## Report (round 1) in the PR body). CI on this head is still running; I'll add its result here when it finishes.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP list and search tools return priority and metadata with different JSON types than kb_get

2 participants