Release 0.6.0: reliable ingestion and auditable retrieval

CHANGELOG.md

@@ -1,5 +1,34 @@
 # Changelog
 
+## 0.6.0
+
+### Added
+
+- Episode ingestion retries transient model and database failures with bounded
+  exponential backoff. Every attempt reuses a content-derived episode UUID, successful
+  writes return and persist canonical episode/node/edge receipts, and exhausted failures
+  are appended to a durable failure ledger for reconciliation.
+- Staged extraction writes an `accept` / `refuse` / `contested` decision template for
+  every fact edge. Promotion refuses unresolved decisions, filters refused facts,
+  re-seals the approved snapshot, and records both the source and decision digests.
+- `kg-ingest` accepts Markdown, text, JSON, JSONL, and directory trees. Text is split at
+  stable paragraph and sentence boundaries before it reaches the model; empty or
+  unsupported inputs fail explicitly instead of reporting a zero-record success.
+- CLI and MCP retrieval responses include candidate, eligible, returned, and suppressed
+  counts plus `recall_may_be_incomplete` when the bounded candidate ceiling was reached.
+- OpenAI-compatible local models receive schemas whose nullable properties are present
+  in `required`, including nested temporal fields, so constrained decoding emits a key
+  with `null` rather than silently omitting it.
+- Entity fuzzy-match ties use a stable UUID ordering, independent of
+  `PYTHONHASHSEED` and worker process.
+
+### Changed
+
+- **`graphiti-core` is pinned to 0.30.2** for request-scoped FalkorDB routing and the
+  indexed edge-search fixes in that release.
+- `kg-ingest` splits source bodies above 12,000 characters by default. Pass
+  `--max-chars 0` to retain one episode per input record.
+
 ## 0.5.0
 
 ### Added

README.md

@@ -52,10 +52,17 @@ to propose PostgreSQL, review and apply that update, and retrieve it from an MCP
 Model extraction can be wrong; inspect the returned facts and validity timestamps.
 
 To review extraction before it touches the configured graph, run
-`kg-ingest INPUT --review-output review.jsonl`, inspect the snapshot, then use the
-printed restore command to promote those exact records. Searches return current facts
+`kg-ingest INPUT --review-output review.jsonl`, inspect the snapshot, and mark every
+fact in the generated `review.jsonl.review.jsonl` file as `accept`, `refuse`, or
+`contested`. The printed restore command refuses unresolved decisions and promotes a
+checksum-sealed snapshot containing accepted facts only. Searches return current facts
 by default; `kg ask --history` is the explicit historical view.
 
+`kg-ingest` also accepts a Markdown/text file or a directory tree directly. Large
+documents split at stable text boundaries. Transient writes retry with one stable
+episode id; `ingest-receipts.jsonl` and `ingest-failures.jsonl` in the workspace show
+exactly what landed and what still needs attention.
+
 ## Multiple users on Ladybug
 
 Give each user a group, and each group its own Ladybug file:

config/falkordb.example.yaml

@@ -20,6 +20,8 @@ graph:
   # when a backend does not answer in time.
   query_timeout_seconds: 30
   writer_lock_timeout_seconds: 30
+  ingest_max_attempts: 3
+  ingest_retry_base_seconds: 1
 
 database:
   provider: falkordb
@@ -38,6 +40,7 @@ llm:
   api_mode: auto
   # Hard ceiling for every extraction call, including larger upstream prompt requests.
   max_tokens: 4096
+  require_all_schema_properties: true
 
 embedder:
   model: ${KG_EMBEDDER_MODEL:text-embedding-3-small}

config/ladybug.example.yaml

@@ -18,6 +18,8 @@ graph:
   # when a backend does not answer in time.
   query_timeout_seconds: 30
   writer_lock_timeout_seconds: 30
+  ingest_max_attempts: 3
+  ingest_retry_base_seconds: 1
 
 database:
   provider: ladybug
@@ -34,6 +36,7 @@ llm:
   structured_output_mode: json_schema
   api_mode: auto
   max_tokens: 4096
+  require_all_schema_properties: true
 
 embedder:
   model: ${KG_EMBEDDER_MODEL:text-embedding-3-small}

config/neo4j.example.yaml

@@ -18,6 +18,8 @@ graph:
   # when a backend does not answer in time.
   query_timeout_seconds: 30
   writer_lock_timeout_seconds: 30
+  ingest_max_attempts: 3
+  ingest_retry_base_seconds: 1
 
 database:
   provider: neo4j
@@ -33,6 +35,7 @@ llm:
   structured_output_mode: json_schema
   api_mode: auto
   max_tokens: 4096
+  require_all_schema_properties: true
 
 embedder:
   model: ${KG_EMBEDDER_MODEL:text-embedding-3-small}

config/ollama.example.yaml

@@ -31,6 +31,8 @@ graph:
   # when a backend does not answer in time.
   query_timeout_seconds: 30
   writer_lock_timeout_seconds: 30
+  ingest_max_attempts: 3
+  ingest_retry_base_seconds: 1
 
 database:
   provider: ladybug
@@ -45,6 +47,7 @@ llm:
   temperature: 0
   api_mode: chat
   max_tokens: 4096
+  require_all_schema_properties: true
 
 embedder:
   model: ${KG_EMBEDDER_MODEL:nomic-embed-text}

docs/reference.md

@@ -135,30 +135,45 @@ Ingestion is a separate command and is dry-run by default:
 ```bash
 uv run kg-ingest examples/local_memory_demo.jsonl
 uv run kg-ingest examples/local_memory_demo.jsonl --apply
+uv run kg-ingest ./docs --domain example --apply
 ```
 
-Inputs are UTF-8 JSONL objects with `name` and `body`; `domain`, `valid_at`, and
-`provenance` are optional. A requested domain must be in `graph.groups`.
+Inputs may be UTF-8 JSONL/JSON records, Markdown/text files, or a directory containing
+those types. JSON records need `name` and `body`; `domain`, `valid_at`, and `provenance`
+are optional. `--domain` supplies the graph group for plain documents. A requested
+domain must be in `graph.groups`. Bodies split at stable paragraph/sentence boundaries
+above 12,000 characters; choose another bound with `--max-chars`, or pass `0` to keep
+one episode per source record. Empty and unsupported inputs fail explicitly.
 
 Ingestion is resumable. Each applied record is written to a content-keyed ledger in
 the workspace, so re-running the same file ingests only what has not landed yet
-instead of duplicating it. A record that fails is isolated and reported; the rest of
-the batch still lands, and the failure sets a non-zero exit code. Use `--no-resume`
-to ignore the ledger and `--fail-fast` for the old stop-at-first-error behaviour.
+instead of duplicating it. Each record has a content-derived episode UUID. Transient
+model and backend errors retry with bounded backoff; a record that still fails is
+isolated and reported, the rest of the batch lands, and the command exits non-zero.
+`ingest-receipts.jsonl` records canonical episode ids and node/edge counts;
+`ingest-failures.jsonl` records exhausted attempts for reconciliation. Use
+`--no-resume` to ignore the ledger and `--fail-fast` to stop after the first exhausted
+record.
 
 To inspect model extraction before it reaches the configured graph, stage it:
 
 ```bash
 kg-ingest notes.jsonl --review-output ./review.jsonl
 # inspect the entity_node, entity_edge, and episodic records in review.jsonl
-kg-ingest ./review.jsonl --restore --group team-a --apply
+# edit review.jsonl.review.jsonl: review -> accept, refuse, or contested
+kg-ingest ./review.jsonl --restore \
+  --review-decisions ./review.jsonl.review.jsonl \
+  --approved-output ./review.approved.jsonl --group team-a --apply
 ```
 
 The first command creates a disposable embedded graph, runs extraction there, exports
 the exact resolved records, and removes the temporary database. It never opens the
 configured production backend. One review snapshot accepts one domain so its promotion
-target remains explicit. The printed `promote` command restores the reviewed records
-without running extraction again.
+target remains explicit. The decision file names the source snapshot digest and one
+typed decision per fact. Promotion refuses missing, `review`, or `contested` rows,
+drops refused fact edges, updates episode provenance lists, and writes a new canonical
+snapshot carrying source and decision digests. The printed `promote` command restores
+that sealed result without running extraction again.
 
 `SIGTERM` and `SIGINT` stop it at a record boundary rather than mid-write: it finishes
 the record in flight, closes the driver, and reports `interrupted`. This matters when a
@@ -228,7 +243,10 @@ other correction.
 `reranker.candidate_multiplier` controls how many RRF candidates reach the configured
 cross encoder, and `reranker.min_score` filters only its final scores. Equal fact text
 does not collapse distinct edge UUIDs. `passthrough` keeps the original order without
-an extra model call.
+an extra model call. CLI and MCP search payloads include a `retrieval` object with the
+requested limit, candidate ceiling, candidates seen, eligible and returned counts, and
+`recall_may_be_incomplete`. The flag is true whenever the upstream candidate ceiling
+was reached, so an empty or short filtered result cannot masquerade as exhaustive.
 
 FalkorDB queries remove only a standalone `_` token, which RediSearch reserves, and a
 multi-label node search fans out by label before merging UUIDs. Identifiers such as
@@ -238,7 +256,10 @@ multi-label node search fans out by label before merging UUIDs. Identifiers such
 upstream prompt that requests a larger budget. `llm.api_mode` selects `responses`,
 `chat`, or `auto`; `auto` uses Responses on the official OpenAI endpoint and Chat
 Completions for compatible endpoints. Set it explicitly for a proxy whose URL does
-not reveal which API family it implements.
+not reveal which API family it implements. For grammar-backed compatible endpoints,
+`llm.require_all_schema_properties` defaults to true: nullable schema fields remain
+nullable but their keys are required, preventing temporal keys from vanishing during
+constrained decoding.
 
 Run the release gate before sharing:
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "graphiti-local"
-version = "0.5.0"
+version = "0.6.0"
 description = "A local-first temporal knowledge graph with a read-only MCP server and CLI"
 readme = "README.md"
 requires-python = ">=3.10,<4"
@@ -20,7 +20,7 @@ classifiers = [
 ]
 dependencies = [
   "filelock>=3.16,<4",
-  "graphiti-core[falkordb]==0.30.1",
+  "graphiti-core[falkordb]==0.30.2",
   "ladybug>=0.19.1,<1",
   "mcp>=1.27,<2",
   "pydantic>=2.11,<3",

server.json

@@ -7,12 +7,12 @@
     "url": "https://github.com/renezander030/graphiti-local",
     "source": "github"
   },
-  "version": "0.5.0",
+  "version": "0.6.0",
   "packages": [
     {
       "registryType": "pypi",
       "identifier": "graphiti-local",
-      "version": "0.5.0",
+      "version": "0.6.0",
       "transport": {
         "type": "stdio"
       }

src/kg_mcp/cli.py

@@ -87,14 +87,15 @@ async def _edge_search(
     args: argparse.Namespace,
     groups: list[str] | None,
     settings: Any,
-) -> tuple[list[tuple[Any, float]], int]:
+) -> tuple[list[tuple[Any, float]], dict[str, Any]]:
     """Return current, identity-safe reranked edges."""
     from kg_mcp.retrieval import (
         candidate_limit,
         compatible_query,
         current_edges,
         keyword_edge_search_config,
         rerank_edges,
+        retrieval_stats,
     )
 
     query = compatible_query(args.query, settings)
@@ -111,7 +112,15 @@ async def _edge_search(
     current, suppressed = current_edges(
         list(edges), include_invalidated=getattr(args, "history", False)
     )
-    return await rerank_edges(graph, query, current, settings, args.limit), suppressed
+    ranked = await rerank_edges(graph, query, current, settings, args.limit)
+    return ranked, retrieval_stats(
+        requested=args.limit,
+        candidate_ceiling=candidates,
+        candidates_seen=len(edges),
+        eligible=len(current),
+        returned=len(ranked),
+        suppressed=suppressed,
+    )
 
 
 async def _graph_command(args: argparse.Namespace) -> dict[str, Any]:
@@ -166,7 +175,7 @@ async def _graph_command_on(args: argparse.Namespace, settings: Any) -> dict[str
     try:
         if args.command == "ask":
             groups = allowed_groups(args.groups, settings)
-            results, suppressed = await bounded(
+            results, retrieval = await bounded(
                 _edge_search(graph, args, groups, settings),
                 timeout,
                 "ask",
@@ -186,7 +195,8 @@ async def _graph_command_on(args: argparse.Namespace, settings: Any) -> dict[str
                     }
                     for result, score in results
                 ],
-                "suppressed_invalidated": suppressed,
+                "suppressed_invalidated": retrieval["suppressed"],
+                "retrieval": retrieval,
                 "ranker": settings.reranker.provider,
                 "pending": _pending_items(groups or settings.graph.groups),
             }
@@ -196,26 +206,38 @@ async def _graph_command_on(args: argparse.Namespace, settings: Any) -> dict[str
                 candidate_limit,
                 compatible_query,
                 keyword_node_search_config,
+                retrieval_stats,
             )
 
-            candidate_limit(args.limit, settings)
+            candidates = candidate_limit(args.limit, settings)
             query = compatible_query(args.query, settings)
             from graphiti_core.search.search_config_recipes import NODE_HYBRID_SEARCH_RRF
 
             node_config = (
-                keyword_node_search_config(args.limit) if args.keyword else NODE_HYBRID_SEARCH_RRF
+                keyword_node_search_config(candidates)
+                if args.keyword
+                else NODE_HYBRID_SEARCH_RRF.model_copy(update={"limit": candidates})
             )
             result = await bounded(
                 graph.search_(query, config=node_config, group_ids=groups),
                 timeout,
                 "nodes",
             )
+            nodes = list(result.nodes or [])
+            returned = nodes[: args.limit]
             return {
                 "query": args.query,
                 "nodes": [
                     {"name": node.name, "group_id": node.group_id, "uuid": node.uuid}
-                    for node in result.nodes[: args.limit]
+                    for node in returned
                 ],
+                "retrieval": retrieval_stats(
+                    requested=args.limit,
+                    candidate_ceiling=candidates,
+                    candidates_seen=len(nodes),
+                    eligible=len(nodes),
+                    returned=len(returned),
+                ),
             }
         if args.command == "episodes":
             groups = allowed_groups(args.groups, settings)

src/kg_mcp/config.py

@@ -115,6 +115,10 @@ class GraphConfig(BaseModel):
     # An embedded Ladybug graph has one writer. Wait this long for another ingest,
     # restore, drain, or setup command to finish before failing explicitly.
     writer_lock_timeout_seconds: float = Field(default=30.0, gt=0)
+    # Retry one episode rather than losing it when an inference endpoint or graph backend
+    # fails transiently. A content-derived episode UUID keeps those retries idempotent.
+    ingest_max_attempts: int = Field(default=3, ge=1, le=10)
+    ingest_retry_base_seconds: float = Field(default=1.0, ge=0, le=60)
 
     @model_validator(mode="after")
     def validate_groups(self) -> GraphConfig:
@@ -137,6 +141,10 @@ class LLMConfig(OpenAIConfig):
     max_tokens: int = Field(default=4096, ge=1)
     structured_output_mode: Literal["json_schema", "json_object"] = "json_schema"
     api_mode: Literal["auto", "responses", "chat"] = "auto"
+    # Grammar-backed OpenAI-compatible servers commonly omit schema properties that
+    # are not listed in ``required``. Keep nullable fields nullable, but require their
+    # keys so temporal values such as valid_at/invalid_at cannot disappear silently.
+    require_all_schema_properties: bool = True
 
 
 class EmbedderConfig(OpenAIConfig):