feat(knowledge-base): self-curating knowledge base (OKF pages + folder missions)#2455
Draft
nicoloboschi wants to merge 6 commits into
Draft
feat(knowledge-base): self-curating knowledge base (OKF pages + folder missions)#2455nicoloboschi wants to merge 6 commits into
nicoloboschi wants to merge 6 commits into
Conversation
nicoloboschi
force-pushed
the
feat/knowledge-pages-okf
branch
2 times, most recently
from
3291b45 to
f41f25e
Compare
…r missions)
Server-side knowledge base: a hierarchy of folders and pages over mental
models, projected to the Open Knowledge Format, with a mission-driven curator
that maintains pages automatically after each consolidation.
- knowledge_pages table (PG + Oracle): parent_id tree, kind folder/page,
mission, managed, last_curated_at; partial unique index on (folder, name)
for concurrency-safe dedup; added to BACKUP_TABLES.
- api/okf.py: OKF serializer (frontmatter + body, index/log, constellation graph).
- engine/knowledge_curator.py: folder curator (LLM op plan + safe apply); reads
new memories since last curation (delta, not recall); ops create/merge/delete
page + spawn sub-folder (bounded depth<=3, <=8). Runs as an async curate_folder
task on folder/mission create and after consolidation. Curator pages use an
observation-only delta trigger with exclude_mental_models.
- MemoryEngine: folder/page CRUD, tree, curate, async submit + worker handler.
- /v1/default/banks/{bank}/knowledge-base/* endpoints.
- Control plane: knowledge-base tree view + constellation toggle, missions,
OKF page panel + bundle export; proxies, client, sidebar, i18n.
- Tests: okf unit, knowledge-base HTTP, curator apply + dedup guard, hs_llm_core e2e.
- Regenerated OpenAPI + SDK clients + docs-skill.
Add @vectorize-io/hindsight-fs, a CLI under hindsight-tools/ that mirrors a Hindsight bank's mental models as real markdown files (YAML frontmatter + body) in a local directory, refreshed from the API on an interval. Once mounted, ordinary shell tools (ls, cat, grep, find, ...) work against current memory. - Pull-based sync engine: full list each tick, write changed/new/tampered files, skip unchanged (content-hashed), prune deleted models. Atomic writes; a transient API error never wipes the mirror. - One-way mirror enforced two ways: files are read-only (0444) so agent edits fail with EACCES, plus a tamper-revert backstop that compares on-disk bytes and overwrites drift on the next pass. --writable opts out. - Commands: mount/start/stop/restart/sync/status/list/logs/unmount. Background daemon via detached process + pidfile; per-mount config is remembered. - status doubles as a healthcheck: --json report and a non-zero exit when the mount is dead/failed/stale (--stale-after overrides the threshold). - Tests: unit (sync engine, frontmatter, health) + e2e that spawns the real CLI against a mock API and exercises real bash commands. 26 tests.
…dels Re-point hindsight-fs at the knowledge base so it projects a bank's folder/page hierarchy as nested directories + .md files, instead of a flat list of mental models. - client: fetch GET /knowledge-base/tree + /export (two calls, any bank size) and join by page id; replaces the paginated mental-models list. - format: planMirror() walks the tree into folder dirs + page files at nested paths (slug per segment, collision-safe); pages render the page's OKF doc. - sync: create folder dirs, write pages at nested paths, prune removed pages and emptied folders; state keyed by relative path + tracked dirs. - config/cli: drop the mental-model `detail` flag; `list` prints folders+pages; help/README updated. Tests rewritten for the tree/export model. Verified live against a bank's knowledge base: the `people` folder mirrors to people/anna.md + people/marco.md with OKF frontmatter.
The knowledge base is now purely client-managed (CRUD over folders/pages); the server no longer auto-curates. Removes the folder curator entirely and the folder `mission` concept, and leads the sidebar with Knowledge Base. - Remove engine/knowledge_curator.py, the curate_folder task (handler + dispatch + submit_async_curate_folder / _bank_folders), the post-consolidation curation hook, and the folder-create / mission-update curation triggers. - Remove folder `mission` and `last_curated_at` (columns + engine + API + UI); keep `managed` as a client-set flag. Migration a5b6 now adds `managed` only; the last_curated_at migration is dropped and the unique-index migration repointed. Single alembic head preserved. - API: KnowledgeNode/CreateFolderRequest/UpdateNodeRequest lose `mission`; PATCH node handles name/parent_id only. - Control plane: sidebar leads with Knowledge Base (before Memories); remove the mission field, edit-mission dialog, and mission display from the KB view. - Delete the curator tests; regenerate OpenAPI + SDK clients.
… tokens Client-created pages had no server curation applying a trigger, so they fell back to the plain mental-model default (no refresh, full mode, all fact types). Make a knowledge page a living document by default: when the client omits `trigger`, use observation-only + delta + exclude_mental_models + refresh_after_consolidation; when it omits `max_tokens`, default to 4096 (vs the mental-model 2048). Clients can still override either.
…or + memory Euler graph A large control-plane pass on the knowledge base UX: - Home dashboard (home-view): memory constellation + read-only knowledge-page TOC (reuses the Pages tree) + recent documents + the bank-profile "Memory store" card and "Memories by ingested time" chart (extracted as reusable exports). Fixed-height top row so the constellation fills and the side cards scroll. - Sidebar: add Home (first); order Home → Memories → Knowledge. - Knowledge view: Pages / Mental Models sub-tabs (Mental Models moved out of the Memories view). Pages tab is an Obsidian-style workspace — file-tree sidebar + inline editor with open-page tabs; the generation prompt is tucked behind a "How this page is derived" expander; a "Backed by N memories" line opens the backing model's based_on via the existing mental-model detail modal. First page auto-opens; deep-link via ?page=. - Knowledge graph: reframed as an Euler/Venn of the source memories — nodes are based_on memories, one translucent circle per page (overlaps = shared memories), plus the memory graph's own edges. New Constellation venn mode (nodeGroupsFn / groupColorFn / groupLabelFn) drawing overlapping per-group circles + pill labels. Backend: knowledge_page_memory_graph endpoint (pages' based_on → memory nodes). - Pages default to the living-document trigger (observation-only, delta, exclude mental models, auto-refresh) + 4096 max_tokens. - Documents: metadata badges are expandable (show all keys, not just 3). - Regenerated OpenAPI + docs-skill.
nicoloboschi
force-pushed
the
feat/knowledge-pages-okf
branch
from
f41f25e to
fa60f20
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A server-side Knowledge Base: a hierarchy of folders and pages over mental models, with a mission-driven curator that maintains pages automatically, plus a control-plane UI (file-tree + constellation). Pages project to the Open Knowledge Format (markdown + YAML frontmatter).
Model
mission. Pages are backed by a mental model (mental_model_id) and markedmanaged(curator-owned) vs pinned (human).create_page/merge_pages/delete_page/create_subfolder(bounded: depth ≤ 3, ≤ 8 sub-folders).curate_foldertask), same machinery as the mental-model refresh — no manual trigger.{mode: delta, fact_types: [observation], exclude_mental_models: true, refresh_after_consolidation: true}so they synthesize from consolidated observations and stay current.Changes
Server (
hindsight-api-slim)knowledge_pagestable (PG + Oracle) +mission/managed/last_curated_atcolumns; added toBACKUP_TABLES; partial unique index on (folder, page-name) to make concurrent curator runs dedup-safe.api/okf.py— OKF serializer (frontmatter + body, index.md/log.md, tag/folder constellation graph).engine/knowledge_curator.py— the curator (LLM op plan + safe apply).MemoryEngine— folder/page CRUD, tree, curate, async submit + worker handler./v1/default/banks/{bank}/knowledge-base/*endpoints (tree, folders, pages, nodes, graph, export).Control plane
knowledge-base-view.tsx— folder/file tree (create folder/page, missions, delete) + constellation toggle, OKF page panel, OKF bundle export. Sidebar entry + proxies + i18n (all locales).Generated — OpenAPI + SDK clients + docs-skill regenerated.
Tests
test_okf.py(unit),test_knowledge_base.py(HTTP: tree/move/cascade/graph/export),test_knowledge_curator.py(deterministic apply ops + dedup guard),test_knowledge_curator_e2e.py(hs_llm_core, full pipeline).Open items (why draft)
exclude_mental_modelstrigger mitigates it; a forced recall+synthesize path is the likely follow-up.