Add Kurultai buildroom research compiler - #2
Merged
Danservfinn merged 24 commits intoMay 11, 2026
Conversation
…hermes-autopilot]
… brain-service
Step 6 of the Phase 2.5 carveout (per migrations/neo4j-to-wiki/11-phase-2-5-carveout.md):
brain-service now exposes calendar.* RPC methods backed by gcalcli subprocess and
the calendar_event_cache table (telemetry.db schema migration v3).
New: kublai/calendar.py
- CalendarStore: SQLite cache wrapper for calendar_event_cache
- GcalcliClient: subprocess wrapper for gcalcli list/agenda/add/edit/delete
- CalendarService: high-level API (list_events, create_event, update_event,
cancel_event, list_due_reminders, health)
Modified: kublai/brain_service.py
- Import calendar module
- Wire CalendarService into BrainService.__init__ (uses telemetry_db for cache,
GCALCLI_PATH and GCALCLI_DEFAULT_CALENDAR env vars for runtime config)
- Dispatch calendar.* RPCs in handle_rpc; surface CalendarError via the standard
error envelope
Note: kublai/telemetry.py's list_due_reminders method is also needed for Step 6;
it lives in the working tree alongside earlier Phase 1 trace/audit work that has
not yet been committed. Will be picked up in the consolidating commit.
Verified: brain-service restarted; calendar.health and calendar.list_events
round-trip cleanly over the Unix socket. 7 Neo4j CalendarEvent nodes archived
to ~/backups/neo4j-pre-migration/2026-05-02/calendar-events.{json,cypher}.gz
with SHA-256.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…js routes
Adds the brain-service surface that server.js calls into for Phase 2.5
operational labels (per migrations/neo4j-to-wiki/11-phase-2-5-carveout.md).
New modules:
- kublai/humans.py: HumansStore reads hard-private/human-contacts/*.md and
supports list / get / update_consent (consent block in frontmatter).
- kublai/messages.py: MessagesStore reads ~/.kublai/messages.jsonl with
list_recent + append; respects HARD-private rules (no raw content kept).
Modified modules:
- kublai/telemetry.py: + list_reminders, list_due_reminders, create_reminder,
cancel_reminder, mark_reminder_sent, record_reminder_error,
replace_event_reminders, record_operator_action, list_operator_actions.
Also exposes list_due_reminders (added in Step 6 commit, now landed here).
- kublai/knowledge.py: + KnowledgeStore.list_by_tag — frontmatter tag scan
with public / hard-private privacy_scope filtering. Module-level
_read_frontmatter helper for shared use.
- kublai/brain_service.py: imports humans + messages, instantiates them in
BrainService.__init__, adds RPC dispatch for:
knowledge.list_by_tag
telemetry.{list_reminders,list_due_reminders,create_reminder,
cancel_reminder,mark_reminder_sent,record_reminder_error,
replace_event_reminders,record_operator_action,
list_operator_actions}
humans.{list,get,update_consent}
messages.{list_recent,append}
Adds BrainIndex.list_by_tag_in_wiki delegate.
Verification:
- All 5 modules pass py_compile.
- BrainService instantiation + handle_rpc dispatch tested in-process for
every new method.
- brain-service launchd kickstart succeeds; new RPCs reachable over
/Users/kublai/.kublai/brain-service.sock.
- humans.list returns 22 hard-private/human-contacts pages.
- knowledge.list_by_tag('molt') returns 1 analysis page.
- telemetry.list_due_reminders returns 5 pending reminders;
telemetry.list_operator_actions returns 1 entry.
Pre-step backups:
~/backups/neo4j-pre-migration/2026-05-02/kublai-repo-pre-step8/
Note: this commit does NOT include earlier-uncommitted Phase 1 work
(audit.py, memory_tiers.py, permissions.py, reliability.py, tracing.py,
tests/phase4/) which remains in the working tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the read-side Phase 3 Task-lifecycle RPCs that server.js will use to
migrate off Neo4j :Event, :TaskOutput, :TaskOutcome, and :FailureReport
nodes. Write-side companions for in_flight_tasks (status mutations from
non-claim paths) come later in Phase 3.
SQLite schema migration v4 (`/tmp/migration-4-task-lifecycle.sql`)
introduces four STRICT tables — `task_events`, `task_outputs`,
`task_outcomes`, `failure_reports` — with INTEGER ms timestamps, json_valid
CHECK constraints, and per-task / per-agent indexes. The migration is
recorded in `schema_migrations` (version 4).
`kublai/telemetry.py` (+261 lines) gains 10 methods modeled after
`record_operator_action`/`list_operator_actions`:
- list_tasks / get_task (read in_flight_tasks + claim_locks)
- list_task_events / append_task_event
- list_task_outputs / append_task_output
- list_task_outcomes / append_task_outcome
- list_failure_reports / append_failure_report
All append_* helpers default ids to `uuid.uuid4()`, default `occurred_at`
to `utc_ms()`, and serialize JSON columns with `sort_keys=True`. Read
helpers reuse `_decode_json_fields` so callers receive parsed dict /
list payloads.
`kublai/brain_service.py` (+20 lines, 10 new dispatch branches) wires
each method into `handle_rpc` under the existing `telemetry.*` namespace.
Verification (live Mac mini, telemetry.db backed up to
~/backups/neo4j-pre-migration/2026-05-02/telemetry-before-phase-3-step1.db
with sha256 sidecar):
- py_compile on both files: OK
- In-process: 14/14 assertions pass covering all 10 RPCs (including
list_tasks filter combinations and get_task on existing 230-row
in_flight_tasks fixture).
- Over-socket via /Users/kublai/.kublai/brain-service.sock: 10/10
smokes pass after `launchctl kickstart -k` (pid 94557 -> 99278,
stdout/stderr clean).
Synthetic test rows are deleted after each test pass so no production
data is touched.
No changes outside kublai/telemetry.py and kublai/brain_service.py.
Hermes Phase 4 disables the mcp-neo4j-cypher MCP integration that today
lets six Hermes skills emit arbitrary Cypher (queue diagnostic, codebase
autopilot, pipeline stall, proposal review, rate-limit debug, provider
health). To keep parity, brain-service must expose typed RPCs for the
read+write paths those skills exercise.
This commit adds five telemetry RPCs and one knowledge RPC, modeled on
the existing record_operator_action / list_operator_actions style:
telemetry.task_analytics
Aggregates in_flight_tasks: counts grouped by status / agent /
retry_count, plus duration mean/p50/p90, retry summary, and failure
rate. Replaces pipeline-analytics-queries.md cookbook Cypher.
telemetry.cap_retry_count
{task_ids, max_retries} -> caps retry_count at max_retries for each
task whose retry_count exceeds it. Records a retry_capped event
per modified task. Replaces the LLM-emitted SET t.retry_count
Cypher used by codebase-autopilot retry-overshoot remediation.
telemetry.cancel_tasks
{task_ids, reason, agent?} -> bulk cancellation. Skips already-
terminal tasks (returns them under skipped[]); records a
cancelled task_event with reason+previous_status; clears any
claim_lock. Replaces the LLM-emitted SET t.status=CANCELLED
Cypher used by pipeline-stall-diagnosis and the daily queue patrol.
telemetry.reset_to_pending
{task_ids, agent?, reason?} -> recovers orphaned in_progress tasks:
drops the claim_lock, clears claimed_by/claimed_at/active_claim_
token/active_lease_version, sets status=pending, records a
reset_to_pending task_event. Skips tasks not in in_progress.
telemetry.list_terminal_tasks
{since_ms, status_filter?, limit=100} -> tasks with terminal
status (completed|failed|cancelled by default) since since_ms,
ordered DESC by completion/failure time. Replaces the obsolete
Cypher pattern in kurultai_terminal_task_notifier.py:107 and gives
Hermes a uniform RPC surface instead of a direct SQLite read.
knowledge.list_proposals
{status?, since_ms?, limit=50} -> wiki pages with frontmatter
type:proposal (or proposal in tags as a legacy fallback). Honors
the hard-private boundary; sorts by mtime DESC. Implementation
choice: scan wiki pages directly rather than aliasing to
list_by_tag, because (a) we want the type:proposal frontmatter to
take precedence over a possibly-missing tag, (b) we need
status filtering on frontmatter, and (c) the response shape adds
updated/created/mtime_ms which list_by_tag does not surface.
Plus a small _percentile helper at module scope (no numpy dep).
brain_service.handle_rpc gains six dispatch branches matching the
existing telemetry.* / knowledge.* namespaces.
Verification (live Mac mini):
- py_compile on all three changed files: OK
- In-process unit tests: 35/35 assertions pass across all 6 RPCs
including edge cases (empty inputs, idempotent re-runs, terminal
status skipping, bad enum values, hard-private exclusion).
- Restart: launchctl kickstart -k ai.kurultai.brain-service
(pid 99278 -> 8711, socket alive).
- Over-socket smoke (using brain-service.sock): all 6 RPCs respond
with expected shapes against synthetic test rows; rows hard-deleted
in the smoke teardown.
Backups before edit:
~/backups/neo4j-pre-migration/2026-05-02/kublai-pre-phase-4/
{telemetry,brain_service,knowledge}.py.bak
… CRUD RPCs
Schema migration v5 ALTERs in_flight_tasks to add the 24+ columns the
the-kurultai dashboards Kanban view needs (title, prompt, dispatch_phase,
paused, sort_order, domain, source, parent_task, pipeline_id, etc.) and
widens the status CHECK to accept the openclaw wide enum
(PENDING/WORKING/COMPLETED/FAILED/ORPHANED/OBSOLETE/DONE/STALE/CANCELLED/
IN_PROGRESS in addition to the legacy lowercase set).
13 task CRUD RPCs (plus 4 bonus) added to kublai/telemetry.py and
dispatched in brain_service.py. Each emits task_events for audit trail:
- create_task_full, set_task_status, retry_task, redo_task,
retry_all_tasks, set_task_obsolete, revert_task_status,
revert_task_prompt, reassign_task, update_task_prompt, delete_task,
pause_task, unpause_task, reorder_tasks, move_task_to_top,
bulk_reassign_tasks, get_task_with_output
list_tasks extended with status_in / pipeline_id / parent_task / paused
filters and an order_by="dashboard" mode that matches the Kanban GET
ordering (sort_order DESC, priority bucket, created_at DESC).
Verified in-process (55/55 pass) and over-socket (17/17 pass).
brain-service restarted clean (pid 8711->13738).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…se 2.5 residual Replaces server.js's Neo4j fulltext-index query (`message_text_search`) at lines 5723-5777 — the last Phase 2.5 Cypher residual that Wave 5 surfaced during the Neo4j decommission gate check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
Notes