Skip to content

feat(meta,daemon): meta-orchestrator + cluster-leader foundation skeleton - #101

Merged
chitcommit merged 3 commits into
mainfrom
feat/meta-orchestrator-extension
Jun 3, 2026
Merged

chitcommit merged 3 commits into
mainfrom
feat/meta-orchestrator-extension

Conversation

@chitcommit

Copy link
Copy Markdown
Contributor

Summary

Implements the Foundation PR scope of ADR-001 — Extend ChittyCommand into the Tier-2 meta-orchestrator (chittycanon://docs/architecture/chittycommand/ADR-001).

Skeleton only: every endpoint executes real SQL or makes real HTTP calls. No mocks, no fake data, no placeholder route bodies. Existing src/, agents/, and ui/ are untouched — the Tier-5 dashboard surface stays intact.

What's in this PR

Schema additions (in src/db/schema.ts + generated migration migrations/0002_naive_mac_gargan.sql):

  • cc_goals — Goals (top of intent ladder, owned by a ChittyID actor)
  • cc_plans — Plans (strategies advancing a Goal; capture sovereignty assessment at authoring time)
  • cc_intents — Intents (executable units; sovereignty gate frozen at enqueue; status pending → claimed → running → done/failed/blocked_human)
  • cc_node_leases — Cluster lease table mirroring chittyentity/workers/shared/agent-tasks.ts task_leases shape; role PK + atomic UPDATE...RETURNING

meta/ — meta-orchestrator layer:

  • meta/intent.ts — Goal/Plan/Intent CRUD + claimNextIntent (atomic via FOR UPDATE SKIP LOCKED)
  • meta/sovereignty.tsassessSovereignty(actor, intent, env) → calls https://trust.chitty.cc/v1/reckon/:chittyId; returns { decision, trustScore, reasoning }. Conservative decision matrix; richer semantics deferred per ADR out-of-scope.
  • meta/channels.tslistRegisteredChannels(env) → reads agent.chitty.cc/api/v1/channels. Throws on non-2xx / unreachable; never synthesizes a stub list.
  • meta/context.ts — Forever-context wrapper following the chittyconnect-client.ts primary + service-binding fallback pattern.

daemon/ — persistent cluster process:

  • daemon/leader.tsclaimLeadership, heartbeat, releaseLeadership, describeLease. Single-statement atomic UPDATE on cc_node_leases is the gate.
  • daemon/loop.tsrunLeaderLoop(env, options) skeleton: claim → inner loop (heartbeat + claim intent + dispatch via injected executor + mark done/failed) → on lease loss park + retry. AbortSignal-driven clean exit + best-effort release.
  • daemon/supervisor.md — launchd (macOS Minis) + systemd (Ubuntu Minis + chittyserv-vm) supervision plan. Doc only.

Tests:

  • tests/daemon/leader.spec.ts — Real Neon integration test. Verifies (1) first claim wins, (2) concurrent claim from different node is rejected while lease is live, (3) holder can heartbeat, (4) non-holder cannot heartbeat, (5) post-release, previously-rejected node can claim. Skips automatically without DATABASE_URL or with SKIP_INTEGRATION=1.

No-mocks evidence

Generated migration applied to a disposable Neon branch (br-misty-lake-aklabdcz on project cool-bar-13270800 / ChittyCommand); lease semantics exercised end-to-end via the Neon MCP. Sample SQL output:

-- Node A (chittymini-03, 01-A-MIN-0003-L-66-3-7) claims:
RETURNING role, node_id, lease_expires_at
=> [{"role":"meta-orchestrator-leader-validation","node_id":"01-A-MIN-0003-L-66-3-7","lease_expires_at":"2026-06-03T03:49:35.320Z"}]

-- Node B (chittymini-05, 01-A-MIN-0005-L-66-2-1) tries while A's lease is live:
=> []  -- 0 rows, rejected

-- After A releases, B claims:
=> [{"role":"meta-orchestrator-leader-validation","node_id":"01-A-MIN-0005-L-66-2-1"}]

Schema columns verified via information_schema.columns — all 4 new tables present with the expected types.

Lint + tests

  • npm run typecheck — clean
  • SKIP_INTEGRATION=1 npm test — 15 passed, 5 skipped (the integration test). Pre-existing tests still pass.

What's NOT in this PR

Verbatim from the ADR's out-of-scope list:

  • Actual deployment / wrangler changes / domain routing.
  • Refactoring shared types.
  • The autonomous-loop policy semantics beyond the interface.
  • Multi-node deployment scripts.
  • Cluster daemon hardening (Neon-loss handling beyond "park the node").

Also not in this PR:

  • CHARTER.md / CHITTY.md classification update from Tier-5 to "Tier-2 platform with Tier-5 dashboard surface" — deferred per the ADR's Consequences section.
  • Wiring daemon/loop.ts to the existing ActionAgent — the loop takes an injected executor so this PR introduces no coupling.

Test plan

  • Maintainer review of meta/sovereignty.ts decision matrix thresholds
  • Maintainer review of meta/channels.ts throw-on-unreachable behavior (vs returning empty list)
  • Optional: run integration test against a fresh Neon branch via DATABASE_URL=... npx vitest run tests/daemon/leader.spec.ts
  • Verify no existing routes in src/routes/ are affected (they aren't — no files touched there)

Do NOT enable auto-merge. Do NOT deploy. Follow-up PRs will wire the executor and update the CHARTER.

ADR reference: chittycanon://docs/architecture/chittycommand/ADR-001

Generated with Claude Code

…eton

Implements the foundation PR scope from ADR-001 (chittycanon://docs/architecture/chittycommand/ADR-001):

- meta/intent.ts — Goal/Plan/Intent ladder with real Drizzle-backed CRUD
- meta/sovereignty.ts — Trust-gated decision via live ChittyTrust reckon
- meta/channels.ts — Real list of registered channels (throws on unreachable)
- meta/context.ts — Forever-context wrapper with primary+fallback pattern
- daemon/leader.ts — cc_node_leases atomic claim / heartbeat / release
- daemon/loop.ts — Persistent leader loop skeleton with injected executor
- daemon/supervisor.md — launchd + systemd supervision plan (doc only)
- migrations/0002_naive_mac_gargan.sql — Drizzle-generated DDL
- tests/daemon/leader.spec.ts — Real Neon integration test (skips without DATABASE_URL)

No mocks, no fake data, no placeholder endpoints. Schema validated against a
disposable Neon branch (br-misty-lake-aklabdcz on project cool-bar-13270800);
lease claim/reject/release semantics validated end-to-end via SQL on the same
branch.

Existing src/, agents/, ui/ unchanged. Tier-5 surface untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 3, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
chittycommand 203fe36 Jun 03 2026, 07:17 PM

@coderabbitai

coderabbitai Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@chitcommit, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 10 minutes. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 606e567a-0907-4b11-a37b-c6cdb849173f

📥 Commits

Reviewing files that changed from the base of the PR and between d19c22e and 203fe36.

📒 Files selected for processing (15)
  • daemon/leader.ts
  • daemon/loop.ts
  • daemon/supervisor.md
  • docs/architecture/ADR-001-meta-orchestrator-extension.md
  • meta/channels.ts
  • meta/context.ts
  • meta/intent.ts
  • meta/sovereignty.ts
  • migrations/0002_naive_mac_gargan.sql
  • migrations/meta/0002_snapshot.json
  • migrations/meta/_journal.json
  • src/db/schema.ts
  • tests/daemon/leader.spec.ts
  • tsconfig.json
  • vitest.config.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/meta-orchestrator-extension

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
  1. @coderabbitai review
  2. @copilot review
  3. @codex review
  4. @claude review
    Adversarial review request: evaluate security, policy bypass paths, regression risk, and merge-gating bypass attempts.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1b4912b37e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread meta/intent.ts
Comment thread daemon/leader.ts
Comment thread daemon/loop.ts
Comment thread src/db/schema.ts
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
  1. @coderabbitai review
  2. @copilot review
  3. @codex review
  4. @claude review
    Adversarial review request: evaluate security, policy bypass paths, regression risk, and merge-gating bypass attempts.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chitcommit

chitcommit commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

@claude[agent] review codex recommendations and make neccessary updates and resolve comments

@Claude

Claude AI commented Jun 3, 2026

Copy link
Copy Markdown

@chitcommit Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: ae1e5275-3e73-4c2f-8707-9eebf7ffd003

Sorry for the inconvenience!

5 similar comments
@Claude

Claude AI commented Jun 3, 2026

Copy link
Copy Markdown

@chitcommit Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: ae1e5275-3e73-4c2f-8707-9eebf7ffd003

Sorry for the inconvenience!

@Claude

Claude AI commented Jun 3, 2026

Copy link
Copy Markdown

@chitcommit Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: ae1e5275-3e73-4c2f-8707-9eebf7ffd003

Sorry for the inconvenience!

@Claude

Claude AI commented Jun 3, 2026

Copy link
Copy Markdown

@chitcommit Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: ae1e5275-3e73-4c2f-8707-9eebf7ffd003

Sorry for the inconvenience!

@Claude

Claude AI commented Jun 3, 2026

Copy link
Copy Markdown

@chitcommit Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: ae1e5275-3e73-4c2f-8707-9eebf7ffd003

Sorry for the inconvenience!

@Claude

Claude AI commented Jun 3, 2026

Copy link
Copy Markdown

@chitcommit Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: ae1e5275-3e73-4c2f-8707-9eebf7ffd003

Sorry for the inconvenience!

@chitcommit

Copy link
Copy Markdown
Contributor Author

@claude review open comments making necessary updates to resolve

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dd9c9dad33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread daemon/leader.ts
Comment thread meta/intent.ts
Comment thread meta/context.ts
Comment thread meta/context.ts
@chitcommit
chitcommit enabled auto-merge (squash) June 3, 2026 19:17
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
  1. @coderabbitai review
  2. @copilot review
  3. @codex review
  4. @claude review
    Adversarial review request: evaluate security, policy bypass paths, regression risk, and merge-gating bypass attempts.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chitcommit
chitcommit merged commit 03e8fc9 into main Jun 3, 2026
18 checks passed
@chitcommit
chitcommit deleted the feat/meta-orchestrator-extension branch June 3, 2026 19:33
chitcommit added a commit that referenced this pull request Jun 4, 2026
* feat(meta,daemon): meta-orchestrator + cluster-leader foundation skeleton

Implements the foundation PR scope from ADR-001 (chittycanon://docs/architecture/chittycommand/ADR-001):

- meta/intent.ts — Goal/Plan/Intent ladder with real Drizzle-backed CRUD
- meta/sovereignty.ts — Trust-gated decision via live ChittyTrust reckon
- meta/channels.ts — Real list of registered channels (throws on unreachable)
- meta/context.ts — Forever-context wrapper with primary+fallback pattern
- daemon/leader.ts — cc_node_leases atomic claim / heartbeat / release
- daemon/loop.ts — Persistent leader loop skeleton with injected executor
- daemon/supervisor.md — launchd + systemd supervision plan (doc only)
- migrations/0002_naive_mac_gargan.sql — Drizzle-generated DDL
- tests/daemon/leader.spec.ts — Real Neon integration test (skips without DATABASE_URL)

No mocks, no fake data, no placeholder endpoints. Schema validated against a
disposable Neon branch (br-misty-lake-aklabdcz on project cool-bar-13270800);
lease claim/reject/release semantics validated end-to-end via SQL on the same
branch.

Existing src/, agents/, ui/ unchanged. Tier-5 surface untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(schema): composite FK on cc_intents + reclaim_count column

Codex P2 PR#101 findings 1 and 4.

- F4: cc_intents now references cc_plans via composite FK
  (plan_id, goal_id) -> cc_plans(id, goal_id) so an intent's goal_id
  MUST match its plan's goal_id. Adds UNIQUE(id, goal_id) on cc_plans
  to back the composite reference.
- F1: adds cc_intents.reclaim_count for stuck-intent bookkeeping
  surfaced by reclaimStuckIntents() in the next commit.

Validated on disposable Neon branch br-delicate-mode-akkgde73 off
project cool-bar-13270800: matching (plan_id, goal_id) insert succeeds;
mismatched goal_id insert is rejected with
"violates foreign key constraint cc_intents_plan_goal_cc_plans_fk".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(meta/intent): reclaim stuck intents + guard terminal states

Codex P2 PR#101 findings 1 and 6.

- F1: adds reclaimStuckIntents(maxRunningSeconds) — atomically resets
  intents whose status='claimed'|'running' updated_at is older than the
  threshold back to 'pending', increments reclaim_count, and clears
  dispatched_task_id + error_message. Idempotent; returns rowcount.
  cc_intents has no claimed_by/claimed_at; staleness is measured via
  updated_at, which dispatch and (next-commit) heartbeat paths bump.
- F6: completeIntent now requires status='running'; failIntent allows
  ('claimed','running') but never overwrites a terminal state. Prevents
  parallel cancellation paths from being silently clobbered.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(daemon): session-scoped lease + reclaim + executor heartbeat

Codex P2 PR#101 findings 2, 3, and 5.

- F5: heartbeat() now matches WHERE session_id IS NOT DISTINCT FROM
  $sessionId. A restarted process with the same nodeId cannot extend
  a lease that already belongs to a newer leader.
- F2: releaseLeadership() applies the same session-ownership guard.
- F3: runLeaderLoop wraps each executor.execute(intent) in a
  setInterval heartbeat ticker at lease/2 cadence (min 1s). The
  interval is cleared in finally. Ticker uses the same session token
  so stale executors cannot keep a foreign lease alive.
- Also: calls reclaimStuckIntents(leaseSeconds * 2) once per tick
  before claiming new work (wires up F1 from the previous commit).
- Updates tests/daemon/leader.spec.ts to pass sessionId on heartbeat
  and release so it remains green under the new ownership guard.

Schema column is session_id; the original task spec said
"session_token". Used the real column.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(meta/context): reuse canonical connect token + service-binding fallback

Codex P2 PR#101 findings 7 and 8.

- F8: ContextEnv now prefers CHITTY_CONNECT_TOKEN — the binding already
  used by src/lib/cron.ts, src/routes/bridge/credentials.ts, and the
  rest of the worker. Legacy CHITTYCONNECT_TOKEN is kept as a
  back-compat fallback via resolveConnectToken().
- F7: getEcosystemAwareness now falls back to the AGENT_CONNECT service
  binding when the HTTPS path has no URL/token or the upstream fetch
  fails — matching the persist/recall fallback pattern already in
  place. Avoids hard failure when the worker only has the binding.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: real Neon coverage for codex-p2 intent + leader-session guards

- tests/meta/intent-lifecycle.spec.ts — exercises F1 (reclaim round-
  trip, reclaim_count increment, fresh-running rows untouched) and F6
  (completeIntent rejects non-running; failIntent cannot overwrite a
  terminal state).
- tests/daemon/leader-session.spec.ts — exercises F2 + F5 by letting
  an old session's lease expire, having a new session reclaim, then
  proving the old session can neither heartbeat nor release.

Both tests follow the existing pattern in tests/daemon/leader.spec.ts:
real Neon, skip with `describe.skipIf(!DATABASE_URL)` so CI without a
Neon branch URL doesn't fail. Realistic ChittyOS-shaped IDs only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chitcommit added a commit that referenced this pull request Jun 4, 2026
… start)

Stacked on #101. Adds the runtime supervisor surface for the meta-orchestrator
daemon foundation:

- daemon/runtime/entrypoint.ts — Node entrypoint wiring runLeaderLoop with
  SIGTERM/SIGINT graceful shutdown that releases the lease before exit.
- daemon/runtime/chittycommand-daemon.service — hardened systemd unit
  (Restart=always, dedicated chittycommand user, EnvironmentFile from
  /etc/chittycommand/env, journal logging).
- daemon/runtime/env.tmpl — op inject template; only op:// references, no
  secret values. 1Password stays the cold source of truth.
- daemon/runtime/launchd/com.chittyos.chittycommand-daemon.plist — macOS
  artifact for the Mac Mini cluster (not used on the VM, included so
  next-node bring-up doesn't need a separate PR).
- daemon/runtime/tsconfig.daemon.json + package.json build:daemon script —
  emits dist/daemon/runtime/entrypoint.js for the service ExecStart.
- scripts/install-daemon-vm.sh — idempotent VM bootstrap; supports --dry-run;
  creates user, builds, syncs to /opt/chittycommand, renders env via op
  inject, installs+enables the unit, STOPS WITHOUT STARTING.
- docs/runbooks/daemon-bring-up-vm.md — operator runbook (mint flow,
  verify, heartbeat, failure modes, rollback).
- daemon/supervisor.md — links to the runbook and pins chittyserv-vm as
  first-node target.

Operator action required: sudo systemctl start chittycommand-daemon.service
(after sudo -E ./scripts/install-daemon-vm.sh). See PR body for dry-run output.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chitcommit added a commit that referenced this pull request Jun 4, 2026
…l consumer (#109)

* feat(daemon): loop body wires executeIntent end-to-end (claim → dispatch → heartbeat)

Stacked on #106. Replaces the injected-executor abstraction in daemon/loop.ts
with a direct call to meta/intent.ts::executeIntent, closing the
meta-orchestrator loop. Status transitions, audit-row writes, and the second
sovereignty gate are all owned by executeIntent → dispatch; the loop's
responsibility is leader lifecycle, intent claiming, heartbeats, and outcome
classification.

Four outcomes are handled distinctly:

  - ok=true (executed)      → bump processed counter, reset error backoff
  - ok=true (replayed)      → bump replayed counter, no backoff, no double-count
  - ok=false (refused)      → bump refused counter, no backoff (steady-state)
  - ok=false (exec error)   → bump errored counter, bounded exp backoff

Sovereignty refusals are identified by canonical error prefixes emitted by
meta/executors/dispatch.ts ("sovereignty re-reckon:" /
"sovereignty snapshot stale ..."). Refusals are NOT treated as transient
faults — they are valid outcomes and do not trigger backoff.

The loop honors options.signal via AbortController throughout, including
inside the sleep helper, so SIGTERM from daemon/runtime/entrypoint.ts (PR #105)
unwinds cleanly through releaseLeadership.

tests/daemon/loop.spec.ts — real Neon integration. Seeds two pending intents
against the update_obligation_status executor, runs runLeaderLoop with
maxIntents=2, asserts:
  - both intents reach status='done'
  - each produces exactly one cc_actions_log row (attempt=1, key set, status='completed')
  - cc_obligations rows actually moved to 'deferred'
  - cc_node_leases shows leadership released on clean exit
  - log stream contains intent_heartbeat_before / intent_heartbeat_after pairs

Out of scope (not in this PR):
  - new executors (mercury, etc.)
  - production deploy
  - multi-node coordination beyond single-node leader
  - schema additions on cc_intents / cc_actions_log

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(health,wrangler): real-dependency /health probe + chittytrack tail consumer

The previous /health returned a static {status:"ok",...} regardless of
whether the worker could reach its real dependencies — a direct violation
of the chittyentity CLAUDE.md "No Mocks, Fake Data, or Placeholder
Endpoints" binding rule ("every endpoint must return real results against
a real datastore on the day it is committed").

This change replaces /health with a real probe that executes against the
worker's actual dependencies:

  * db            — SELECT 1 via Neon HTTP driver. Critical: failure -> 503.
  * chittyconnect — GET ${CHITTYCONNECT_URL}/health. Degraded if unreachable.
  * daemon        — newest cc_node_leases.heartbeat_at, stale if older than
                    2x daemon/loop.ts default heartbeatMs (10000ms ->
                    20000ms cutoff). Missing table -> not_provisioned
                    (degraded, NOT down) so deploys against bases without
                    #101 don't 503.

Per-dep timeout 2000ms; total probe bounded ≤ 5000ms via Promise.all.
Runs unauthenticated (no auth middleware on /health).

The probe handler is extracted to src/routes/health.ts so it can be
integration-tested in pure Node — importing src/index.ts directly drags
in `cloudflare:` modules (Agents SDK / DOs) that only resolve under
workerd. Integration test exercises the handler against a real Neon
branch (no mocks), validates DB probe ok and shape of all three probes,
and asserts 503 + status=down when DB is unreachable.

wrangler.jsonc already declares `tail_consumers: [{service: chittytrack}]`
(present on the base branch) — no change required there.

Verified against Neon branch br-spring-queen-akggkmso of project
cool-bar-13270800 (ChittyCommand). Sample real response:

{
  "status": "degraded",
  "service": "chittycommand",
  "version": "0.1.0",
  "timestamp": "2026-06-04T05:27:16.111Z",
  "probes": {
    "db": { "status": "ok", "latency_ms": 226 },
    "chittyconnect": {
      "status": "degraded", "latency_ms": 0,
      "error": "CHITTYCONNECT_URL not configured"
    },
    "daemon": {
      "status": "not_provisioned", "newest_heartbeat_age_ms": null,
      "error": "relation \"cc_node_leases\" does not exist"
    }
  }
}

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(health): query active leases by heartbeat_at, not released_at

Addresses Codex P2 on PR #109. The cc_node_leases schema in
src/db/schema.ts has no released_at column — release is represented by
NULLing heartbeat_at/lease_expires_at in daemon/leader.ts. The previous
WHERE released_at IS NULL clause raised 'column released_at does not
exist', which the catch block then mis-classified as 'not_provisioned',
so /health would never report ok/stale based on real heartbeats.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: chitcommit <noreply@chitty.cc>
chitcommit added a commit that referenced this pull request Jun 10, 2026
…l consumer (#109)

* feat(daemon): loop body wires executeIntent end-to-end (claim → dispatch → heartbeat)

Stacked on #106. Replaces the injected-executor abstraction in daemon/loop.ts
with a direct call to meta/intent.ts::executeIntent, closing the
meta-orchestrator loop. Status transitions, audit-row writes, and the second
sovereignty gate are all owned by executeIntent → dispatch; the loop's
responsibility is leader lifecycle, intent claiming, heartbeats, and outcome
classification.

Four outcomes are handled distinctly:

  - ok=true (executed)      → bump processed counter, reset error backoff
  - ok=true (replayed)      → bump replayed counter, no backoff, no double-count
  - ok=false (refused)      → bump refused counter, no backoff (steady-state)
  - ok=false (exec error)   → bump errored counter, bounded exp backoff

Sovereignty refusals are identified by canonical error prefixes emitted by
meta/executors/dispatch.ts ("sovereignty re-reckon:" /
"sovereignty snapshot stale ..."). Refusals are NOT treated as transient
faults — they are valid outcomes and do not trigger backoff.

The loop honors options.signal via AbortController throughout, including
inside the sleep helper, so SIGTERM from daemon/runtime/entrypoint.ts (PR #105)
unwinds cleanly through releaseLeadership.

tests/daemon/loop.spec.ts — real Neon integration. Seeds two pending intents
against the update_obligation_status executor, runs runLeaderLoop with
maxIntents=2, asserts:
  - both intents reach status='done'
  - each produces exactly one cc_actions_log row (attempt=1, key set, status='completed')
  - cc_obligations rows actually moved to 'deferred'
  - cc_node_leases shows leadership released on clean exit
  - log stream contains intent_heartbeat_before / intent_heartbeat_after pairs

Out of scope (not in this PR):
  - new executors (mercury, etc.)
  - production deploy
  - multi-node coordination beyond single-node leader
  - schema additions on cc_intents / cc_actions_log

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(health,wrangler): real-dependency /health probe + chittytrack tail consumer

The previous /health returned a static {status:"ok",...} regardless of
whether the worker could reach its real dependencies — a direct violation
of the chittyentity CLAUDE.md "No Mocks, Fake Data, or Placeholder
Endpoints" binding rule ("every endpoint must return real results against
a real datastore on the day it is committed").

This change replaces /health with a real probe that executes against the
worker's actual dependencies:

  * db            — SELECT 1 via Neon HTTP driver. Critical: failure -> 503.
  * chittyconnect — GET ${CHITTYCONNECT_URL}/health. Degraded if unreachable.
  * daemon        — newest cc_node_leases.heartbeat_at, stale if older than
                    2x daemon/loop.ts default heartbeatMs (10000ms ->
                    20000ms cutoff). Missing table -> not_provisioned
                    (degraded, NOT down) so deploys against bases without
                    #101 don't 503.

Per-dep timeout 2000ms; total probe bounded ≤ 5000ms via Promise.all.
Runs unauthenticated (no auth middleware on /health).

The probe handler is extracted to src/routes/health.ts so it can be
integration-tested in pure Node — importing src/index.ts directly drags
in `cloudflare:` modules (Agents SDK / DOs) that only resolve under
workerd. Integration test exercises the handler against a real Neon
branch (no mocks), validates DB probe ok and shape of all three probes,
and asserts 503 + status=down when DB is unreachable.

wrangler.jsonc already declares `tail_consumers: [{service: chittytrack}]`
(present on the base branch) — no change required there.

Verified against Neon branch br-spring-queen-akggkmso of project
cool-bar-13270800 (ChittyCommand). Sample real response:

{
  "status": "degraded",
  "service": "chittycommand",
  "version": "0.1.0",
  "timestamp": "2026-06-04T05:27:16.111Z",
  "probes": {
    "db": { "status": "ok", "latency_ms": 226 },
    "chittyconnect": {
      "status": "degraded", "latency_ms": 0,
      "error": "CHITTYCONNECT_URL not configured"
    },
    "daemon": {
      "status": "not_provisioned", "newest_heartbeat_age_ms": null,
      "error": "relation \"cc_node_leases\" does not exist"
    }
  }
}

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(health): query active leases by heartbeat_at, not released_at

Addresses Codex P2 on PR #109. The cc_node_leases schema in
src/db/schema.ts has no released_at column — release is represented by
NULLing heartbeat_at/lease_expires_at in daemon/leader.ts. The previous
WHERE released_at IS NULL clause raised 'column released_at does not
exist', which the catch block then mis-classified as 'not_provisioned',
so /health would never report ok/stale based on real heartbeats.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: chitcommit <noreply@chitty.cc>
chitcommit added a commit that referenced this pull request Jun 10, 2026
… start)

Stacked on #101. Adds the runtime supervisor surface for the meta-orchestrator
daemon foundation:

- daemon/runtime/entrypoint.ts — Node entrypoint wiring runLeaderLoop with
  SIGTERM/SIGINT graceful shutdown that releases the lease before exit.
- daemon/runtime/chittycommand-daemon.service — hardened systemd unit
  (Restart=always, dedicated chittycommand user, EnvironmentFile from
  /etc/chittycommand/env, journal logging).
- daemon/runtime/env.tmpl — op inject template; only op:// references, no
  secret values. 1Password stays the cold source of truth.
- daemon/runtime/launchd/com.chittyos.chittycommand-daemon.plist — macOS
  artifact for the Mac Mini cluster (not used on the VM, included so
  next-node bring-up doesn't need a separate PR).
- daemon/runtime/tsconfig.daemon.json + package.json build:daemon script —
  emits dist/daemon/runtime/entrypoint.js for the service ExecStart.
- scripts/install-daemon-vm.sh — idempotent VM bootstrap; supports --dry-run;
  creates user, builds, syncs to /opt/chittycommand, renders env via op
  inject, installs+enables the unit, STOPS WITHOUT STARTING.
- docs/runbooks/daemon-bring-up-vm.md — operator runbook (mint flow,
  verify, heartbeat, failure modes, rollback).
- daemon/supervisor.md — links to the runbook and pins chittyserv-vm as
  first-node target.

Operator action required: sudo systemctl start chittycommand-daemon.service
(after sudo -E ./scripts/install-daemon-vm.sh). See PR body for dry-run output.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants