fix(health,wrangler): real-dependency /health probe + chittytrack tail consumer - #109
Merged
chitcommit merged 4 commits intoJun 4, 2026
Conversation
…tch → heartbeat) Stacked on #106. Replaces the injected-executor abstraction in daemon/loop.ts with a direct call to meta/intent.ts::executeIntent, closing the meta-orchestrator loop. Status transitions, audit-row writes, and the second sovereignty gate are all owned by executeIntent → dispatch; the loop's responsibility is leader lifecycle, intent claiming, heartbeats, and outcome classification. Four outcomes are handled distinctly: - ok=true (executed) → bump processed counter, reset error backoff - ok=true (replayed) → bump replayed counter, no backoff, no double-count - ok=false (refused) → bump refused counter, no backoff (steady-state) - ok=false (exec error) → bump errored counter, bounded exp backoff Sovereignty refusals are identified by canonical error prefixes emitted by meta/executors/dispatch.ts ("sovereignty re-reckon:" / "sovereignty snapshot stale ..."). Refusals are NOT treated as transient faults — they are valid outcomes and do not trigger backoff. The loop honors options.signal via AbortController throughout, including inside the sleep helper, so SIGTERM from daemon/runtime/entrypoint.ts (PR #105) unwinds cleanly through releaseLeadership. tests/daemon/loop.spec.ts — real Neon integration. Seeds two pending intents against the update_obligation_status executor, runs runLeaderLoop with maxIntents=2, asserts: - both intents reach status='done' - each produces exactly one cc_actions_log row (attempt=1, key set, status='completed') - cc_obligations rows actually moved to 'deferred' - cc_node_leases shows leadership released on clean exit - log stream contains intent_heartbeat_before / intent_heartbeat_after pairs Out of scope (not in this PR): - new executors (mercury, etc.) - production deploy - multi-node coordination beyond single-node leader - schema additions on cc_intents / cc_actions_log Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…l consumer
The previous /health returned a static {status:"ok",...} regardless of
whether the worker could reach its real dependencies — a direct violation
of the chittyentity CLAUDE.md "No Mocks, Fake Data, or Placeholder
Endpoints" binding rule ("every endpoint must return real results against
a real datastore on the day it is committed").
This change replaces /health with a real probe that executes against the
worker's actual dependencies:
* db — SELECT 1 via Neon HTTP driver. Critical: failure -> 503.
* chittyconnect — GET ${CHITTYCONNECT_URL}/health. Degraded if unreachable.
* daemon — newest cc_node_leases.heartbeat_at, stale if older than
2x daemon/loop.ts default heartbeatMs (10000ms ->
20000ms cutoff). Missing table -> not_provisioned
(degraded, NOT down) so deploys against bases without
#101 don't 503.
Per-dep timeout 2000ms; total probe bounded ≤ 5000ms via Promise.all.
Runs unauthenticated (no auth middleware on /health).
The probe handler is extracted to src/routes/health.ts so it can be
integration-tested in pure Node — importing src/index.ts directly drags
in `cloudflare:` modules (Agents SDK / DOs) that only resolve under
workerd. Integration test exercises the handler against a real Neon
branch (no mocks), validates DB probe ok and shape of all three probes,
and asserts 503 + status=down when DB is unreachable.
wrangler.jsonc already declares `tail_consumers: [{service: chittytrack}]`
(present on the base branch) — no change required there.
Verified against Neon branch br-spring-queen-akggkmso of project
cool-bar-13270800 (ChittyCommand). Sample real response:
{
"status": "degraded",
"service": "chittycommand",
"version": "0.1.0",
"timestamp": "2026-06-04T05:27:16.111Z",
"probes": {
"db": { "status": "ok", "latency_ms": 226 },
"chittyconnect": {
"status": "degraded", "latency_ms": 0,
"error": "CHITTYCONNECT_URL not configured"
},
"daemon": {
"status": "not_provisioned", "newest_heartbeat_age_ms": null,
"error": "relation \"cc_node_leases\" does not exist"
}
}
}
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
chittycommand | 35506e4 | Jun 04 2026, 12:59 PM |
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
To use Codex here, create a Codex account and connect to github. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0a11bbd89
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const rows = (await withTimeout( | ||
| sql`SELECT EXTRACT(EPOCH FROM (NOW() - max(heartbeat_at))) * 1000 AS age_ms | ||
| FROM cc_node_leases | ||
| WHERE released_at IS NULL`, |
There was a problem hiding this comment.
In environments where cc_node_leases is provisioned, this query still fails because the table defined in src/db/schema.ts has no released_at column; releases are represented by nulling heartbeat_at/lease_expires_at in daemon/leader.ts. The catch block then matches the resulting “column released_at does not exist” error as not_provisioned, so /health will never report the daemon as ok or stale based on real heartbeats once the table exists.
Useful? React with 👍 / 👎.
Contributor
Author
There was a problem hiding this comment.
Fixed in e476115. Replaced WHERE released_at IS NULL with WHERE heartbeat_at IS NOT NULL to match the actual release semantics in daemon/leader.ts (release = NULL out heartbeat_at + lease_expires_at). Once cc_node_leases is provisioned with held leases the probe will now return ok/stale based on real heartbeat age instead of mis-classifying the SQL error as not_provisioned.
Addresses Codex P2 on PR #109. The cc_node_leases schema in src/db/schema.ts has no released_at column — release is represented by NULLing heartbeat_at/lease_expires_at in daemon/leader.ts. The previous WHERE released_at IS NULL clause raised 'column released_at does not exist', which the catch block then mis-classified as 'not_provisioned', so /health would never report ok/stale based on real heartbeats. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
To use Codex here, create a Codex account and connect to github. |
Base automatically changed from
feat/daemon-loop-executes-intents
to
feat/meta-executors-registry
June 4, 2026 12:58
…e-and-tail-consumer
chitcommit
merged commit Jun 4, 2026
45d4dff
into
feat/meta-executors-registry
9 of 11 checks passed
|
|
To use Codex here, create a Codex account and connect to github. |
chitcommit
added a commit
that referenced
this pull request
Jun 10, 2026
…l consumer (#109) * feat(daemon): loop body wires executeIntent end-to-end (claim → dispatch → heartbeat) Stacked on #106. Replaces the injected-executor abstraction in daemon/loop.ts with a direct call to meta/intent.ts::executeIntent, closing the meta-orchestrator loop. Status transitions, audit-row writes, and the second sovereignty gate are all owned by executeIntent → dispatch; the loop's responsibility is leader lifecycle, intent claiming, heartbeats, and outcome classification. Four outcomes are handled distinctly: - ok=true (executed) → bump processed counter, reset error backoff - ok=true (replayed) → bump replayed counter, no backoff, no double-count - ok=false (refused) → bump refused counter, no backoff (steady-state) - ok=false (exec error) → bump errored counter, bounded exp backoff Sovereignty refusals are identified by canonical error prefixes emitted by meta/executors/dispatch.ts ("sovereignty re-reckon:" / "sovereignty snapshot stale ..."). Refusals are NOT treated as transient faults — they are valid outcomes and do not trigger backoff. The loop honors options.signal via AbortController throughout, including inside the sleep helper, so SIGTERM from daemon/runtime/entrypoint.ts (PR #105) unwinds cleanly through releaseLeadership. tests/daemon/loop.spec.ts — real Neon integration. Seeds two pending intents against the update_obligation_status executor, runs runLeaderLoop with maxIntents=2, asserts: - both intents reach status='done' - each produces exactly one cc_actions_log row (attempt=1, key set, status='completed') - cc_obligations rows actually moved to 'deferred' - cc_node_leases shows leadership released on clean exit - log stream contains intent_heartbeat_before / intent_heartbeat_after pairs Out of scope (not in this PR): - new executors (mercury, etc.) - production deploy - multi-node coordination beyond single-node leader - schema additions on cc_intents / cc_actions_log Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(health,wrangler): real-dependency /health probe + chittytrack tail consumer The previous /health returned a static {status:"ok",...} regardless of whether the worker could reach its real dependencies — a direct violation of the chittyentity CLAUDE.md "No Mocks, Fake Data, or Placeholder Endpoints" binding rule ("every endpoint must return real results against a real datastore on the day it is committed"). This change replaces /health with a real probe that executes against the worker's actual dependencies: * db — SELECT 1 via Neon HTTP driver. Critical: failure -> 503. * chittyconnect — GET ${CHITTYCONNECT_URL}/health. Degraded if unreachable. * daemon — newest cc_node_leases.heartbeat_at, stale if older than 2x daemon/loop.ts default heartbeatMs (10000ms -> 20000ms cutoff). Missing table -> not_provisioned (degraded, NOT down) so deploys against bases without #101 don't 503. Per-dep timeout 2000ms; total probe bounded ≤ 5000ms via Promise.all. Runs unauthenticated (no auth middleware on /health). The probe handler is extracted to src/routes/health.ts so it can be integration-tested in pure Node — importing src/index.ts directly drags in `cloudflare:` modules (Agents SDK / DOs) that only resolve under workerd. Integration test exercises the handler against a real Neon branch (no mocks), validates DB probe ok and shape of all three probes, and asserts 503 + status=down when DB is unreachable. wrangler.jsonc already declares `tail_consumers: [{service: chittytrack}]` (present on the base branch) — no change required there. Verified against Neon branch br-spring-queen-akggkmso of project cool-bar-13270800 (ChittyCommand). Sample real response: { "status": "degraded", "service": "chittycommand", "version": "0.1.0", "timestamp": "2026-06-04T05:27:16.111Z", "probes": { "db": { "status": "ok", "latency_ms": 226 }, "chittyconnect": { "status": "degraded", "latency_ms": 0, "error": "CHITTYCONNECT_URL not configured" }, "daemon": { "status": "not_provisioned", "newest_heartbeat_age_ms": null, "error": "relation \"cc_node_leases\" does not exist" } } } Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(health): query active leases by heartbeat_at, not released_at Addresses Codex P2 on PR #109. The cc_node_leases schema in src/db/schema.ts has no released_at column — release is represented by NULLing heartbeat_at/lease_expires_at in daemon/leader.ts. The previous WHERE released_at IS NULL clause raised 'column released_at does not exist', which the catch block then mis-classified as 'not_provisioned', so /health would never report ok/stale based on real heartbeats. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: chitcommit <noreply@chitty.cc>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Compliance audit verdict (chittyos-compliance):
This PR fixes that.
What
/healthis now realOld (
src/index.ts, before):New (
src/index.ts, after) — delegates tosrc/routes/health.ts:Probes (each per-dep timeout 2000ms, total ≤ 5000ms via
Promise.all):dbSELECT 1via Neon HTTP driverstatus: down, HTTP 503chittyconnectGET ${CHITTYCONNECT_URL}/healthdegraded(200) — chittycommand survives brief outagesdaemonmax(heartbeat_at) FROM cc_node_leases WHERE released_at IS NULLstaleif > 2×daemon/loop.tsdefault heartbeatMs (10000 → 20000ms cutoff). Missing table →not_provisioned(degraded, NOT down) — so deploys against bases without #101 don't 503Runs unauthenticated —
/healthlives above theapp.use('/api/*', authMiddleware)mount, so the probe never touches auth.The probe handler is extracted to
src/routes/health.tsso the integration test can exercise it in pure Node — importingsrc/index.tsdirectly drags incloudflare:namespaced modules (Agents SDK / Durable Objects) that only resolve under workerd.wrangler.jsonctail_consumers: [{ "service": "chittytrack" }]is already present on the base branch (lines 84–86). No change needed — verified.Real Neon validation
Validated via the Neon MCP against a disposable branch of the ChittyCommand project (
cool-bar-13270800/br-spring-queen-akggkmso):This exercises both the happy path (db probe ok) and the
not_provisionedfallback (parent branch has nocc_node_leasessince #101 hasn't merged yet).Sample real
/healthresponse captured by runningrunHealthProbesagainst the live Neon branch (no mocks):{ "status": "degraded", "service": "chittycommand", "version": "0.1.0", "timestamp": "2026-06-04T05:27:16.111Z", "probes": { "db": { "status": "ok", "latency_ms": 226 }, "chittyconnect": { "status": "degraded", "latency_ms": 0, "error": "CHITTYCONNECT_URL not configured" }, "daemon": { "status": "not_provisioned", "newest_heartbeat_age_ms": null, "error": "relation \"cc_node_leases\" does not exist" } } }HTTP status: 200 (db ok → not down; chittyconnect/daemon degraded surfaces as
degradednotdown).Test plan
npm run typecheckcleanSKIP_INTEGRATION=1 npm test— 15 passed / 15 skipped (no regressions)DATABASE_URL=<neon-branch> npx vitest run tests/health/health-probe.spec.ts— 2/2 pass against real Neonokwithlatency_ms > 0against the real Neon branchok | stale | not_provisioned(under-asserted — depends on whether the branch has cc_node_leases provisioned and an active node)status: down, HTTP 503,probes.db.status === 'down'Not in this PR (explicitly)
/healthremains unauthenticated, same as before)meta/,daemon/, or the executor registrywrangler.jsoncchanges —tail_consumers: chittytrackwas already presentAssumptions documented inline
daemon/loop.tsdefaultheartbeatMs(10000ms) = 20000ms. If the daemon is run with a differentheartbeatMs, this cutoff would need to be reconsidered. Documented insrc/routes/health.ts:25-26.Auto-merge OFF. No deploy.
🤖 Generated with Claude Code