Skip to content

feat(traffic): Cloudflare Worker connect + ingest API surface#636

Draft
arberx wants to merge 2 commits into
arberx/server-log-integrationsfrom
arberx/cloudflare-worker-routes
Draft

feat(traffic): Cloudflare Worker connect + ingest API surface#636
arberx wants to merge 2 commits into
arberx/server-log-integrationsfrom
arberx/cloudflare-worker-routes

Conversation

@arberx
Copy link
Copy Markdown
Member

@arberx arberx commented May 27, 2026

Summary

Wires the Cloudflare Worker traffic source end-to-end on the server side. Stacked on top of #635 — the foundation PR carrying the contracts, schema, and integration package.

After this PR:

  • Operator can POST /api/v1/projects/:name/traffic/connect/cloudflare and get back a generated Worker script + wrangler.toml
  • Operator deploys the Worker to their Cloudflare zone
  • Worker forwards filtered requests to POST /api/v1/projects/:name/traffic/cloudflare/ingest
  • Events land in the same classifier + hourly-rollup pipeline as the existing pull adapters

Connect endpoint

POST /api/v1/projects/:name/traffic/connect/cloudflare

  • Generates a fresh bearer token + HMAC secret + Worker script + wrangler.toml
  • Stores sha256(bearer) on traffic_sources.ingest_token_hash; cleartext secrets in ~/.canonry/config.yaml under cloudflareTraffic.connections.<sourceId>
  • Idempotent — reconnect rotates both secrets and re-emits the script, reusing the existing source row so rollups stay attached
  • Writes audit log entry traffic.cloudflare.connected
  • No upstream probe — the Worker is deployed by the operator after this call (push model has nothing to probe)

Ingest endpoint

POST /api/v1/projects/:name/traffic/cloudflare/ingest

The unique endpoint in canonry — opts out of the global cnry_* bearer check via shouldSkipAuth and carries its own per-source auth:

  1. Resolve source by X-Canonry-Source-Id header
  2. Constant-time bearer hash comparison
  3. verifyRequestSignature from integration-cloudflare-worker — ±300s timestamp window + HMAC-SHA256 over \${timestamp}.\${body}
  4. Any failure returns a single 401 envelope (never disambiguate which leg failed — defends against credential-leg enumeration)
  5. After auth: parses payload, normalizes via normalizeCloudflareWorkerEvent, runs buildTrafficProbeReport, upserts hourly rollups in a single transaction, updates last_worker_version

Response includes acceptedEvents, droppedEvents, workerVersionAck, plus per-bucket row counts so the Worker (or a debug curl) can see what landed.

What's plumbed

  • New package dep: api-routesintegration-cloudflare-worker
  • auth.tsshouldSkipAuth now skips /traffic/cloudflare/ingest URLs
  • traffic.tsCloudflareTrafficCredentialStore interface, both routes, CLOUDFLARE_WORKER_VERSION constant
  • index.ts — wires cloudflareTrafficCredentialStore + cloudflareTrafficIngestUrl through TrafficRoutesOptions
  • canonry/src/config.tsCloudflareTrafficConnectionConfigEntry
  • canonry/src/cloudflare-traffic-config.ts — CRUD helpers (NEW)
  • canonry/src/server.ts — constructs the store from config, derives ingest URL from publicUrl/apiUrl + basePath
  • canonry/src/client.tsApiClient.trafficConnectCloudflare()
  • MCP: new tool canonry_traffic_connect_cloudflare (write, idempotent). Ingest is classified excluded-protocol — wrong auth shape for an MCP caller
  • OpenAPI + regenerated SDK

Tests (TDD)

  • 19 new tests in packages/api-routes/test/traffic-cloudflare.test.ts:
    • Connect: source row + secrets persisted, embedded constants land in script, audit log written, idempotent reconnect rotates secrets, config persists bot list version, 404 on unknown project
    • Ingest: well-signed event lands in crawler bucket; last_worker_version updated; missing/wrong bearer/expired ts/unknown source-id/tampered body all return 401; empty events array / wrong schemaVersion return 400; AI-referral → referral bucket; AI-user-fetch UA → ai-user-fetch bucket; raw_event_samples row written
  • 12 new tests in packages/canonry/test/cloudflare-traffic-config.test.ts
  • Updated counts in mcp-registry.test.ts (85 → 86, traffic toolkit 10 → 11) and mcp-stdio.test.ts (87 → 88)
  • Updated DTO coverage explanations for the new traffic_sources columns in the foundation PR

Full workspace test passes (3412/3412), workspace typecheck clean, workspace lint clean (only pre-existing apps/web warnings).

Out of scope (next stacks)

  • Stack 3: CLI commands (`canonry traffic connect cloudflare`, `rotate`, `verify`)
  • Stack 4: Rotate endpoint + doctor checks (`cloudflare.worker.last-seen`, `version-stale`, `signature-failures`)
  • Stack 5: Dashboard connect-modal entry
  • Phase 2 (deferred): auto-deploy via Cloudflare API token, Cloudflare-as-proxy zone provisioning, Logpush sibling adapter

🤖 Generated with Claude Code

arberx and others added 2 commits May 27, 2026 12:59
Wires the Cloudflare Worker traffic source end-to-end on the server side
— the operator can now POST /traffic/connect/cloudflare to get a generated
Worker script and the deployed Worker can POST /traffic/cloudflare/ingest
to push events into the rollup pipeline.

Connect endpoint:
- Generates per-source bearer token + HMAC secret + Worker script
- Stores bearer hash on traffic_sources.ingest_token_hash; cleartext
  secrets to ~/.canonry/config.yaml under cloudflareTraffic.connections
- Idempotent — reconnect rotates secrets, reuses source row
- Writes audit log entry traffic.cloudflare.connected
- No upstream probe (push model — Worker is deployed by the operator)

Ingest endpoint:
- Opts out of the global cnry_* bearer check (custom per-source auth)
- Verifies bearer hash + HMAC-SHA256 signature + ±300s timestamp window
- Single 401 envelope on any auth failure (no leg-disambiguation)
- Normalizes events, runs the shared classifier + rollup pipeline,
  updates last_worker_version
- Returns accepted/dropped counts + per-bucket counters

Plus:
- ApiClient.trafficConnectCloudflare()
- MCP tool canonry_traffic_connect_cloudflare (ingest is excluded —
  wrong auth shape for an MCP caller)
- OpenAPI spec + regenerated SDK
- 19 new tests in packages/api-routes/test/traffic-cloudflare.test.ts
  (connect + ingest happy paths, every auth-failure leg, payload
  validation, AI-referral / AI-user-fetch routing, sample row write)
- 12 new tests for the cloudflare-traffic-config helpers

Stacked on arberx/server-log-integrations (foundation PR #635).
Next stack: CLI commands + doctor checks + dashboard.

Full workspace test passes (3412/3412).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
observedAt was set inside buildEvent, which only ran from forward() after
event.waitUntil fired — i.e. after the upstream response resolved or
rejected. The field reflected response completion (or full upstream
timeout on the error path), not request entry, and could push events
into the wrong hourly rollup bucket on slow origins.

Capture observedAt synchronously at the top of the fetch handler and
thread it through forward() → buildEvent() so the field matches its
name.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant