-
-
Notifications
You must be signed in to change notification settings - Fork 90
feat(selfhost): add Gittensory Orb outcome signal collector (#1219) #1224
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
6f5ce0c
feat(selfhost): add Gittensory Orb outcome signal collector (#1219)
JSONbored 72bf610
test(selfhost): fix TS cast errors + close 100% branch coverage on ne…
JSONbored 7d692b7
fix(selfhost): require PUBLIC_API_ORIGIN for Orb setup wizard — preve…
JSONbored e1c0dd8
fix(selfhost): use ORB_WEBHOOK_SECRET/ORB_APP_ID in Orb collector + d…
JSONbored a9fca4f
fix(selfhost): replace strftime with CURRENT_TIMESTAMP in Orb migrations
JSONbored File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| -- Gittensory Orb (#1219): local outcome-signal store. Records the gate verdict and | ||
| -- final outcome (merged / closed) for every PR the engine reviewed. Used by the Orb | ||
| -- export job to batch-send calibration signals to the central collector (opt-in) or | ||
| -- to keep them local for operator-only analysis (ORB_AIR_GAP=true). | ||
| CREATE TABLE IF NOT EXISTS orb_events ( | ||
| id INTEGER PRIMARY KEY AUTOINCREMENT, | ||
| repo TEXT NOT NULL, | ||
| pr_number INTEGER NOT NULL, | ||
| head_sha TEXT NOT NULL, | ||
| outcome TEXT NOT NULL CHECK (outcome IN ('merged', 'closed')), | ||
| gate_verdict TEXT, -- 'approve' | 'block' | 'comment' | NULL (no review recorded) | ||
| time_to_close_ms INTEGER, -- ms from PR open to close; NULL if opened_at unavailable | ||
| created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), | ||
| exported_at TEXT, -- NULL = pending export; set when batch-sent to collector | ||
| UNIQUE (repo, pr_number, head_sha) -- idempotent: same close event may arrive more than once | ||
| ); | ||
| CREATE INDEX IF NOT EXISTS orb_events_repo_pr ON orb_events (repo, pr_number); | ||
| CREATE INDEX IF NOT EXISTS orb_events_export_pending ON orb_events (exported_at) WHERE exported_at IS NULL; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| -- Gittensory Orb (#1219): tracks which repos have the Orb GitHub App installed. | ||
| -- `removed_at IS NULL` = currently installed; set on uninstall/removal events. | ||
| CREATE TABLE IF NOT EXISTS orb_installations ( | ||
| id INTEGER PRIMARY KEY AUTOINCREMENT, | ||
| installation_id INTEGER NOT NULL, | ||
| repo TEXT NOT NULL, | ||
| installed_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')), | ||
| removed_at TEXT, -- NULL = still installed | ||
| UNIQUE (installation_id, repo) | ||
| ); | ||
| CREATE INDEX IF NOT EXISTS orb_installations_repo ON orb_installations (repo, removed_at); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| // Gittensory Orb (#1219) — local outcome-signal collector. Records gate verdict + final PR | ||
| // outcome (merged/closed) for every PR the engine reviewed, enabling calibration of gate | ||
| // thresholds and AI prompts from real-world feedback signals. | ||
| // | ||
| // Collection is always local (DB only). Export to the central collector is opt-in: | ||
| // ORB_ENABLED=true — activates collection (off by default) | ||
| // ORB_COLLECTOR_URL=<url> — endpoint to export batches to (default: https://orb.gittensory.app/v1/ingest) | ||
| // ORB_AIR_GAP=true — keep all events local, never send externally | ||
| // ORB_ANONYMIZE=true — HMAC-hash repo/owner before export (default: true) | ||
| // | ||
| // Nothing is ever sent without ORB_ENABLED=true. No diffs, no code, no comments, no user | ||
| // identifiers — only aggregate outcome metadata (repo-hash, verdict, outcome, timing). | ||
| import { createHash, createHmac } from "node:crypto"; | ||
| import { incr } from "./metrics"; | ||
|
|
||
| export interface OrbEvent { | ||
| repo: string; | ||
| pr_number: number; | ||
| head_sha: string; | ||
| outcome: "merged" | "closed"; | ||
| gate_verdict?: string; | ||
| time_to_close_ms?: number; | ||
| } | ||
|
|
||
| interface OrbRow { | ||
| id: number; | ||
| repo: string; | ||
| pr_number: number; | ||
| head_sha: string; | ||
| outcome: string; | ||
| gate_verdict: string | null; | ||
| time_to_close_ms: number | null; | ||
| created_at: string; | ||
| exported_at: string | null; | ||
| } | ||
|
|
||
| interface OrbExportPayload { | ||
| instance_id: string; | ||
| events: Array<{ | ||
| repo_hash: string; | ||
| pr_hash: string; | ||
| outcome: string; | ||
| gate_verdict: string | null; | ||
| time_to_close_ms: number | null; | ||
| created_at: string; | ||
| }>; | ||
| } | ||
|
|
||
| /** Stable instance identifier (hash of the App ID — no PII). */ | ||
| function instanceId(): string { | ||
| return createHash("sha256").update(process.env.GITHUB_APP_ID ?? "unknown").digest("hex").slice(0, 16); | ||
| } | ||
|
|
||
| /** HMAC a string with the webhook secret for anonymized export. */ | ||
| function hmacField(value: string, secret: string): string { | ||
| return createHmac("sha256", secret).update(value).digest("hex").slice(0, 24); | ||
| } | ||
|
|
||
| /** Returns true only when Orb collection is explicitly enabled. */ | ||
| export function orbEnabled(): boolean { | ||
| const v = (process.env.ORB_ENABLED ?? "").toLowerCase(); | ||
| return v === "true" || v === "1" || v === "yes"; | ||
| } | ||
|
|
||
| /** Record a single outcome event in the local DB. No-op when ORB_ENABLED is false. */ | ||
| export async function recordOrbEvent(db: D1Database, event: OrbEvent): Promise<void> { | ||
| if (!orbEnabled()) return; | ||
| try { | ||
| await db | ||
| .prepare( | ||
| `INSERT OR IGNORE INTO orb_events (repo, pr_number, head_sha, outcome, gate_verdict, time_to_close_ms) | ||
| VALUES (?, ?, ?, ?, ?, ?)`, | ||
| ) | ||
| .bind(event.repo, event.pr_number, event.head_sha, event.outcome, event.gate_verdict ?? null, event.time_to_close_ms ?? null) | ||
| .run(); | ||
| incr("gittensory_orb_events_recorded_total"); | ||
| } catch { | ||
| // best-effort — never let Orb collection crash job processing | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Export pending Orb events to the central collector. Called periodically (e.g. hourly). | ||
| * Reads up to `batchSize` unexported events, signs and POSTs them, marks them as exported. | ||
| * Returns the number of events exported (0 if air-gap, disabled, or nothing pending). | ||
| */ | ||
| export async function exportOrbBatch( | ||
| db: D1Database, | ||
| batchSize = 200, | ||
| fetchFn: typeof fetch = fetch, | ||
| ): Promise<number> { | ||
| if (!orbEnabled()) return 0; | ||
| if ((process.env.ORB_AIR_GAP ?? "").toLowerCase() === "true") return 0; | ||
|
|
||
| const collectorUrl = process.env.ORB_COLLECTOR_URL ?? "https://orb.gittensory.app/v1/ingest"; | ||
| const secret = process.env.GITHUB_WEBHOOK_SECRET ?? ""; | ||
|
JSONbored marked this conversation as resolved.
Outdated
|
||
| const anonymize = (process.env.ORB_ANONYMIZE ?? "true").toLowerCase() !== "false"; | ||
|
JSONbored marked this conversation as resolved.
|
||
|
|
||
| const { results } = await db | ||
| .prepare(`SELECT * FROM orb_events WHERE exported_at IS NULL ORDER BY id LIMIT ?`) | ||
| .bind(batchSize) | ||
| .all<OrbRow>(); | ||
|
|
||
| if (!results || results.length === 0) return 0; | ||
|
|
||
| const payload: OrbExportPayload = { | ||
| instance_id: instanceId(), | ||
| events: results.map((r) => ({ | ||
| repo_hash: anonymize ? hmacField(r.repo, secret) : r.repo, | ||
| pr_hash: anonymize ? hmacField(`${r.repo}#${r.pr_number}`, secret) : String(r.pr_number), | ||
| outcome: r.outcome, | ||
| gate_verdict: r.gate_verdict, | ||
| time_to_close_ms: r.time_to_close_ms, | ||
| created_at: r.created_at, | ||
| })), | ||
| }; | ||
|
|
||
| const body = JSON.stringify(payload); | ||
| const signature = createHmac("sha256", secret).update(body).digest("hex"); | ||
|
|
||
| try { | ||
| const res = await fetchFn(collectorUrl, { | ||
| method: "POST", | ||
| headers: { | ||
| "content-type": "application/json", | ||
| "x-orb-signature": `sha256=${signature}`, | ||
| "x-orb-instance": instanceId(), | ||
| }, | ||
| body, | ||
| }); | ||
| if (!res.ok) { | ||
| incr("gittensory_orb_export_errors_total"); | ||
| return 0; | ||
| } | ||
| } catch { | ||
| incr("gittensory_orb_export_errors_total"); | ||
| return 0; | ||
| } | ||
|
|
||
| // Mark all exported events | ||
| const ids = results.map((r) => r.id); | ||
| const placeholders = ids.map(() => "?").join(","); | ||
| const now = new Date().toISOString(); | ||
| await db.prepare(`UPDATE orb_events SET exported_at=? WHERE id IN (${placeholders})`).bind(now, ...ids).run(); | ||
|
|
||
| incr("gittensory_orb_events_exported_total", {}, ids.length); | ||
| return ids.length; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| // Gittensory Orb (#1219) setup wizard. Mirrors setup-wizard.ts but for the lightweight | ||
| // "Gittensory Orb" GitHub App — pull_requests:read + metadata:read + pull_request + | ||
| // installation events only. Creates a separate App so operators can install Orb | ||
| // independently of the main review App, and revoke data collection without touching reviews. | ||
| // | ||
| // Routes (server.ts): GET /orb/setup → form page; GET /orb/setup/callback → exchange code. | ||
|
|
||
| export interface OrbCredentials { | ||
| id: number; | ||
| slug: string; | ||
| webhook_secret: string; | ||
| pem: string; | ||
| } | ||
|
|
||
| /** Minimal Orb App manifest — read-only permissions, no write capabilities. */ | ||
| export function buildOrbManifest(origin: string, state: string): Record<string, unknown> { | ||
| const base = origin.replace(/\/+$/, ""); | ||
| return { | ||
| name: "Gittensory Orb", | ||
| url: base, | ||
| hook_attributes: { url: `${base}/orb/webhook` }, | ||
| redirect_url: `${base}/orb/setup/callback?state=${encodeURIComponent(state)}`, | ||
| public: false, | ||
| default_permissions: { | ||
| pull_requests: "read", | ||
| metadata: "read", | ||
| }, | ||
| default_events: ["pull_request", "installation", "installation_repositories"], | ||
| }; | ||
| } | ||
|
|
||
| /** HTML page with a single button that POSTs the manifest to GitHub's App-creation flow. */ | ||
| export function renderOrbSetupPage(origin: string, state: string): string { | ||
| const manifest = JSON.stringify(buildOrbManifest(origin, state)).replace(/'/g, "'"); | ||
| return `<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Gittensory Orb setup</title></head> | ||
| <body style="font-family:system-ui;max-width:40rem;margin:4rem auto;padding:0 1rem"> | ||
| <h1>Gittensory Orb setup</h1> | ||
| <p>This creates a lightweight read-only GitHub App that observes PR outcomes for local calibration | ||
| and optional aggregate telemetry. Install it on the same repositories as your main Gittensory App. | ||
| GitHub will redirect back here with the credentials — then restart the container to activate collection.</p> | ||
| <form action="https://github.com/settings/apps/new" method="post"> | ||
| <input type="hidden" name="manifest" value='${manifest}'> | ||
| <button type="submit" style="padding:.6rem 1.2rem;font-size:1rem;cursor:pointer">Create Gittensory Orb App →</button> | ||
| </form> | ||
| </body></html>`; | ||
| } | ||
|
|
||
| /** Exchange a one-time manifest code (from GitHub's callback) for the App's credentials. */ | ||
| export async function exchangeOrbManifestCode(code: string, fetchImpl: typeof fetch = fetch): Promise<OrbCredentials> { | ||
| const res = await fetchImpl(`https://api.github.com/app-manifests/${encodeURIComponent(code)}/conversions`, { | ||
| method: "POST", | ||
| headers: { accept: "application/vnd.github+json", "user-agent": "gittensory-selfhost" }, | ||
| }); | ||
| if (!res.ok) throw new Error(`orb_manifest_exchange_http_${res.status}`); | ||
| return (await res.json()) as OrbCredentials; | ||
| } | ||
|
|
||
| /** Serialize Orb credentials as env-file lines for the operator to load. */ | ||
| export function orbCredentialsToEnv(creds: OrbCredentials): string { | ||
| return [ | ||
| `ORB_APP_ID=${creds.id}`, | ||
| `ORB_APP_SLUG=${creds.slug}`, | ||
| `ORB_WEBHOOK_SECRET=${creds.webhook_secret}`, | ||
| `ORB_PRIVATE_KEY=${JSON.stringify(creds.pem)}`, | ||
| ].join("\n") + "\n"; | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.