Orq — AI Incident Response Operator

Durable, auditable, AI-driven incident response. Production alert fires → Claude analyzes logs → Slack approval gate → postmortem drafted. Runs on Temporal Cloud so every step is retried, observable, and resumable.

---

What Orq does

When a production incident fires, Orq runs a durable 10-step workflow automatically:

1. Fetch service logs — real logs from the affected service (via Datadog or direct webhook)
2. AI root cause analysis — Claude reads the logs and produces a structured RCA with evidence
3. Find service owner — identifies the team responsible
4. Search similar incidents — finds past incidents and resolutions for context
5. Create PagerDuty incident — pages the on-call engineer
6. Create Jira ticket — opens a tracked ops ticket
7. Post Slack alert — structured Block Kit message to #incidents with the RCA summary
8. Request human approval — interactive Slack gate: Approve / Reject the AI remediation plan
9. Draft postmortem — Claude writes a blameless postmortem once approved
10. Post postmortem to Slack — notifies the team it's ready

Every step runs as a Temporal activity — retried automatically on failure, durably paused at the approval gate, and fully observable in the execution timeline.

---

Architecture

checkout-service (Railway)
   │  POST /simulate/{scenario}
   │  → ships logs to Datadog (backdated 5 min for query window)
   │  → fires Orq webhook with logs[] in payload
   ▼
Orq API  (Railway — FastAPI)
   │  normalizes webhook → starts Temporal workflow
   ▼
Temporal Cloud
   │  orchestrates durable DAG execution
   ▼
Orq Worker  (Railway — Temporal worker)
   ├─ logs.fetch       → real logs from webhook payload (source: checkout-service)
   │                      OR Datadog search (source: datadog)
   │                      OR scenario logs (source: demo)
   ├─ ai.root_cause_analysis  → Claude Sonnet 4.6
   ├─ ai.find_service_owner   → Claude Haiku
   ├─ incidents.search_history
   ├─ pagerduty.create_incident
   ├─ jira.create_incident_ticket
   ├─ slack.send_incident_alert
   ├─ slack.request_approval  → durable pause in Temporal
   │       ↑ human clicks Approve / Reject in Slack
   ├─ ai.draft_postmortem     → Claude Sonnet 4.6
   └─ slack.send_message
   ▼
Orq Frontend (Railway — Next.js)
   └─ Live execution timeline, provenance strip, approval inbox

---

Demo flow

1. Open Orq
2. Select AI Incident Response Operator template → Use
3. Scroll to the Simulate Incident panel
4. Click DB Migration Failure
5. Watch the execution appear and run live (10 nodes, ~2 minutes)
6. Approve the remediation in Slack (#incidents)
7. See the postmortem drafted and posted

The full chain — trigger → logs → RCA → approval → postmortem — completes without human intervention except the deliberate approval gate.

---

What is real

Component	Status	Notes
checkout-service	✅ Real deployed service	FastAPI on Railway — real HTTP endpoints, real failure simulation
Logs	✅ Real service logs	Generated by checkout-service, sent in webhook payload and to Datadog
Datadog	✅ Real ingestion	Logs shipped to real Datadog account (backdated 5 min for query window)
Temporal Cloud	✅ Real orchestration	Actual Temporal Cloud namespace, durable workflows, real retry/resume
Claude RCA	✅ Real AI	Claude Sonnet 4.6 via Anthropic API — real model, real analysis
Slack	✅ Real integration	Real Slack workspace, Block Kit messages, interactive approval buttons
Approval gate	✅ Real durable pause	Temporal workflow durably suspended until Slack button clicked

What is simulated

Component	Status	Notes
checkout-service failure	⚠️ Controlled	In-memory state flag — not a real production system going down
PagerDuty	⚠️ Simulated	No real PD key configured — returns mock incident ID. Labeled Simulated in UI
Jira	⚠️ Simulated	No real Jira credentials — returns mock ticket key. Labeled Simulated in UI
Similar incidents	⚠️ Hardcoded	Returns plausible historical incidents, not a live incident database
Metrics bar	⚠️ Demo workspace	Counts are from demo runs in this workspace, not real production incidents

The UI clearly labels simulated integrations with a Simulated badge on each node row. We are not hiding this.

---

Why Temporal matters

Orq's value depends on durability — an incident response workflow that can:

• Retry a failed Slack API call without restarting the whole workflow
• Pause durably at a human approval gate — even if the server restarts, the workflow resumes from exactly where it left off
• Audit every step — Temporal's event history is the source of truth for what happened and when
• Parallelize — Find Owner and Search Similar Incidents run concurrently in the DAG

A naive async task queue cannot provide durability + observability + human-in-the-loop in one primitive. Temporal can.

---

Execution provenance

Every run shows a full provenance strip in the UI:

TRIGGER: checkout-service → LOGS: checkout-service · 10 logs → RCA: Claude AI → APPROVAL: Slack → RUNTIME: Temporal

The logs badge is honest about its source:

Badge	Color	Meaning
checkout-service	🟢 Teal	Real logs from the deployed service, passed directly in webhook payload
Datadog	🟣 Purple	Real logs fetched from Datadog search API (requires DATADOG_APP_KEY)
Demo scenario	🟡 Amber	Scenario logs generated by Orq (no real app logs in Datadog)

---

Stack

Layer	Technology
Orchestration	Temporal Cloud
AI	Anthropic Claude Sonnet 4.6 + Haiku
API	FastAPI (Python 3.11)
Worker	Temporal Python SDK
Frontend	Next.js 14, Tailwind CSS
Log ingestion	Datadog Logs API
Notifications	Slack (Block Kit + interactive buttons)
Demo service	FastAPI (checkout-service)
Hosting	Railway (4 services)

---

How to run locally

Prerequisites

• Python 3.11+, Node.js 20+
• Temporal CLI (brew install temporal or docs)
• API keys: ANTHROPIC_API_KEY, SLACK_BOT_TOKEN
• Optional: DATADOG_API_KEY, DATADOG_APP_KEY for real Datadog integration

Backend

cd backend
pip install -r requirements.txt
cp .env.example .env          # fill in your keys

# Terminal 1 — Temporal dev server
temporal server start-dev

# Terminal 2 — FastAPI
uvicorn app.main:app --reload --port 8000

# Terminal 3 — Temporal worker
python -m app.temporal.run_worker

Frontend

cd frontend
npm install
cp .env.local.example .env.local
# NEXT_PUBLIC_API_URL=http://localhost:8000/api/v1
npm run dev

checkout-service (optional — for full demo)

cd checkout-service
pip install -r requirements.txt
cp .env.example .env
# DATADOG_API_KEY=... ORQ_WEBHOOK_URL=http://localhost:8000/api/v1/webhooks/datadog/tpl-incident-response
uvicorn main:app --reload --port 8001

Set NEXT_PUBLIC_CHECKOUT_SERVICE_URL=http://localhost:8001 in frontend/.env.local.

---

Triggering via webhook (no checkout-service needed)

curl -X POST https://orq-runtime-production.up.railway.app/api/v1/webhooks/datadog/tpl-incident-response \
  -H "Content-Type: application/json" \
  -d '{
    "title": "DB migration dropped orders table — checkout 100% down",
    "service": "checkout-service",
    "environment": "production",
    "severity": "critical",
    "source": "datadog",
    "alert_type": "error"
  }'

---

Roadmap

• Real Datadog monitor integration — Orq as a webhook destination triggered by real DD monitors
• Multi-tenant workspaces — separate keys, execution history, and channels per team
• Real PagerDuty + Jira — replace simulated integrations with production-grade ones
• Auto-remediation actions — kubectl rollout restart, feature flag toggle, cache flush
• Runbook library — attach runbooks to services; AI picks the right one per incident
• MTTR analytics — track resolution time across teams and services over time
• On-call calendar integration — PagerDuty / OpsGenie schedule awareness

---

Built with Temporal · Anthropic Claude · Railway