Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Before designing, understand what you are building observability for. The metric

**If the user points you to a codebase**: Read it. Look at the entry points, the API routes, the database models, the queue consumers, the external service calls. Understanding the code gives you the context to choose metrics that actually matter — not just generic RED/USE signals.

**If the user describes the business**: Use that context to tailor the Customer-Facing section. An e-commerce service cares about checkout success rates. A messaging service cares about delivery latency. A payment service cares about transaction completion. Generic "request rate" and "error rate" are a starting point, but the real value comes from metrics that map to business outcomes.
**If the user describes the business**: Use that context to tailor the Business (`B`) section. An e-commerce service cares about checkout success rates. A messaging service cares about delivery latency. A payment service cares about transaction completion. Generic "request rate" and "error rate" are a starting point, but the real value comes from metrics that map to customer-visible outcomes.

**Skip domain discovery if**: You already have deep context about the service from prior conversations or the user has provided detailed specifications.

Expand Down Expand Up @@ -81,11 +81,11 @@ Using your domain understanding and the chosen framework, design the group struc
- **[thresholds.md](references/thresholds.md)** — Alert threshold markers, threshold proximity, Y-axis configuration

**Key principles** (not rigid rules — use judgment):
- **Prefix every widget title** with its layer and priority: `I0:` (most critical infra), `P0:` (most critical platform), `D0:` (most critical domain). See [widgets.md](references/widgets.md) for the full prefix system.
- Start with a **Customer-Facing** group (5-8 metrics) so someone with zero service knowledge can tell if customers are affected within 5 seconds. Tailor the metrics to the domain.
- **Prefix every widget title** with its layer and priority: `I0:` (most critical infra), `P0:` (most critical platform), `D0:` (most critical domain), `B0:` (most critical business). See [widgets.md](references/widgets.md) for the full prefix system.
- Start with a **Business** group (5-8 `B`-prefixed metrics) so someone with zero service knowledge can tell if customers are affected within 5 seconds. Tailor the metrics to the domain.
- Timeseries widgets should have **alert threshold markers** (red lines) with thresholds close to normal traffic. If a metric doesn't warrant an alert, question whether it belongs — but context-providing metrics can earn their place.
- Set **Y-axis max** explicitly near the threshold — don't let auto-scaling compress the normal range.
- Order groups macro-to-micro: customer-facing → overview → domain-specific → infrastructure.
- Order groups macro-to-micro: business → overview → domain-specific → infrastructure.

### 4. Write the output

Expand Down Expand Up @@ -130,12 +130,12 @@ pup metrics list --filter="trace.http.request.*"
- [ ] Dashboard reflects the actual product and business — metrics tailored to the domain
- [ ] Dashboard title is concise (no environment, region, or version)
- [ ] Template variables defined for env, service, and relevant scopes (default `*`)
- [ ] **Customer-Facing group** with 5-8 metrics tailored to the service's business impact
- [ ] Groups ordered macro-to-micro (customer-facing → overview → details)
- [ ] **Business group** with 5-8 `B`-prefixed metrics tailored to the service's customer-visible outcomes
- [ ] Groups ordered macro-to-micro (business → overview → details)
- [ ] **Timeseries widgets have alert threshold markers** (red lines) where the metric is alertable
- [ ] **Thresholds close to normal traffic** — no excessive whitespace
- [ ] **Zero-knowledge readability** — someone with no service knowledge can spot problems via red indicators
- [ ] **Widget titles prefixed** with layer and priority (`I0:`, `P1:`, `D0:`, etc.)
- [ ] **Widget titles prefixed** with layer and priority (`I0:`, `P1:`, `D0:`, `B0:`, etc.)
- [ ] Widget titles use sentence case, don't repeat group name
- [ ] Every metric earns its place — if it spikes, someone can act on it

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ Template variables make one dashboard serve many contexts. Define them before la

| Approach | Strengths | Weaknesses |
|----------|-----------|------------|
| **Per-service** | Focused, fast to scan during incidents. Each team owns their dashboard. Customer-Facing section is specific and actionable. Ops reviews can go service-by-service. | More dashboards to maintain. Cross-service correlation requires switching dashboards. |
| **Consolidated** | Single pane of glass for multiple services. Good for seeing cross-service dependencies. Fewer dashboards to maintain. | Can become overwhelming (100+ metrics). Customer-Facing section becomes diluted. Slower to load and harder to scan during incidents. |
| **Hybrid** | Per-service dashboards for depth + one top-level dashboard with only the Customer-Facing section from each service. Best of both worlds. | Requires maintaining both levels. Customer-Facing metrics duplicated across dashboards. |
| **Per-service** | Focused, fast to scan during incidents. Each team owns their dashboard. Business section is specific and actionable. Ops reviews can go service-by-service. | More dashboards to maintain. Cross-service correlation requires switching dashboards. |
| **Consolidated** | Single pane of glass for multiple services. Good for seeing cross-service dependencies. Fewer dashboards to maintain. | Can become overwhelming (100+ metrics). Business section becomes diluted. Slower to load and harder to scan during incidents. |
| **Hybrid** | Per-service dashboards for depth + one top-level dashboard with only the Business section from each service. Best of both worlds. | Requires maintaining both levels. Business metrics duplicated across dashboards. |

---

Expand All @@ -41,7 +41,7 @@ Organize widgets into collapsible groups. Groups are the primary navigation mech

**Recommended groups** (in order):

1. **Customer-Facing** — 5-8 metrics that answer "are customers affected?" within 5 seconds. Should be the first group. The specific metrics depend on the product. Design so someone with zero service knowledge can spot problems via red indicators.
1. **Business** — 5-8 `B`-prefixed metrics that answer "are customers affected?" within 5 seconds. Should be the first group. The specific metrics depend on the product. Design so someone with zero service knowledge can spot problems via red indicators.
2. **Overview** — Service checks, key health indicators, monitor summaries.
3. **Domain-specific groups** — Organized by the chosen framework (e.g., Rate / Errors / Duration for RED), adapted to the service's actual architecture and concerns.

Expand All @@ -66,12 +66,12 @@ The most common dashboard type. Monitors a single service's request-level health
┌─────────────────────────────────────────────────────┐
│ Template Variables: env | service | region │
├─────────────────────────────────────────────────────┤
│ Group: Customer-Facing
│ Group: Business
│ ┌──────────┬──────────┬──────────┬──────────┐ │
│ │ P0:Req/s │ P0:Errs │ P0:p99 │ P0:Apdex │ │
│ │ B0:Req/s │ B0:Errs │ B0:p99 │ B0:Apdex │ │
│ │ (QV+bg) │ (QV+bg) │ (QV+bg) │ (QV+bg) │ │
│ ├──────────────────────┬────────────────────┤ │
│ │ D0: Key txn success │ I0: DB conn pool │ │
│ │ B0: Key txn success │ B1: DB conn pool │ │

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Infrastructure metrics incorrectly re-prefixed as Business layer

Medium Severity

Several infrastructure metrics in layout templates were blindly re-prefixed from I to B when the "Customer-Facing" group was renamed to "Business." Specifically: B1: DB conn pool (lines 74, 250) was formerly I0: DB conn pool, and B0: Host avail. / B1: Network errors (line 136) were formerly I0: prefixed. These are infrastructure metrics per the classification guide in widgets.md (databases, networks, OS resources), not business outcomes. The widgets.md examples were updated correctly to exclude non-B metrics from the Business group, but the layout templates were not — creating an internal inconsistency that will confuse the AI skill and produce incorrectly classified dashboards.

Additional Locations (2)
Fix in Cursor Fix in Web

│ │ (TS + red threshold) │ (TS + red thresh.) │ │
│ └──────────────────────┴────────────────────┘ │
│ 5-8 metrics · zero-knowledge readable │
Expand Down Expand Up @@ -112,7 +112,7 @@ The most common dashboard type. Monitors a single service's request-level health
└─────────────────────────────────────────────────────┘
```

All widget titles use the layer-priority prefix system (`I0:`, `P0:`, `D0:`, etc.) — see [widgets.md](widgets.md) for details.
All widget titles use the layer-priority prefix system (`I0:`, `P0:`, `D0:`, `B0:`, etc.) — see [widgets.md](widgets.md) for details.
All timeseries widgets include red alert threshold markers set close to normal traffic.

**Widget count**: 20-24
Expand All @@ -128,12 +128,12 @@ Monitors host, container, or VM resource health.
┌─────────────────────────────────────────────────────┐
│ Template Variables: env | host | availability_zone │
├─────────────────────────────────────────────────────┤
│ Group: Customer-Facing
│ Group: Business
│ ┌──────────┬──────────┬──────────┬──────────┐ │
│ │P0:Svc Rq │P0:Svc Err│P0:Svc p99│P0:Apdex │ │
│ │B0:Svc Rq │B0:Svc Err│B0:Svc p99│B0:Apdex │ │
│ │ (QV+bg) │ (QV+bg) │ (QV+bg) │ (QV+bg) │ │
│ ├──────────────────────┬────────────────────┤ │
│ │ I0: Host avail. │ I0: Network errors │ │
│ │ B0: Host avail. │ B1: Network errors │ │
│ │ (TS + red threshold) │ (TS + red thresh.) │ │
│ └──────────────────────┴────────────────────┘ │
│ 5-8 metrics · zero-knowledge readable │
Expand Down Expand Up @@ -186,12 +186,12 @@ High-level view across multiple services for leadership.
┌─────────────────────────────────────────────────────┐
│ Template Variables: env | team | region │
├─────────────────────────────────────────────────────┤
│ Group: Customer-Facing
│ Group: Business
│ ┌──────────┬──────────┬──────────┬──────────┐ │
│ │P0:Uptime │P0:Cst Err│P0:p99 Lat│P0:Traffic│ │
│ │B0:Uptime │B0:Cst Err│B0:p99 Lat│B0:Traffic│ │
│ │ (QV+bg) │ (QV+bg) │ (QV+bg) │ (QV+bg) │ │
│ ├──────────────────────┬────────────────────┤ │
│ │ D0: Revenue txn succ │ D0: Checkout lat │ │
│ │ B0: Revenue txn succ │ B0: Checkout lat │ │
│ │ (TS + red threshold) │ (TS + red thresh.) │ │
│ └──────────────────────┴────────────────────┘ │
│ 5-8 metrics · zero-knowledge readable │
Expand Down Expand Up @@ -242,12 +242,12 @@ Deep-dive dashboard for active incident investigation.
┌─────────────────────────────────────────────────────┐
│ Template Variables: env | service | host | endpoint │
├─────────────────────────────────────────────────────┤
│ Group: Customer-Facing
│ Group: Business
│ ┌──────────┬──────────┬──────────┬──────────┐ │
│ │P0:Cst Err│P0:p99 Lat│P0:Req/s │P0:Apdex │ │
│ │B0:Cst Err│B0:p99 Lat│B0:Req/s │B0:Apdex │ │
│ │ (QV+bg) │ (QV+bg) │ (QV+bg) │ (QV+bg) │ │
│ ├──────────────────────┬────────────────────┤ │
│ │ D0: Key txn success │ I0: DB conn pool │ │
│ │ B0: Key txn success │ B1: DB conn pool │ │
│ │ (TS + red threshold) │ (TS + red thresh.) │ │
│ └──────────────────────┴────────────────────┘ │
│ 5-8 metrics · zero-knowledge readable │
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,8 @@ Every widget title starts with a layer-priority prefix so anyone can immediately
|--------|-------|---------------|
| `I` | **Infrastructure** | Load balancers, databases, networks, DNS, CDN, storage — shared infrastructure that the service depends on but doesn't own |
| `P` | **Platform** | Service-specific platform components from the codebase — gRPC servers, connection pools, cache clients, queue consumers, circuit breakers |
| `D` | **Domain** | Business and domain metrics — checkout success rate, order throughput, payment completion, delivery latency, user sign-ups |
| `D` | **Domain** | Technical health of domain processes — saga failures, aggregate timeouts, domain event lag (tech stuff) |
| `B` | **Business** | Business outcomes — checkout success rate, payment completion, on-time delivery (business stuff) |

### Priority Numbers

Expand All @@ -249,21 +250,20 @@ The number after the layer letter indicates priority within that layer. `0` is t
### Examples

```text
Group: "Customer-Facing"
D0: Checkout success rate
D0: Order throughput
I0: Load balancer 5xx
P0: API p99 latency
I1: Database connection pool
P1: gRPC client errors
Group: "Business"
B0: Checkout success rate
B0: Order throughput
B1: API p99 latency
B1: Customer-visible error rate
B2: Failed payment rate

Group: "Rate"
P0: Requests per second
P1: By endpoint

Group: "Errors"
P0: Error rate over time
D1: Failed payment rate
D1: Order saga failures
P2: Top errors by endpoint

Group: "Infrastructure"
Expand All @@ -279,7 +279,8 @@ When assigning prefixes, use the domain discovery context:

- **I (Infrastructure)**: Would this metric exist even if your code didn't? Load balancer, database engine, OS resources, network — things the ops team manages.
- **P (Platform)**: Is this about how your code runs? Connection pools your code configures, gRPC channels your code opens, cache hit rates for caches your code uses — the technical platform layer.
- **D (Domain)**: Does this metric map to a business outcome? Checkout completions, delivery SLA, payment success — things a product manager would understand.
- **D (Domain)**: Is this technical health of a domain process? Saga step failures, aggregate timeouts, domain event processing lag — tech stuff that a domain engineer cares about.
- **B (Business)**: Is this a business outcome? Payment success rate, checkout completion, on-time delivery — business stuff that a product manager or customer cares about.

The priority number comes from the ops review order: what do you look at first when paged at 3am? That's `0`.

Expand All @@ -291,7 +292,7 @@ The priority number comes from the ops review order: what do you look at first w

**Widget titles**: Prefix + sentence case, concise, action-oriented.

- Always start with the layer-priority prefix (`I0:`, `P1:`, `D0:`, etc.)
- Always start with the layer-priority prefix (`I0:`, `P1:`, `D0:`, `B0:`, etc.)
- Do not repeat the group title after the prefix
- Do not repeat the integration name if it is obvious from context
- Alias all formulas so legends are readable
Loading