Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@
"description": "NATS messaging skills for designing subject hierarchies with naming conventions and wildcard patterns",
"source": "./plugins/trogonstack-nats",
"category": "development"
},
{
"name": "trogonstack-datadog",
"description": "Datadog observability skills for designing dashboards with proper widget selection, layout patterns, and template variables using the pup CLI",
"source": "./plugins/trogonstack-datadog",
"category": "development"
}
]
}
3 changes: 3 additions & 0 deletions .github/release-please-config.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@
},
"plugins/trogonstack-nats": {
"component": "trogonstack-nats"
},
"plugins/trogonstack-datadog": {
"component": "trogonstack-datadog"
}
},
"plugins": [
Expand Down
3 changes: 2 additions & 1 deletion .github/release-please-manifest.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"plugins/trogonstack-diataxis": "0.0.2",
"plugins/trogonstack-gh": "0.0.2",
"plugins/trogonstack-nats": "0.0.2"
"plugins/trogonstack-nats": "0.0.2",
"plugins/trogonstack-datadog": "0.0.1"
}
9 changes: 9 additions & 0 deletions plugins/trogonstack-datadog/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"name": "trogonstack-datadog",
"description": "Datadog observability skills for designing dashboards with proper widget selection, layout patterns, and template variables using the pup CLI",
"version": "0.0.1",
"author": {
"name": "TrogonStack",
"url": "https://github.com/TrogonStack"
}
}
1 change: 1 addition & 0 deletions plugins/trogonstack-datadog/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Changelog
147 changes: 147 additions & 0 deletions plugins/trogonstack-datadog/skills/datadog-design-dashboard/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
---
name: datadog-design-dashboard
description: >-
Design Datadog dashboards with proper widget selection, layout organization,
template variables, and group structure. Uses pup CLI for inspecting existing
dashboards and validating designs. Use when designing new dashboards, auditing
existing ones, or planning observability layouts for services. Do not use for:
(1) Datadog agent installation or configuration, (2) monitor/alert rule design,
(3) APM instrumentation or tracing setup, (4) log pipeline configuration.
allowed-tools: AskUserQuestion, Write, Read, Shell
---

# Design Datadog Dashboard

Design a dashboard layout that tells a clear story — from high-level health signals down to granular diagnostics — using proper widget types, group organization, and template variables for reusability.

**Important**: Always check for existing dashboards first with `pup dashboards list`. Do not create a new dashboard if one already exists for the same service or purpose — update the existing one instead. Only create a new dashboard when no relevant one exists or the user explicitly asks for a new one.

**Philosophy**: The frameworks, layouts, and widget guides in this skill are starting points — not rigid rules. Every product and business is different. Understand the domain first, then adapt the frameworks to fit. The best dashboards reflect how the business actually works, not how a generic template says they should.

## Domain Discovery

Before designing, understand what you are building observability for. The metrics that matter depend entirely on the product and business context.

**Ask the user**:
- "Can you tell me about the product and what this service does for the business? What does a customer experience when they interact with it?"
- "What does a bad day look like for this service? What breaks, and how do customers feel it?"
- "Are there codebases, architecture docs, or README files I can read to understand the service and its dependencies?"

**If the user points you to a codebase**: Read it. Look at the entry points, the API routes, the database models, the queue consumers, the external service calls. Understanding the code gives you the context to choose metrics that actually matter — not just generic RED/USE signals.

**If the user describes the business**: Use that context to tailor the Customer-Facing section. An e-commerce service cares about checkout success rates. A messaging service cares about delivery latency. A payment service cares about transaction completion. Generic "request rate" and "error rate" are a starting point, but the real value comes from metrics that map to business outcomes.

**Skip domain discovery if**: You already have deep context about the service from prior conversations or the user has provided detailed specifications.

## Interview

**Skip if ALL of these are already specified**: dashboard purpose, target audience, data sources, template variable needs, dashboard strategy.

**Always interview if**: Auditing or redesigning an existing dashboard (needs current state review first).

1. **Purpose** — "What is this dashboard for? Service overview, infrastructure, executive KPIs, debugging, or SLO tracking?"
2. **Audience** — "Who will use this? On-call engineers, platform team, leadership, or mixed?"
3. **Data Sources** — "Which Datadog products are involved? Metrics only, APM + Metrics, Logs + Metrics, or full stack?"
4. **Scope** — "Is this for a single service, a group of services, or infrastructure-wide?"
5. **Dashboard Strategy** — "One dashboard per service, or a consolidated view?" — share the trade-offs from [references/layouts.md](references/layouts.md) to help them decide. If unsure, ask: "During an outage, does your team investigate one service at a time, or do they need to see all services simultaneously?"
6. **Existing Dashboard** — "Is there an existing dashboard to audit or redesign?" If yes, fetch with `pup dashboards get <id>` before designing.

---

## Workflow

### 1. Gather existing context

```bash
pup dashboards list
pup dashboards get <dashboard-id>
pup dashboards url <dashboard-id>
```

If auditing an existing dashboard, fetch its definition first and analyze its current structure before redesigning.

### 2. Choose a framework

Match the dashboard purpose to a framework. Read [references/frameworks.md](references/frameworks.md) for detailed metric mappings and group structures.

| Purpose | Framework |
|---------|-----------|
| Service overview | RED (Rate, Errors, Duration) |
| Infrastructure | USE (Utilization, Saturation, Errors) |
| Executive/business | Golden Signals |
| SLO tracking | SLI/SLO |
| Debugging | Drill-down |

### 3. Design the layout

Using your domain understanding and the chosen framework, design the group structure and select widgets. Read these references as needed:

- **[layouts.md](references/layouts.md)** — Template variable conventions, group structure patterns, dashboard strategy trade-offs, grid sizing, anti-patterns
- **[widgets.md](references/widgets.md)** — Widget selection guide, display options, sizing, naming conventions
- **[thresholds.md](references/thresholds.md)** — Alert threshold markers, threshold proximity, Y-axis configuration

**Key principles** (not rigid rules — use judgment):
- **Prefix every widget title** with its layer and priority: `I0:` (most critical infra), `P0:` (most critical platform), `D0:` (most critical domain). See [widgets.md](references/widgets.md) for the full prefix system.
- Start with a **Customer-Facing** group (5-8 metrics) so someone with zero service knowledge can tell if customers are affected within 5 seconds. Tailor the metrics to the domain.
- Timeseries widgets should have **alert threshold markers** (red lines) with thresholds close to normal traffic. If a metric doesn't warrant an alert, question whether it belongs — but context-providing metrics can earn their place.
- Set **Y-axis max** explicitly near the threshold — don't let auto-scaling compress the normal range.
- Order groups macro-to-micro: customer-facing → overview → domain-specific → infrastructure.

### 4. Write the output

Present the design using this template:

```markdown
# Dashboard Design: [Dashboard Title]

## Purpose
[1-2 sentences: what this monitors, who uses it]

## Template Variables
| Variable | Tag | Default |
|----------|-----|---------|
| ... | ... | `*` |

## Layout

### Group: [Group Title]
| Widget | Type | Query/Metric | Width | Alert Threshold |
|--------|------|-------------|-------|----------------|
| ... | ... | ... | ... | ... |

[Repeat for each group]

## Quality Validation
[Run quality principles below]
```

### 5. Validate

```bash
pup dashboards list
pup dashboards get <dashboard-id>
pup metrics list --filter="trace.http.request.*"
```

---

## Quality Principles

- [ ] Dashboard reflects the actual product and business — metrics tailored to the domain
- [ ] Dashboard title is concise (no environment, region, or version)
- [ ] Template variables defined for env, service, and relevant scopes (default `*`)
- [ ] **Customer-Facing group** with 5-8 metrics tailored to the service's business impact
- [ ] Groups ordered macro-to-micro (customer-facing → overview → details)
- [ ] **Timeseries widgets have alert threshold markers** (red lines) where the metric is alertable
- [ ] **Thresholds close to normal traffic** — no excessive whitespace
- [ ] **Zero-knowledge readability** — someone with no service knowledge can spot problems via red indicators
- [ ] **Widget titles prefixed** with layer and priority (`I0:`, `P1:`, `D0:`, etc.)
- [ ] Widget titles use sentence case, don't repeat group name
- [ ] Every metric earns its place — if it spikes, someone can act on it

## References

- **[Observability Frameworks](references/frameworks.md)** — RED, USE, Golden Signals, SLI/SLO with metric mappings
- **[Layout & Structure](references/layouts.md)** — Template variables, group patterns, dashboard strategy, grid sizing, anti-patterns
- **[Widgets](references/widgets.md)** — Widget types, display options, sizing, naming conventions
- **[Alert Thresholds](references/thresholds.md)** — Threshold markers, proximity guide, Y-axis configuration
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Observability Frameworks

Choose the framework that matches the dashboard purpose. Each framework defines which metrics to prioritize and how to organize them.

---

## RED Method (Request-Driven Services)

Best for: microservices, APIs, web applications — anything that handles requests.

| Signal | Metric | Example Query |
|--------|--------|---------------|
| **Rate** | Requests per second | `sum:trace.http.request.hits{$service,$env}.as_rate()` |
| **Errors** | Error rate (%) | `sum:trace.http.request.errors{$service,$env} / sum:trace.http.request.hits{$service,$env} * 100` |
| **Duration** | Latency percentiles | `p50:trace.http.request.duration{$service,$env}`, `p90:trace.http.request.duration{$service,$env}`, `p99:trace.http.request.duration{$service,$env}` (one query per percentile, overlaid on the same timeseries widget) |

Comment thread
yordis marked this conversation as resolved.
### Group Structure

```text
Overview → Rate → Errors → Duration → [Dependencies] → [Infrastructure]
```

### When to Use RED

- Service receives external or internal HTTP/gRPC requests
- You care about user-facing latency and availability
- On-call engineers need to quickly identify degradation

### Drill-Down Pattern

1. **Overview**: Query Value widgets showing current Rate, Error %, p99 Latency
2. **Rate**: Timeseries of request rate, Top List by endpoint
3. **Errors**: Timeseries of error rate, Top List of failing endpoints, Log Stream of errors
4. **Duration**: Timeseries of latency percentiles, Heatmap of latency distribution, Top List of slowest endpoints

---

## USE Method (Resource-Oriented)

Best for: infrastructure, databases, queues — anything with finite capacity.

| Signal | Metric | Example Query |
|--------|--------|---------------|
| **Utilization** | Resource usage (%) | `avg:system.cpu.user{$host,$env}` |
| **Saturation** | Queue depth / waiting | `avg:system.load.1{$host,$env}` |
| **Errors** | Hardware/resource errors | `sum:system.disk.error{$host,$env}` |

### Group Structure

```text
Overview → CPU → Memory → Disk → Network → [Application-specific]
```

### When to Use USE

- Monitoring hosts, containers, or VMs
- Database or cache performance
- Queue/worker infrastructure
- Capacity planning

### Resource Mapping

| Resource | Utilization | Saturation | Errors |
|----------|-------------|------------|--------|
| CPU | `system.cpu.user` | `system.load.1` | — |
| Memory | `system.mem.pct_usable` | `system.swap.used` | OOM events |
| Disk | `system.disk.in_use` | `system.io.await` | `system.disk.error` |
| Network | `system.net.bytes_sent` | `system.net.packets_dropped` | `system.net.errors` |

---

## Golden Signals (SRE / Executive)

Best for: executive dashboards, cross-service views, SRE-level monitoring.

| Signal | Description | Example Metric |
|--------|-------------|----------------|
| **Latency** | Time to serve requests | `p99:trace.http.request.duration{$service,$env}` |
| **Traffic** | Request volume | `sum:trace.http.request.hits{$service,$env}.as_rate()` |
| **Errors** | Rate of failed requests | `sum:trace.http.request.errors{$service,$env} / sum:trace.http.request.hits{$service,$env}` |
| **Saturation** | Resource fullness | `avg:system.cpu.user{$service,$env}`, `avg:system.mem.pct_usable{$service,$env}` |

### Group Structure

```text
Executive Summary → Latency → Traffic → Errors → Saturation → [Business KPIs]
```

### When to Use Golden Signals

- Cross-service or platform-wide view
- Executive or leadership audience
- SRE team situational awareness
- Incident command dashboards

### Presentation Style

- Executive dashboards favor **Query Value** and **Timeseries** over detailed Top Lists
- Use wider time windows (1h, 4h, 1d) for trend visibility
- Include week-over-week comparisons where possible

---

## SLI/SLO Tracking

Best for: SLO compliance, error budget monitoring, reliability reviews.

| Signal | Description | Example Metric |
|--------|-------------|----------------|
| **SLI** | Service Level Indicator | Availability %, latency p99 < threshold |
| **Error Budget** | Remaining budget before SLO breach | `(1 - SLI) / (1 - SLO_target)` |
| **Burn Rate** | How fast budget is consumed | Error budget consumed / time elapsed |

### Group Structure

```text
SLO Summary → Error Budget → Burn Rate Alerts → Historical Compliance
```

### When to Use SLI/SLO

- Tracking reliability commitments
- Error budget-driven development decisions
- Reliability review meetings
- Incident post-mortem context

### Widget Recommendations

- **SLO widget**: Native Datadog SLO widget for compliance tracking
- **Query Value**: Current error budget remaining (color-coded)
- **Timeseries**: Burn rate over time with threshold lines
- **Change Widget**: Week-over-week SLI comparison
Loading