Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions starlight-docs/src/content/docs/dashboards/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,11 @@ Each panel has a query editor where you write PPL or PromQL. The query determine
For logs and traces (PPL):
```sql
search earliest=-6h source = logs-otel-v1*
| where severity.text = 'ERROR'
| timechart span=5m count() by instrumentationScope.name
| where severityNumber >= 17
| timechart span=5m count() by `resource.attributes.service.name`
```

For metrics (PromQL):
For metrics (PromQL — adjust metric names to match your environment):
```promql
sum by (service_name) (rate(http_server_request_duration_seconds_count[5m]))
```
Expand Down
4 changes: 2 additions & 2 deletions starlight-docs/src/content/docs/dashboards/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Filters let you narrow the data across all panels at once without editing indivi
### Adding filters

1. Select **Add filter** in the filter bar
2. Choose a field (e.g., `instrumentationScope.name`, `severity.text`, `service_name`)
2. Choose a field (e.g., `resource.attributes.service.name`, `severity.text`, `service_name`)
3. Pick an operator — `is`, `is not`, `is one of`, `exists`, etc.
4. Set the value (e.g., `checkout-service`)
5. The filter applies to every panel on the dashboard immediately
Expand All @@ -57,7 +57,7 @@ You can stack multiple filters. They combine with AND logic — all conditions m

| Scenario | Filter |
|---|---|
| Focus on one service during an incident | `instrumentationScope.name` is `checkout-service` |
| Focus on one service during an incident | `resource.attributes.service.name` is `checkout-service` |
| Exclude noisy debug logs | `severity.text` is not `DEBUG` |
| Show only error-related data | `severity.text` is one of `ERROR, FATAL` |
| Filter to a specific environment | `resource.attributes.deployment.environment` is `production` |
Expand Down
68 changes: 34 additions & 34 deletions starlight-docs/src/content/docs/investigate/explore-logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ Discover shows results as a table of log events. You can add or remove columns,

```sql
source = logs-otel-v1*
| where severity.text = 'ERROR'
| fields @timestamp, body, instrumentationScope.name, traceId
| where severityNumber >= 17
| fields @timestamp, body, `resource.attributes.service.name`, traceId
| sort - @timestamp
| head 50
```
Expand All @@ -43,17 +43,17 @@ source = logs-otel-v1*

```sql
source = logs-otel-v1*
| where body LIKE '%timeout%'
| fields @timestamp, severity.text, body, instrumentationScope.name
| where body LIKE '%HTTP%' AND severityNumber >= 17
| fields @timestamp, severity.text, body, `resource.attributes.service.name`
| sort - @timestamp
```

### Filter by service and time

```sql
search earliest=-1h source = logs-otel-v1*
| where instrumentationScope.name = 'my-service'
| where severity.text IN ('ERROR', 'WARN')
| where `resource.attributes.service.name` = 'my-service'
| where severityNumber >= 13
| fields @timestamp, severity.text, body, traceId
| sort - @timestamp
```
Expand All @@ -69,7 +69,7 @@ source = logs-otel-v1*
| where body LIKE '%HTTP%'
| rex field=body "(?<httpMethod>GET|POST|PUT|DELETE)\s+(?<httpPath>/\S+)\s+HTTP"
| rex field=body "HTTP/\d\.\d\"\s+(?<statusCode>\d{3})"
| fields @timestamp, httpMethod, httpPath, statusCode, severity.text
| fields @timestamp, httpMethod, httpPath, statusCode, severityNumber
| sort - @timestamp
| head 30
```
Expand All @@ -78,7 +78,7 @@ source = logs-otel-v1*

```sql
source = logs-otel-v1*
| where severity.text = 'ERROR' AND body LIKE '%Exception%'
| where severityNumber >= 17 AND body LIKE '%Exception%'
| rex field=body "(?<exClass>[A-Za-z.]+Exception)"
| stats count() as occurrences by exClass
| sort - occurrences
Expand All @@ -93,7 +93,7 @@ source = logs-otel-v1*
| where body LIKE '%"status"%'
| spath input=body path=status output=responseStatus
| spath input=body path=service output=serviceName
| fields @timestamp, serviceName, responseStatus, severity.text
| fields @timestamp, serviceName, responseStatus, severityNumber
| sort - @timestamp
```

Expand All @@ -103,8 +103,8 @@ source = logs-otel-v1*

```sql
search earliest=-6h source = logs-otel-v1*
| where severity.text = 'ERROR'
| stats count() as errorCount by instrumentationScope.name
| where severityNumber >= 17
| stats count() as errorCount by `resource.attributes.service.name`
| sort - errorCount
```

Expand All @@ -122,10 +122,10 @@ Pipe this into a stacked bar chart in Discover to see how severity distribution
```sql
search earliest=-6h source = logs-otel-v1*
| eventstats count() as totalLogs,
count(severity.text = 'ERROR') as errorCount
by instrumentationScope.name
count(severityNumber >= 17) as errorCount
by `resource.attributes.service.name`
| eval errorRate = errorCount * 100.0 / totalLogs
| stats avg(errorRate) as avgErrorRate by instrumentationScope.name
| stats avg(errorRate) as avgErrorRate by `resource.attributes.service.name`
| sort - avgErrorRate
```

Expand All @@ -138,7 +138,7 @@ One of the most powerful aspects of OTEL-instrumented logs is the `traceId` fiel
```sql
source = logs-otel-v1*
| where traceId = '<your-trace-id>'
| fields @timestamp, spanId, severity.text, body, instrumentationScope.name
| fields @timestamp, spanId, severity.text, body, `resource.attributes.service.name`
| sort @timestamp
```

Expand All @@ -150,10 +150,10 @@ Use a subquery to first identify traces with errors, then pull all logs from tho
source = logs-otel-v1*
| where traceId in [
source = logs-otel-v1*
| where severity.text = 'ERROR'
| where severityNumber >= 17
| fields traceId
]
| fields @timestamp, traceId, spanId, severity.text, body, instrumentationScope.name
| fields @timestamp, traceId, spanId, severity.text, body, `resource.attributes.service.name`
| sort traceId, @timestamp
```

Expand All @@ -165,12 +165,12 @@ Join error logs with span data to see whether errors correlate with slow request

```sql
source = logs-otel-v1* as a
| where severity.text = 'ERROR'
| where severityNumber >= 17
| left join ON a.traceId = b.traceId
[ source = otel-v1-apm-span-*
| stats max(durationInNanos) as maxSpanDuration by traceId
] as b
| fields a.@timestamp, a.body, a.instrumentationScope.name, a.traceId, b.maxSpanDuration
| fields a.@timestamp, a.body, a.`resource.attributes.service.name`, a.traceId, b.maxSpanDuration
| sort - b.maxSpanDuration
| head 20
```
Expand All @@ -181,9 +181,9 @@ See which services produce errors together — a sign of cascading failures:

```sql
search earliest=-2h source = logs-otel-v1*
| where severity.text = 'ERROR'
| stats dc(instrumentationScope.name) as serviceCount,
values(instrumentationScope.name) as services
| where severityNumber >= 17
| stats dc(`resource.attributes.service.name`) as serviceCount,
values(`resource.attributes.service.name`) as services
by traceId
| where serviceCount > 1
| sort - serviceCount
Expand All @@ -201,10 +201,10 @@ Use eventstats to compare each service's error rate against the overall average:
```sql
search earliest=-6h source = logs-otel-v1*
| eventstats count() as totalLogs,
count(severity.text = 'ERROR') as errorCount
by instrumentationScope.name
count(severityNumber >= 17) as errorCount
by `resource.attributes.service.name`
| eval errorRate = errorCount * 100.0 / totalLogs
| stats avg(errorRate) as avgErrorRate by instrumentationScope.name
| stats avg(errorRate) as avgErrorRate by `resource.attributes.service.name`
| eventstats avg(avgErrorRate) as globalAvgRate, stddev(avgErrorRate) as stddevRate
| eval zScore = (avgErrorRate - globalAvgRate) / stddevRate
| where zScore > 2
Expand Down Expand Up @@ -244,12 +244,12 @@ Find exception types that appeared recently but weren't present in the prior per

```sql
search earliest=-1h source = logs-otel-v1*
| where severity.text = 'ERROR'
| where severityNumber >= 17
| rex field=body "(?<exClass>[A-Za-z.]+Exception)"
| stats count() as recentCount by exClass
| left join ON a.exClass = b.exClass
[ search earliest=-24h latest=-1h source = logs-otel-v1*
| where severity.text = 'ERROR'
| where severityNumber >= 17
| rex field=body "(?<exClass>[A-Za-z.]+Exception)"
| stats count() as priorCount by exClass
] as b
Expand All @@ -265,16 +265,16 @@ Exception classes that appear in the last hour but not in the prior 23 hours are

```sql
search earliest=-1h source = logs-otel-v1*
| where severity.text = 'ERROR'
| stats count() as currentErrors by instrumentationScope.name
| left join ON a.instrumentationScope.name = b.instrumentationScope.name
| where severityNumber >= 17
| stats count() as currentErrors by `resource.attributes.service.name`
| left join ON a.`resource.attributes.service.name` = b.`resource.attributes.service.name`
[ search earliest=-2h latest=-1h source = logs-otel-v1*
| where severity.text = 'ERROR'
| stats count() as previousErrors by instrumentationScope.name
| where severityNumber >= 17
| stats count() as previousErrors by `resource.attributes.service.name`
] as b
| fillnull using currentErrors = 0, previousErrors = 0
| eval changePercent = ((currentErrors - b.previousErrors) * 100.0) / b.previousErrors
| fields instrumentationScope.name, currentErrors, b.previousErrors, changePercent
| fields `resource.attributes.service.name`, currentErrors, b.previousErrors, changePercent
| sort - changePercent
```

Expand All @@ -299,7 +299,7 @@ search earliest=-4h source = logs-otel-v1*
| where severity.text IN ('ERROR', 'WARN')
| eval period = if(@timestamp < TIMESTAMP('2025-03-04T14:00:00'), 'before', 'after')
| stats count() as logCount,
dc(instrumentationScope.name) as affectedServices
dc(`resource.attributes.service.name`) as affectedServices
by period, severity.text
| sort period, severity.text
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ title: "Explore Metrics"
description: "Investigate time-series metrics in Discover using PromQL — query rates, latencies, resource usage, detect anomalies, and compare time ranges"
---

:::caution[Placeholder queries]
The PromQL queries on this page are representative examples based on standard OpenTelemetry metric names. Your environment may use different metric names, labels, or configurations. Treat these as starting points — adjust metric names and label selectors to match your actual data.
:::

Metrics provide a continuous, low-overhead view of system health. In OpenSearch, the Discover experience for metrics uses PromQL — the industry-standard query language for Prometheus-compatible time-series data. Use it to monitor request rates, latencies, error ratios, and resource consumption across your services.

## Getting started in Discover
Expand Down Expand Up @@ -201,7 +205,7 @@ Metrics tell you *what* is happening; logs and traces tell you *why*. Use metric
source = otel-v1-apm-span-*
| where serviceName = 'checkout-service'
AND durationInNanos > 5000000000
| fields traceId, operationName, durationInNanos, status.code
| fields traceId, name, durationInNanos, status.code
| sort - durationInNanos
| head 10
```
Expand Down
38 changes: 18 additions & 20 deletions starlight-docs/src/content/docs/investigate/explore-traces.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,8 @@ Each row in the results represents a single span. Expand a row to see all span a
| `spanId` | Unique identifier for this span |
| `parentSpanId` | The parent span (empty for root spans) |
| `serviceName` | Service that produced this span |
| `operationName` | The operation or endpoint name |
| `name` | The operation or endpoint name |
| `durationInNanos` | Span duration in nanoseconds |
| `startTime` | When the span started |
| `endTime` | When the span ended |
| `status.code` | Span status — UNSET, OK, or ERROR |
| `kind` | Span kind — SERVER, CLIENT, INTERNAL, PRODUCER, CONSUMER |
| `resource.attributes.*` | Resource-level attributes |
Expand All @@ -38,7 +36,7 @@ Each row in the results represents a single span. Expand a row to see all span a
```sql
source = otel-v1-apm-span-*
| where durationInNanos > 5000000000
| fields traceId, serviceName, operationName, durationInNanos, status.code
| fields traceId, serviceName, name, durationInNanos, status.code
| sort - durationInNanos
| head 20
```
Expand All @@ -50,8 +48,8 @@ source = otel-v1-apm-span-*
```sql
search earliest=-1h source = otel-v1-apm-span-*
| where serviceName = 'frontend'
AND operationName LIKE '%/api/%'
| fields traceId, operationName, durationInNanos, status.code
AND name LIKE '%/api/%'
| fields traceId, name, durationInNanos, status.code
| sort - durationInNanos
```

Expand All @@ -60,7 +58,7 @@ search earliest=-1h source = otel-v1-apm-span-*
```sql
source = otel-v1-apm-span-*
| where status.code = 'ERROR'
| fields traceId, serviceName, operationName, durationInNanos
| fields traceId, serviceName, name, durationInNanos
| sort - durationInNanos
| head 50
```
Expand All @@ -70,7 +68,7 @@ source = otel-v1-apm-span-*
```sql
source = otel-v1-apm-span-*
| where parentSpanId = '' OR parentSpanId IS NULL
| fields traceId, serviceName, operationName, durationInNanos, status.code
| fields traceId, serviceName, name, durationInNanos, status.code
| sort - durationInNanos
| head 30
```
Expand All @@ -84,8 +82,8 @@ Root spans represent the entry point of a request — typically the first servic
```sql
source = otel-v1-apm-span-*
| where traceId = '<your-trace-id>'
| fields startTime, serviceName, operationName, spanId, parentSpanId, durationInNanos, status.code
| sort startTime
| fields serviceName, name, spanId, parentSpanId, durationInNanos, status.code
| sort - durationInNanos
```

This gives you the full span tree for a trace, ordered chronologically. Walk through it to see exactly which services were called, in what order, and how long each took.
Expand Down Expand Up @@ -127,7 +125,7 @@ search earliest=-6h source = otel-v1-apm-span-*
| stats avg(durationMs) as p50,
max(durationMs) as pMax,
count() as spanCount
by serviceName, operationName
by serviceName, name
| where spanCount > 10
| sort - p50
| head 20
Expand Down Expand Up @@ -164,7 +162,7 @@ The `traceId` field is the bridge between traces and logs. Use it to get the ful
```sql
source = logs-otel-v1*
| where traceId = '<your-trace-id>'
| fields @timestamp, spanId, severity.text, body, instrumentationScope.name
| fields @timestamp, spanId, severity.text, body, `resource.attributes.service.name`
| sort @timestamp
```

Expand All @@ -175,13 +173,13 @@ source = otel-v1-apm-span-* as a
| where status.code = 'ERROR'
| left join ON a.traceId = b.traceId
[ source = logs-otel-v1*
| where severity.text = 'ERROR'
| where severityNumber >= 17
| stats count() as logErrorCount,
values(instrumentationScope.name) as errorServices
values(`resource.attributes.service.name`) as errorServices
by traceId
] as b
| where b.logErrorCount > 0
| fields a.traceId, a.serviceName, a.operationName, a.durationInNanos, b.logErrorCount, b.errorServices
| fields a.traceId, a.serviceName, a.name, a.durationInNanos, b.logErrorCount, b.errorServices
| sort - b.logErrorCount
| head 20
```
Expand Down Expand Up @@ -219,10 +217,10 @@ search earliest=-6h source = otel-v1-apm-span-*
| eval durationMs = durationInNanos / 1000000
| eventstats avg(durationMs) as avgDuration,
stddev(durationMs) as stddevDuration
by serviceName, operationName
by serviceName, name
| eval zScore = (durationMs - avgDuration) / stddevDuration
| where zScore > 3
| fields traceId, serviceName, operationName, durationMs, avgDuration, zScore
| fields traceId, serviceName, name, durationMs, avgDuration, zScore
| sort - zScore
| head 20
```
Expand Down Expand Up @@ -263,11 +261,11 @@ Flags time windows where average latency is 50% or more above the moving average
```sql
search earliest=-1h source = otel-v1-apm-span-*
| where status.code = 'ERROR'
| stats count() as recentErrors by serviceName, operationName
| left join ON a.serviceName = b.serviceName AND a.operationName = b.operationName
| stats count() as recentErrors by serviceName, name
| left join ON a.serviceName = b.serviceName AND a.name = b.name
[ search earliest=-24h latest=-1h source = otel-v1-apm-span-*
| where status.code = 'ERROR'
| stats count() as priorErrors by serviceName, operationName
| stats count() as priorErrors by serviceName, name
] as b
| where b.priorErrors IS NULL
| sort - recentErrors
Expand Down
Loading
Loading