diff --git a/starlight-docs/src/content/docs/dashboards/build.md b/starlight-docs/src/content/docs/dashboards/build.md index 3358652..09f96bb 100644 --- a/starlight-docs/src/content/docs/dashboards/build.md +++ b/starlight-docs/src/content/docs/dashboards/build.md @@ -70,11 +70,11 @@ Each panel has a query editor where you write PPL or PromQL. The query determine For logs and traces (PPL): ```sql search earliest=-6h source = logs-otel-v1* -| where severity.text = 'ERROR' -| timechart span=5m count() by instrumentationScope.name +| where severityNumber >= 17 +| timechart span=5m count() by `resource.attributes.service.name` ``` -For metrics (PromQL): +For metrics (PromQL — adjust metric names to match your environment): ```promql sum by (service_name) (rate(http_server_request_duration_seconds_count[5m])) ``` diff --git a/starlight-docs/src/content/docs/dashboards/index.md b/starlight-docs/src/content/docs/dashboards/index.md index 9f37bec..40d12d4 100644 --- a/starlight-docs/src/content/docs/dashboards/index.md +++ b/starlight-docs/src/content/docs/dashboards/index.md @@ -46,7 +46,7 @@ Filters let you narrow the data across all panels at once without editing indivi ### Adding filters 1. Select **Add filter** in the filter bar -2. Choose a field (e.g., `instrumentationScope.name`, `severity.text`, `service_name`) +2. Choose a field (e.g., `resource.attributes.service.name`, `severity.text`, `service_name`) 3. Pick an operator — `is`, `is not`, `is one of`, `exists`, etc. 4. Set the value (e.g., `checkout-service`) 5. The filter applies to every panel on the dashboard immediately @@ -57,7 +57,7 @@ You can stack multiple filters. They combine with AND logic — all conditions m | Scenario | Filter | |---|---| -| Focus on one service during an incident | `instrumentationScope.name` is `checkout-service` | +| Focus on one service during an incident | `resource.attributes.service.name` is `checkout-service` | | Exclude noisy debug logs | `severity.text` is not `DEBUG` | | Show only error-related data | `severity.text` is one of `ERROR, FATAL` | | Filter to a specific environment | `resource.attributes.deployment.environment` is `production` | diff --git a/starlight-docs/src/content/docs/investigate/explore-logs.md b/starlight-docs/src/content/docs/investigate/explore-logs.md index e719163..1c235ae 100644 --- a/starlight-docs/src/content/docs/investigate/explore-logs.md +++ b/starlight-docs/src/content/docs/investigate/explore-logs.md @@ -33,8 +33,8 @@ Discover shows results as a table of log events. You can add or remove columns, ```sql source = logs-otel-v1* -| where severity.text = 'ERROR' -| fields @timestamp, body, instrumentationScope.name, traceId +| where severityNumber >= 17 +| fields @timestamp, body, `resource.attributes.service.name`, traceId | sort - @timestamp | head 50 ``` @@ -43,8 +43,8 @@ source = logs-otel-v1* ```sql source = logs-otel-v1* -| where body LIKE '%timeout%' -| fields @timestamp, severity.text, body, instrumentationScope.name +| where body LIKE '%HTTP%' AND severityNumber >= 17 +| fields @timestamp, severity.text, body, `resource.attributes.service.name` | sort - @timestamp ``` @@ -52,8 +52,8 @@ source = logs-otel-v1* ```sql search earliest=-1h source = logs-otel-v1* -| where instrumentationScope.name = 'my-service' -| where severity.text IN ('ERROR', 'WARN') +| where `resource.attributes.service.name` = 'my-service' +| where severityNumber >= 13 | fields @timestamp, severity.text, body, traceId | sort - @timestamp ``` @@ -69,7 +69,7 @@ source = logs-otel-v1* | where body LIKE '%HTTP%' | rex field=body "(?GET|POST|PUT|DELETE)\s+(?/\S+)\s+HTTP" | rex field=body "HTTP/\d\.\d\"\s+(?\d{3})" -| fields @timestamp, httpMethod, httpPath, statusCode, severity.text +| fields @timestamp, httpMethod, httpPath, statusCode, severityNumber | sort - @timestamp | head 30 ``` @@ -78,7 +78,7 @@ source = logs-otel-v1* ```sql source = logs-otel-v1* -| where severity.text = 'ERROR' AND body LIKE '%Exception%' +| where severityNumber >= 17 AND body LIKE '%Exception%' | rex field=body "(?[A-Za-z.]+Exception)" | stats count() as occurrences by exClass | sort - occurrences @@ -93,7 +93,7 @@ source = logs-otel-v1* | where body LIKE '%"status"%' | spath input=body path=status output=responseStatus | spath input=body path=service output=serviceName -| fields @timestamp, serviceName, responseStatus, severity.text +| fields @timestamp, serviceName, responseStatus, severityNumber | sort - @timestamp ``` @@ -103,8 +103,8 @@ source = logs-otel-v1* ```sql search earliest=-6h source = logs-otel-v1* -| where severity.text = 'ERROR' -| stats count() as errorCount by instrumentationScope.name +| where severityNumber >= 17 +| stats count() as errorCount by `resource.attributes.service.name` | sort - errorCount ``` @@ -122,10 +122,10 @@ Pipe this into a stacked bar chart in Discover to see how severity distribution ```sql search earliest=-6h source = logs-otel-v1* | eventstats count() as totalLogs, - count(severity.text = 'ERROR') as errorCount - by instrumentationScope.name + count(severityNumber >= 17) as errorCount + by `resource.attributes.service.name` | eval errorRate = errorCount * 100.0 / totalLogs -| stats avg(errorRate) as avgErrorRate by instrumentationScope.name +| stats avg(errorRate) as avgErrorRate by `resource.attributes.service.name` | sort - avgErrorRate ``` @@ -138,7 +138,7 @@ One of the most powerful aspects of OTEL-instrumented logs is the `traceId` fiel ```sql source = logs-otel-v1* | where traceId = '' -| fields @timestamp, spanId, severity.text, body, instrumentationScope.name +| fields @timestamp, spanId, severity.text, body, `resource.attributes.service.name` | sort @timestamp ``` @@ -150,10 +150,10 @@ Use a subquery to first identify traces with errors, then pull all logs from tho source = logs-otel-v1* | where traceId in [ source = logs-otel-v1* - | where severity.text = 'ERROR' + | where severityNumber >= 17 | fields traceId ] -| fields @timestamp, traceId, spanId, severity.text, body, instrumentationScope.name +| fields @timestamp, traceId, spanId, severity.text, body, `resource.attributes.service.name` | sort traceId, @timestamp ``` @@ -165,12 +165,12 @@ Join error logs with span data to see whether errors correlate with slow request ```sql source = logs-otel-v1* as a -| where severity.text = 'ERROR' +| where severityNumber >= 17 | left join ON a.traceId = b.traceId [ source = otel-v1-apm-span-* | stats max(durationInNanos) as maxSpanDuration by traceId ] as b -| fields a.@timestamp, a.body, a.instrumentationScope.name, a.traceId, b.maxSpanDuration +| fields a.@timestamp, a.body, a.`resource.attributes.service.name`, a.traceId, b.maxSpanDuration | sort - b.maxSpanDuration | head 20 ``` @@ -181,9 +181,9 @@ See which services produce errors together — a sign of cascading failures: ```sql search earliest=-2h source = logs-otel-v1* -| where severity.text = 'ERROR' -| stats dc(instrumentationScope.name) as serviceCount, - values(instrumentationScope.name) as services +| where severityNumber >= 17 +| stats dc(`resource.attributes.service.name`) as serviceCount, + values(`resource.attributes.service.name`) as services by traceId | where serviceCount > 1 | sort - serviceCount @@ -201,10 +201,10 @@ Use eventstats to compare each service's error rate against the overall average: ```sql search earliest=-6h source = logs-otel-v1* | eventstats count() as totalLogs, - count(severity.text = 'ERROR') as errorCount - by instrumentationScope.name + count(severityNumber >= 17) as errorCount + by `resource.attributes.service.name` | eval errorRate = errorCount * 100.0 / totalLogs -| stats avg(errorRate) as avgErrorRate by instrumentationScope.name +| stats avg(errorRate) as avgErrorRate by `resource.attributes.service.name` | eventstats avg(avgErrorRate) as globalAvgRate, stddev(avgErrorRate) as stddevRate | eval zScore = (avgErrorRate - globalAvgRate) / stddevRate | where zScore > 2 @@ -244,12 +244,12 @@ Find exception types that appeared recently but weren't present in the prior per ```sql search earliest=-1h source = logs-otel-v1* -| where severity.text = 'ERROR' +| where severityNumber >= 17 | rex field=body "(?[A-Za-z.]+Exception)" | stats count() as recentCount by exClass | left join ON a.exClass = b.exClass [ search earliest=-24h latest=-1h source = logs-otel-v1* - | where severity.text = 'ERROR' + | where severityNumber >= 17 | rex field=body "(?[A-Za-z.]+Exception)" | stats count() as priorCount by exClass ] as b @@ -265,16 +265,16 @@ Exception classes that appear in the last hour but not in the prior 23 hours are ```sql search earliest=-1h source = logs-otel-v1* -| where severity.text = 'ERROR' -| stats count() as currentErrors by instrumentationScope.name -| left join ON a.instrumentationScope.name = b.instrumentationScope.name +| where severityNumber >= 17 +| stats count() as currentErrors by `resource.attributes.service.name` +| left join ON a.`resource.attributes.service.name` = b.`resource.attributes.service.name` [ search earliest=-2h latest=-1h source = logs-otel-v1* - | where severity.text = 'ERROR' - | stats count() as previousErrors by instrumentationScope.name + | where severityNumber >= 17 + | stats count() as previousErrors by `resource.attributes.service.name` ] as b | fillnull using currentErrors = 0, previousErrors = 0 | eval changePercent = ((currentErrors - b.previousErrors) * 100.0) / b.previousErrors -| fields instrumentationScope.name, currentErrors, b.previousErrors, changePercent +| fields `resource.attributes.service.name`, currentErrors, b.previousErrors, changePercent | sort - changePercent ``` @@ -299,7 +299,7 @@ search earliest=-4h source = logs-otel-v1* | where severity.text IN ('ERROR', 'WARN') | eval period = if(@timestamp < TIMESTAMP('2025-03-04T14:00:00'), 'before', 'after') | stats count() as logCount, - dc(instrumentationScope.name) as affectedServices + dc(`resource.attributes.service.name`) as affectedServices by period, severity.text | sort period, severity.text ``` diff --git a/starlight-docs/src/content/docs/investigate/explore-metrics.md b/starlight-docs/src/content/docs/investigate/explore-metrics.md index ada4143..ee4392f 100644 --- a/starlight-docs/src/content/docs/investigate/explore-metrics.md +++ b/starlight-docs/src/content/docs/investigate/explore-metrics.md @@ -3,6 +3,10 @@ title: "Explore Metrics" description: "Investigate time-series metrics in Discover using PromQL — query rates, latencies, resource usage, detect anomalies, and compare time ranges" --- +:::caution[Placeholder queries] +The PromQL queries on this page are representative examples based on standard OpenTelemetry metric names. Your environment may use different metric names, labels, or configurations. Treat these as starting points — adjust metric names and label selectors to match your actual data. +::: + Metrics provide a continuous, low-overhead view of system health. In OpenSearch, the Discover experience for metrics uses PromQL — the industry-standard query language for Prometheus-compatible time-series data. Use it to monitor request rates, latencies, error ratios, and resource consumption across your services. ## Getting started in Discover @@ -201,7 +205,7 @@ Metrics tell you *what* is happening; logs and traces tell you *why*. Use metric source = otel-v1-apm-span-* | where serviceName = 'checkout-service' AND durationInNanos > 5000000000 - | fields traceId, operationName, durationInNanos, status.code + | fields traceId, name, durationInNanos, status.code | sort - durationInNanos | head 10 ``` diff --git a/starlight-docs/src/content/docs/investigate/explore-traces.md b/starlight-docs/src/content/docs/investigate/explore-traces.md index 96c0150..43438ce 100644 --- a/starlight-docs/src/content/docs/investigate/explore-traces.md +++ b/starlight-docs/src/content/docs/investigate/explore-traces.md @@ -22,10 +22,8 @@ Each row in the results represents a single span. Expand a row to see all span a | `spanId` | Unique identifier for this span | | `parentSpanId` | The parent span (empty for root spans) | | `serviceName` | Service that produced this span | -| `operationName` | The operation or endpoint name | +| `name` | The operation or endpoint name | | `durationInNanos` | Span duration in nanoseconds | -| `startTime` | When the span started | -| `endTime` | When the span ended | | `status.code` | Span status — UNSET, OK, or ERROR | | `kind` | Span kind — SERVER, CLIENT, INTERNAL, PRODUCER, CONSUMER | | `resource.attributes.*` | Resource-level attributes | @@ -38,7 +36,7 @@ Each row in the results represents a single span. Expand a row to see all span a ```sql source = otel-v1-apm-span-* | where durationInNanos > 5000000000 -| fields traceId, serviceName, operationName, durationInNanos, status.code +| fields traceId, serviceName, name, durationInNanos, status.code | sort - durationInNanos | head 20 ``` @@ -50,8 +48,8 @@ source = otel-v1-apm-span-* ```sql search earliest=-1h source = otel-v1-apm-span-* | where serviceName = 'frontend' - AND operationName LIKE '%/api/%' -| fields traceId, operationName, durationInNanos, status.code + AND name LIKE '%/api/%' +| fields traceId, name, durationInNanos, status.code | sort - durationInNanos ``` @@ -60,7 +58,7 @@ search earliest=-1h source = otel-v1-apm-span-* ```sql source = otel-v1-apm-span-* | where status.code = 'ERROR' -| fields traceId, serviceName, operationName, durationInNanos +| fields traceId, serviceName, name, durationInNanos | sort - durationInNanos | head 50 ``` @@ -70,7 +68,7 @@ source = otel-v1-apm-span-* ```sql source = otel-v1-apm-span-* | where parentSpanId = '' OR parentSpanId IS NULL -| fields traceId, serviceName, operationName, durationInNanos, status.code +| fields traceId, serviceName, name, durationInNanos, status.code | sort - durationInNanos | head 30 ``` @@ -84,8 +82,8 @@ Root spans represent the entry point of a request — typically the first servic ```sql source = otel-v1-apm-span-* | where traceId = '' -| fields startTime, serviceName, operationName, spanId, parentSpanId, durationInNanos, status.code -| sort startTime +| fields serviceName, name, spanId, parentSpanId, durationInNanos, status.code +| sort - durationInNanos ``` This gives you the full span tree for a trace, ordered chronologically. Walk through it to see exactly which services were called, in what order, and how long each took. @@ -127,7 +125,7 @@ search earliest=-6h source = otel-v1-apm-span-* | stats avg(durationMs) as p50, max(durationMs) as pMax, count() as spanCount - by serviceName, operationName + by serviceName, name | where spanCount > 10 | sort - p50 | head 20 @@ -164,7 +162,7 @@ The `traceId` field is the bridge between traces and logs. Use it to get the ful ```sql source = logs-otel-v1* | where traceId = '' -| fields @timestamp, spanId, severity.text, body, instrumentationScope.name +| fields @timestamp, spanId, severity.text, body, `resource.attributes.service.name` | sort @timestamp ``` @@ -175,13 +173,13 @@ source = otel-v1-apm-span-* as a | where status.code = 'ERROR' | left join ON a.traceId = b.traceId [ source = logs-otel-v1* - | where severity.text = 'ERROR' + | where severityNumber >= 17 | stats count() as logErrorCount, - values(instrumentationScope.name) as errorServices + values(`resource.attributes.service.name`) as errorServices by traceId ] as b | where b.logErrorCount > 0 -| fields a.traceId, a.serviceName, a.operationName, a.durationInNanos, b.logErrorCount, b.errorServices +| fields a.traceId, a.serviceName, a.name, a.durationInNanos, b.logErrorCount, b.errorServices | sort - b.logErrorCount | head 20 ``` @@ -219,10 +217,10 @@ search earliest=-6h source = otel-v1-apm-span-* | eval durationMs = durationInNanos / 1000000 | eventstats avg(durationMs) as avgDuration, stddev(durationMs) as stddevDuration - by serviceName, operationName + by serviceName, name | eval zScore = (durationMs - avgDuration) / stddevDuration | where zScore > 3 -| fields traceId, serviceName, operationName, durationMs, avgDuration, zScore +| fields traceId, serviceName, name, durationMs, avgDuration, zScore | sort - zScore | head 20 ``` @@ -263,11 +261,11 @@ Flags time windows where average latency is 50% or more above the moving average ```sql search earliest=-1h source = otel-v1-apm-span-* | where status.code = 'ERROR' -| stats count() as recentErrors by serviceName, operationName -| left join ON a.serviceName = b.serviceName AND a.operationName = b.operationName +| stats count() as recentErrors by serviceName, name +| left join ON a.serviceName = b.serviceName AND a.name = b.name [ search earliest=-24h latest=-1h source = otel-v1-apm-span-* | where status.code = 'ERROR' - | stats count() as priorErrors by serviceName, operationName + | stats count() as priorErrors by serviceName, name ] as b | where b.priorErrors IS NULL | sort - recentErrors diff --git a/starlight-docs/src/content/docs/investigate/index.md b/starlight-docs/src/content/docs/investigate/index.md index 7198764..d5331eb 100644 --- a/starlight-docs/src/content/docs/investigate/index.md +++ b/starlight-docs/src/content/docs/investigate/index.md @@ -25,25 +25,16 @@ Logs and traces share PPL as their query language, giving analysts a consistent PPL uses a pipe-delimited syntax where each command transforms the result set and passes it to the next. It reads naturally from left to right. -**Sample queries:** +**Log queries** (run in the Logs Discover view against `logs-otel-v1*`): Filter error logs and count by service: ```sql source = logs-otel-v1* -| where severity.text = 'ERROR' -| stats count() as errorCount by instrumentationScope.name +| where severityNumber >= 17 +| stats count() as errorCount by `resource.attributes.service.name` | sort - errorCount ``` -Find the slowest traces in the last hour: -```sql -source = otel-v1-apm-span-* -| where durationInNanos > 5000000000 -| fields traceId, serviceName, operationName, durationInNanos -| sort - durationInNanos -| head 10 -``` - Extract HTTP status codes from log bodies: ```sql source = logs-otel-v1* @@ -53,12 +44,27 @@ source = logs-otel-v1* | sort statusCode ``` +**Trace queries** (run in the Traces Discover view against `otel-v1-apm-span-*`): + +Find the slowest traces in the last hour: +```sql +source = otel-v1-apm-span-* +| where durationInNanos > 5000000000 +| fields traceId, serviceName, name, durationInNanos +| sort - durationInNanos +| head 10 +``` + For the full PPL command reference, see the [PPL documentation](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.md). For hands-on examples using OTEL data, see [Explore Logs](/opensearch-agentops-website/docs/investigate/explore-logs/) and [Explore Traces](/opensearch-agentops-website/docs/investigate/explore-traces/). ### PromQL PromQL is a functional query language for selecting and aggregating time-series metrics. It supports instant queries, range queries, and built-in functions for rates, aggregations, and mathematical operations. +:::caution[Placeholder queries] +The PromQL examples below use standard OpenTelemetry metric names. Your environment may use different metric names and labels — adjust accordingly. +::: + **Sample queries:** Request rate per service over 5 minutes: