Skip to content

[9.1] (backport #10141) [otel] - Add Diagnostics Extension #10201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: 9.1
Choose a base branch
from
Open

[9.1] (backport #10141) [otel] - Add Diagnostics Extension #10201

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
527 changes: 527 additions & 0 deletions NOTICE-fips.txt
View file
Open in desktop

Large diffs are not rendered by default.

527 changes: 527 additions & 0 deletions NOTICE.txt
View file
Open in desktop

Large diffs are not rendered by default.

75 changes: 75 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,12 +78,31 @@ require (
go.elastic.co/apm/v2 v2.7.1
go.elastic.co/ecszap v1.0.3
go.elastic.co/go-licence-detector v0.7.0
<<<<<<< HEAD
go.opentelemetry.io/collector/component/componentstatus v0.132.0
go.opentelemetry.io/collector/connector/forwardconnector v0.132.0
go.opentelemetry.io/collector/pipeline v1.38.0
go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.132.0
go.opentelemetry.io/collector/receiver/nopreceiver v0.132.0
=======
go.opentelemetry.io/collector/component/componentstatus v0.135.0
go.opentelemetry.io/collector/component/componenttest v0.135.0
go.opentelemetry.io/collector/connector/forwardconnector v0.135.0
go.opentelemetry.io/collector/exporter/debugexporter v0.135.0
go.opentelemetry.io/collector/exporter/nopexporter v0.135.0
go.opentelemetry.io/collector/exporter/otlpexporter v0.135.0
go.opentelemetry.io/collector/exporter/otlphttpexporter v0.135.0
go.opentelemetry.io/collector/extension/extensiontest v0.135.0
go.opentelemetry.io/collector/extension/memorylimiterextension v0.135.0
go.opentelemetry.io/collector/pipeline v1.41.0
go.opentelemetry.io/collector/processor/batchprocessor v0.135.0
go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.135.0
go.opentelemetry.io/collector/receiver/nopreceiver v0.135.0
go.opentelemetry.io/collector/receiver/otlpreceiver v0.135.0
go.opentelemetry.io/ebpf-profiler v0.0.202536
>>>>>>> cac88a467 ([otel] - Add Diagnostics Extension (#10141))
go.uber.org/zap v1.27.0
go.yaml.in/yaml/v3 v3.0.4
golang.org/x/crypto v0.41.0
golang.org/x/exp v0.0.0-20250215185904-eff6e970281f
golang.org/x/mod v0.27.0
Expand Down Expand Up @@ -634,6 +653,7 @@ require (
go.etcd.io/bbolt v1.4.2 // indirect
go.mongodb.org/mongo-driver v1.17.4 // indirect
go.opencensus.io v0.24.0 // indirect
<<<<<<< HEAD
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/collector v0.132.0 // indirect
go.opentelemetry.io/collector/client v1.38.0 // indirect
Expand Down Expand Up @@ -684,6 +704,57 @@ require (
go.opentelemetry.io/collector/receiver/xreceiver v0.132.0 // indirect
go.opentelemetry.io/collector/scraper v0.132.0 // indirect
go.opentelemetry.io/collector/scraper/scraperhelper v0.132.0 // indirect
=======
go.opentelemetry.io/auto/sdk v1.2.0 // indirect
go.opentelemetry.io/collector v0.135.0 // indirect
go.opentelemetry.io/collector/client v1.41.0 // indirect
go.opentelemetry.io/collector/config/configauth v0.135.0 // indirect
go.opentelemetry.io/collector/config/configcompression v1.41.0 // indirect
go.opentelemetry.io/collector/config/configgrpc v0.135.0 // indirect
go.opentelemetry.io/collector/config/confighttp v0.135.0 // indirect
go.opentelemetry.io/collector/config/configmiddleware v1.41.0 // indirect
go.opentelemetry.io/collector/config/confignet v1.41.0 // indirect
go.opentelemetry.io/collector/config/configopaque v1.41.0 // indirect
go.opentelemetry.io/collector/config/configoptional v0.135.0 // indirect
go.opentelemetry.io/collector/config/configretry v1.41.0 // indirect
go.opentelemetry.io/collector/config/configtelemetry v0.135.0 // indirect
go.opentelemetry.io/collector/config/configtls v1.41.0 // indirect
go.opentelemetry.io/collector/confmap/xconfmap v0.135.0 // indirect
go.opentelemetry.io/collector/connector/connectortest v0.135.0 // indirect
go.opentelemetry.io/collector/connector/xconnector v0.135.0 // indirect
go.opentelemetry.io/collector/consumer v1.41.0 // indirect
go.opentelemetry.io/collector/consumer/consumererror v0.135.0 // indirect
go.opentelemetry.io/collector/consumer/consumererror/xconsumererror v0.135.0 // indirect
go.opentelemetry.io/collector/consumer/consumertest v0.135.0 // indirect
go.opentelemetry.io/collector/consumer/xconsumer v0.135.0 // indirect
go.opentelemetry.io/collector/exporter/exporterhelper v0.135.0 // indirect
go.opentelemetry.io/collector/exporter/exporterhelper/xexporterhelper v0.135.0 // indirect
go.opentelemetry.io/collector/exporter/exportertest v0.135.0 // indirect
go.opentelemetry.io/collector/exporter/xexporter v0.135.0 // indirect
go.opentelemetry.io/collector/extension/extensionauth v1.41.0 // indirect
go.opentelemetry.io/collector/extension/extensioncapabilities v0.135.0 // indirect
go.opentelemetry.io/collector/extension/extensionmiddleware v0.135.0 // indirect
go.opentelemetry.io/collector/extension/xextension v0.135.0 // indirect
go.opentelemetry.io/collector/filter v0.135.0 // indirect
go.opentelemetry.io/collector/internal/fanoutconsumer v0.135.0 // indirect
go.opentelemetry.io/collector/internal/memorylimiter v0.135.0 // indirect
go.opentelemetry.io/collector/internal/sharedcomponent v0.135.0 // indirect
go.opentelemetry.io/collector/internal/telemetry v0.135.0 // indirect
go.opentelemetry.io/collector/pdata v1.41.0 // indirect
go.opentelemetry.io/collector/pdata/pprofile v0.135.0 // indirect
go.opentelemetry.io/collector/pdata/testdata v0.135.0 // indirect
go.opentelemetry.io/collector/pdata/xpdata v0.135.0 // indirect
go.opentelemetry.io/collector/pipeline/xpipeline v0.135.0 // indirect
go.opentelemetry.io/collector/processor/processorhelper v0.135.0 // indirect
go.opentelemetry.io/collector/processor/processorhelper/xprocessorhelper v0.135.0 // indirect
go.opentelemetry.io/collector/processor/processortest v0.135.0 // indirect
go.opentelemetry.io/collector/processor/xprocessor v0.135.0 // indirect
go.opentelemetry.io/collector/receiver/receiverhelper v0.135.0 // indirect
go.opentelemetry.io/collector/receiver/receivertest v0.135.0 // indirect
go.opentelemetry.io/collector/receiver/xreceiver v0.135.0 // indirect
go.opentelemetry.io/collector/scraper v0.135.0 // indirect
go.opentelemetry.io/collector/scraper/scraperhelper v0.135.0 // indirect
>>>>>>> cac88a467 ([otel] - Add Diagnostics Extension (#10141))
go.opentelemetry.io/collector/semconv v0.128.1-0.20250610090210-188191247685 // indirect
go.opentelemetry.io/collector/service v0.132.0 // indirect
go.opentelemetry.io/collector/service/hostcapabilities v0.132.0 // indirect
Expand Down Expand Up @@ -721,7 +792,11 @@ require (
go.uber.org/ratelimit v0.3.1 // indirect
go.uber.org/zap/exp v0.3.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
<<<<<<< HEAD
go.yaml.in/yaml/v3 v3.0.4 // indirect
=======
golang.org/x/arch v0.20.0 // indirect
>>>>>>> cac88a467 ([otel] - Add Diagnostics Extension (#10141))
golang.org/x/oauth2 v0.30.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
gonum.org/v1/gonum v0.16.0 // indirect
Expand Down
16 changes: 16 additions & 0 deletions internal/pkg/otel/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ This section provides a summary of components included in the Elastic Distributi

| Component | Version |
|---|---|
<<<<<<< HEAD
| [apikeyauthextension](https://github.com/elastic/opentelemetry-collector-components/blob/extension/apikeyauthextension/v0.4.1/extension/apikeyauthextension/README.md) | v0.4.1 |
| [apmconfigextension](https://github.com/elastic/opentelemetry-collector-components/blob/extension/apmconfigextension/v0.6.0/extension/apmconfigextension/README.md) | v0.6.0 |
| [bearertokenauthextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/bearertokenauthextension/v0.132.0/extension/bearertokenauthextension/README.md) | v0.132.0 |
Expand All @@ -97,6 +98,21 @@ This section provides a summary of components included in the Elastic Distributi
| [k8sobserver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/observer/k8sobserver/v0.132.0/extension/observer/k8sobserver/README.md) | v0.132.0 |
| [memorylimiterextension](https://github.com/open-telemetry/opentelemetry-collector/blob/extension/memorylimiterextension/v0.132.0/extension/memorylimiterextension/README.md) | v0.132.0 |
| [pprofextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/pprofextension/v0.132.0/extension/pprofextension/README.md) | v0.132.0 |
=======
| [apikeyauthextension](https://github.com/elastic/opentelemetry-collector-components/blob/extension/apikeyauthextension/v0.5.0/extension/apikeyauthextension/README.md) | v0.5.0 |
| [apmconfigextension](https://github.com/elastic/opentelemetry-collector-components/blob/extension/apmconfigextension/v0.7.0/extension/apmconfigextension/README.md) | v0.7.0 |
| [bearertokenauthextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/bearertokenauthextension/v0.135.0/extension/bearertokenauthextension/README.md) | v0.135.0 |
| [beatsauthextension](https://github.com/elastic/opentelemetry-collector-components/blob/extension/beatsauthextension/v0.3.0/extension/beatsauthextension/README.md) | v0.3.0 |
| [extensiontest](https://github.com/open-telemetry/opentelemetry-collector/blob/extension/extensiontest/v0.135.0/extension/extensiontest/README.md) | v0.135.0 |
| [filestorage](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/storage/filestorage/v0.135.0/extension/storage/filestorage/README.md) | v0.135.0 |
| [headerssetterextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/headerssetterextension/v0.135.0/extension/headerssetterextension/README.md) | v0.135.0 |
| [healthcheckextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/healthcheckextension/v0.135.0/extension/healthcheckextension/README.md) | v0.135.0 |
| [healthcheckv2extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/healthcheckv2extension/v0.135.0/extension/healthcheckv2extension/README.md) | v0.135.0 |
| [k8sleaderelector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/k8sleaderelector/v0.135.0/extension/k8sleaderelector/README.md) | v0.135.0 |
| [k8sobserver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/observer/k8sobserver/v0.135.0/extension/observer/k8sobserver/README.md) | v0.135.0 |
| [memorylimiterextension](https://github.com/open-telemetry/opentelemetry-collector/blob/extension/memorylimiterextension/v0.135.0/extension/memorylimiterextension/README.md) | v0.135.0 |
| [pprofextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/pprofextension/v0.135.0/extension/pprofextension/README.md) | v0.135.0 |
>>>>>>> cac88a467 ([otel] - Add Diagnostics Extension (#10141))

### Connectors

Expand Down
64 changes: 64 additions & 0 deletions internal/pkg/otel/extension/elasticdiagnostics/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# elasticdiagnosticsextension

`elasticdiagnosticsextension` is an internal package for peforming diagnostics and is used in conjunction with EDOT.
The extension is designed to return diagnostics in a format compatible with the [ActionDiagnosticUnitResult](https://github.com/elastic/elastic-agent-client/blob/888026ef85e1c9190fe76eb158cf21d9c9c02920/elastic-agent-client.proto#L424-L437) type defined in the control protocol.


## Configuration

The extension accepts the `endpoint` as a sole parameter. The endpoint should begin with a valid protocol and it valid values are `unix` and `npipe` for now. Here are a few examples:

- `unix:///tmp/elastic-agent/xyz.soc`
- `npipe:///elastic-agent`

## Features

- Acts as a registrar and keeps track of common diagnostic hooks.
- Collects profiles using `runtime/pprof`.
- Collects internal telemetry exposed by the OTeL Collector.
- Implements the `extensioncapabilities.ConfigWatcher` interface and stores the latest configuration of the running collector.
- Listens for diagnostic requests and provides diagnostic data.

## Design

### Diagnostic hooks:
- Individual beats register custom diagnostic hooks and these hooks are called when we run the elastic-agent diagnostics command.
- Our extension stores these hooks and executes them everytime it gets a "diagnostics" request.

### Request/Response format:
- This extension runs an HTTP server and listens to new requests on `/diagnostics` path.
- The following query parameters are optional:
- `cpu`
- If `true`, the extension will also collect cpu profile of EDOT.
- By default, the extension doesn't collect the CPU profile unless explicitly specified.
- `cpuduration`:
- Specifies the time duration over which the CPU profile should be collected.
- Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`
- Default: `30s`.
- The response format is defined in [response.go](./response.go).
- `GlobalDiagnostics`: Data related to the overall process:
1. Profiles.
2. Internal telemetry.
3. latest collector configuration.
- `ComponentDiagnostics`: Data from individual receivers, collected via registered diagnostic hooks.

### Interaction with Elastic-Agent service in hybrid mode.

- When the user triggers the diagnostic request, EDOT diagnostics are injected at two levels:
1. At top-Level:
- When `DiagnosticAgent()` is called in [server.go](https://github.com/elastic/elastic-agent/blob/710c49f45433e2f136a6e41cae980c1aa37dabdd/pkg/control/v2/server/server.go#L197).
- Diagnostics are captured at the global level and stored under the `edot/*` directory in the resulting ZIP archive.
2. At component-level:
- When `otelMgr.PerformComponentsDiagnostics()` is called in [coordinator.go](https://github.com/elastic/elastic-agent/blob/710c49f45433e2f136a6e41cae980c1aa37dabdd/internal/pkg/agent/application/coordinator/coordinator.go#L863).
- Diagnostics are added per component and stored under the `components/{comp}/*` directory in the resulting ZIP archive.

#### Diagram

```mermaid
graph LR
A[elastic-agent install ...] --> B[Runs the service in hybrid mode<br/>and we inject agentdiagnosticsextension]
B -->|listens to| D[edot-diagnostics-extension.sock]

C[elastic-agent diagnostics] --> E[Extracts diagnostics socket path via the binary]
E -->|requests OTeL diagnostics| D
```
26 changes: 26 additions & 0 deletions internal/pkg/otel/extension/elasticdiagnostics/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License 2.0;
// you may not use this file except in compliance with the Elastic License 2.0.

package elasticdiagnostics

import (
"errors"

"go.opentelemetry.io/collector/component"
)

type Config struct {
Endpoint string `mapstructure:"endpoint"`
}

func createDefaultConfig() component.Config {
return &Config{}
}

func (c *Config) Validate() error {
if c.Endpoint == "" {
return errors.New("endpoint is a required field")
}
return nil
}
Loading
Loading