[9.1] (backport #10052) [otel] Implement EDOT diagnostics extension

[mergify]

What does this PR do?

This PR add a new elasticdiagnsotics extension. This new extension will be used in hybrid mode and will perform diagnostics for running otel collector. It will:

• Acts as a registrar and keeps track of common diagnostic hooks.
• Collects profiles using runtime/pprof.
• Collects internal telemetry exposed by the OTeL Collector.
• Implements the extensioncapabilities.ConfigWatcher interface and stores the latest configuration of the running collector.
• Listens for diagnostic requests and provides diagnostic data.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None that i can think of. I have to do some final testing on windows to verify this.

Related issues

• Relates [beat_receiver] manager interface beats#44604

---

This is an automatic backport of pull request #10052 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 47112bd has failed:

On branch mergify/bp/9.1/pr-10052
Your branch is up to date with 'origin/9.1'.

You are currently cherry-picking commit 47112bda4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/agent/application/paths/common.go
	new file:   internal/pkg/agent/cmd/diagnostics_otel.go
	modified:   internal/pkg/agent/cmd/run.go
	modified:   internal/pkg/diagnostics/diagnostics.go
	new file:   internal/pkg/otel/diagnostics.go
	modified:   internal/pkg/otel/manager/force_extension_converter.go
	modified:   internal/pkg/otel/translate/otelconfig_test.go
	modified:   pkg/control/v2/server/server.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   internal/pkg/agent/cmd/otel.go
	both modified:   internal/pkg/otel/components.go
	deleted by us:   internal/pkg/otel/extension/elasticdiagnostics/extension.go
	deleted by us:   internal/pkg/otel/manager/diagnostics.go
	deleted by us:   internal/pkg/otel/manager/diagnostics_test.go
	deleted by us:   internal/pkg/otel/manager/execution_embedded.go
	both modified:   internal/pkg/otel/manager/manager.go
	both modified:   internal/pkg/otel/manager/manager_test.go
	both modified:   internal/pkg/otel/translate/otelconfig.go
	both modified:   testing/integration/ess/diagnostics_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pierrehilbert]

@VihasMakwana you have some conflicts to fix.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: dad57ad

Failed CI Steps

• check-ci
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 2022
• Unit tests - Windows 2022
• Unit tests - Windows 2016
• Unit tests - Windows 2016
• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• Unit tests - Windows 10
• Unit tests - Windows 10
• Unit tests - Windows 11
• Unit tests - Windows 11
• Packaging: linux/amd64 rpm
• Packaging: linux/amd64 rpm
• Packaging: Ubuntu x86_64 FIPS
• Packaging: Ubuntu x86_64 FIPS
• Packaging: linux/arm64 tar.gz
• Packaging: linux/arm64 tar.gz
• Packaging: Ubuntu arm64 FIPS
• Packaging: Ubuntu arm64 FIPS
• Packaging: windows/amd64 zip
• Packaging: windows/amd64 zip
• Packaging: Containers linux/amd64
• Packaging: Containers linux/arm64
• Packaging: Containers linux/amd64 FIPS
• Packaging: Containers linux/arm64 FIPS

History

• 💔 Build #28759 failed 3332b8a
• 💔 Build #28606 failed 96359d7

cc @VihasMakwana

[VihasMakwana]

Not needed on 9.1.