Disable sending local collection metrics to batch processor

[carsonip]

Motivation/summary

Disable default instrumentation metrics sending to batch processor by removing local exporter.

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Ensure that no metrics-apm.app.apm_server-* is automatically created when apm-server runs.

Related issues

Closes #12225

[mergify]

This pull request does not have a backport label. Could you fix it @carsonip? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.

NOTE: backport-skip has been added to this pull request.

[carsonip]

We'll have to update https://github.com/elastic/observability-docs/blob/main/docs/en/observability/apm/monitoring/monitoring-local-collection.asciidoc

[kruskall]

I think we still need this for the otel metrics. Added in #11470

[carsonip]

I thought metric.WithReader(exporter), is good enough for otel. localExporter is for exporting to the batchprocessor interface.

[dmathieu]

The first reader should be enough for otel, yes.

[dmathieu]

The first reader should be enough for otel, yes.

[carsonip]

Testing notes

✔️ test-plan-ok
local collection metrics are no longer available on 8.12.1

8.12.0

Dev tools:

GET /metrics-apm.app.apm_server-*/_search

returns

Details

```json { "took": 20, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 10000, "relation": "gte" }, "max_score": 1, "hits": [ { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "ptEQfo0BV9mN7pGUT0y7", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:56:53.712Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:56:54Z", "received": "2024-02-06T10:56:53.712Z" }, "http": { "server": { "request": { "count": 12 } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "p9EQfo0BV9mN7pGUT0y7", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:56:53.712Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:56:54Z", "received": "2024-02-06T10:56:53.712Z" }, "http": { "server": { "request": { "duration": { "values": [ 0.001953125 ], "counts": [ 12 ] } } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "qNEQfo0BV9mN7pGUT0y7", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:56:53.712Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:56:54Z", "received": "2024-02-06T10:56:53.712Z" }, "http": { "server": { "response": { "valid": { "count": 12 } } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "oNEOfo0BV9mN7pGUekz8", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:54:53.713Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:54:54Z", "received": "2024-02-06T10:54:53.713Z" }, "http": { "server": { "request": { "count": 12 } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "odEOfo0BV9mN7pGUekz8", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:54:53.713Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:54:54Z", "received": "2024-02-06T10:54:53.713Z" }, "http": { "server": { "request": { "duration": { "values": [ 0.001953125 ], "counts": [ 12 ] } } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "otEOfo0BV9mN7pGUekz8", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:54:53.713Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:54:54Z", "received": "2024-02-06T10:54:53.713Z" }, "http": { "server": { "response": { "valid": { "count": 12 } } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "o9EPfo0BV9mN7pGUZUxc", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:55:53.712Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:55:54Z", "received": "2024-02-06T10:55:53.712Z" }, "http": { "server": { "request": { "count": 12 } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "pNEPfo0BV9mN7pGUZUxc", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:55:53.712Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:55:54Z", "received": "2024-02-06T10:55:53.712Z" }, "http": { "server": { "request": { "duration": { "values": [ 0.001953125 ], "counts": [ 12 ] } } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "pdEPfo0BV9mN7pGUZUxc", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:55:53.712Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:55:54Z", "received": "2024-02-06T10:55:53.712Z" }, "http": { "server": { "response": { "valid": { "count": 12 } } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } }, { "_index": ".ds-metrics-apm.app.apm_server-default-2024.02.01-000001", "_id": "mtEMfo0BV9mN7pGUpkw7", "_score": 1, "_source": { "@timestamp": "2024-02-06T10:52:53.712Z", "data_stream": { "dataset": "apm.app.apm_server", "namespace": "default", "type": "metrics" }, "event": { "agent_id_status": "missing", "ingested": "2024-02-06T10:52:54Z", "received": "2024-02-06T10:52:53.712Z" }, "http": { "server": { "request": { "duration": { "values": [ 0.001953125 ], "counts": [ 12 ] } } } }, "metricset": { "name": "app" }, "observer": { "hostname": "54dd7c335acb", "type": "apm-server", "version": "8.12.0" }, "processor": { "event": "metric" }, "service": { "language": { "name": "go" }, "name": "apm-server" } } } ] } } ```

8.12.1

Dev tools:

GET /metrics-apm.app.apm_server-*/_search

returns

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0,
    "hits": []
  }
}