-
Notifications
You must be signed in to change notification settings - Fork 50
Description
Motivation
If instrumenting our SFPAY SpringBoot application with micrometer metrics it’s not possible to filter out custom metrics on a fine granular way via config.
This is only possible for non-custom metrics like jvm, tomcat, jdbc, hikaricp via application properties, e.g. management.metrics.enable.hikaricp=false
.
Furthermore, we are now ingesting all micrometer metrics to CLS via otlp and openTelemetry Java Agent and here we also can't suppress some custom micrometer metrics via config (JVM parameters or environment variable).
Finally the otel-metrics-v1-*
index gets flooded with the too many metrics and the promised retention period of 21 days is no longer guaranteed for observability purposes.
Proposal
JVM Parameters
-Dotel.instrumentation.micrometer.enabled=true|false
-Dotel.instrumentation.micrometer.disable.name=<comma-separated list of custom micrometer metric names>
Or (my favorite)
-Dotel.instrumentation.micrometer.enabled=true|false
-Dotel.instrumentation.micrometer.enabled.<custom micrometer metric name>=true|false
Example - Drop by Name
Given there is a metric
- name: sfpay.processtracker.atomicprocesses.accepted
- instrumentationScope.name: io.opentelemetry.micrometer-1.5
and given the below configuration, note that not the full name was configured:
-Dotel.instrumentation.micrometer.enabled=true
-Dotel.instrumentation.micrometer.enabled.sfpay.processtracker.atomicprocesses=false
Then the given metric should be dropped.
Related Requirement
In future there might be a requirement to not expose metrics for some dedicated attribute values but here I guess its always related to a dedicated metric. Hence, a key-value map would be required...
JVM Parameters
-Dotel.instrumentation.micrometer.enabled=true|false
-Dotel.instrumentation.micrometer.enabled.<custom micrometer metric name>={"<resource or metric attribute name>":"<resource or metric attribute value>", ...}
Example - Drop by Resource or Metric Attribute value
Given there is a metric
- name: sfpay.processtracker.atomicprocesses.accepted
- instrumentationScope.name: io.opentelemetry.micrometer-1.5
- metric.attributes.status: OK
and given the below configuration, note that not the full name was configured:
-Dotel.instrumentation.micrometer.enabled=true
-Dotel.instrumentation.micrometer.enabled.sfpay.processtracker.atomicprocesses={"metric.attributes.status":"ERROR"}
Then the given metric should be dropped.