diff --git a/proposals/0086-semconv-internal-telemetry.md b/proposals/0086-semconv-internal-telemetry.md new file mode 100644 index 0000000..cbcebf4 --- /dev/null +++ b/proposals/0086-semconv-internal-telemetry.md @@ -0,0 +1,255 @@ +# Prometheus Internal Telemetry as an OTel Semantic Convention Registry + +* **Owners:** + * Nicolas Takashi [@nicolastakashi](https://github.com/nicolastakashi) [nicolas.takashi@coralogix.com](mailto:nicolas.takashi@coralogix.com) + * Arthur Silva Sens [@ArthurSens](https://github.com/ArthurSens) [arthursens2005@gmail.com](mailto:arthursens2005@gmail.com) + +* **Implementation Status:** `Partially implemented` + +* **Related Issues and PRs:** + * [WIP: Prometheus semconvs](https://github.com/prometheus/prometheus/pull/17868) + * [Weaver: Implement `weaver registry infer` command](https://github.com/open-telemetry/weaver/pull/1138) + +* **Other docs or links:** + * [Dev Summit notes: internal telemetry consensus](https://docs.google.com/document/d/1uurQCi5iVufhYHGlBZ8mJMK_freDFKPG0iYBQqJ9fvA/edit?tab=t.0#bookmark=id.ojugisgspwvq) + * [OpenTelemetry Weaver](https://github.com/open-telemetry/weaver) + * [OTel Semantic Convention specification](https://opentelemetry.io/docs/specs/semconv/) + +> TL;DR: We propose defining all metrics exported by the Prometheus binary as a formal [OTel semantic convention registry](https://opentelemetry.io/docs/specs/semconv/). One machine-readable schema serves as the single source of truth, enabling auto-generated instrumentation code, always-up-to-date metric documentation, contract testing against a live instance with `promtool`, and a surface downstream projects can check their own metric references against. + +## Why + +Prometheus defines its own internal metrics as scattered `prometheus/client_golang` constructor calls across dozens of files. There is no machine-readable description of what Prometheus emits. + +This creates the following concrete problems: + +* Dashboard and alert authors who depend on these metrics have no canonical reference. They reverse-engineer what Prometheus emits by reading Go source or running a live instance. +* There is no contract distinguishing stable public API metrics from internal implementation details. Metrics can be renamed, removed, or semantically changed without versioned signals. +* Documentation, if it exists, is written separately from the code and drifts. There is no mechanism to keep it in sync. +* There is no regression safety. Nothing prevents a code change from silently altering which metrics Prometheus emits or what labels they carry. +* As Prometheus increasingly interoperates with OTel (OTLP ingestion, remote write, collectors), its own telemetry remains outside the OTel schema world and is invisible to tools that understand semantic conventions. + +### Pitfalls of the current solution + +* Metric help strings are inconsistent: some describe counter semantics, others describe the event being counted. +* Histogram bucket configurations are chosen ad hoc with no enforcement. +* Units are absent from most metric definitions. +* No lifecycle model exists: no way to mark a metric as experimental, stable, or deprecated in a way that tooling understands. +* Ecosystem consumers (Thanos, Mimir, Grafana dashboards, alerting rules) have no authoritative reference for what a running Prometheus exposes. + +## Goals + +* [Required] Define Prometheus' internal telemetry as a formal OTel semantic convention registry (a single `registry.yaml`), making it the single machine-readable source of truth for every metric Prometheus exposes. +* [Required] Generate instrumentation code from the registry, eliminating hand-written metric definitions in Go. +* [Required] Generate metric documentation from the registry that cannot drift from the implementation. +* [Nice to have] Enable contract testing through `promtool`: validate that a running Prometheus exposes exactly what the registry says, with no OTel Collector and no Weaver binary in the test path. +* [Required] Record metric stability (`development`, `stable`, `deprecated`) as first-class schema information, so lifecycle can be expressed and changes to it reviewed. +* [Nice to have] Lay the foundation for multi-language instrumentation code generation and ecosystem tooling (dashboards, alerting rules) derived from the same registry. + +### Audience + +Prometheus maintainers and contributors. Operators and SREs who build dashboards and alerts on top of Prometheus' internal metrics. OTel ecosystem tools that consume or validate Prometheus telemetry. + +## Non-Goals + +* Changing existing metric names, label names, or semantics. This is a schema-first refactor of how metrics are defined, not what they measure. +* Adopting the OTel SDK for instrumentation. `prometheus/client_golang` remains the instrumentation layer. +* Migrating exporters or other ecosystem projects (a natural follow-on, out of scope here). +* Publishing the registry as part of OTel upstream semantic conventions (possible long term, not required now). + +## How + +### The registry + +A single `registry.yaml` at the repository root describes every metric the Prometheus binary exposes. One file keeps global name uniqueness and stability audits trivial, and gives `promtool` and downstream consumers one artifact to point at. If the file becomes unwieldy, splitting it per package is a later refactor. + +```yaml +groups: + - id: metric.prometheus_tsdb_compaction_duration_seconds + type: metric + stability: stable + brief: Duration of compaction runs. + metric_name: prometheus_tsdb_compaction_duration_seconds + instrument: histogram + unit: s + annotations: + prometheus: + histogram_type: mixed_histogram + exponential_buckets: {start: 1, factor: 2, count: 14} + bucket_factor: 1.1 + max_bucket_number: 100 + min_reset_duration: "1h" +``` + +This file is the contract. Go code, documentation, and contract tests are all derived from it. The `annotations.prometheus` block carries Prometheus-specific details invisible to OTel (histogram variant, bucket configuration, callback-based gauges, labels fixed at construction time). + +The repository structure is: + +``` +semconv/ + registry.yaml ← source of truth, hand-authored, one file for the whole repository + +/internal/semconv/ + metrics.gen.go ← generated, DO NOT EDIT + README.md ← generated, DO NOT EDIT +``` + +Generated packages are `internal`, so they cannot be imported from outside the package that owns the metrics. Generated files carry a `.gen.go` suffix so their origin is unambiguous to tooling and contributors. Both follow review feedback on the proof-of-concept. + +The Weaver templates and Rego policies used during generation are not committed to this repository. Where they live is an open question discussed below; the generation step resolves them at build time. + +### Instrumentation code generation + +[OTel Weaver](https://github.com/open-telemetry/weaver) renders `registry.yaml` into typed Go code via Jinja2 templates. A Makefile target regenerates every `metrics.gen.go` across the repository. CI enforces that generated files stay in sync with the registry. + +A metric with dynamic labels generates a typed `.With()` method that accepts a sealed per-metric interface, so passing the wrong label is a compile error rather than a silent runtime mismatch: + +```go +func (m PrometheusTargetIntervalLengthSeconds) With( + interval IntervalAttr, + extra ...PrometheusTargetIntervalLengthSecondsAttr, +) prometheus.Observer { ... } +``` + +Metrics without labels generate a plain constructor, and labels fixed at construction time (`const_labels`) become typed constructor parameters. Callback gauges (`prometheus.GaugeFunc`) are the exception. Their value is pulled from a closure at scrape time rather than set by the program, so the registry marks them `only_opts: true` and Weaver generates an `Opts()` accessor instead of a constructor. The schema owns the name, help, and unit; the closure stays in hand-written code. + +Package code imports the generated types directly: + +```go +import semconv "github.com/prometheus/prometheus/tsdb/internal/semconv" + +duration: semconv.NewPrometheusTSDBCompactionDurationSeconds(), +``` + +The exact shape of the generated API is not settled. It is decided while migrating the first package, before the rest follow. + +### Documentation generation + +The same Weaver invocation that produces `metrics.gen.go` also produces a `README.md` per package: a complete structured reference of every metric, including name, type, unit, label semantics, stability level, and examples. Because both files are rendered from the same `registry.yaml`, documentation cannot drift from the code. + +### Contract testing + +Contract testing checks a running Prometheus against the registry, with no OTel Collector and no Weaver binary in the test path. + +The primary surface is `promtool`. It already reads text exposition from stdin, already runs `promlint` over it, and already exits 3 when it finds problems: + +``` +curl -s http://localhost:9090/metrics | promtool check metrics --schema registry.yaml +``` + +A `--schema` flag points it at the registry and adds the checks `promlint` cannot make on its own: that every declared metric is present, that no undeclared metric appears, and that types, units, and label names match. This works for any exporter in any language, needs no Go, and ships through a binary Prometheus already releases. It is also the surface downstream projects would use, since a mixin or alert collection can check its metric references against a release's registry in its own CI. + +`promtool` sees only metrics that produced a sample, because that is all text exposition contains. A vector that never received a child is invisible to it, for the same reason it is invisible to OTLP. + +Prometheus' own tests therefore also validate in-process, where the full declared surface is reachable. `Collector.Describe()` yields a descriptor for every metric a collector declares, sample or no sample. This needs a small addition to `client_golang`: `prometheus.Desc` currently exposes only `Err()` and `String()`, with all fields unexported, so a declared metric's name, help, unit, and label names cannot be read structurally. Adding those accessors introduces no new dependency. + +Both paths share the same registry parsing and the same `promlint` extension point, `AddCustomValidations`, which takes `func(*dto.MetricFamily) []error`. Prometheus already vendors `promlint` for `promtool check metrics`, but no test currently lints Prometheus' own `/metrics`. + +The alternative is to route a running Prometheus through an OTel Collector (Prometheus receiver → OTLP exporter) into `live-check`. This proposal does not take that path. It asserts against post-translation telemetry, where name normalization, unit suffixing, and `_total` handling have already been applied, so a translation bug surfaces as a schema violation. It also puts a Collector in the test harness that every consumer not already running one would have to adopt. + +The validator does not implement semantic convention resolution. It reads a flat, already-resolved registry; `ref`, `extends`, imports, and group merging stay Weaver's job at authoring time. `opentelemetry-collector-contrib`'s `schemaprocessor` is precedent for this split: it consumes resolved schema artifacts in Go with no Weaver dependency. + +Contract testing detects a changed metric surface before it ships. A rename, a new label, or a changed type or unit fails CI, and the change becomes visible in review. It does not tell an already-deployed downstream how to follow a rename. That needs Prometheus to publish a versioned telemetry schema describing renames between releases, a format `go.opentelemetry.io/otel/schema` already parses in Go. That is a natural follow-on and is not proposed here. + +### Relationship to `client_golang` + +Three separate suggestions raised in review have all been described as "host this in `client_golang`". They are distinct decisions: + +| Suggestion | Layer | Status | +|----------------------------------------------------------|--------------------------------|----------------------------------------------| +| Host the Weaver Jinja2 templates and Rego policies there | Generation assets | Open (see template and policy hosting below) | +| Replace Weaver with a lighter native Go implementation | Instrumentation and generation | Not chosen (see Alternatives) | +| Add `Desc` introspection to support contract testing | Verification | Proposed above | + +Only the third changes `client_golang` itself, and it is additive: a few accessors on an existing type, no new dependencies, and useful to any project that wants to inspect what a collector declares. + +The validator module stays outside `client_golang`. `client_golang` is a single Go module with no nested `go.mod`, so a schema package inside it would add YAML and OTel-schema dependencies to the module that nearly every Prometheus exporter imports. Where the module does live is an open question below. + +### Metric lifecycle and evolution + +OTel stability levels are first-class fields in `registry.yaml`: + +* `development`: internal or experimental, may change without notice. +* `stable`: public API, changes require a deprecation cycle. +* `deprecated`: kept for backward compatibility, will be removed in a future major version. + +Weaver's code generator can use these levels to emit deprecation warnings in generated code and to omit deprecated metrics from new generation targets. A stable metric cannot then be removed without a schema change, and that change is visible in review and auditable in git history. + +This proposal adds the *field* and makes changes to it reviewable. It does not define what `stable` obligates Prometheus to, how long a deprecation cycle runs, or who decides which of today's metrics are stable. Marking a metric `stable` would be a backwards-compatibility commitment Prometheus has not previously made, and that is a policy decision rather than a schema one. Assigning stability levels to existing metrics should be a follow-on proposal; until then, migrated metrics carry `development`, which describes the status quo rather than granting a guarantee by default. + +### What "across the ecosystem" means + +Downstream projects hardcode Prometheus' internal metric names and have no way to learn that one changed, short of a user reporting a broken panel or a silent alert. + +[`samber/awesome-prometheus-alerts`](https://github.com/samber/awesome-prometheus-alerts) hardcodes 37 distinct `prometheus_*` metric names across roughly 1,150 rule entries in a single data file. They span `tsdb`, `target`, `rule`, `notifications`, `sd`, and `engine`, the same packages this proposal migrates, and include `prometheus_target_interval_length_seconds`, the metric used as the labelled example above. [`perses/community-mixins`](https://github.com/perses/community-mixins) builds dashboards and recording rules for Prometheus, Thanos, `node_exporter`, Alertmanager, and others, with metric names written into queries in Go. Neither project can validate those references against anything. + +A machine-readable registry gives them something to read. A mixin or rule collection can compare the metric names it references against the declared surface and fail its own CI when a reference no longer resolves, when a metric changed type or unit, or when a label it groups by is gone. The check runs in the consumer's CI on the consumer's schedule, and needs nothing from Prometheus beyond a registry that can be fetched and parsed. + +The same applies to exporters. An exporter that declares its metrics in this schema gets generated documentation and a machine-readable surface, and every mixin or alert collection targeting that exporter gains the same drift detection. Migrating exporters is out of scope here, but adopting the format is what makes it possible later. + +This is what "safe metric evolution across the ecosystem" means in this proposal: the registry is published and consumable, so downstream projects can detect drift themselves. It stops short of Prometheus publishing a versioned schema of renames between releases, which is what would let a downstream migrate automatically rather than merely notice. That is a larger commitment, since it adds a release artifact and a compatibility contract, and is left to a follow-on proposal. The format for it already exists and is parseable in Go via `go.opentelemetry.io/otel/schema`. + +### Rego validation policies + +A set of [OPA Rego](https://www.openpolicyagent.org/docs/latest/policy-language/) policies validates each `registry.yaml` before code generation. Where those policies are hosted is part of the open question on template hosting below. + +* Every `histogram` instrument must declare `annotations.prometheus.histogram_type`. +* Valid values for `histogram_type` are `classic_histogram`, `native_histogram`, `mixed_histogram`, `summary`. +* Classic and mixed histograms must declare `buckets` or `exponential_buckets`. +* Native and mixed histograms must declare `bucket_factor`, `max_bucket_number`, `min_reset_duration`. + +Validation runs as part of the generation step and fails the build if any registry violates the policies. + +### Open questions + +* **`.With()` allocations on hot paths**: The generated `.With()` method allocates a `prometheus.Labels` map on each call. For high-frequency paths such as per-scrape counters or per-sample append metrics, this may be too expensive. A typed `WithX(value string)` fast path should be benchmarked before broad rollout. See the related [reviewer comment](https://github.com/prometheus/prometheus/pull/17868#discussion_r2716984753). + +* **Validator module home**: The contract-testing module needs somewhere to live that is not `client_golang`'s main module. Options include a nested module inside the `client_golang` repository with its own `go.mod` (keeps it under the same maintainers and release cadence without touching the main module's dependency graph), a new repository in the `prometheus` organization, or incubation elsewhere with donation once it has users. This does not block the rest of the proposal. + +* **Mapping registry entries to packages**: With a single root registry, generation needs to know which package each metric belongs to so it can emit that package's `metrics.gen.go`. An annotation on each group is the obvious mechanism, but the exact shape is a generation detail to settle during implementation. + +* **Template and policy hosting**: The Jinja2 templates and Rego policies that drive code generation need to live somewhere accessible at build time. Three options are under consideration: (1) in this repository under a `build/` directory, keeping everything self-contained but coupling the templates to the Prometheus server; (2) in `prometheus/client_golang`, making them reusable across the ecosystem; (3) bundled into the Weaver binary itself, which Weaver is actively developing ([weaver#1145](https://github.com/open-telemetry/weaver/pull/1145)) and would remove the hosting question entirely. This decision needs to be made before the migration can be considered stable. + +## Alternatives + +### Hand-written definitions with linting only + +Keep metric definitions as `prometheus.NewCounter(...)` calls and enforce naming conventions, unit presence, and histogram rules with a linter. No new tool is needed to try this: `client_golang` already ships `promlint`, which validates help strings, units, counter naming, type-in-name, reserved characters, and histogram conventions, and `testutil.GatherAndLint` wires any registry into it. Prometheus imports it today only in `promtool`, where it lints other projects' metrics; pointing it at Prometheus' own `/metrics` would cost a single test. + +That test is worth adding regardless of this proposal, and the contract testing described above runs its schema checks through the same `promlint` extension point rather than duplicating it. + +Linting alone is not sufficient. A linter has no notion of which metrics are *supposed* to exist, so it cannot detect a removed metric, a renamed metric, or a metric that gained a label, which are the changes that break downstream consumers. It also cannot generate documentation or express a lifecycle. + +### Generate the registry from instrumentation code + +Invert the direction. Instead of authoring `registry.yaml` and generating Go from it, use the `Desc` introspection described above to emit the registry *from* the binary and commit it as a golden file. CI fails when the emitted registry differs from the committed one. Go uses this arrangement for its own public API in `api/go1.*.txt`. + +This costs contributors almost nothing, since metric definitions stay ordinary `client_golang` calls, and it delivers regression safety on its own: any change to the metric surface appears as a reviewable diff. + +It is not chosen because the registry stops being authoritative. Help text, units, and stability levels revert to being written inline in Go, free to drift, with nothing to enforce them. There is no schema to generate documentation from and nowhere for a lifecycle model to live. + +It shares the same `Desc` introspection, so it remains available as a fallback if the schema-first direction does not reach consensus. + +### Adopt OTel SDK for instrumentation + +Replace `prometheus/client_golang` with the OTel Go SDK and emit metrics via OTLP natively, using OTel's toolchain end-to-end. + +This solution is not chosen because Prometheus is the reference implementation of the Prometheus data model. Using a different SDK for its own instrumentation would be surprising to contributors and would add a heavyweight dependency. This proposal deliberately keeps `client_golang` as the instrumentation layer and uses Weaver only at the schema and generation layer. The two concerns are separable. + +### Publish registry as upstream OTel semantic conventions + +Contribute `prometheus_*` metric definitions directly to `open-telemetry/semantic-conventions`. + +This solution is not chosen because Prometheus' internal metrics describe Prometheus' own implementation, not a general convention for other software to follow. If the schema matures and becomes relevant for compatible implementations (e.g. Thanos, Mimir), contributing upstream is a natural follow-on. It is not a prerequisite. + +## Action Plan + +* [ ] Get consensus on this proposal. +* [ ] Add `Desc` accessors to `client_golang` so the full declared metric surface can be read. +* [ ] Write `semconv/registry.yaml` by hand, package by package. +* [ ] Build contract testing, one package at a time, and run it in CI. +* [ ] Add `--schema` to `promtool check metrics` so any exporter can run the same check. +* [ ] Generate code for one small package and settle the generated API there. +* [ ] Benchmark `.With()` before touching hot paths. +* [ ] Generate the rest, one pull request per package. +* [ ] Add the Makefile target and CI check that generated files match the registry.