The PodMonitoring metricRelabeling do not work correctly

[almeida-silas]

While migrating from OperatorConfig to PodMonitoring, I’m using metricRelabeling to keep a small allowlist of metric names. Most names are scraped as expected, but a few specific ones are not scraped when the same pattern works with OperatorConfig.

These two metric families are missing when configured via PodMonitoring:

workqueue_(queue_duration_seconds_bucket|work_duration_seconds_bucket)
controller_runtime_(duration_seconds_bucket)

Other names in the same alternation are scraped correctly (e.g., go_info, rest_client_requests_total, leader_election_master_status, controller_runtime_reconcile_total, controller_runtime_max_concurrent_reconciles, workqueue_depth, workqueue_adds_total, workqueue_retries_total).

What did I expect to happen?

The PodMonitoring metricRelabeling regex should keep the exact same set of metric names that are successfully kept by the OperatorConfig filter.matchOneOf configuration (shown below), including the two families above.

How to reproduce

Create a PodMonitoring with metricRelabeling that keeps the following set of metric names:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: example
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s
      metricRelabeling:
        - sourceLabels: [__name__]
          action: keep
          # also tried without anchors
          regex: ^(go_info| rest_client_requests_total|leader_election_master_status|workqueue_(queue_duration_seconds_bucket|work_duration_seconds_bucket|depth|adds_total|retries_total)|controller_runtime_(duration_seconds_bucket|reconcile_total|max_concurrent_reconciles))$

Observe that:
go_info, rest_client_requests_total, leader_election_master_status are present.

These are missing:
workqueue_queue_duration_seconds_bucket,
workqueue_work_duration_seconds_bucket,
controller_runtime_duration_seconds_bucket.

Try a simplified rule to isolate one family (still missing):

metricRelabeling:
  - sourceLabels: [__name__]
    action: keep
    regex: workqueue_(queue_duration_seconds_bucket|work_duration_seconds_bucket)

Compare with OperatorConfig, where the same allowlist works as expected:

apiVersion: monitoring.googleapis.com/v1
kind: OperatorConfig
# ...
collection:
  filter:
    matchOneOf:
      - '{__name__=~"controller_runtime_(duration_seconds_bucket|reconcile_total|max_concurrent_reconciles)"}'
      - '{__name__=~"go_info"}'
      - '{__name__=~"leader_election_master_status"}'
      - '{__name__=~"rest_client_requests_total"}'
      - '{__name__=~"workqueue_(queue_duration_seconds_bucket|work_duration_seconds_bucket|depth|adds_total|retries_total)"}'

What I’ve already tried

Removing ^/$ anchors.
Using a single-family regex (e.g., only workqueue_(queue_duration_seconds_bucket|work_duration_seconds_bucket)).
Matching the exact metric name directly, (e.g., workqueue_queue_duration_seconds_bucket or controller_runtime_duration_seconds_bucket)
Verifying the metrics do exist on the target endpoint via

curl -s http://localhost:8080/metrics | grep -E 'workqueue_work_duration_seconds_bucket'

Confirmed that other families in the exact same alternation are kept.

Environment

GKE / K8s version: v1.32.2-gke.1182003
prometheus-engine / GMP collector version: operator:v0.15.3-gke.0

---

The behavior difference is only between OperatorConfig.filter.matchOneOf and PodMonitoring.spec.endpoints[*].metricRelabeling.

It looks like metricRelabeling is evaluated, but certain names that should match the alternation aren’t being kept, while others are.

If there are known limitations or differences in regex dialect/evaluation order between OperatorConfig and PodMonitoring relabeling, guidance would be appreciated.

Thanks for taking a look!

[almeida-silas]

Would you mind taking a look, @lyanco and @bwplotka?

[bwplotka]

Hi!

I don't see anything immediately wrong in the metricRelabeling you are trying, but let me outline some tips and suggestions for investigating deeper:

1. Double check if collector DaemonSet YAML does not have EXTRA_ARGS env var configured. And if it has, it does not have ANY export.matchers set that would override whatever relabeling you are adding. We are investigating a bug that causes this variable to be left over from the old GKE versions.
2. I would check if this relabelling is propagated as you expect to the generated Prometheus config in "collector" configMap in gmp-system or gke-gmp-system (autopilot) namespace. I would assume your relabelling rules are in the correct job you expect it to be.
3. This field is already anchored, and it's using RE2. I recommend https://regex101.com/ with Golang flavour to test your regex.
4. I recommend quoting strings in YAML (", just to be safe) and simple regexes first (e.g. don't match partial names, but full names with "(name1|name2|name3)"

Hope that helps!

[almeida-silas]

Hi @bwplotka, I hope you’re doing well.
Thanks a lot for responding to me and for your tips!

I went through your suggestions:

1. There’s no EXTRA_ARGS env var configured.
2. Propagation is working correctly.
3. I had already checked this on the regex testing website (with the Golang flavor), my regex works there.
4. I also tried the approach you suggested.

What’s interesting is that these regex patterns do work with OperatorConfig:
workqueue_(queue_duration_seconds_bucket|work_duration_seconds_bucket|depth|adds_total|retries_total)
controller_runtime_(duration_seconds_bucket|reconcile_total|max_concurrent_reconciles)

And they also match correctly on my terminal:

curl -s http://localhost:8080/metrics | grep -E '(workqueue_queue_duration_seconds_bucket|workqueue_work_duration_seconds_bucket)'
# Or even:
grep -E 'workqueue_(queue_duration_seconds_bucket|work_duration_seconds_bucket)'

But the same regex fails when used in PodMonitoring.

I’ve also tried the variations you suggested:
(workqueue_queue_duration_seconds_bucket|workqueue_work_duration_seconds_bucket)
(workqueue_queue_duration_seconds_(.+)|workqueue_work_duration_seconds_(.+))
(workqueue_queue_duration_seconds_bucket)

But they still don’t match in PodMonitoring.

It only works if I configure PodMonitoring like this:
workqueue_queue_duration_seconds_(.+)
But not like this:
workqueue_queue_duration_seconds_bucket(.+)
or anything similar.

[bwplotka]

Ack, will try to reproduce this on our end, thanks!

[bwplotka]

👋🏽

I checked deeper and you are hitting a small nuance around Prometheus classic histograms.

I tried to explain this for you and others in the new example (#1754). Let me know if this helps!

TL;DR:

Looks like the workqueue_queue_duration in your data semantically represents the classic histogram (is of type=histogram), which means it consist of 3 metric names:

• workqueue_queue_duration_seconds_bucket (multiple series per histogram),
• workqueue_queue_duration_seconds_sum
• workqueue_queue_duration_seconds_count

OSS Prometheus allows you to ingest partial histogram, even if it makes little sense (histogram is not usable).

Google Cloud Monitoring rejects such partial histograms, so to keep full histogram you need to remember about all series matching either by explicit (workqueue_queue_duration_seconds_bucket|workqueue_queue_duration_seconds_sum|workqueue_queue_duration_seconds_count) or regex matching workqueue_queue_duration_seconds_.*.

This is a common gotcha, but it should be fixed with nhcb concept slowly getting added to OSS Prometheus.

Compare with OperatorConfig, where the same allowlist works as expected

It's likely due to the fact we have an ongoing issue with operatorconfig.collection.filter.matchOneOf (in some versions the filtering is essentially ignored) .

[almeida-silas]

Google Cloud Monitoring rejects such partial histograms, so to keep full histogram you need to remember about all series matching either by explicit (workqueue_queue_duration_seconds_bucket|workqueue_queue_duration_seconds_sum|workqueue_queue_duration_seconds_count) or regex matching workqueue_queue_duration_seconds_.*.

I actually fixed it this way, but your explanation makes it much clearer. Thanks a lot for the detailed answer!