Label mismatch when alerting on GKE system metrics

[falau]

When setting up alerts with GMP by rules custom resources, the PromQL expressions of alerts would be "scoped" with project_id, location, cluster, and namespace (depending on the rule scope). This is a behaviour hardcoded in the code that can't be configured by users as far as I can see.

However, for alerting on built-in GKE metrics, e.g. container/cpu/limit_utilization, the labels for cluster and namespace are cluster_name and namespace_name, according to the type spec. These labels are also not configurable.

Therefore, if I would like to to make alerts with Rules or ClusterRules on those built-in metrics, the alerts will never work due to the mismatch in labels.

The only workaround I can think of so far is to use GlobalRules to avoid scoping on cluster and namespace, but is this the only way to configure alerts on these GKE metrics, or is there any better suggestions?

Thanks :)

[lyanco]

You're right - if using the in-cluster alerting, you have to use GlobalRules. This is documented here: https://docs.cloud.google.com/stackdriver/docs/managed-prometheus/rules-managed#globalrules

That being said, we highly recommend using Cloud Alerting's PromQL alerting whenever possible, as it solves a lot of nasty issues with scoping, permissions, service accounts, cross-project alert evaluation, etc. Documentation is here and you can even migrate Prometheus rules and Prometheus AlertManager config. For GMP rules you'll have to slightly modify them, but it's mostly a copy-paste job.