argocd_cluster_events_total metric is high for ConfigMaps

[dee-kryvenko]

See screenshot - this look suspicious to me. Can someone please explain what does that metric tracking, exactly? I don't see kubernetes events anywhere near that numbers, so it must be about resource updates then? How do I identify where and why it all comes from? ConfigMaps are far from the most frequently edited resource in my clusters...

[crenshaw-dev]

it must be about resource updates then

Yep!

How do I identify where and why it all comes from?

Check your Kubernetes API audit logs.

[dee-kryvenko]

Thanks! Looks like it is caused by control-plane.alpha.kubernetes.io/leader lease updates from all the controllers that are still using configmaps for that... I am researching in the context of this issue #15464 (comment), could this be in any way related? Can argo be configured to ignore configmaps that are used for leader elections?

[dee-kryvenko]

  resource.customizations.ignoreResourceUpdates.ConfigMap: |
    jqPathExpressions:
    - '.metadata.annotations."control-plane.alpha.kubernetes.io/leader"'

Something like this should work I suppose...

[dee-kryvenko]

Well, it didn't. This metric is still high with that ignoreResourceUpdates from above.

[dee-kryvenko]

Ok so I am reading this https://argo-cd.readthedocs.io/en/stable/operator-manual/reconcile/#ignoring-updates-for-untracked-resources - sounds like my customizations are not going to be picked up unless resources are directly tracked, otherwise resources needs to be annotated with argocd.argoproj.io/ignore-resource-updates: "true". How's that even supposed to work? In many cases I don't even have any control over these resources. I seem to be having same problem with all the Crossplane resources that are failing to compose and causing .spec.resourceRefs change hundreds of times per minute. This is not practical to require to annotate every resource to get these rules to work. What else can be done @crenshaw-dev?

[dee-kryvenko]

Anyway I think this actually is a bug, so I am creating #21506 and closing this question