Chore(Helm): Align Helm with Kustomize by enabling leader-election.

[Baarsgaard]

Helm has leader election disabled per default.
That is not the case for the Kustomize deployment

[weisdd]

Leader election is needed to prevent two controllers from trying to manage the same set of resources at the same time.
If you disable elections, better to have spec.strategy.type set to Recreate. Although, I think it's rather helm chart that has suboptimal configuration.

[theSuess]

We talked about this in the sync call after you left. We concluded that regular deployments should be fine as status updates no longer trigger reconciliation. So the only way simultaneous reconciliation can happen is by coinciding resyncPeriods

[weisdd]

@theSuess I was thinking rather about briefly having two pods during Rollout. - We have readiness probes, so the old pod will not be terminated for up to 15 seconds (initialDelaySeconds: 5 + periodSeconds: 10). Upon start, controllers would receive all CRs and watched owned resources.

[weisdd]

Briefly discussed this with @Baarsgaard over Slack. - There's an old bug in our chart which causes an additional ConfigMap with controller_manager_config.yaml to get created when leader election is on. - The values there are hardcoded, so they do not match whatever is passed through values.yaml.
We agreed that, as part of the PR, we should address templates/cm.yaml as well.

[Baarsgaard]

After some digging I found both some solid documentation on Leader election and a bug 😅
Leader election is handled through leases per default according to the Manager docs

The bug is that we always trigger status sync on Grafanas regardless of the instance being elected as the leader.
I have a couple of ideas on fixing that in a separate PR.

PS: Transitioning from my fork to the main repo, accidentally deleted the branch in the wrong repository.