Manage deployment of resources for MCOA and metrics collector

[thibaultmg]

When enabling the MCOA feature, we must disable the current metrics-collector related resources. However, some resources, like certificates, must be preserved/created. And eventually, they will probably be handled by MCOA directly at a later stage.

Because the necessary changes were complex to implement, I ended up refactoring the high-level logic of the placementrule_controller to improve readability and maintainability. For this part I tried to keep identical processing.
The only breaking change is the end of OCP 3 support.

Here are some of the refactoring actions done:

• Simplify logging: remove error logs when not needed (error is returned), remove noisy doing nothing logs (only keeping high level ones in the Reconcile function), log write operations
• Wrap returned errors to provide more context
• Avoid ignoring errors, especially in the certificates package
• Forward context to functions calling the k8s api (replacing .TODO())
• Stop processing on errors if possible (some were just logged)
• Extract processing in coherent sub functions
• Reduce nested if statements
• Reduce deep function call chains with long parameter lists
• Remove processing specific to the upgrade from 2.9 to 2.10
• Remove some shared variables
• Remove setting variables in predicates when possible

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: thibaultmg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [thibaultmg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[thibaultmg]

/retest

[sonarqubecloud]

Quality Gate failed

Failed conditions
32.0% Coverage on New Code (required ≥ 70%)

See analysis details on SonarQube Cloud

[openshift-ci]

@thibaultmg: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/sonarcloud	d6a16dd	link	false	/test sonarcloud

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.