DatadogMonitor Finalizers get removed regardless of failure

[paulbrassard-figure]

Describe what happened:
Deleting a DatadogMonitor that is referenced by a DatadogSLO results in the monitor resource being deleted from the kubernetes cluster but not from Datadog. This results in the monitor existing still, outside of source control.

Describe what you expected:
Finalizers are not removed until the datadog-operator is able to successfully finalize the DatadogMonitor, resulting in the kubernetes object DatadogMonitor remaining until the resource is deleted from Datadog.

Steps to reproduce the issue:

• Create a DatadogMonitor
• Create a DatadogSLO that references the DatadogMonitor
• Delete the DatadogMonitor

Additional information:

• Function finalizeDatadogMonitor only logs an error in case of a failure
  

  datadog-operator/controllers/datadogmonitor/finalizer.go

  Lines 54 to 66 in c72fc48

  	func (r *Reconciler) finalizeDatadogMonitor(logger logr.Logger, dm *datadoghqv1alpha1.DatadogMonitor) {
  	if dm.Status.Primary {
  	err := deleteMonitor(r.datadogAuth, r.datadogClient, dm.Status.ID)
  	if err != nil {
  	logger.Error(err, "failed to finalize monitor", "Monitor ID", fmt.Sprint(dm.Status.ID))
  	return
  	}
  	logger.Info("Successfully finalized DatadogMonitor", "Monitor ID", fmt.Sprint(dm.Status.ID))
  	event := buildEventInfo(dm.Name, dm.Namespace, datadog.DeletionEvent)
  	r.recordEvent(dm, event)
  	}
  	}

• Function handleFinalizer will remove the finalizers regardless of failure
  

  datadog-operator/controllers/datadogmonitor/finalizer.go

  Lines 28 to 31 in c72fc48

  	r.finalizeDatadogMonitor(logger, dm)
  	dm.SetFinalizers(utils.RemoveString(dm.GetFinalizers(), datadogMonitorFinalizer))
  	err := r.client.Update(context.TODO(), dm)

• Example log with SLO related failure:

  datadog-operator-576d8ccd98-ftftr {"level":"ERROR","ts":"2024-07-11T19:54:35Z","logger":"controllers.DatadogMonitor","msg":"failed to finalize monitor","datadogmonitor":"default/sandbox-slo-generation-verification-datadog-monitor","Monitor ID":"149070491","error":"error deleting monitor: 400 Bad Request: {\"errors\":[\"defaultdict(<class 'list'>, {149070491: ['monitor [149070491,Sandbox SLO generation monitor] is referenced in slos: [7716c0b1451055bd87688d5733f4e6d2,sandbox-slo-generation-verification-datadog-monitor slo]']})\"]}"}
  

Environment details (Operating System, Cloud provider, etc):
We are on GKE, deployed the latest datadog operator with helm chart 1.8.3 as detailed in the documentation.