Bug DatadogAgent CRD status agent list bloated causing etcdserver request too large

[ii64]

Encountered an issue where Terraform is unable to delete DatadogAgent object that has been error'd out for couple of hours, turns out that was caused by the admission controller delete request returned an error due to the etcdserver rejecting request because of the request is too large. Checked out the CRD object and found that there are a lot of .Status.AgentList.

# wc -l a.log
112276 a.log

# kubectl delete -n dd-redacted-app-production datadogagents.datadoghq.com dd-redacted-app
Error from server: etcdserver: request is too large

# kubectl apply -f ./b.json
Warning: resource datadogagents/dd-redacted-app is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
Error from server: error when applying patch:
[... redacted ...]
to:
Resource: "datadoghq.com/v2alpha1, Resource=datadogagents", GroupVersionKind: "datadoghq.com/v2alpha1, Kind=DatadogAgent"
Name: "dd-redacted-app", Namespace: "dd-redacted-app-production"
for: "./b.json": error when patching "./b.json": etcdserver: request is too large

Name:         dd-redacted-app
Namespace:    dd-redacted-app-production
Labels:       <none>
Annotations:  <none>
API Version:  datadoghq.com/v2alpha1
Kind:         DatadogAgent
Metadata:
  Creation Timestamp:  2024-12-11T21:10:35Z
  Finalizers:
    finalizer.agent.datadoghq.com
  Generation:        1
  Resource Version:  1234472018
  UID:               85e7b088-f8d6-47e3-96f7-054dc7621c64
Spec:
  Global:
    Cluster Name:  redacted-app
    Credentials:
      API Secret:
        Key Name:     api-key
        Secret Name:  dd-redacted-app-secrets
    Site:             us5.datadoghq.com
Status:
  Agent:
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed (0/0/0)
    Up To Date:  0
  Agent List:
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed
    Up To Date:  0
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed
    Up To Date:  0
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed
    Up To Date:  0
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed
    Up To Date:  0
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed
    Up To Date:  0
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed
    Up To Date:  0
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed
    Up To Date:  0
    Available:   0
    Current:     0
    Desired:     0
    Ready:       0
    State:       Failed
    Status:      Failed
    Up To Date:  0
    Available:   0
    Current:     0
[... truncated ...]

[levan-m]

Hello @ii64,
Thanks for submitting the issue. Are you able to reproduce the issue?
Could you please share DatadogAgent manifest, Operator, chart version and helm overrides?