Logs API Stops Functioning When a Pod Fails

[Fluder-Paradyne]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
When a pod in a deployment fails, the logs API returns a 500 status code when attempting to fetch logs for the entire deployment. This behavior has been observed when one or more pods are in one of the following states: Completed, Failed, or Error.

To Reproduce
Deploy this chart https://github.com/Fluder-Paradyne/hello-world-helm/tree/main/charts/hello-world

and run this command, this cmd will cause the pod to run out off storage and get evicted.

kubectl exec $POD_NAME -c storage-consumer -- sh -c "dd if=/dev/zero of=/tmp/dummy bs=1M count=5000"

Kubernetes logs api gives ,
unable to retrieve container logs for containerd://a0002996fc1eb2a973c4708fa4f4f5f1b9e643114322c71fe54dbbfef27148d5

This causes a parsing error here

Expected behavior
Show logs of other pods

Screenshots

Version

{
    "Version": "v2.10.7+b060053",
    "BuildDate": "2024-04-15T08:45:08Z",
    "GitCommit": "b060053b099b4c81c1e635839a309c9c8c1863e9",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/arm64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.3+gf03cc04",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}

Logs

19:10:24                api-server | INFO[0025] received streaming call /application.ApplicationService/PodLogs  grpc.method=PodLogs grpc.request.content="name:\"hello\" namespace:\"argocd\" container:\"hello-world\" sinceSeconds:0 tailLines:1000 follow:true kind:\"ReplicaSet\" group:\"apps\" resourceName:\"hello-hello-world-79fcc7594\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2024-12-21T19:10:24+05:30" span.kind=server system=grpc
19:10:24                api-server | ERRO[0025] finished streaming call with code Unknown     error="parsing time \"unable\" as \"2006-01-02T15:04:05.999999999Z07:00\": cannot parse \"unable\" as \"2006\"" grpc.code=Unknown grpc.method=PodLogs grpc.service=application.ApplicationService grpc.start_time="2024-12-21T19:10:24+05:30" grpc.time_ms=33.48 span.kind=server system=grpc

[Fluder-Paradyne]

I have already started debugging this issue,
I would like to fix this.

[Fluder-Paradyne]

Hi @andrii-korotkov-verkada, this is issue is present in latest version, I am able to replicate this locally in master