[TCL 2925] Modify removeNADFinalizerIfSafe to ignore pods without GUID protection finalizer instead of pods with deletionTimestamp set

[Part-TimeWizard]

Instead of checking if the pod has a deletion timestamp, we will instead check if the pod has the GUID protection finalizer. If it doesn’t, we know the GUID no longer exists in UFM and therefore no longer needs the NAD for pkey lookup. The flow then becomes:

1. Lookup NAD to get PKey and guid list
2. Remove GUIDs from UFM
3. Remove pod finalizer for successful deletions
4. Attempt to remove NAD finalizer if all pods pointing to the NAD no longer have finalizers.

Consistency Guarantee Note: This gets hairy from a consistency perspective because we require read after write consistency to ensure that pods we just removed the finalizers from show without finalizers in the subsequent List call in checkIfAnyPodsUsingNetwork(). From what I could find, it seems like Kubernetes versions ≥v1.31 provide strong consistency guarantees for List calls by default by doing quorum reads.

Most recent # Mode when no configs are set
Return data at the most recent resource version. The returned data must be consistent (in detail: served from etcd via a quorum read). For etcd v3.4.31+ and v3.5.13+, Kubernetes 1.34 serves “most recent” reads from the watch cache: an internal, in-memory store within the API server that caches and mirrors the state of data persisted into etcd. Kubernetes requests progress notification to maintain cache consistency against the etcd persistence layer. Kubernetes v1.28 through to v1.30 also supported this feature, although as Alpha it was not recommended for production nor enabled by default until the v1.31 release.

[Part-TimeWizard]

Linear issue w/ debug log: https://linear.app/together-ai/issue/TCL-2925/vmi-finalizer-hang-in-oh1-nad-removed-before-its-safe-to