User problem
TauGrid uses Kueue to admit Jobs and RayJobs based on configured quota, resource flavors, topology, and priority. Kueue quota, however, is not proof that the corresponding physical nodes can be created. A workload can be admitted and consume queue quota while its pods remain pending for cluster autoscaling, and a distributed or gang-scheduled workload can wait on partial capacity without a clear admission-time result.
TauGrid needs a coherent way for Kueue admission to account for capacity that must be provisioned by a cluster autoscaler. Users should not need to manually coordinate Kueue objects, autoscaler-specific APIs, placeholder workloads, retries, or cleanup to safely run scale-from-zero CPU and GPU workloads.
Desired outcome
Produce a system design for capacity-aware Kueue admission in TauGrid, with Cluster Autoscaler as the first concrete integration.
The design should establish the user experience and system contract, explain the ownership boundaries between TauGrid, Kueue, the Kubernetes scheduler, and the autoscaler, and fit TauGrid's existing CLI, TauCluster/TauWorkspace APIs, workload profiles, queues, Job/RayJob rendering, status, and lifecycle commands.
A reasonable design should include sequence and component diagrams, API and configuration examples, controller reconciliation boundaries, failure-mode analysis, and alternatives considered. It should also propose a thin end-to-end prototype that starts with insufficient or zero worker capacity, submits both a Kubernetes Job and a multi-node RayJob through Kueue, observes autoscaler behavior, and demonstrates success, timeout, cancellation, and cleanup paths.
User problem
TauGrid uses Kueue to admit Jobs and RayJobs based on configured quota, resource flavors, topology, and priority. Kueue quota, however, is not proof that the corresponding physical nodes can be created. A workload can be admitted and consume queue quota while its pods remain pending for cluster autoscaling, and a distributed or gang-scheduled workload can wait on partial capacity without a clear admission-time result.
TauGrid needs a coherent way for Kueue admission to account for capacity that must be provisioned by a cluster autoscaler. Users should not need to manually coordinate Kueue objects, autoscaler-specific APIs, placeholder workloads, retries, or cleanup to safely run scale-from-zero CPU and GPU workloads.
Desired outcome
Produce a system design for capacity-aware Kueue admission in TauGrid, with Cluster Autoscaler as the first concrete integration.
The design should establish the user experience and system contract, explain the ownership boundaries between TauGrid, Kueue, the Kubernetes scheduler, and the autoscaler, and fit TauGrid's existing CLI, TauCluster/TauWorkspace APIs, workload profiles, queues, Job/RayJob rendering, status, and lifecycle commands.
A reasonable design should include sequence and component diagrams, API and configuration examples, controller reconciliation boundaries, failure-mode analysis, and alternatives considered. It should also propose a thin end-to-end prototype that starts with insufficient or zero worker capacity, submits both a Kubernetes Job and a multi-node RayJob through Kueue, observes autoscaler behavior, and demonstrates success, timeout, cancellation, and cleanup paths.