User problem
Platform operators running TauGrid for customers need stronger assurance that GPU nodes are safe, healthy, and performant throughout their lifecycle—not only after a fault is already visible to a workload. Today, lifecycle responsibilities such as onboarding validation, burn-in, readiness qualification, continuous verification, maintenance handling, and return-to-service decisions are not represented as one end-to-end TauGrid capability.
Without a more intensive validation and lifecycle model, customers or operators must assemble these processes themselves, and TauGrid cannot consistently determine when a node should enter service, remain schedulable, be isolated, be remediated, or return to the fleet.
Desired outcome
Explore and define how TauGrid could handle more of the node lifecycle on behalf of customers, with intensive and repeatable testing across phases such as:
- initial node discovery, configuration, and onboarding;
- pre-production qualification and burn-in before workloads are admitted;
- continuous production health, performance, topology, and connectivity validation;
- safe cordon, drain, quarantine, diagnosis, and remediation workflows;
- post-maintenance or post-failure requalification before a node returns to service;
- lifecycle state, evidence, history, and fleet-level visibility for operators.
The result should clarify the customer experience, lifecycle states and transitions, ownership boundaries, safety controls, and what can be automated versus requiring operator approval.
Proposed approach
Keep the initial investigation open ended and use it to build a test and product model before committing to a specific API or controller design. Candidate work includes:
- inventory existing TauGrid node-health, GPU-health, topology, scheduling, and observability capabilities;
- define representative failure and degradation scenarios across GPUs, networking, storage, firmware, host configuration, Kubernetes, and workload execution;
- design increasingly intensive qualification suites, from fast readiness gates to longer reliability and performance exercises;
- establish measurable pass/fail criteria, baselines, evidence retention, retry policy, and escalation behavior;
- prototype a node lifecycle state machine and determine how it interacts with Kubernetes conditions, taints, cordon/drain, Kueue admission, maintenance windows, and customer workloads;
- test recovery and return-to-service behavior, including partial failures, interrupted remediation, controller restarts, and multi-node or correlated faults;
- evaluate opt-in policy, disruption budgets, auditability, and controls needed for TauGrid to take lifecycle actions safely on a customer's behalf;
- produce recommendations and follow-up issues for the smallest useful product increment.
This issue is intentionally exploratory. The investigation should avoid prematurely fixing the lifecycle phases, test inventory, or implementation boundary until experiments identify the right model.
Alternatives considered
Continue exposing health signals and leave onboarding, qualification, remediation, and return-to-service orchestration entirely to each operator. This minimizes TauGrid's responsibility but duplicates substantial operational work and produces inconsistent reliability outcomes.
Implement only additional health checks without lifecycle state or automated actions. This improves detection but does not address qualification, coordinated workload handling, remediation, or safe return to service.
Additional context
TauGrid already describes GPU health monitoring, automated drain on hardware faults, and fleet health visibility. This investigation should determine how those capabilities can evolve into a broader, customer-facing node lifecycle service backed by substantially more intensive testing.
Related: #282 covers operator-injected validation checks in the GPU dashboard and may provide one input surface, but this issue is broader and includes lifecycle policy, automated handling, qualification, remediation, and return to service.
Confirmation
User problem
Platform operators running TauGrid for customers need stronger assurance that GPU nodes are safe, healthy, and performant throughout their lifecycle—not only after a fault is already visible to a workload. Today, lifecycle responsibilities such as onboarding validation, burn-in, readiness qualification, continuous verification, maintenance handling, and return-to-service decisions are not represented as one end-to-end TauGrid capability.
Without a more intensive validation and lifecycle model, customers or operators must assemble these processes themselves, and TauGrid cannot consistently determine when a node should enter service, remain schedulable, be isolated, be remediated, or return to the fleet.
Desired outcome
Explore and define how TauGrid could handle more of the node lifecycle on behalf of customers, with intensive and repeatable testing across phases such as:
The result should clarify the customer experience, lifecycle states and transitions, ownership boundaries, safety controls, and what can be automated versus requiring operator approval.
Proposed approach
Keep the initial investigation open ended and use it to build a test and product model before committing to a specific API or controller design. Candidate work includes:
This issue is intentionally exploratory. The investigation should avoid prematurely fixing the lifecycle phases, test inventory, or implementation boundary until experiments identify the right model.
Alternatives considered
Continue exposing health signals and leave onboarding, qualification, remediation, and return-to-service orchestration entirely to each operator. This minimizes TauGrid's responsibility but duplicates substantial operational work and produces inconsistent reliability outcomes.
Implement only additional health checks without lifecycle state or automated actions. This improves detection but does not address qualification, coordinated workload handling, remediation, or safe return to service.
Additional context
TauGrid already describes GPU health monitoring, automated drain on hardware faults, and fleet health visibility. This investigation should determine how those capabilities can evolve into a broader, customer-facing node lifecycle service backed by substantially more intensive testing.
Related: #282 covers operator-injected validation checks in the GPU dashboard and may provide one input surface, but this issue is broader and includes lifecycle policy, automated handling, qualification, remediation, and return to service.
Confirmation