User problem
Infrastructure engineers need to validate environment-specific properties such as inter-node GPU connectivity, storage paths, topology boundaries, or accelerator networking and expose the results beside fleet health and capacity. TauGrid currently has built-in node health signals and a fixed RDMA validation contract, but there is no supported way for an operator to register an additional check without modifying the Portal or granting the Portal permission to execute arbitrary privileged workloads.
Desired outcome
Provide a supported extension contract for operator-owned, run-based validation checks. Authorized operators can define how a check is launched and how its durable results are published; the GPU dashboard can discover and render those results with explicit Passed, Failed, Running, Unknown, and Stale semantics. Missing, malformed, incomplete, unauthorized, or unsupported results must never appear healthy.
Proposed approach
- Define a versioned validation result envelope with check type/version, run identity, workspace/cluster scope, target inventory identities, start/end/observed timestamps, state/reason, cleanup state, evidence hashes, and an opaque typed details payload.
- Add a registration mechanism, likely a Tau core CRD plus controller/CLI integration, that references an approved workload template or controller implementation. The Portal must remain a read-only consumer and must not execute arbitrary scripts or hold workload credentials.
- Run checks through existing Kubernetes/Kueue authorization and scheduling boundaries. Persist scalar lifecycle events through existing telemetry and larger evidence through the approved artifact path.
- Expose a Portal API that validates schema versions, authorization scope, freshness, pagination, and unknown-schema behavior before returning results.
- Render a compact check summary only when a check is registered or has a result. Put technical evidence in an on-demand detail view rather than the fleet first viewport.
- Include deterministic examples and tests for an inter-node NCCL/InfiniBand check, including socket fallback, partial targets, stale evidence, cleanup failure, and node-identity replacement.
Alternatives considered
- Hard-code each new validation into the Portal. This tightly couples operational checks to UI releases and does not scale.
- Let the Portal launch arbitrary scripts directly. This creates an unacceptable privileged execution and secret-management boundary.
- Treat static device resources or node-local health scripts as proof of inter-node connectivity. Those signals establish local capability but do not prove an application data path.
Additional context
The initial use case is a two-node, one-GPU-per-node NCCL validation that proves NET/IB transport without socket fallback. Existing GPU monitoring already checks local InfiniBand device presence, ACTIVE/LinkUp state, expected rate, PKey, link flapping, and error counters; injected checks should complement rather than replace those continuous node signals.
Confirmation
User problem
Infrastructure engineers need to validate environment-specific properties such as inter-node GPU connectivity, storage paths, topology boundaries, or accelerator networking and expose the results beside fleet health and capacity. TauGrid currently has built-in node health signals and a fixed RDMA validation contract, but there is no supported way for an operator to register an additional check without modifying the Portal or granting the Portal permission to execute arbitrary privileged workloads.
Desired outcome
Provide a supported extension contract for operator-owned, run-based validation checks. Authorized operators can define how a check is launched and how its durable results are published; the GPU dashboard can discover and render those results with explicit Passed, Failed, Running, Unknown, and Stale semantics. Missing, malformed, incomplete, unauthorized, or unsupported results must never appear healthy.
Proposed approach
Alternatives considered
Additional context
The initial use case is a two-node, one-GPU-per-node NCCL validation that proves
NET/IBtransport without socket fallback. Existing GPU monitoring already checks local InfiniBand device presence,ACTIVE/LinkUpstate, expected rate, PKey, link flapping, and error counters; injected checks should complement rather than replace those continuous node signals.Confirmation