feat(portal): add unified workload diagnostics - #326
Kevin Cho (chokevin) wants to merge 4 commits into
Conversation
| if !scope.Managed && namespace == "" { | ||
| namespace = s.runs.Namespace | ||
| } | ||
| live, err := runs.Board(ctx, s.runs.Reader, runs.Options{Namespace: namespace}) |
There was a problem hiding this comment.
When two managed workspaces share a namespace but use different LocalQueues, this lookup accepts a known workload UID from either queue. The new logs handler trusts this result, so a viewer authorized only for one workspace can read the other queue's container logs when the Portal has log access. Pass Queue: scope.LocalQueue before accepting a live workload, matching the existing runs endpoint and the Kueue fallback, and add detail/log tests for a same-namespace, different-queue workload. The queue filter already exists in runs.filterQueue.
| LimitBytes: &limitBytes, | ||
| Timestamps: true, | ||
| } | ||
| data, err := c.core.Pods(namespace).GetLogs(pod, opts).DoRaw(ctx) |
There was a problem hiding this comment.
The default chart-managed Portal ServiceAccount cannot make this request: its ClusterRole grants reads on pods, but not the separate pods/log subresource. Every valid current/previous container-log request therefore returns a Kubernetes authorization error and becomes HTTP 502. Add a dedicated get permission for pods/log to the generated role and cover it with a chart RBAC assertion.
| if (snap.objectState === 'deleted') return <Empty>Logs are not retained after the Kubernetes workload and pods are deleted.</Empty>; | ||
| if (!pod || !container) return <><Note>Choose a workload-owned container. Logs are fetched on demand as bounded snapshots and are not streamed or stored by the Portal.</Note> | ||
| {!snap.pods?.length ? <Empty>No live pods are available.</Empty> : <Table headers={['Pod', 'Container', 'State', 'Snapshots']} rows={snap.pods.flatMap(p => (p.containers || []).map(c => [ | ||
| p.name, c.name, <Status value={c.state}/>, <><ScopedLink to={`?view=logs&pod=${encodeURIComponent(p.name)}&container=${encodeURIComponent(c.name)}`}>current</ScopedLink> |
There was a problem hiding this comment.
These query-only links navigate away from the workload page. scopedURL resolves them against window.location.origin, so ?view=logs&pod=... becomes /?view=logs&pod=...; the server then redirects to /portal instead of opening the selected logs. Prefix the current workload pathname, as Subtabs does, for the pod/container, current, previous, and "Choose another container" links. Add click-navigation coverage so the workload UID and workspace remain selected.
This branch has not been deployed
2 participants
What
Adds a unified Portal workload details page keyed by immutable Kubernetes resource UID. The page separates Kueue admission, Kubernetes scheduling, application readiness, and experiment tracking; provides bounded pod/container log snapshots and event evidence; and shows workload-scoped GPU utilization, temperature, power, memory, and health signals with a link to the experiment dashboard.
The overview retains the coordinated live-source refresh and priority queueing views from the lower stack layers, keeps Active jobs as a distinct runtime concept, and adds links from pending/admitted workload rows into diagnostics.
Closes #324. Incorporates the workload-scoped portion of #322 without duplicating the full experiment dashboard.
Stack
This is stack layer 3:
chokevin-priority-queueingand should be reviewed/merged after the lower layers.Why
Users need one coherent workload lifecycle story: why quota has or has not been admitted, whether Kubernetes and the application are actually running, what failure evidence exists, whether the assigned GPUs look healthy and efficient, and where to continue deeper experiment analysis. Admission is deliberately not treated as proof of execution.
Non-goals
Testing
cd portal && make testcd portal && make lintcd portal && make buildcd portal/frontend && npm test -- --run(41 tests)cd portal && go test ./internal/portal/links ./internal/portalapipython3 scripts/check-license-headers.pygit diff --checkNo live cluster was contacted; end-to-end behavior is covered by repository fixtures and a session-local UI fixture rather than a deployed environment.
Risk
The main risk is correlation or authorization drift across Kueue Workloads, owner UIDs, pods, logs, telemetry, and durable history. The implementation keeps trusted workspace/namespace scope, UID-fences runtime evidence, bounds and redacts log snapshots, and explicitly marks unavailable/deleted evidence rather than guessing. Rollback is isolated to this stack layer.
AI assistance
Implementation, conflict resolution, and validation were performed with GitHub Copilot assistance.