feat: standalone GPU claim scheduler (OpenRLWorker) - #178
Open
ShubyM wants to merge 5 commits into
Open
Conversation
ShubyM
force-pushed
the
feat/gpu-scheduler
branch
2 times, most recently
from
August 13, 2026 02:37
910416e to
3d78087
Compare
ShubyM
force-pushed
the
feat/gpu-scheduler
branch
from
August 13, 2026 19:00
00ebcfb to
8fd7789
Compare
The contract: a worker says which role it is, which model it serves, which owner it belongs to, and how much accelerator memory the estimator says it needs. Everything else -- device count, claim, node -- is the controller's decision, reported back in status. Specs are immutable (CEL-enforced): every field either places the worker or renders its pod, and V1 does neither twice. Identity is metadata.name; modelId is configuration for the worker process, never identity. docs/design.md is the full specification, including the V1 limitation this system is built around -- several workers may be assigned to one claim, but exactly one is resident in accelerator memory at a time -- and Appendix C records the exact inputs, outputs, and reconcile sequence.
Decide is the policy in one sentence: spread onto a free pool while one exists, share an allocated claim only under contention. Joining takes three checks -- the worker fits a device by itself (nothing is ever summed; only one worker is resident), the claim is below max-workers-per-claim, and the node has host memory for every worker that may be parked -- ranked by fewest workers, then name. Unallocated claims reserve the pool they were sized for but are never joined: nothing about a claim with no node can be checked against anything real, so a burst waits a retry instead of guessing. behavior_test.go is the acceptance suite: arrivals and departures with the estimator's real tier figures on the hardware we run, played through the same Decide the controller calls.
The Kubernetes half: one reconcile reads the fleet fresh -- the DRA driver's latest complete ResourceSlice pools intersected with the operator's node labels, plus managed claims and every worker's booking -- makes one placement decision, and does at most three writes: create a ResourceClaim, create the worker pod, patch status. Identity is layered so nothing can alias: the CR name within an incarnation, the UID across incarnations. Claim names derive from the UID, so a recreated worker never collides with or adopts its predecessor's claim; a pod owned by another incarnation is replaced, never adopted; and a finalizer holds a deleting worker -- and its memory booking -- until its pod is verifiably gone, so a seat cannot free while the process still holds the device. Claims carry their shape as labels and CEL bounds: the floor is the worker's per-device share, the ceiling the device size the claim was priced against, so DRA cannot substitute a bigger device placement never chose. Pods select nodes with two ORed affinity terms, making the documented role-label default actually schedulable; labels carry sanitized identities while env vars carry exact ones. Claims have no owner reference -- a shared claim belongs to no single worker -- so a periodic sweep reclaims the ones nobody references. Applying the manifests is inert: nothing creates OpenRLWorker objects yet. Labeled nodes are OpenRL-exclusive by contract.
hack/kind-smoke.sh runs the pipeline no unit test can: real API server, real kube-scheduler, real DRA -- fake GPUs from the DRA example driver by default, the same script against real hardware via env. It asks for one more worker than the node has devices and asserts spread-then-share with every claim genuinely allocated. hack/stress.sh churns workers against a live cluster and checks the books every round: running == min(live, seats), every pending worker names its reason, never more claims than GPUs, and a full teardown leaves nothing behind. The scheduler-smoke overlay keeps the smoke on the shipped manifests instead of shell fixups. CI runs gofmt/vet/go test on pull requests and builds and publishes the placement-controller image alongside the existing ones. Workflow steps are SHA-pinned with persist-credentials off, per the zizmor scan.
ShubyM
force-pushed
the
feat/gpu-scheduler
branch
from
August 13, 2026 20:44
6df05cc to
96f772a
Compare
ShubyM
marked this pull request as ready for review
August 14, 2026 19:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A self-contained Go controller that turns OpenRLWorker requests into DRA ResourceClaims and worker pods. A worker states its role, model, owner, and estimated peak accelerator memory. The controller then spreads workers onto free devices while any exist, shares claims under contention, and enforces the V1 rule: several workers may be assigned to a claim, but exactly one is resident in accelerator memory at a time.
Full spec in controller/docs/design.md; reading order in controller/docs/layout.md.
Structure (one commit per layer): API + spec → placement as pure functions (+ behavior tests using the estimator's real tier figures) → controller + manifests → smoke/stress/CI → docs. Applying the manifests is inert until something creates OpenRLWorkers.
Verified: unit + behavior suites in CI; a kind smoke (make smoke, fake GPUs via the DRA example driver — no hardware) asserting spread-then-share with genuinely allocated claims; a churn stress (24 workers / 16 seats / 5 rounds, clean teardown incl. claim reclamation); and the identical script against real 2× L4s under the NVIDIA DRA driver.
V1 limits, on purpose: no elasticity (pending workers wait; nothing autoscales), no GPU-memory co-residency or bin-packing, no migration/rebalancing after placement, one node per worker, fairness is cooperative per-claim
round-robin by owner, specs are immutable (replace, don't mutate), and openrl.io/enabled nodes are OpenRL-exclusive by contract.