Repository navigation
Conversation
…node clusters
Everything under OPEN_RL_TMP_DIR lives on the shared volume: model weights, the vLLM compile cache, a trainer's sparse
deltas and LoRA adapters, the samplers' weights, checkpoints. On one node, every worker reaches the shared volume as a
single NFS client, and on Filestore that client's limits bind: a 207 MiB delta write took 31 s at the median and a LoRA
adapter save 17 s per step.
With OPEN_RL_LOCAL_HOST_PATH set, worker pods and sampler-pool dispatchers mount that host directory (for example the
node's local SSDs) over OPEN_RL_TMP_DIR, and mount back only OPEN_RL_SHARED_DIRS (default: runs) from the shared volume.
It is one mount, not one per directory, so files still move between directories by rename and hard link: a LoRA save
moves the adapter from peft into sampler_lora, which fails across mounts with EXDEV. A dispatcher writes its pool's
revision file, which its samplers' readiness probes read, so OPEN_RL_LOCAL_NODE_SELECTOR (key=value) places it on the
GPU node. OPEN_RL_TMP_DIR must be under /mnt/shared.
A pod sees only what was written on its own node, so this is for clusters whose workers all run on one node, and
nothing on the host directory survives the node. Unset, nothing changes.
The API server reads and writes the same directories (revision metadata, adapters, checkpoints), so it needs the same
mount and must run on the GPU node; that is a patch to its Deployment, not part of this change:
env:
- {name: OPEN_RL_LOCAL_HOST_PATH, value: /mnt/stateful_partition/kube-ephemeral-ssd/openrl}
- {name: OPEN_RL_LOCAL_NODE_SELECTOR, value: cloud.google.com/gke-accelerator=nvidia-h100-80gb}
volumeMounts:
- {name: local-tmp, mountPath: /mnt/shared/open-rl}
- {name: shared-storage, mountPath: /mnt/shared/open-rl/runs, subPath: open-rl/runs}
volumes:
- {name: local-tmp, hostPath: {path: /mnt/stateful_partition/kube-ephemeral-ssd/openrl, type: DirectoryOrCreate}}
nodeSelector: {cloud.google.com/gke-accelerator: nvidia-h100-80gb}
tolerations: [{key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}]
Measured on 1 x a3-highgpu-8g (8 x H100, 16 local NVMe SSDs), mixed LoRA and full fine-tuning sweeps:
- LoRA: time between steps 61 s -> 29 s, optim_step to adapter saved 17 s -> 7 s.
- Full fine-tuning: delta write 31 s -> 0.5 s, first step 295 s -> 170 s with warm model and compile caches.
- Sweep wall clock 14.7 min (20 jobs, deltas already local) -> 6.8 min (16 jobs, everything local).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…node clusters
Everything under OPEN_RL_TMP_DIR lives on the shared volume: model weights, the vLLM compile cache, a trainer's sparse deltas and LoRA adapters, the samplers' weights, checkpoints. On one node, every worker reaches the shared volume as a single NFS client, and on Filestore that client's limits bind: a 207 MiB delta write took 31 s at the median and a LoRA adapter save 17 s per step.
With OPEN_RL_LOCAL_HOST_PATH set, worker pods and sampler-pool dispatchers mount that host directory (for example the node's local SSDs) over OPEN_RL_TMP_DIR, and mount back only OPEN_RL_SHARED_DIRS (default: runs) from the shared volume. It is one mount, not one per directory, so files still move between directories by rename and hard link: a LoRA save moves the adapter from peft into sampler_lora, which fails across mounts with EXDEV. A dispatcher writes its pool's revision file, which its samplers' readiness probes read, so OPEN_RL_LOCAL_NODE_SELECTOR (key=value) places it on the GPU node. OPEN_RL_TMP_DIR must be under /mnt/shared.
A pod sees only what was written on its own node, so this is for clusters whose workers all run on one node, and nothing on the host directory survives the node. Unset, nothing changes.
The API server reads and writes the same directories (revision metadata, adapters, checkpoints), so it needs the same mount and must run on the GPU node; that is a patch to its Deployment, not part of this change:
env:
- {name: OPEN_RL_LOCAL_HOST_PATH, value: /mnt/stateful_partition/kube-ephemeral-ssd/openrl}
- {name: OPEN_RL_LOCAL_NODE_SELECTOR, value: cloud.google.com/gke-accelerator=nvidia-h100-80gb} volumeMounts:
- {name: local-tmp, mountPath: /mnt/shared/open-rl}
- {name: shared-storage, mountPath: /mnt/shared/open-rl/runs, subPath: open-rl/runs} volumes:
- {name: local-tmp, hostPath: {path: /mnt/stateful_partition/kube-ephemeral-ssd/openrl, type: DirectoryOrCreate}} nodeSelector: {cloud.google.com/gke-accelerator: nvidia-h100-80gb} tolerations: [{key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}]
Measured on 1 x a3-highgpu-8g (8 x H100, 16 local NVMe SSDs), mixed LoRA and full fine-tuning sweeps: