Skip to content

workers: an experimental temp root on a node's local SSD, for single-… - #440

Draft
droot wants to merge 1 commit into
openrl-dev:mainfrom
droot:pr/local-tmp-single-node
Draft

droot wants to merge 1 commit into
openrl-dev:mainfrom
droot:pr/local-tmp-single-node

Conversation

@droot

@droot droot commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

…node clusters

Everything under OPEN_RL_TMP_DIR lives on the shared volume: model weights, the vLLM compile cache, a trainer's sparse deltas and LoRA adapters, the samplers' weights, checkpoints. On one node, every worker reaches the shared volume as a single NFS client, and on Filestore that client's limits bind: a 207 MiB delta write took 31 s at the median and a LoRA adapter save 17 s per step.

With OPEN_RL_LOCAL_HOST_PATH set, worker pods and sampler-pool dispatchers mount that host directory (for example the node's local SSDs) over OPEN_RL_TMP_DIR, and mount back only OPEN_RL_SHARED_DIRS (default: runs) from the shared volume. It is one mount, not one per directory, so files still move between directories by rename and hard link: a LoRA save moves the adapter from peft into sampler_lora, which fails across mounts with EXDEV. A dispatcher writes its pool's revision file, which its samplers' readiness probes read, so OPEN_RL_LOCAL_NODE_SELECTOR (key=value) places it on the GPU node. OPEN_RL_TMP_DIR must be under /mnt/shared.

A pod sees only what was written on its own node, so this is for clusters whose workers all run on one node, and nothing on the host directory survives the node. Unset, nothing changes.

The API server reads and writes the same directories (revision metadata, adapters, checkpoints), so it needs the same mount and must run on the GPU node; that is a patch to its Deployment, not part of this change:

env:
- {name: OPEN_RL_LOCAL_HOST_PATH, value: /mnt/stateful_partition/kube-ephemeral-ssd/openrl}
- {name: OPEN_RL_LOCAL_NODE_SELECTOR, value: cloud.google.com/gke-accelerator=nvidia-h100-80gb} volumeMounts:
- {name: local-tmp, mountPath: /mnt/shared/open-rl}
- {name: shared-storage, mountPath: /mnt/shared/open-rl/runs, subPath: open-rl/runs} volumes:
- {name: local-tmp, hostPath: {path: /mnt/stateful_partition/kube-ephemeral-ssd/openrl, type: DirectoryOrCreate}} nodeSelector: {cloud.google.com/gke-accelerator: nvidia-h100-80gb} tolerations: [{key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}]

Measured on 1 x a3-highgpu-8g (8 x H100, 16 local NVMe SSDs), mixed LoRA and full fine-tuning sweeps:

  • LoRA: time between steps 61 s -> 29 s, optim_step to adapter saved 17 s -> 7 s.
  • Full fine-tuning: delta write 31 s -> 0.5 s, first step 295 s -> 170 s with warm model and compile caches.
  • Sweep wall clock 14.7 min (20 jobs, deltas already local) -> 6.8 min (16 jobs, everything local).

…node clusters

Everything under OPEN_RL_TMP_DIR lives on the shared volume: model weights, the vLLM compile cache, a trainer's sparse
deltas and LoRA adapters, the samplers' weights, checkpoints. On one node, every worker reaches the shared volume as a
single NFS client, and on Filestore that client's limits bind: a 207 MiB delta write took 31 s at the median and a LoRA
adapter save 17 s per step.

With OPEN_RL_LOCAL_HOST_PATH set, worker pods and sampler-pool dispatchers mount that host directory (for example the
node's local SSDs) over OPEN_RL_TMP_DIR, and mount back only OPEN_RL_SHARED_DIRS (default: runs) from the shared volume.
It is one mount, not one per directory, so files still move between directories by rename and hard link: a LoRA save
moves the adapter from peft into sampler_lora, which fails across mounts with EXDEV. A dispatcher writes its pool's
revision file, which its samplers' readiness probes read, so OPEN_RL_LOCAL_NODE_SELECTOR (key=value) places it on the
GPU node. OPEN_RL_TMP_DIR must be under /mnt/shared.

A pod sees only what was written on its own node, so this is for clusters whose workers all run on one node, and
nothing on the host directory survives the node. Unset, nothing changes.

The API server reads and writes the same directories (revision metadata, adapters, checkpoints), so it needs the same
mount and must run on the GPU node; that is a patch to its Deployment, not part of this change:

  env:
    - {name: OPEN_RL_LOCAL_HOST_PATH, value: /mnt/stateful_partition/kube-ephemeral-ssd/openrl}
    - {name: OPEN_RL_LOCAL_NODE_SELECTOR, value: cloud.google.com/gke-accelerator=nvidia-h100-80gb}
  volumeMounts:
    - {name: local-tmp, mountPath: /mnt/shared/open-rl}
    - {name: shared-storage, mountPath: /mnt/shared/open-rl/runs, subPath: open-rl/runs}
  volumes:
    - {name: local-tmp, hostPath: {path: /mnt/stateful_partition/kube-ephemeral-ssd/openrl, type: DirectoryOrCreate}}
  nodeSelector: {cloud.google.com/gke-accelerator: nvidia-h100-80gb}
  tolerations: [{key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}]

Measured on 1 x a3-highgpu-8g (8 x H100, 16 local NVMe SSDs), mixed LoRA and full fine-tuning sweeps:
- LoRA: time between steps 61 s -> 29 s, optim_step to adapter saved 17 s -> 7 s.
- Full fine-tuning: delta write 31 s -> 0.5 s, first step 295 s -> 170 s with warm model and compile caches.
- Sweep wall clock 14.7 min (20 jobs, deltas already local) -> 6.8 min (16 jobs, everything local).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant