|
1 | | -# Harvey LAB RL on the GKE scheduler stack |
| 1 | +# Harvey LAB RL on GKE |
2 | 2 |
|
3 | | -How to run the Harvey LAB LoRA recipe against an OpenRL cluster deployed with |
4 | | -the scheduler (`OPEN_RL_WORKER_MANAGER=scheduler`, see |
5 | | -[gke-fft-timeslice.md](gke-fft-timeslice.md)), with one GPU for the trainer and |
6 | | -one for the sampler. Verified on 2026-09-07 with Qwen3.5-9B on two H100 nodes. |
| 3 | +This guide runs the [Harvey LAB recipe](../../examples/harvey_labs/README.md) |
| 4 | +at full scale on GKE. The recipe's driver runs as a Kubernetes Job in the |
| 5 | +OpenRL training cluster, episode sandboxes run on Agent Sandbox in a separate |
| 6 | +cluster, and the judge runs behind an OpenAI-compatible endpoint that the |
| 7 | +driver can reach. The sandboxes have no network egress, so only the driver |
| 8 | +talks to the judge. |
7 | 9 |
|
8 | | -## What the cluster needs |
| 10 | +``` |
| 11 | +training cluster sandbox cluster |
| 12 | + openrl-system: API server, scheduler, workers Agent Sandbox warm pool |
| 13 | + harvey-labs Job (driver) ──claims, tool calls──────▶ LAB sandbox pods |
| 14 | + │ |
| 15 | + └──grading──▶ judge (OpenAI-compatible, e.g. GLM-5.2 on vLLM) |
| 16 | +``` |
9 | 17 |
|
10 | | -The stack on `main` runs the recipe end to end only with the server fixes on |
11 | | -the `harvey/k8s-stack` branch (all in `src/server` and `src/training`): |
| 18 | +## What you need |
12 | 19 |
|
13 | | -| Fix | Why the recipe needs it | |
| 20 | +| Component | What the reference run used | |
14 | 21 | | --- | --- | |
15 | | -| API server env passthrough to worker pods | Scheduler-placed samplers otherwise start at `max_model_len=8192`; LAB episodes need 64k or more. | |
16 | | -| Per-session adapter snapshots | The trainer rewrote one adapter directory in place while the sampler read it over the shared volume. | |
17 | | -| Adapter keys remapped to the hub layout | Qwen3.5 and Gemma 4 hubs wrap the text model; vLLM silently applies no adapter otherwise. | |
18 | | -| `train_unembed` ignored by default | The tinker client asks for an `lm_head` adapter; vLLM rejects the whole adapter for these model families. | |
19 | | -| Sampler host memory: 6 B/param, 3x limit | vLLM 0.25 peaks near 170 GiB of host memory for a minute while warming up Qwen3.5-9B. | |
20 | | -| Chunked target-logprob projection | The full-logits path needs ~60 GiB at 64k tokens; one H100 cannot hold it beside the model. | |
21 | | -| `VLLM_MAX_MODEL_LEN` capped at the model's limit | One API-server-wide value must not crash-loop smaller models. | |
22 | | -| LoRA targets found through PEFT wrappers | The LoRA runtime is shared per base model; the second model to arrive otherwise fails `create_model`. | |
| 22 | +| OpenRL on GKE | [GKE setup](gke-setup.md) with an 8 × H100 80 GB GPU node pool. Four GPUs run the Automodel trainer with context parallelism across 4 GPUs, and four GPUs run vLLM samplers. | |
| 23 | +| Sandbox cluster | [LAB sandboxes on Agent Sandbox](../../examples/harvey_labs/agent-sandbox/README.md) in a separate cluster on the same VPC. | |
| 24 | +| Judge | GLM-5.2-FP8 on vLLM, served as `gpt-glm-5.2` and reachable from pods in the training cluster. | |
| 25 | +| Shared volume | A `ReadWriteMany` PVC in the Job's namespace for logs and checkpoints. `cluster-job.yaml` uses `open-rl-shared-pvc`. | |
23 | 26 |
|
24 | | -Build and roll them from that branch (Cloud Build must be enabled in the project): |
| 27 | +## 1. Set the context window on the API server |
| 28 | + |
| 29 | +By default the recipe uses the model's full context window: 262,144 tokens for |
| 30 | +Qwen3.5-9B. Because the OpenRL API server passes its token limits to worker |
| 31 | +pods only when creating them, set these environment variables on the API |
| 32 | +server before starting the run: |
25 | 33 |
|
26 | 34 | ```bash |
27 | | -make cloud-build-api-server cloud-build-server GCP_PROJECT=<project> |
28 | | -kubectl set image deployment/open-rl-api-server API server=gcr.io/<project>/open-rl-api-server:<tag> |
29 | | -kubectl set env deployment/open-rl-api-server OPEN_RL_WORKER_IMAGE=gcr.io/<project>/open-rl-server:<tag> |
| 35 | +kubectl -n openrl-system set env deployment/open-rl-api-server \ |
| 36 | + VLLM_MAX_MODEL_LEN=262144 OPEN_RL_TRAIN_TOKEN_BUDGET=262144 |
30 | 37 | ``` |
31 | 38 |
|
32 | | -Then set the run knobs on the API server. They ride along into every worker pod: |
| 39 | +We ran it on hardware that fits that window. On GPUs with less memory, pass a |
| 40 | +smaller `max_trajectory_tokens` in the Job and lower both limits to match. |
33 | 41 |
|
34 | | -```bash |
35 | | -kubectl set env deployment/open-rl-api-server \ |
36 | | - VLLM_MAX_MODEL_LEN=131072 OPEN_RL_TRAIN_TOKEN_BUDGET=81920 \ |
37 | | - VLLM_MAX_LORA_RANK=64 VLLM_GPU_MEMORY_UTILIZATION=0.90 \ |
38 | | - ENABLE_GRADIENT_CHECKPOINTING=1 FLA_TILELANG=1 |
39 | | -``` |
| 42 | +If Workloads for the run already exist, delete them after updating the API |
| 43 | +server so OpenRL recreates the worker pods with the new limits. |
40 | 44 |
|
41 | | -`FLA_TILELANG=1` is for Hopper (H100/H200); leave it unset elsewhere. Worker |
42 | | -pods read these when they are created, so change them before a run, or delete |
43 | | -the Workloads so the next run recreates the pods. |
| 45 | +The Job requests its trainer and samplers through `TINKER_TAGS` with |
| 46 | +`openrl.trainer_backend=automodel,openrl.trainer_gpus=4,openrl.trainer_cp=4,openrl.sampler_replicas=4`. |
| 47 | +Without context parallelism, the trainer runs out of GPU memory. One 80 GB H100 |
| 48 | +fits about 80k tokens of Qwen3.5-9B, and data parallelism does not help because |
| 49 | +each GPU still holds a full sequence. |
44 | 50 |
|
45 | | -## One GPU per role |
| 51 | +## 2. Build the driver image |
46 | 52 |
|
47 | | -The scheduler's default `binpack` strategy seats the sampler on the trainer's |
48 | | -claim when a node accepts both roles. Give each role its own node with labels; |
49 | | -unlabeled nodes count as both roles, so disable the rest for the run: |
| 53 | +[Dockerfile](../../examples/harvey_labs/Dockerfile) packages the recipe client, |
| 54 | +the LAB fork at a pinned revision, and LAB's grading environment. Build the |
| 55 | +image from the repository root: |
50 | 56 |
|
51 | 57 | ```bash |
52 | | -kubectl label node <trainer-node> openrl.io/sampler=false --overwrite |
53 | | -kubectl label node <sampler-node> openrl.io/trainer=false --overwrite |
54 | | -kubectl label node <every-other-gpu-node> openrl.io/enabled=false --overwrite |
| 58 | +gcloud builds submit --config cloudbuild.yaml \ |
| 59 | + --substitutions=_IMAGE=<registry>/harvey-labs-client,_DOCKERFILE=examples/harvey_labs/Dockerfile,_TAG=<tag> . |
55 | 60 | ``` |
56 | 61 |
|
57 | | -Qwen3.5-9B needs 80 GB GPUs. The estimator sizes its LoRA trainer at ~21 GiB, |
58 | | -so an enabled L4 node would take it and OOM on the first batch. |
| 62 | +## 3. Launch the Job |
59 | 63 |
|
60 | | -## Reaching the API server |
| 64 | +Update [cluster-job.yaml](../../examples/harvey_labs/cluster-job.yaml) with |
| 65 | +the driver image you built, `OPENAI_BASE_URL` for your judge, your PVC name, |
| 66 | +and `RUN`, the run name used for the log directory. The manifest writes out the |
| 67 | +recipe's default arguments explicitly so the Job spec alone reproduces the run. |
61 | 68 |
|
62 | | -The API server service is ClusterIP. From a workstation: |
| 69 | +Apply the Job manifest and stream the driver logs: |
63 | 70 |
|
64 | 71 | ```bash |
65 | | -nohup sh -c 'while true; do kubectl port-forward svc/open-rl-api-server-service 9003:8000 >/dev/null 2>&1; sleep 1; done' & |
| 72 | +kubectl apply -f examples/harvey_labs/cluster-job.yaml |
| 73 | +kubectl logs job/harvey-labs -f |
66 | 74 | ``` |
67 | 75 |
|
68 | | -If the driver runs on another machine (it needs Podman and the LAB checkout), |
69 | | -forward the port on: `ssh -N -R 9003:127.0.0.1:9003 <driver-host>`. |
70 | | - |
71 | | -`kubectl port-forward` keeps a dead session when the API server pod is replaced |
72 | | -(every `kubectl set env` or `set image` on the deployment does that). Kill and |
73 | | -restart the loop after an API server rollout, then check |
74 | | -`curl http://127.0.0.1:9003/api/v1/healthz` from the driver host; the tinker |
75 | | -client gives up after a few minutes of connection errors. |
| 76 | +The first `create_model` call starts the trainer, and the first sampling call |
| 77 | +starts the samplers. Add `openrl.recipe=rl` to `TINKER_TAGS` to start the |
| 78 | +samplers together with the trainer instead. Watch the Workloads and pods start: |
76 | 79 |
|
77 | | -## Driver host |
78 | | - |
79 | | -Follow the recipe [README](../../examples/harvey_labs/README.md): `uv sync` |
80 | | -in `examples/`, run `harvey_labs/setup_lab.sh`, and export the judge |
81 | | -credentials. The default judge is GLM (`judge_model=gpt-glm-5.2`) through an |
82 | | -OpenAI-compatible endpoint: `OPENAI_BASE_URL` and `OPENAI_API_KEY`. For Gemini, |
83 | | -pass `judge_model=gemini-3.5-flash` and export `GEMINI_API_KEY`. |
| 80 | +```bash |
| 81 | +kubectl -n openrl-system get workloads,pods -o wide |
| 82 | +``` |
84 | 83 |
|
85 | | -## Run |
| 84 | +While the Job runs, inspect progress from a pod that runs the driver image and |
| 85 | +mounts the shared volume. The driver image has the recipe installed as a module, |
| 86 | +not as the `harvey-results` command: |
86 | 87 |
|
87 | 88 | ```bash |
88 | | -cd examples |
89 | | -TINKER_API_KEY=tml-dummy-key uv run harvey-train \ |
90 | | - base_url=http://127.0.0.1:9003 \ |
91 | | - model_name=Qwen/Qwen3.5-9B renderer_name=qwen3_5 \ |
92 | | - max_trajectory_tokens=81920 max_tool_result_tokens=4096 max_tokens=16384 \ |
93 | | - batch_size=1 rollouts_per_example=4 max_steps=2 task=<area>/<task> \ |
94 | | - log_path=artifacts/harvey-labs/k8s-smoke |
| 89 | +/app/.venv/bin/python -m harvey_labs.results log_dir=/mnt/shared/harvey-labs/<RUN> |
95 | 90 | ``` |
96 | 91 |
|
97 | | -`max_trajectory_tokens` must not exceed the API server's `VLLM_MAX_MODEL_LEN`. |
98 | | -Drop `task=` for the seeded 300/50 split. Context and generation budgets |
99 | | -decide whether episodes reach the judge at all: |
| 92 | +## What to expect |
100 | 93 |
|
101 | | -| Setting | What happened | |
102 | | -| --- | --- | |
103 | | -| 32k or 64k context | Every episode ended by `stop/context_overflow` before writing a deliverable: reward floor, nothing graded. | |
104 | | -| `max_tokens=4096` per turn | The turn that writes the deliverable is long; hitting the cap ends the episode ungraded. Use 16k. | |
105 | | -| 96k and 128k context | All episodes completed and graded (pass fraction 0.54 to 0.74 untrained), then the trainer step OOMed on one 80 GB H100. Activation CPU offload does not help: the failing 11.7 GiB allocation is a per-layer transient of the eager GDN path, which the server image falls back to without causal-conv1d and the fla kernels. | |
106 | | -| 80k context | Works on one H100 as configured above: 3 of 4 episodes graded, mean pass fraction 0.52, reward 0.49, optimizer step done. | |
| 94 | +On the reference 8 × H100 setup, a full run progresses as follows: |
107 | 95 |
|
108 | | -Per-episode `stop/<reason>` metrics in the cookbook output say how episodes |
109 | | -ended; `lab/failed_before_grading=1` means the reward function never ran. The first `create_model` creates a |
110 | | -trainer Workload, the first sampling session a sampler Workload; watch them: |
| 96 | +- Once the node has cached the container image and the shared Hugging Face |
| 97 | + cache holds the model weights, the trainer starts within a few minutes. |
| 98 | + Samplers take longer on their first start while vLLM compiles Qwen3.5 |
| 99 | + kernels. |
| 100 | +- The step-0 evaluation of 50 held-out tasks runs first and takes the longest. |
| 101 | +- A training step of 48 episodes took 18 to 35 minutes on 8 H100 GPUs. Sampling |
| 102 | + dominates step time. Routing all turns of an episode to the sampler that |
| 103 | + already holds its prefix cut step time in half. On one 8 × B200 node with the |
| 104 | + same `TINKER_TAGS`, a step after the first took about 29 minutes. |
| 105 | +- The held-out pooled criterion pass rate rose from 0.54 at step 0 to 0.78 at |
| 106 | + step 35. |
111 | 107 |
|
112 | | -```bash |
113 | | -kubectl get workloads |
114 | | -kubectl get pods -l app=open-rl-worker -o wide |
115 | | -``` |
| 108 | +## When episodes score zero |
116 | 109 |
|
117 | | -## What to expect |
| 110 | +Per-episode `stop/<reason>` metrics record how episodes ended. |
| 111 | +`lab/no_output=1` means the episode wrote no deliverable, so there was nothing |
| 112 | +to grade. `lab/failed_before_grading=1` means the episode ended before grading |
| 113 | +started, for example on a parse error. |
118 | 114 |
|
119 | | -- Trainer up in 2-3 minutes (image already on the node, model in the shared HF cache). |
120 | | -- Sampler up in 8-10 minutes: vLLM's warmup of the Qwen3.5 GDN kernels takes |
121 | | - about 8 minutes and peaks near 170 GiB of host memory on the way. The |
122 | | - API server waits 300 s for sampler readiness on the first session; the client |
123 | | - retries, so the first step is slow but does not fail. |
124 | | -- An 80k step with 4 rollouts of one task took 11 minutes end to end: 8.5 |
125 | | - sampling and grading, 2.8 training. The recipe defaults to 128k; on one |
126 | | - H100 that needs the server image built with the `fastpath` extra |
127 | | - (causal-conv1d plus the fla kernels) so Qwen3.5's GDN layers stop running |
128 | | - the eager fallback, or a second trainer GPU. |
| 115 | +| Symptom | Likely cause | |
| 116 | +| --- | --- | |
| 117 | +| Every episode ends with `context_overflow` before writing deliverables | The context window is too small. A 32k or 64k context window is too small for LAB, and the default recipe uses the model's full window (262,144 tokens for Qwen3.5-9B). | |
| 118 | +| Many turns hit the per-turn cap (`max_tokens_reached`) | The per-turn token limit is too small for the turn that writes the deliverable. Keep `max_tokens` at 16k or higher. | |
| 119 | +| `lab/reward_error` is high | The judge endpoint is unreachable or failing. Check `grading.log` under `<lab_root>/results/<run-id>/` and verify firewall rules between the training cluster and the judge. | |
| 120 | +| The trainer is OOM-killed on the first step | The context window exceeds trainer GPU memory. Use the Automodel trainer with `openrl.trainer_cp`, or lower `max_trajectory_tokens` and the API server's limits together. | |
| 121 | +| Episodes fail before the first tool call | The sandbox claim failed. Check that the warm pool can hold a step's training episodes plus evaluation tasks, and verify that the driver pod can reach the sandbox pods. | |
129 | 122 |
|
130 | 123 | ## Cleanup |
131 | 124 |
|
132 | | -LoRA Workloads outlive the run (the runtime is shared per base model): |
| 125 | +Delete the driver Job and remove the run's trainer and sampler Workloads: |
133 | 126 |
|
134 | 127 | ```bash |
135 | | -kubectl delete workload lora-<base-model>-0-trainer lora-<base-model>-0-sampler |
| 128 | +kubectl delete job harvey-labs |
| 129 | +kubectl -n openrl-system get workloads # delete the run's trainer and sampler Workloads |
136 | 130 | ``` |
137 | 131 |
|
138 | | -Restore the node labels you changed (`openrl.io/enabled=true`, roles to `true`). |
| 132 | +The driver releases its sandbox claims when it exits. Any claims that it could |
| 133 | +not release expire after 15 minutes. |
0 commit comments