Skip to content

Commit 38aa335

Browse files
authored
harvey_labs: rewrite the recipe README and GKE guide (#433)
* harvey_labs: rewrite the recipe README and GKE guide around how the recipe runs today * harvey_labs: tighten the prose in the recipe README and GKE guide * harvey_labs: correct the docs where they overstated what the code does * harvey_labs: list every setting of the reference run in the README * harvey_labs: document how to choose and run the judge * harvey_labs: document the Substrate backend, fair grading, and the recipe's tests Fold the Agent Substrate section into the Sandbox backends table, describe the per-turn budget floor and judge-failure accounting from the fair-grading change, list every recipe unit test under Development, and ignore the LAB checkout and run logs that the README's commands create inside the repo. * harvey_labs: note the step time measured on an 8 x B200 node * harvey_labs: update README test invocation for simplified unit suites * harvey_labs: document the full-window default and what to change on smaller GPUs
1 parent 3a22b21 commit 38aa335

4 files changed

Lines changed: 414 additions & 166 deletions

File tree

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,9 @@ Thumbs.db
6060

6161
# Local benchmark run outputs
6262
runs/
63+
# Harvey LAB checkout (setup_lab.sh) and run logs (default log_path)
64+
examples/harvey_labs/harvey-labs/
65+
examples/artifacts/
6366
*.pyc
6467
docs/fft/2026-08-20-*.html
6568
docs/fft/2026-08-20-*.md

‎docs/setup/harvey-lab-on-gke.md‎

Lines changed: 91 additions & 96 deletions
Original file line numberDiff line numberDiff line change
@@ -1,138 +1,133 @@
1-
# Harvey LAB RL on the GKE scheduler stack
1+
# Harvey LAB RL on GKE
22

3-
How to run the Harvey LAB LoRA recipe against an OpenRL cluster deployed with
4-
the scheduler (`OPEN_RL_WORKER_MANAGER=scheduler`, see
5-
[gke-fft-timeslice.md](gke-fft-timeslice.md)), with one GPU for the trainer and
6-
one for the sampler. Verified on 2026-09-07 with Qwen3.5-9B on two H100 nodes.
3+
This guide runs the [Harvey LAB recipe](../../examples/harvey_labs/README.md)
4+
at full scale on GKE. The recipe's driver runs as a Kubernetes Job in the
5+
OpenRL training cluster, episode sandboxes run on Agent Sandbox in a separate
6+
cluster, and the judge runs behind an OpenAI-compatible endpoint that the
7+
driver can reach. The sandboxes have no network egress, so only the driver
8+
talks to the judge.
79

8-
## What the cluster needs
10+
```
11+
training cluster sandbox cluster
12+
openrl-system: API server, scheduler, workers Agent Sandbox warm pool
13+
harvey-labs Job (driver) ──claims, tool calls──────▶ LAB sandbox pods
14+
│
15+
└──grading──▶ judge (OpenAI-compatible, e.g. GLM-5.2 on vLLM)
16+
```
917

10-
The stack on `main` runs the recipe end to end only with the server fixes on
11-
the `harvey/k8s-stack` branch (all in `src/server` and `src/training`):
18+
## What you need
1219

13-
| Fix | Why the recipe needs it |
20+
| Component | What the reference run used |
1421
| --- | --- |
15-
| API server env passthrough to worker pods | Scheduler-placed samplers otherwise start at `max_model_len=8192`; LAB episodes need 64k or more. |
16-
| Per-session adapter snapshots | The trainer rewrote one adapter directory in place while the sampler read it over the shared volume. |
17-
| Adapter keys remapped to the hub layout | Qwen3.5 and Gemma 4 hubs wrap the text model; vLLM silently applies no adapter otherwise. |
18-
| `train_unembed` ignored by default | The tinker client asks for an `lm_head` adapter; vLLM rejects the whole adapter for these model families. |
19-
| Sampler host memory: 6 B/param, 3x limit | vLLM 0.25 peaks near 170 GiB of host memory for a minute while warming up Qwen3.5-9B. |
20-
| Chunked target-logprob projection | The full-logits path needs ~60 GiB at 64k tokens; one H100 cannot hold it beside the model. |
21-
| `VLLM_MAX_MODEL_LEN` capped at the model's limit | One API-server-wide value must not crash-loop smaller models. |
22-
| LoRA targets found through PEFT wrappers | The LoRA runtime is shared per base model; the second model to arrive otherwise fails `create_model`. |
22+
| OpenRL on GKE | [GKE setup](gke-setup.md) with an 8 × H100 80 GB GPU node pool. Four GPUs run the Automodel trainer with context parallelism across 4 GPUs, and four GPUs run vLLM samplers. |
23+
| Sandbox cluster | [LAB sandboxes on Agent Sandbox](../../examples/harvey_labs/agent-sandbox/README.md) in a separate cluster on the same VPC. |
24+
| Judge | GLM-5.2-FP8 on vLLM, served as `gpt-glm-5.2` and reachable from pods in the training cluster. |
25+
| Shared volume | A `ReadWriteMany` PVC in the Job's namespace for logs and checkpoints. `cluster-job.yaml` uses `open-rl-shared-pvc`. |
2326

24-
Build and roll them from that branch (Cloud Build must be enabled in the project):
27+
## 1. Set the context window on the API server
28+
29+
By default the recipe uses the model's full context window: 262,144 tokens for
30+
Qwen3.5-9B. Because the OpenRL API server passes its token limits to worker
31+
pods only when creating them, set these environment variables on the API
32+
server before starting the run:
2533

2634
```bash
27-
make cloud-build-api-server cloud-build-server GCP_PROJECT=<project>
28-
kubectl set image deployment/open-rl-api-server API server=gcr.io/<project>/open-rl-api-server:<tag>
29-
kubectl set env deployment/open-rl-api-server OPEN_RL_WORKER_IMAGE=gcr.io/<project>/open-rl-server:<tag>
35+
kubectl -n openrl-system set env deployment/open-rl-api-server \
36+
VLLM_MAX_MODEL_LEN=262144 OPEN_RL_TRAIN_TOKEN_BUDGET=262144
3037
```
3138

32-
Then set the run knobs on the API server. They ride along into every worker pod:
39+
We ran it on hardware that fits that window. On GPUs with less memory, pass a
40+
smaller `max_trajectory_tokens` in the Job and lower both limits to match.
3341

34-
```bash
35-
kubectl set env deployment/open-rl-api-server \
36-
VLLM_MAX_MODEL_LEN=131072 OPEN_RL_TRAIN_TOKEN_BUDGET=81920 \
37-
VLLM_MAX_LORA_RANK=64 VLLM_GPU_MEMORY_UTILIZATION=0.90 \
38-
ENABLE_GRADIENT_CHECKPOINTING=1 FLA_TILELANG=1
39-
```
42+
If Workloads for the run already exist, delete them after updating the API
43+
server so OpenRL recreates the worker pods with the new limits.
4044

41-
`FLA_TILELANG=1` is for Hopper (H100/H200); leave it unset elsewhere. Worker
42-
pods read these when they are created, so change them before a run, or delete
43-
the Workloads so the next run recreates the pods.
45+
The Job requests its trainer and samplers through `TINKER_TAGS` with
46+
`openrl.trainer_backend=automodel,openrl.trainer_gpus=4,openrl.trainer_cp=4,openrl.sampler_replicas=4`.
47+
Without context parallelism, the trainer runs out of GPU memory. One 80 GB H100
48+
fits about 80k tokens of Qwen3.5-9B, and data parallelism does not help because
49+
each GPU still holds a full sequence.
4450

45-
## One GPU per role
51+
## 2. Build the driver image
4652

47-
The scheduler's default `binpack` strategy seats the sampler on the trainer's
48-
claim when a node accepts both roles. Give each role its own node with labels;
49-
unlabeled nodes count as both roles, so disable the rest for the run:
53+
[Dockerfile](../../examples/harvey_labs/Dockerfile) packages the recipe client,
54+
the LAB fork at a pinned revision, and LAB's grading environment. Build the
55+
image from the repository root:
5056

5157
```bash
52-
kubectl label node <trainer-node> openrl.io/sampler=false --overwrite
53-
kubectl label node <sampler-node> openrl.io/trainer=false --overwrite
54-
kubectl label node <every-other-gpu-node> openrl.io/enabled=false --overwrite
58+
gcloud builds submit --config cloudbuild.yaml \
59+
--substitutions=_IMAGE=<registry>/harvey-labs-client,_DOCKERFILE=examples/harvey_labs/Dockerfile,_TAG=<tag> .
5560
```
5661

57-
Qwen3.5-9B needs 80 GB GPUs. The estimator sizes its LoRA trainer at ~21 GiB,
58-
so an enabled L4 node would take it and OOM on the first batch.
62+
## 3. Launch the Job
5963

60-
## Reaching the API server
64+
Update [cluster-job.yaml](../../examples/harvey_labs/cluster-job.yaml) with
65+
the driver image you built, `OPENAI_BASE_URL` for your judge, your PVC name,
66+
and `RUN`, the run name used for the log directory. The manifest writes out the
67+
recipe's default arguments explicitly so the Job spec alone reproduces the run.
6168

62-
The API server service is ClusterIP. From a workstation:
69+
Apply the Job manifest and stream the driver logs:
6370

6471
```bash
65-
nohup sh -c 'while true; do kubectl port-forward svc/open-rl-api-server-service 9003:8000 >/dev/null 2>&1; sleep 1; done' &
72+
kubectl apply -f examples/harvey_labs/cluster-job.yaml
73+
kubectl logs job/harvey-labs -f
6674
```
6775

68-
If the driver runs on another machine (it needs Podman and the LAB checkout),
69-
forward the port on: `ssh -N -R 9003:127.0.0.1:9003 <driver-host>`.
70-
71-
`kubectl port-forward` keeps a dead session when the API server pod is replaced
72-
(every `kubectl set env` or `set image` on the deployment does that). Kill and
73-
restart the loop after an API server rollout, then check
74-
`curl http://127.0.0.1:9003/api/v1/healthz` from the driver host; the tinker
75-
client gives up after a few minutes of connection errors.
76+
The first `create_model` call starts the trainer, and the first sampling call
77+
starts the samplers. Add `openrl.recipe=rl` to `TINKER_TAGS` to start the
78+
samplers together with the trainer instead. Watch the Workloads and pods start:
7679

77-
## Driver host
78-
79-
Follow the recipe [README](../../examples/harvey_labs/README.md): `uv sync`
80-
in `examples/`, run `harvey_labs/setup_lab.sh`, and export the judge
81-
credentials. The default judge is GLM (`judge_model=gpt-glm-5.2`) through an
82-
OpenAI-compatible endpoint: `OPENAI_BASE_URL` and `OPENAI_API_KEY`. For Gemini,
83-
pass `judge_model=gemini-3.5-flash` and export `GEMINI_API_KEY`.
80+
```bash
81+
kubectl -n openrl-system get workloads,pods -o wide
82+
```
8483

85-
## Run
84+
While the Job runs, inspect progress from a pod that runs the driver image and
85+
mounts the shared volume. The driver image has the recipe installed as a module,
86+
not as the `harvey-results` command:
8687

8788
```bash
88-
cd examples
89-
TINKER_API_KEY=tml-dummy-key uv run harvey-train \
90-
base_url=http://127.0.0.1:9003 \
91-
model_name=Qwen/Qwen3.5-9B renderer_name=qwen3_5 \
92-
max_trajectory_tokens=81920 max_tool_result_tokens=4096 max_tokens=16384 \
93-
batch_size=1 rollouts_per_example=4 max_steps=2 task=<area>/<task> \
94-
log_path=artifacts/harvey-labs/k8s-smoke
89+
/app/.venv/bin/python -m harvey_labs.results log_dir=/mnt/shared/harvey-labs/<RUN>
9590
```
9691

97-
`max_trajectory_tokens` must not exceed the API server's `VLLM_MAX_MODEL_LEN`.
98-
Drop `task=` for the seeded 300/50 split. Context and generation budgets
99-
decide whether episodes reach the judge at all:
92+
## What to expect
10093

101-
| Setting | What happened |
102-
| --- | --- |
103-
| 32k or 64k context | Every episode ended by `stop/context_overflow` before writing a deliverable: reward floor, nothing graded. |
104-
| `max_tokens=4096` per turn | The turn that writes the deliverable is long; hitting the cap ends the episode ungraded. Use 16k. |
105-
| 96k and 128k context | All episodes completed and graded (pass fraction 0.54 to 0.74 untrained), then the trainer step OOMed on one 80 GB H100. Activation CPU offload does not help: the failing 11.7 GiB allocation is a per-layer transient of the eager GDN path, which the server image falls back to without causal-conv1d and the fla kernels. |
106-
| 80k context | Works on one H100 as configured above: 3 of 4 episodes graded, mean pass fraction 0.52, reward 0.49, optimizer step done. |
94+
On the reference 8 × H100 setup, a full run progresses as follows:
10795

108-
Per-episode `stop/<reason>` metrics in the cookbook output say how episodes
109-
ended; `lab/failed_before_grading=1` means the reward function never ran. The first `create_model` creates a
110-
trainer Workload, the first sampling session a sampler Workload; watch them:
96+
- Once the node has cached the container image and the shared Hugging Face
97+
cache holds the model weights, the trainer starts within a few minutes.
98+
Samplers take longer on their first start while vLLM compiles Qwen3.5
99+
kernels.
100+
- The step-0 evaluation of 50 held-out tasks runs first and takes the longest.
101+
- A training step of 48 episodes took 18 to 35 minutes on 8 H100 GPUs. Sampling
102+
dominates step time. Routing all turns of an episode to the sampler that
103+
already holds its prefix cut step time in half. On one 8 × B200 node with the
104+
same `TINKER_TAGS`, a step after the first took about 29 minutes.
105+
- The held-out pooled criterion pass rate rose from 0.54 at step 0 to 0.78 at
106+
step 35.
111107

112-
```bash
113-
kubectl get workloads
114-
kubectl get pods -l app=open-rl-worker -o wide
115-
```
108+
## When episodes score zero
116109

117-
## What to expect
110+
Per-episode `stop/<reason>` metrics record how episodes ended.
111+
`lab/no_output=1` means the episode wrote no deliverable, so there was nothing
112+
to grade. `lab/failed_before_grading=1` means the episode ended before grading
113+
started, for example on a parse error.
118114

119-
- Trainer up in 2-3 minutes (image already on the node, model in the shared HF cache).
120-
- Sampler up in 8-10 minutes: vLLM's warmup of the Qwen3.5 GDN kernels takes
121-
about 8 minutes and peaks near 170 GiB of host memory on the way. The
122-
API server waits 300 s for sampler readiness on the first session; the client
123-
retries, so the first step is slow but does not fail.
124-
- An 80k step with 4 rollouts of one task took 11 minutes end to end: 8.5
125-
sampling and grading, 2.8 training. The recipe defaults to 128k; on one
126-
H100 that needs the server image built with the `fastpath` extra
127-
(causal-conv1d plus the fla kernels) so Qwen3.5's GDN layers stop running
128-
the eager fallback, or a second trainer GPU.
115+
| Symptom | Likely cause |
116+
| --- | --- |
117+
| Every episode ends with `context_overflow` before writing deliverables | The context window is too small. A 32k or 64k context window is too small for LAB, and the default recipe uses the model's full window (262,144 tokens for Qwen3.5-9B). |
118+
| Many turns hit the per-turn cap (`max_tokens_reached`) | The per-turn token limit is too small for the turn that writes the deliverable. Keep `max_tokens` at 16k or higher. |
119+
| `lab/reward_error` is high | The judge endpoint is unreachable or failing. Check `grading.log` under `<lab_root>/results/<run-id>/` and verify firewall rules between the training cluster and the judge. |
120+
| The trainer is OOM-killed on the first step | The context window exceeds trainer GPU memory. Use the Automodel trainer with `openrl.trainer_cp`, or lower `max_trajectory_tokens` and the API server's limits together. |
121+
| Episodes fail before the first tool call | The sandbox claim failed. Check that the warm pool can hold a step's training episodes plus evaluation tasks, and verify that the driver pod can reach the sandbox pods. |
129122

130123
## Cleanup
131124

132-
LoRA Workloads outlive the run (the runtime is shared per base model):
125+
Delete the driver Job and remove the run's trainer and sampler Workloads:
133126

134127
```bash
135-
kubectl delete workload lora-<base-model>-0-trainer lora-<base-model>-0-sampler
128+
kubectl delete job harvey-labs
129+
kubectl -n openrl-system get workloads # delete the run's trainer and sampler Workloads
136130
```
137131

138-
Restore the node labels you changed (`openrl.io/enabled=true`, roles to `true`).
132+
The driver releases its sandbox claims when it exits. Any claims that it could
133+
not release expire after 15 minutes.

0 commit comments

Comments
 (0)