Skip to content

Latest commit

 

History

History
285 lines (227 loc) · 11.9 KB

railway-deploy.md

File metadata and controls

285 lines (227 loc) · 11.9 KB

Railway Parallel Benchmark Runner

The evaluation/ harness can run as a fleet of Railway-hosted workers, each pulling tasks from a shared Postgres queue. This page covers the provisioning flow end to end.

Architecture

+----------------------------+      +-------------------+
| developer machine          |      | Railway project   |
|                            |      |   "kai-bench"     |
|  evaluation.cli enqueue    +----->+                   |
|  (writes bench_runs +      |      | Postgres (plugin) |
|   bench_tasks rows)        |      |                   |
|                            |      | bench-worker x N  |
|                            |<-----+   replicas        |
|  evaluation.cli watch <id> |      |   (Dockerfile)    |
+----------------------------+      +-------------------+

The worker (python -m evaluation.worker) is the long-running process the Docker image starts. It:

  1. Connects to Postgres via DATABASE_URL.
  2. Resolves the adapter named by BENCHMARK_ADAPTER.
  3. Loops over claim_next() -> _run_task() -> complete().
  4. Exits cleanly on SIGTERM so Railway can recycle the container.

Operational golden rules (read these first)

These are the lessons that cost the most time. Internalise them before you run a fleet — most "lost run" and "it didn't take" surprises trace back to one of these.

  1. A redeploy wipes replica disk. railway up — and any replica recycle — rebuilds the containers, so every rollout under output/ on the old container is gone. Rollouts live only on the replica's ephemeral disk, never in Postgres. Always pull rollouts before you redeploy, scale down, or change a variable that needs a redeploy. See Pulling rollouts.
  2. Env changes do not reach a running worker. The worker reads its environment once, at process start. railway variables --set … updates stored config but not the live process — you must railway up to apply it (and, per rule 1, pull first).
  3. Replica count lives in railway.json, not in a scale command. Edit deploy.numReplicas and railway up. Ad-hoc scale commands / the dashboard slider did not reliably take in our runs. railway down stops the fleet.
  4. Enqueue into the same Postgres the workers poll. If the project has more than one Postgres, enqueueing into the wrong one leaves tasks pending forever while the workers sit idle. The DATABASE_URL you enqueue with must equal the workers' DATABASE_URL.
  5. Pull per replica, keyed by worker_id. Each task ran on exactly one replica. A plain railway ssh only reaches the first instance and silently misses the rest — target the exact replica that ran the task.

One-time setup

1. Create the project

# from the repo root
railway login
railway init             # interactive: pick your workspace, name it (e.g. kai-bench)

2. Add a Postgres plugin

In the Railway dashboard for kai-bench:

  • + New -> Database -> Postgres.
  • Wait for provisioning.
  • Copy the resulting DATABASE_URL (Railway exposes it as a reference variable named ${{Postgres.DATABASE_URL}}).

3. Create the worker service

railway service create bench-worker

Then in the dashboard for bench-worker -> Settings -> Build:

  • Builder: Dockerfile (Railway will pick up Dockerfile automatically).
  • Watch path: / (default).

4. Configure required env vars

Variable Purpose Source
DATABASE_URL Shared task queue. ${{Postgres.DATABASE_URL}}
OPENROUTER_API_KEY LLM credentials. You paste it.
KAI_BACKEND Backend selector. openrouter (already in Dockerfile)
BENCHMARK_ADAPTER Which adapter to run. cybergym / bountybench / etc.
BENCHMARK_CONFIG Adapter config JSON. e.g. {"server_url":"http://...","data_dir":"/mnt/cg"}
BENCHMARK_RUN_ID (optional) Limit a replica to one run. Generated by enqueue.
BENCHMARK_PIPELINE_ARGS (optional) Extra flags as JSON list. e.g. ["--skip-fixer"]
BENCHMARK_TASK_TIMEOUT_S Wall-clock per task. Default 3600
BENCHMARK_POLL_SECONDS Sleep between claim attempts. Default 5
BENCHMARK_IDLE_EXIT_AFTER Auto-exit after N idle seconds. Default: never

Anything that looks like a secret must be set via the dashboard or railway variables set ... -- never baked into the image.

Beyond the required minimum above, see .env.example at the repo root for the broader set of optional knobs the worker + agent read — the opt-in cybergym strict-verify tool (KAI_CYBERGYM_HARNESS_URL), the per-LLM-call timeout (KAI_LLM_REQUEST_TIMEOUT_S), per-agent model and iteration overrides (KAI_<AGENT>_MODEL, KAI_<AGENT>_ITERS), and worker tuning (BENCHMARK_POLL_SECONDS, BENCHMARK_IDLE_EXIT_AFTER, BENCHMARK_ENV_OVERRIDES, …). The authoritative consumer is evaluation/worker.py; .env.example documents the common ones with defaults and short comments.

For cybergym strict scoring, the verifier sub-agent submits candidate PoC bytes to a CyberGym server you run yourself (e.g. on your own machine, exposed via an ngrok tunnel). Point the cybergym worker at it, and rotate the URL whenever the tunnel reconnects (free-tier ngrok subdomains change):

railway variables --service <your-cybergym-worker> \
  --set KAI_CYBERGYM_HARNESS_URL=https://<new-subdomain>.ngrok-free.app
railway up --service <your-cybergym-worker> --detach

5. Deploy

railway up

railway up uploads the working tree (honouring .dockerignore), builds the image from Dockerfile, and rolls out the result. The .env, output/, tests/, and other heavy / sensitive paths are excluded by .dockerignore, so the image is reproducible across machines.

Enqueueing work

From your laptop, point DATABASE_URL at the Railway Postgres reference (use railway variables --service Postgres to copy it):

export DATABASE_URL='postgres://...'

uv run python -m evaluation.cli enqueue \
    --adapter cybergym \
    --adapter-config @cybergym.json \
    --limit 10

The CLI prints the generated run_id. Then watch progress:

uv run python -m evaluation.cli watch <run_id>
uv run python -m evaluation.cli status <run_id>

Both subcommands fall back to the local output/bench/... summary when DATABASE_URL is unset, so single-machine usage is unchanged.

BENCHMARK_RUN_ID scopes a worker to one run; leave it unset and the worker claims tasks from any run. The worker rebuilds its adapter per-task from the run's stored config, so a single fleet can serve several runs (even at different difficulties) concurrently — pin a fleet to one run only when you need the isolation.

Pulling rollouts

Rollouts live on each replica's ephemeral disk under output/bench/<benchmark>/run_<run_id>/<task_id>/. They are never written to Postgres and they do not survive a redeploy — so pull them to your machine the moment a run goes all-terminal, and before any railway up (golden rules 1 & 5).

1. Find which replica ran each task. The worker stamps its replica id (RAILWAY_REPLICA_ID) onto every row as worker_id:

select task_id, status, worker_id from bench_tasks where run_id = '<run_id>';

2. Pull from that exact replica. A target-less railway ssh only lands on the first instance and silently misses the rest; pin the replica with --deployment-instance <worker_id>:

railway ssh --service <worker> --deployment-instance <worker_id> -- \
  "tar czf - --exclude='*repo-vul.tar.gz' \
     -C /app/output/bench/<benchmark>/run_<run_id> <task_id> | base64 -w0" \
  > <task_id>.b64

python3 - "$task_id" <<'PY'   # decode + extract locally
import base64, io, sys, tarfile
raw = base64.b64decode(open(f"{sys.argv[1]}.b64").read())
tarfile.open(fileobj=io.BytesIO(raw), mode="r:gz").extractall("rollouts/")
PY

Why the dance:

  • tar | base64, not raw tar — a raw binary stream gets corrupted over the SSH websocket; base64 rides through intact.
  • --exclude the large task inputs (e.g. cybergym's repo-vul.tar.gz) — they are tens of MB of input, not rollout, and big enough to overrun the channel and make the pull look "flaky".

Then build a browsable gallery over the pulled dir with uv run python -m evaluation.cli index <dir>/ (see evaluation/README.md).

Replicas, redeploys & teardown

Set the replica count in railway.json, then apply it with railway up:

{ "deploy": { "numReplicas": 3, "startCommand": "python -m evaluation.worker" } }

Each replica claims one task at a time; replicas never race on a row because claim_next uses Postgres' FOR UPDATE SKIP LOCKED.

Every railway up is a fresh rollout. It rebuilds the image and replaces the running containers, wiping their output/ disk — and it is the only way to push an env-var change into running workers. Pull rollouts first (golden rule 1).

Stop the fleet once a run is done and pulled:

railway down --service <worker>   # removes the running deployment (→ 0 workers)

Cost guardrails

  • Cap concurrent tasks per worker. The worker loop always processes one task at a time. If a benchmark adapter needs more, prefer scaling replicas over concurrency inside a single container.
  • Set per-task wall-clock caps. Default BENCHMARK_TASK_TIMEOUT_S = 3600. Lower it during development to fail fast.
  • Set Railway resource limits. From the dashboard -> Settings -> Resources, cap memory (2 GB is enough for one CyberGym task) and CPU (1 vCPU).
  • Auto-exit when idle. Set BENCHMARK_IDLE_EXIT_AFTER=300 to have workers exit after 5 minutes of empty queue and stop paying for idle containers. Caveat: an exited worker frees its container, so its ephemeral disk — and any rollouts you have not pulled — can be reclaimed. For long unattended runs, leave idle-exit unset so workers stay up holding their disk until you have pulled (golden rule 1).
  • Use BENCHMARK_RUN_ID to scope a replica to a single run so a runaway batch can't pull tasks from an unrelated run.

Troubleshooting

  • Worker can't reach Postgres. Check that DATABASE_URL is set via the ${{Postgres.DATABASE_URL}} reference, not pasted verbatim -- pasting tends to omit sslmode=require.
  • All tasks marked failed with worker_exception. Check BENCHMARK_CONFIG parses as JSON, and the adapter's required paths (e.g. CyberGym's data_dir) exist inside the image. Persistent state needed at runtime must come from a Railway volume or be baked in at build time.
  • HEALTHCHECK keeps failing. Usually means the build copied the wrong source tree (missing evaluation/). Verify .dockerignore isn't excluding it.
  • Tasks stay pending while workers sit idle. The enqueue and the workers point at different Postgres instances (golden rule 4), or a worker's BENCHMARK_RUN_ID pins it to a run with no pending tasks. Confirm both DATABASE_URLs match and the run id lines up.
  • A variable change had no effect. The running worker read its env at startup; railway variables --set alone never reaches it. railway up to redeploy — after pulling rollouts (golden rules 1 & 2).
  • Rollouts are missing after a run. Either a railway up / recycle wiped them before they were pulled (gone — pull before redeploying), or the pull only hit the first replica. Re-pull per replica by worker_id (golden rule 5).
  • A pulled archive is tiny or won't extract. It was streamed as raw binary tar and the SSH websocket corrupted it. Use tar | base64 and decode locally, and --exclude large task-input blobs (see Pulling rollouts).

Where you provide secrets

Only one secret is mandatory: OPENROUTER_API_KEY. Set it in the Railway dashboard for bench-worker (Service -> Variables). Optional secrets such as JINA_API_KEY follow the same path.