The evaluation/ harness can run as a fleet of Railway-hosted workers,
each pulling tasks from a shared Postgres queue. This page covers the
provisioning flow end to end.
+----------------------------+ +-------------------+
| developer machine | | Railway project |
| | | "kai-bench" |
| evaluation.cli enqueue +----->+ |
| (writes bench_runs + | | Postgres (plugin) |
| bench_tasks rows) | | |
| | | bench-worker x N |
| |<-----+ replicas |
| evaluation.cli watch <id> | | (Dockerfile) |
+----------------------------+ +-------------------+
The worker (python -m evaluation.worker) is the long-running process
the Docker image starts. It:
- Connects to Postgres via
DATABASE_URL. - Resolves the adapter named by
BENCHMARK_ADAPTER. - Loops over
claim_next() -> _run_task() -> complete(). - Exits cleanly on
SIGTERMso Railway can recycle the container.
These are the lessons that cost the most time. Internalise them before you run a fleet — most "lost run" and "it didn't take" surprises trace back to one of these.
- A redeploy wipes replica disk.
railway up— and any replica recycle — rebuilds the containers, so every rollout underoutput/on the old container is gone. Rollouts live only on the replica's ephemeral disk, never in Postgres. Always pull rollouts before you redeploy, scale down, or change a variable that needs a redeploy. See Pulling rollouts. - Env changes do not reach a running worker. The worker reads its
environment once, at process start.
railway variables --set …updates stored config but not the live process — you mustrailway upto apply it (and, per rule 1, pull first). - Replica count lives in
railway.json, not in a scale command. Editdeploy.numReplicasandrailway up. Ad-hoc scale commands / the dashboard slider did not reliably take in our runs.railway downstops the fleet. - Enqueue into the same Postgres the workers poll. If the project has
more than one Postgres, enqueueing into the wrong one leaves tasks
pendingforever while the workers sit idle. TheDATABASE_URLyou enqueue with must equal the workers'DATABASE_URL. - Pull per replica, keyed by
worker_id. Each task ran on exactly one replica. A plainrailway sshonly reaches the first instance and silently misses the rest — target the exact replica that ran the task.
# from the repo root
railway login
railway init # interactive: pick your workspace, name it (e.g. kai-bench)In the Railway dashboard for kai-bench:
+ New->Database->Postgres.- Wait for provisioning.
- Copy the resulting
DATABASE_URL(Railway exposes it as a reference variable named${{Postgres.DATABASE_URL}}).
railway service create bench-workerThen in the dashboard for bench-worker -> Settings -> Build:
- Builder: Dockerfile (Railway will pick up
Dockerfileautomatically). - Watch path:
/(default).
| Variable | Purpose | Source |
|---|---|---|
DATABASE_URL |
Shared task queue. | ${{Postgres.DATABASE_URL}} |
OPENROUTER_API_KEY |
LLM credentials. | You paste it. |
KAI_BACKEND |
Backend selector. | openrouter (already in Dockerfile) |
BENCHMARK_ADAPTER |
Which adapter to run. | cybergym / bountybench / etc. |
BENCHMARK_CONFIG |
Adapter config JSON. | e.g. {"server_url":"http://...","data_dir":"/mnt/cg"} |
BENCHMARK_RUN_ID (optional) |
Limit a replica to one run. | Generated by enqueue. |
BENCHMARK_PIPELINE_ARGS (optional) |
Extra flags as JSON list. | e.g. ["--skip-fixer"] |
BENCHMARK_TASK_TIMEOUT_S |
Wall-clock per task. | Default 3600 |
BENCHMARK_POLL_SECONDS |
Sleep between claim attempts. | Default 5 |
BENCHMARK_IDLE_EXIT_AFTER |
Auto-exit after N idle seconds. | Default: never |
Anything that looks like a secret must be set via the dashboard or
railway variables set ...-- never baked into the image.
Beyond the required minimum above, see .env.example at
the repo root for the broader set of optional knobs the worker + agent read —
the opt-in cybergym strict-verify tool (KAI_CYBERGYM_HARNESS_URL), the
per-LLM-call timeout (KAI_LLM_REQUEST_TIMEOUT_S), per-agent model and
iteration overrides (KAI_<AGENT>_MODEL, KAI_<AGENT>_ITERS), and worker
tuning (BENCHMARK_POLL_SECONDS, BENCHMARK_IDLE_EXIT_AFTER,
BENCHMARK_ENV_OVERRIDES, …). The authoritative consumer is
evaluation/worker.py; .env.example documents the common ones with defaults
and short comments.
For cybergym strict scoring, the verifier sub-agent submits candidate PoC bytes to a CyberGym server you run yourself (e.g. on your own machine, exposed via an ngrok tunnel). Point the cybergym worker at it, and rotate the URL whenever the tunnel reconnects (free-tier ngrok subdomains change):
railway variables --service <your-cybergym-worker> \
--set KAI_CYBERGYM_HARNESS_URL=https://<new-subdomain>.ngrok-free.app
railway up --service <your-cybergym-worker> --detachrailway uprailway up uploads the working tree (honouring .dockerignore), builds
the image from Dockerfile, and rolls out the result. The .env,
output/, tests/, and other heavy / sensitive paths are excluded by
.dockerignore, so the image is reproducible across machines.
From your laptop, point DATABASE_URL at the Railway Postgres reference
(use railway variables --service Postgres to copy it):
export DATABASE_URL='postgres://...'
uv run python -m evaluation.cli enqueue \
--adapter cybergym \
--adapter-config @cybergym.json \
--limit 10The CLI prints the generated run_id. Then watch progress:
uv run python -m evaluation.cli watch <run_id>
uv run python -m evaluation.cli status <run_id>Both subcommands fall back to the local output/bench/... summary when
DATABASE_URL is unset, so single-machine usage is unchanged.
BENCHMARK_RUN_ID scopes a worker to one run; leave it unset and the
worker claims tasks from any run. The worker rebuilds its adapter
per-task from the run's stored config, so a single fleet can serve
several runs (even at different difficulties) concurrently — pin a fleet to
one run only when you need the isolation.
Rollouts live on each replica's ephemeral disk under
output/bench/<benchmark>/run_<run_id>/<task_id>/. They are never written
to Postgres and they do not survive a redeploy — so pull them to your
machine the moment a run goes all-terminal, and before any railway up
(golden rules 1 & 5).
1. Find which replica ran each task. The worker stamps its replica id
(RAILWAY_REPLICA_ID) onto every row as worker_id:
select task_id, status, worker_id from bench_tasks where run_id = '<run_id>';2. Pull from that exact replica. A target-less railway ssh only lands
on the first instance and silently misses the rest; pin the replica with
--deployment-instance <worker_id>:
railway ssh --service <worker> --deployment-instance <worker_id> -- \
"tar czf - --exclude='*repo-vul.tar.gz' \
-C /app/output/bench/<benchmark>/run_<run_id> <task_id> | base64 -w0" \
> <task_id>.b64
python3 - "$task_id" <<'PY' # decode + extract locally
import base64, io, sys, tarfile
raw = base64.b64decode(open(f"{sys.argv[1]}.b64").read())
tarfile.open(fileobj=io.BytesIO(raw), mode="r:gz").extractall("rollouts/")
PYWhy the dance:
tar | base64, not rawtar— a raw binary stream gets corrupted over the SSH websocket; base64 rides through intact.--excludethe large task inputs (e.g. cybergym'srepo-vul.tar.gz) — they are tens of MB of input, not rollout, and big enough to overrun the channel and make the pull look "flaky".
Then build a browsable gallery over the pulled dir with uv run python -m evaluation.cli index <dir>/ (see evaluation/README.md).
Set the replica count in railway.json, then apply it with railway up:
{ "deploy": { "numReplicas": 3, "startCommand": "python -m evaluation.worker" } }Each replica claims one task at a time; replicas never race on a row
because claim_next uses Postgres' FOR UPDATE SKIP LOCKED.
Every
railway upis a fresh rollout. It rebuilds the image and replaces the running containers, wiping theiroutput/disk — and it is the only way to push an env-var change into running workers. Pull rollouts first (golden rule 1).
Stop the fleet once a run is done and pulled:
railway down --service <worker> # removes the running deployment (→ 0 workers)- Cap concurrent tasks per worker. The worker loop always processes one task at a time. If a benchmark adapter needs more, prefer scaling replicas over concurrency inside a single container.
- Set per-task wall-clock caps. Default
BENCHMARK_TASK_TIMEOUT_S = 3600. Lower it during development to fail fast. - Set Railway resource limits. From the dashboard -> Settings -> Resources, cap memory (2 GB is enough for one CyberGym task) and CPU (1 vCPU).
- Auto-exit when idle. Set
BENCHMARK_IDLE_EXIT_AFTER=300to have workers exit after 5 minutes of empty queue and stop paying for idle containers. Caveat: an exited worker frees its container, so its ephemeral disk — and any rollouts you have not pulled — can be reclaimed. For long unattended runs, leave idle-exit unset so workers stay up holding their disk until you have pulled (golden rule 1). - Use
BENCHMARK_RUN_IDto scope a replica to a single run so a runaway batch can't pull tasks from an unrelated run.
- Worker can't reach Postgres. Check that
DATABASE_URLis set via the${{Postgres.DATABASE_URL}}reference, not pasted verbatim -- pasting tends to omitsslmode=require. - All tasks marked
failedwithworker_exception. CheckBENCHMARK_CONFIGparses as JSON, and the adapter's required paths (e.g. CyberGym'sdata_dir) exist inside the image. Persistent state needed at runtime must come from a Railway volume or be baked in at build time. HEALTHCHECKkeeps failing. Usually means the build copied the wrong source tree (missingevaluation/). Verify.dockerignoreisn't excluding it.- Tasks stay
pendingwhile workers sit idle. The enqueue and the workers point at different Postgres instances (golden rule 4), or a worker'sBENCHMARK_RUN_IDpins it to a run with no pending tasks. Confirm bothDATABASE_URLs match and the run id lines up. - A variable change had no effect. The running worker read its env at
startup;
railway variables --setalone never reaches it.railway upto redeploy — after pulling rollouts (golden rules 1 & 2). - Rollouts are missing after a run. Either a
railway up/ recycle wiped them before they were pulled (gone — pull before redeploying), or the pull only hit the first replica. Re-pull per replica byworker_id(golden rule 5). - A pulled archive is tiny or won't extract. It was streamed as raw
binary
tarand the SSH websocket corrupted it. Usetar | base64and decode locally, and--excludelarge task-input blobs (see Pulling rollouts).
Only one secret is mandatory: OPENROUTER_API_KEY. Set it in the
Railway dashboard for bench-worker (Service -> Variables). Optional
secrets such as JINA_API_KEY follow the same path.