GEAK kernel-build failures block E2E: CK/.cu preprocess, sglang harness-gen, rag-mcp env crash, timeouts

[iraj465]

GEAK-internal kernel-build failures blocking E2E wins (CK/.cu + sglang harness + env + budget)

Observed across the last-30h GEAKv3+HL fleet. When HL dispatches a hot kernel to GEAK,
a large fraction of attempts never produce a usable patch — failing inside GEAK
before any micro/correctness result. These are GEAK-side (harness-gen / preprocess /
runtime env / budget), distinct from HL's promotion logic (fixed in Hyperloom #759).

Evidence (fleet sessions)

• zai-org-GLM-4.7-FP8 (sglang): 8/8 GEAK kernels failed → all REVERT, E2E 0%.
  • int8_kernel.py: [harness_gen] no @perftest/@benchmark decorated functions found
    → falls back to timeout 600 python bench_int8_quant.py; then
    rag-mcp package not found, installing automatically... → agent exits (log ends).
• openai-gpt-oss-120b (vllm): 7 GEAK kernels, all on .cu/.cuh:
  • custom_all_reduce.cuh, quant_kernels.cu → error_class=preprocess_failed
    ("preprocess reported 1 error(s)"), produced_artifact=false.
  • large kernels → error_class=agent_error / exit code 124 (timeout) at ~130 min.

Failure modes → proposed fixes (v4 parity)

1. .cu/CK preprocess_failed — GEAK preprocess can't build a harness for CK .cu
   kernels. → Emit a buildable .cu harness (v4's .cu harness path).
2. harness-gen "no @perftest/@benchmark decorators" (sglang) — generator can't find
   a microbench entrypoint and degrades to raw-script execution. → Synthesize a harness
   from the kernel signature / live shape capture instead of requiring decorators.
3. rag-mcp auto-install crash — agent dies mid-run installing rag-mcp. → Pre-provision
   rag-mcp in the image, or make the agent resilient to install failure (skip RAG, continue).
4. exit 124 timeouts — budget exhausted on large kernels (~130 min observed). → Per-kernel
   adaptive budget / earlier checkpointing so partial-but-correct results are emitted.

Impact if fixed

On these workloads GEAK currently contributes 0% of E2E despite high-GPU% targets
(GLM all-fail; gpt-oss .cu all-fail). Fixing harness/preprocess unblocks the kernels that
HL #759 would then promote to E2E.

[iraj465]

Corrected root-cause breakdown (after deeper log analysis)

The four modes are not a single ".cu harness" bug. Re-reading the fleet logs:

1. GLM-4.7-FP8 (sglang), 8/8 ".cu failures" — actually the rag-mcp crash.
   Every GLM kernel log ends right after resolved_source=..., i.e. the agent died at
   rag-mcp package not found, installing automatically... before .cu preprocess ever ran
   (GLM's config had rag: true). → Fixed by fix(rag): make RAG provisioning best-effort, don't abort the run (#316) #317 (RAG best-effort). Post-fix these kernels
   will actually reach preprocess; whether quant_kernels.cu/attention.cu then build is unknown
   and needs a live re-run.

2. gpt-oss custom_all_reduce.cuh — multi-GPU collective, not a harness bug.
   GEAK did generate a harness (harness_fused_allreduce_rmsnorm.py) but it failed the preflight
   COMMANDMENT contract: an all-reduce cannot be benchmarked on a 1-GPU GEAK slot (the trace
   already flags "comm-bound collective: needs KERNEL_AGENT_NUM_GPUS>=world_size"). → infra
   (multi-GPU GEAK slots) or HL-side dispatch skip for comm-bound collectives. Not a template fix.

3. gemm_a8w8_blockscale.cu — harness works; failure was timeout.
   Confirmed locally: preprocess succeeded, harness generated + ran --correctness/--profile,
   then hit the wall-clock budget (3900s) with micro 0.96. → budget/checkpointing, not harness.

Revised actionable items

• (1) RAG best-effort — fix(rag): make RAG provisioning best-effort, don't abort the run (#316) #317.
• (2) Skip dispatching comm-bound collectives to single-GPU GEAK slots (HL-side), or provision
  multi-GPU GEAK slots for world_size>1 kernels.
• (3) Per-kernel adaptive budget / checkpoint partial-but-correct results before timeout.
• harness-gen "no @perftest/@benchmark decorators" fallback (sglang .py): synthesize a
  harness from the kernel signature instead of degrading to raw-bench.

Net: the .cu harness generation is largely functional for buildable single-GPU kernels;
the blockers are env (rag), infra (multi-GPU collectives), and budget (timeouts) — addressed
incrementally rather than by a harness rewrite.

[iraj465]

Most fundamental finding: GEAK "wins" have NO correctness/E2E evidence and use self-reported speedups

Audited every GEAK "win" (micro>1, NEEDS_REVIEW, status ok) across the last-30h fleet
(vLLM + sglang), reading each kernel's results/<kid>.json verification:

property	result (n=12)
correctness_source	missing on 12/12 (never established — not "failed", missing)
correctness_passed	False on 12/12
micro_speedup_source	report_scan on 12/12 (self-reported, not a controlled paired measurement)
optimized artifact	"missing or invalid" on DeepSeek k002, Qwen3-Next k002, gpt-oss k007
both correctness-verified AND controlled-measured	0 / 12

Examples: DeepSeek gemm_a8w8_blockscale.py 3.68x, MiniMax gemm_a8w8_blockscale.cu 2.91x —
all correctness_source=missing, micro_speedup_source=report_scan, compile_passed=True.
NEEDS_REVIEW reasons: "correctness evidence missing or failed; E2E evidence missing; accuracy
evidence missing".

Implication

The speedups are not trustworthy kernel improvements — they're compiled-and-self-reported
numbers with correctness never verified (and the controlled vs wall/dispatch distinction
unresolved, since report_scan parses GEAK's own log). HL's NEEDS_REVIEW sink is therefore
correct to withhold them; promoting/integrating them would risk shipping an unverified
(possibly incorrect) kernel into the model.

Prerequisite fix (GEAK-side, upstream of #316's harness items)

Per accepted win, GEAK must emit: (a) a valid captured optimized artifact, (b) real
correctness evidence (correctness_source != missing; run the --correctness harness and
record pass/fail with the reference), and (c) a controlled kernel-time speedup (replace
report_scan with a measured kernel-time A/B). Until then there is nothing safe to integrate,
and downstream HL promotion (Hyperloom #759) stays correctly inert (it gates on
correctness_passed is True).

Open question to confirm on a live run: is correctness missing because GEAK's harness never
runs the --correctness step to completion (harness/timeout — ties to the other #316 items),
or because the result is produced but not captured into verification? The GEAK output dir is
an in-pod /hyperloom/... path not persisted to the fleet share, so this needs a live repro.

[iraj465]

CONCLUSIVE root cause: GEAK finds optimizations but is KILLED by the 130-min wrapper timeout before verifying

Read the GEAK kernel-opt logs for all 12 "wins". 11 of 12 end in TimeoutExpired after 7800s
(130 min = HL's geak optimization wrapper budget, _optimization_wrapper_timeout_sec(geak)=130*60+180);
1 "completed". This explains the correctness_source=missing + report_scan finding above.

Evidence (DeepSeek k001, the 3.68x)

[harness_gen] found: ref=run_torch, kernel=run_gemm, test=test_gemm   <- valid correctness harness WITH a reference
[harness_gen] generated harness passed static_check
[geak] test_command=python .../harness_test_gemm_a8w8_blockscale.py --correctness
TimeoutExpired after 7800s                                            <- agent killed before --correctness completes/captures

The correctness step has its own timeout 600 (10 min), so it is not the correctness check
hanging — it is the LLM optimization loop consuming the full 130-min budget before a completed,
verified terminal state is produced.

Interpretation

• GEAK can find optimizations: it builds a valid harness with a reference and self-reports
  1.4x–3.68x.
• GEAK cannot finish: the propose→compile→profile loop (on large MoE/GEMM kernels, each step an
  LLM call) eats the whole wall-clock, and there is no "stop and verify best-so-far before the
  wall" step — so the run is killed with only an unverified report_scan number.
• Therefore correctness_source=missing is a symptom of the timeout, not an optimization-quality
  problem.

Fix (highest leverage)

Checkpoint-and-verify-before-timeout: reserve a budget slice so that before the wall GEAK halts
search and runs --correctness + a controlled kernel-time measurement on its best candidate, and
captures the artifact. A timed-out run then yields a verified result (correctness + controlled
speedup) instead of a report_scan mirage. Secondary: iteration-efficiency in the search, or a
larger budget (costly). This is upstream of the harness/.cu items and of Hyperloom #759.

[iraj465]

Definitive: candidates PASS correctness, but the verified-selection step is starved of budget

Deep-dived a full local GEAK output dir (k004 = gemm_a8w8_blockscale.cu, mode=quick, 60-min budget).

The patches are correct

Round-1 candidate test files contain real correctness results, e.g.:

[correctness] idx=0 shape=(M=64,N=8704,K=5120,splitK=0) rel=0.0000 max_abs=0.0312 finite=True -> PASS
CORRECTNESS: PASS
GEAK_RESULT_LATENCY_MS=0.119890

So GEAK DOES find correct optimizations; per-candidate correctness runs and passes.

Budget breakdown (why no VERIFIED result reaches HL)

phase	wall	note
preprocess (discovery + harness-init)	~33 min	blew past 900s soft AND 1800s hard cap; borrowed from optimization budget
Round 1: 8 parallel rewrite sub-agents	~20 min	consumed the entire remaining ~26 min; patches produced + correctness PASS
post-round FULL_BENCHMARK (verified selection)	0 min left	select_best_verified: no verified round evaluations found -> timeout_auto_finalized

Result: HL receives no verified speedup/correctness -> NEEDS_REVIEW. This is universal on heavy
kernels because preprocess overrun + an 8-wide round starve the verification step every time.
(Also observed: Scoring target: wall — GEAK_RESULT_LATENCY_MS = wall_ms, not kernel time.)

Fix = BOTH (not reserve-&-checkpoint alone)

1. Hard-cap / speed up preprocess so it cannot borrow from and starve optimization+verification
   (33/60 min on harness setup is the single biggest waste).
2. Reserve budget for post-round FULL_BENCHMARK — guarantee the verified measurement runs on the
   best correctness-passing candidate before the wall (checkpoint-and-verify).
3. Cap round fan-out (8 parallel strategies consume the whole opt budget; fewer / per-round wall).
4. Ensure scoring target = kernel (GEAK Make invalid --target fall back to 'kernel' not 'wall' #313 sets default; this run still scored wall).

This is the most upstream cause of the fleet-wide NEEDS_REVIEW: GEAK optimizes correctly but never
gets to emit a VERIFIED result within budget.