Benchmark showing error

[Hun33er]

All the benchmark are showing error,
also running npm run obsidivm:evolve -- --hunter live --judge-model deepseek-v4-pro --accept-threshold 0.7 --max-gens 1 --target obsidivm --prune-after 3 shows FATAL: fetch failed

[lyubomir-bozhinov]

Triaged this locally on current main. Short version: the benchmarks aren't broken — FATAL: fetch failed is a missing-service (and, for the evolve command, missing-LLM-key) problem, and the error message just doesn't say so.

The deterministic benchmarks pass offline — no service or key needed:

npm run binary:bench   # ✅ 4/4
npm run mobile:bench   # ✅ 4/4
npm run cve:bench      # ✅
npm run cloud:bench    # ✅ 4/4

If those error for you too, it's likely a stale install — try npm ci.

The obsidivm:* benchmarks are different: they're a thin HTTP client over the separate OBSIDIVM range service, which has to be running (see docs/OBSIDIVM.md). obsidivm-bench.mjs fetches the target spec from OBSIDIVM_URL (default http://127.0.0.1:4200) before it does anything else; with nothing on that port the fetch throws and you get the opaque FATAL: fetch failed. Note --hunter stub only means "no LLM hunter" — the grading step still needs the OBSIDIVM service, so it isn't a fully offline mode.

To run them:

# 1. start OBSIDIVM (separate project) — listens on :4200
python3 range.py
# 2. then, in this repo:
npm run obsidivm:bench:stub          # plumbing check (needs OBSIDIVM up)

Your evolve --hunter live --judge-model deepseek-v4-pro command needs two things: the OBSIDIVM service and a live LLM key (OPENROUTER_API_KEY / ANTHROPIC_API_KEY / OPENAI_API_KEY in env, .env, or Keychain). fetch failed there is one of those two being absent.

I'm putting up a small PR to make this self-diagnosing — a connection-refused on the OBSIDIVM base URL should tell you the service is unreachable and point you at docs/OBSIDIVM.md, instead of a bare fetch failed.

[jmagly]

The linked fix in #158 has now been reviewed at exact head 13998844a45e474d5f86ff13cd77d7e3b8d5c062: typecheck, all 3 focused network-error regressions, and hosted CI pass. It now distinguishes an unreachable OBSIDIVM service, timeout, and malformed OBSIDIVM_URL, and the help text states that stub mode still needs the range service.

Until that PR lands, the verified workaround remains: start the separate OBSIDIVM service on the configured URL (default http://127.0.0.1:4200) and provide a live LLM key for --hunter live. I’m leaving this issue linked to #158 so it closes with the canonical fix rather than closing from the PR description alone.