Issues

• Portable results: a small jq mapping from results.json (6-field FAIL + 4-class attribution) to EvalPort ResultSet JSON

  nano-step/eval-harness#42 · Filter by author adhabnr-ux opened on Aug 23, 2026

• Add a langgraph-node runner for regression-testing LangGraph agents

  enhancementhelp wantedrunner
  nano-step/eval-harness#36 · Filter by author hoainho opened on Jun 1, 2026

• Add --since filter to eval-harness trend (e.g. --since=7d, --since=2026-05-01)

  enhancementgood first issuehelp wanted
  nano-step/eval-harness#35 · Filter by author hoainho opened on Jun 1, 2026
  1

• Pretty-print eval-harness status output as a colored table

  enhancementgood first issuehelp wanted
  nano-step/eval-harness#34 · Filter by author hoainho opened on Jun 1, 2026

• Document the case YAML with a JSON Schema (for editor autocomplete)

  documentationgood first issuehelp wanted
  nano-step/eval-harness#33 · Filter by author hoainho opened on Jun 1, 2026

• Add eval-harness doctor — preflight diagnostics command

  enhancementgood first issuehelp wanted
  nano-step/eval-harness#32 · Filter by author hoainho opened on Jun 1, 2026

• 🗺 Roadmap: v0.4.3 → v0.5.0 → v0.6.0 → v0.7.0 → v1.0.0

  documentationhelp wanted
  nano-step/eval-harness#26 · Filter by author hoainho opened on May 30, 2026

• Pin and document end-to-end: 'run a real eval-harness against omo-session-distiller'

  documentationgood first issuev0.5.0
  nano-step/eval-harness#25 · Filter by author hoainho opened on May 30, 2026

• Write a GitHub Actions example workflow (.github/workflows/eval.yml)

  ci-integrationdocumentationgood first issuehelp wantedv0.5.0
  nano-step/eval-harness#23 · Filter by author hoainho opened on May 30, 2026

• Document semver + deprecation policy

  documentationgood first issuev0.5.0
  nano-step/eval-harness#22 · Filter by author hoainho opened on May 30, 2026

• Add stochastic pass@k mode (T>0, multiple samples per case)

  enhancementhelp wantedv0.7.0
  nano-step/eval-harness#21 · Filter by author hoainho opened on May 30, 2026

• Cache LLM judge responses by (rubric_hash, artifact_hash)

  enhancementhelp wantedv0.7.0
  nano-step/eval-harness#20 · Filter by author hoainho opened on May 30, 2026

• Anthropic API rate-limit handling + exponential backoff in llm_judge.sh

  enhancementhelp wantedv0.7.0
  nano-step/eval-harness#19 · Filter by author hoainho opened on May 30, 2026

• Add eval-harness ab --base=X --candidate=Y for A/B skill testing

  enhancementhelp wantedv0.6.0
  nano-step/eval-harness#18 · Filter by author hoainho opened on May 30, 2026

• Implement automatic warn-only → blocking promotion after N green days

  enhancementhelp wantedv0.6.0
  nano-step/eval-harness#17 · Filter by author hoainho opened on May 30, 2026

• Cross-skill behavioral interaction diagnosis (which other skill caused this break?)

  enhancementhelp wantedv0.6.0
  nano-step/eval-harness#16 · Filter by author hoainho opened on May 30, 2026

• Add pre-push branch filter (skip WIP/feature/draft branches)

  enhancementgood first issuehelp wantedv0.6.0
  nano-step/eval-harness#15 · Filter by author hoainho opened on May 30, 2026

• Implement auto-fix applier (v0.4.0 only proposes; v0.5+ applies)

  enhancementhelp wantedv0.5.0
  nano-step/eval-harness#14 · Filter by author hoainho opened on May 30, 2026

• Self-eat: add evals/cases for eval-harness itself

  enhancementhelp wantedv0.5.0
  nano-step/eval-harness#13 · Filter by author hoainho opened on May 30, 2026

• Implement shared-state daily budget ledger for EVAL_BUDGET_USD

  enhancementhelp wantedv0.5.0
  nano-step/eval-harness#12 · Filter by author hoainho opened on May 30, 2026

• Add JUnit XML + SARIF reporters (CI-friendly output)

  ci-integrationenhancementhelp wantedv0.5.0
  nano-step/eval-harness#11 · Filter by author hoainho opened on May 30, 2026

• Add --strict mode (flip warn-only off, exit 12 on first regression)

  enhancementhelp wantedv0.5.0
  nano-step/eval-harness#10 · Filter by author hoainho opened on May 30, 2026

• Remove $workdir PATH-prepend in spawn.sh (fixtures can shadow system binaries)

  buggood first issuesecurityv0.4.3
  nano-step/eval-harness#9 · Filter by author hoainho opened on May 30, 2026

• Remove dead code: propose_fixes_for_run with tautology select bug

  buggood first issuev0.4.3
  nano-step/eval-harness#7 · Filter by author hoainho opened on May 30, 2026

• Cap samples: field in llm_judge case YAML (prevent runaway cost)

  enhancementgood first issuev0.4.3
  nano-step/eval-harness#6 · Filter by author hoainho opened on May 30, 2026