Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
122 lines (88 loc) · 6.14 KB

CONTRIBUTING.md

File metadata and controls

122 lines (88 loc) · 6.14 KB
Raw

Contributing to @nano-step/eval-harness

Thanks for considering a contribution. This doc tells you exactly what helps, what doesn't, and how to land a PR fast.

TL;DR for contributors

  1. Pick a good-first-issue from issues
  2. Fork → branch → make change → add or update a test under scripts/eval/tests/
  3. Run npm test — it executes the complete deterministic local suite.
  4. Open a PR with a one-line summary + before/after evidence

What we want

Want Don't want
Bug fixes from documented boundaries in KNOWN_ISSUES.md Refactors with no observable behavior change
New check kinds (with a 3-case demo skill) Features without a regression test
Portability fixes (BSD/macOS, alpine/musl, alternate jq/yq versions) Bumping dependencies "just because"
Faster paths in the critical loop (lock, manifest, score) New runtime dependencies; keep additions to the existing Bash, jq, and Python standard-library toolchain
Documentation that maps invariants to specific file:line Documentation generated by an LLM with no examples
Audit reports that find new bugs (with reproducers) Drive-by lint nits

Project shape

  • The execution path is Bash + jq + OpenCode. Python standard library backs the yq fallback; Node is used only by repository documentation validation.
  • The run adapter is OpenCode-specific. The grade command accepts content-addressed evidence from other runners and does not invoke a model.
  • Honest failure modes: PASS, FAIL, ERROR, NEEDS_REVIEW, and INDETERMINATE are distinct. Missing evidence must not be represented as PASS or numeric zero.
  • Product measurements stay in named dimensions; do not add an uncalibrated global score or claim causal attribution from hash co-occurrence.
  • Shell entrypoints use set -euo pipefail and must work on the minimum supported Bash version.
  • YAML schema changes are additive. New result, baseline, environment-manifest, and grading-manifest schemas require compatibility notes and boundary tests.
  • Every behavioral change needs a consumer-visible regression test and an exercised smoke path. Product scoring needs measured/unavailable boundary coverage.

Running the tests

Product-quality measurements and harness-validity metrics are different records; see POLICY.md — Measurement boundaries.

# Full deterministic local suite
npm test

# One focused integration test
bash scripts/eval/tests/ab_compare.sh

# Dry-run the OpenCode adapter without spawning a model
EVAL_SKIP_AUTH_CHECK=1 eval-harness run --skill=omo-session-distiller --dry-run

Writing a new test

Tests are bash scripts under scripts/eval/tests/. Each one is self-contained:

#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
WORK="$(mktemp -d -t eval-harness-XXX.XXXXXX)"
trap 'rm -rf "$WORK"' EXIT

# ... arrange ephemeral state ...
# ... act (invoke the harness) ...
# ... assert ...

echo "PASS: <what was proven>"
exit 0

Patterns to follow:

  • Stub opencode when you don't need a real LLM call (see regression_inject.sh for the canonical stub)
  • Stub curl when testing llm_judge (see llm_judge_unit.sh)
  • Set EVAL_SKIP_AUTH_CHECK=1 to bypass the API-key preflight in tests
  • Set EVAL_STATE_DIR="$WORK/state" to keep state isolated per test
  • Set OPENCODE_SKILLS_ROOT="$WORK/skills" to keep skill discovery isolated

Coding style

  • Shell: bash, set -euo pipefail, prefer [[ ]] over [ ], quote everything, use local for function vars
  • Avoid eval and shell execution of case input. The implicit-safe grader uses safe_shell.py with an argv allowlist; unsafe_shell: true explicitly opts trusted case YAML into bash -c.
  • Comments only where load-bearing: complex algorithms, security boundaries, portability quirks. Self-documenting code over docstrings.
  • One file = one concern. New libs go in scripts/eval/lib/
  • Cross-platform. Code must work on Linux + macOS (BSD coreutils). When in doubt, test both.

Adding a new check kind

  1. Put grader-specific behavior in scripts/eval/lib/extended_graders.sh; keep shared case/status aggregation in score.sh.
  2. Register the kind in run_check and define its required/optional, dimension, weight, and unavailable-evidence behavior.
  3. Add deterministic positive, negative, malformed-input, and missing-evidence coverage where applicable; do not add an uncalibrated global score.
  4. Update autofix proposal handling only if the new kind can produce an actionable fix.
  5. Document the case contract in README, POLICY, the skill guide, and CHANGELOG; run npm test.

How releases work

  • main is always green (every test in scripts/eval/tests/ passes)
  • Features land on feature/<vX.Y.Z>-<topic> branches
  • Releases happen via merge commit + annotated tag vX.Y.Z + GitHub Release
  • SemVer: PATCH for non-breaking fixes; MINOR for additive changes; 0.x MINOR releases may contain explicitly documented breaking changes with migration paths.

Reporting a bug

Open a new issue with:

  1. One-line summary — what's broken
  2. Reproducer — minimal case YAML / shell commands / env
  3. Expected vs actual — what should happen, what does
  4. Environment — eval-harness --help | head -1, bash --version | head -1, OS

Bonus points if you can point to the file:line where you think it's broken.

Getting help

Forged in the regression furnace.