Add autonomous test discovery and domain evaluation

.github/workflows/discovery-evidence.yml

@@ -0,0 +1,159 @@
+name: discovery-evidence
+
+# Independent-signer acceptance run for Autonomous Test Discovery over the FROZEN dataset
+# (examples/discovery-pilot/frozen): 0 fresh model calls, real Docker sandbox, real StrykerJS, the actual Jev gate in block
+# mode. The signing key is created here, in RUNNER_TEMP, outside the checkout, the output and the eval state; only its
+# paths are exported to later steps (never its contents) and only the PUBLIC allowed_signers file is uploaded.
+# The pinned sandbox image is a single-platform arm64 digest, hence the arm64 runner; there is no emulation.
+
+on:
+  pull_request:
+    paths:
+      - ".github/workflows/discovery-evidence.yml"
+      - "scripts/eval/**"
+      - "examples/discovery-pilot/**"
+  push:
+    branches:
+      - main
+    paths:
+      - ".github/workflows/discovery-evidence.yml"
+      - "scripts/eval/**"
+      - "examples/discovery-pilot/**"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  discovery-evidence:
+    name: discovery evidence (independent signer)
+    runs-on: ubuntu-24.04-arm
+    timeout-minutes: 60
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 20
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Ensure jq is available
+        run: |
+          if ! command -v jq >/dev/null 2>&1; then
+            sudo apt-get update
+            sudo apt-get install -y jq
+          fi
+
+      - name: Install Python YAML dependency
+        run: python -m pip install pyyaml
+
+      - name: Show tool versions
+        run: |
+          bash --version | head -1
+          jq --version
+          python3 --version
+          node --version
+          docker --version
+          ssh -V 2>&1 | head -1 || true
+
+      - name: Pull the pinned sandbox image (explicit; the runner never pulls)
+        run: |
+          set -euo pipefail
+          IMAGE="$(jq -r '.image.ref' scripts/eval/discovery/policy/atd-sandbox.v1.json)"
+          docker pull "$IMAGE"
+          ARCH="$(docker image inspect --format '{{.Architecture}}' "$IMAGE")"
+          case "$(uname -m)" in aarch64|arm64) HOST_ARCH=arm64 ;; x86_64) HOST_ARCH=amd64 ;; *) HOST_ARCH="$(uname -m)" ;; esac
+          if [[ "$ARCH" != "$HOST_ARCH" ]]; then
+            echo "::error::pinned image architecture ($ARCH) does not match the runner ($HOST_ARCH)"
+            exit 1
+          fi
+          echo "image $IMAGE architecture $ARCH"
+
+      - name: Build the trusted dependency snapshot (the only networked step)
+        run: |
+          bash scripts/eval/discovery/setup-deps.sh \
+            --lockfile-dir=examples/discovery-pilot/collision \
+            --allow-network \
+            --out="$RUNNER_TEMP/atd-deps"
+
+      - name: Create the ephemeral CI signing key (outside checkout, output and state)
+        run: |
+          set -euo pipefail
+          umask 077
+          KEYDIR="$RUNNER_TEMP/trusted-signing"
+          mkdir -m 700 "$KEYDIR"
+          ssh-keygen -q -t ed25519 -N '' -C "atd-ci-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" -f "$KEYDIR/signing_key"
+          chmod 600 "$KEYDIR/signing_key"
+          # the principal and namespace come from the actual verifier constants, not from a copy here
+          read -r NAMESPACE PRINCIPAL_PREFIX < <(node --input-type=module -e "
+            import { NAMESPACE, PRINCIPAL_PREFIX } from './scripts/eval/discovery/attest.mjs';
+            console.log(NAMESPACE, PRINCIPAL_PREFIX);")
+          PUB="$(cut -d' ' -f1,2 "$KEYDIR/signing_key.pub")"
+          printf '%sci-secret namespaces="%s" %s\n' "$PRINCIPAL_PREFIX" "$NAMESPACE" "$PUB" > "$KEYDIR/allowed_signers"
+          chmod 600 "$KEYDIR/allowed_signers"
+          rm -f "$KEYDIR/signing_key.pub"
+          {
+            echo "ATD_TRUST=ci-secret"
+            echo "ATD_SIGNING_KEY_FILE=$KEYDIR/signing_key"
+            echo "ATD_ALLOWED_SIGNERS_FILE=$KEYDIR/allowed_signers"
+          } >> "$GITHUB_ENV"
+
+      - name: Run the frozen-dataset evidence acceptance
+        run: |
+          bash scripts/eval/discovery/ci-evidence.sh \
+            --out="$RUNNER_TEMP/atd-ci-evidence" \
+            --dependencies="$RUNNER_TEMP/atd-deps" \
+            --state-dir="$RUNNER_TEMP/atd-eval-state"
+
+      - name: Summarize
+        if: always()
+        run: |
+          set +x
+          EVIDENCE_OUT="$RUNNER_TEMP/atd-ci-evidence"
+          if [[ -f "$EVIDENCE_OUT/ci-observations.json" ]]; then
+            {
+              echo "### discovery evidence"
+              echo
+              jq -r '"- run: \(.ci.runUrl // "n/a") (attempt \(.ci.runAttempt // "n/a"))",
+                     "- commit: `\(.ci.sha // "n/a")`",
+                     "- mode: **\(.mode)** (independent signer claimed: \(.claimsIndependentSigner))",
+                     "- public signer fingerprint: `\(.signer.publicFingerprint // "n/a")`",
+                     "- fresh generation calls: \([.rounds[].freshModelCalls] | add)",
+                     "- gate exit (block mode): \(.gate.exitCode); independently verified: \((.gate.independentVerified // []) | length); repair requests: \(.gate.requests | length)",
+                     "- failed checks: \(if (.failedChecks | length) == 0 then "none" else (.failedChecks | join(", ")) end)",
+                     "", "| check | result |", "| --- | --- |",
+                     (.checks[] | "| \(.id) | \(if .pass then "PASS" else "FAIL" end) |")' "$EVIDENCE_OUT/ci-observations.json"
+            } >> "$GITHUB_STEP_SUMMARY"
+            echo "::notice title=discovery evidence::run ${GITHUB_RUN_ID} commit ${GITHUB_SHA} fingerprint $(jq -r '.signer.publicFingerprint // "n/a"' "$EVIDENCE_OUT/ci-observations.json")"
+          else
+            echo "No ci-observations.json was produced; see the job log." >> "$GITHUB_STEP_SUMMARY"
+          fi
+
+      # Explicit allowlist: reports, case evidence, observations and the PUBLIC signer data. The key directory, the eval state
+      # directory and the dependency snapshot live outside this path and are never uploaded.
+      - name: Upload evidence (reports, case evidence, public signer data)
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: discovery-evidence-${{ github.run_id }}-${{ github.run_attempt }}
+          path: |
+            ${{ runner.temp }}/atd-ci-evidence/ci-observations.json
+            ${{ runner.temp }}/atd-ci-evidence/steps.log
+            ${{ runner.temp }}/atd-ci-evidence/steps
+            ${{ runner.temp }}/atd-ci-evidence/public
+            ${{ runner.temp }}/atd-ci-evidence/clean
+            ${{ runner.temp }}/atd-ci-evidence/defect
+            ${{ runner.temp }}/atd-ci-evidence/mutation
+            ${{ runner.temp }}/atd-ci-evidence/integrity-before.json
+            ${{ runner.temp }}/atd-ci-evidence/integrity-after.json
+          if-no-files-found: warn
+          retention-days: 30

.gitignore

@@ -1,4 +1,5 @@
 node_modules/
+__pycache__/
 .opencode/eval-harness/runs/
 .opencode/eval-harness/locks/
 .opencode/eval-harness/spend.json

CHANGELOG.md

@@ -4,6 +4,26 @@ All notable changes to `@nano-step/eval-harness` are documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
 this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## Unreleased
+
+### Added
+- Autonomous Test Discovery modules for deterministic diff/risk targeting, bounded intent-aware generation requests, untrusted candidate admission and maintained scoped Stryker mutation.
+- `atd-sandbox` runner with independent revision exports, digest-pinned Docker confinement, structured framework outcomes, SSH-signed evidence and fail-closed admission.
+- `discovery_finding` and optional `discovery_test_health` checks integrated with existing stability sampling, grading manifests, typed results, JUnit/SARIF and baseline behavior.
+- Bounded OMP/Sonnet generation and frozen-candidate review replay, executable QA Factory oracle integration and a warn-only Jev test-command gate with bounded, independently verified repair requests.
+- Isolated collision benchmark fixtures, dependency setup, live pilot commands and an operational/security guide.
+- Optional img2threejs product evaluation using a pinned confined Playwright browser, native Three.js structure checks, GLB round-trip validation and separately approved visual baselines.
+
+### Fixed
+- Generic runner stability samples re-execute instead of regrading copied evidence; stale or failed respawns cannot appear independently reproduced.
+- Discovery confirmation preserves existing baseline failures and rejects tampering, missing identities, environment mismatches, runtime faults and unreviewed expectations.
+- Revision export/test overlays reject symlink traversal before host writes and refuse oversized blobs before buffering their contents.
+- Downstream repair/QA/gate consumers recompute findings rather than trusting result JSON; replay admission rejects injected or alternate manifest-selected host checks, and promotion requires the original trusted review.
+- Generated tests retain their revision-relative placement independently of the host input directory, so nested relative imports resolve without rewriting test source.
+- Mutation overlays honor rejected or invalid expectation reviews rather than accepting unsupported assertions merely because the test passes before mutation.
+- Mutation execution and pilot reports verify signed raw evidence before computing counts; unsigned baselines cannot fabricate coverage gains.
+- The Python timeout fallback preserves piped standard input; grading manifests use jq 1.6-compatible object expressions.
+
 ## [0.5.0] — 2026-09-30
 
 ### Added

POLICY.md

@@ -120,6 +120,7 @@ The following variables are active in the v0.5.0 working tree and are not deprec
 - Environment manifests use schema 4. Schema_version is ignored as an environment drift signal; prompt/rubric/tool hashes introduced in schema 4 are ignored when absent from a legacy baseline.
 - Provider-neutral grading manifests use schema 1 and require digests for available case/workdir/transcript/artifact/environment evidence. Unsupported manifest versions or tampered content are rejected.
 - Consumers of results, baseline, or environment-manifest files must ignore unknown additive fields and handle absent optional fields. A breaking change to a required field or its semantics requires a breaking release.
+- Autonomous discovery adds optional finding fields and separately versioned `atd.*` evidence artifacts; it does not change existing case statuses or baseline-regression semantics. Local same-user evidence is reporting-only for downstream blocking and repair. Independent promotion requires explicit opt-in, authenticated evidence and the original trusted expectation review; generation or a matching specification quote alone grants no authority.
 
 ---
 

README.md

@@ -15,8 +15,9 @@ Typed evaluation for OpenCode skills and AI-agent workflows. Run a skill with Op
 - **Provider-neutral grading.** `grade` validates a schema-1 manifest binding the case, workdir tree, transcript, artifacts, environment, and provenance. It does not execute a model or load arbitrary evaluator plugins.
 - **Independent product dimensions.** Numeric measurements stay grouped by their declared dimension; eval-harness does not compute a global product-quality score.
 - **Auditable comparisons and reports.** Baselines, typed A/B results, attribution, token/cost/duration coverage, JUnit XML, and SARIF help explain what changed and what evidence is available.
+- **Autonomous Test Discovery.** Bounded intent-aware test generation, scoped Stryker mutation and isolated base/candidate execution feed the existing graders and reports. Discovery gates are warn-only by default; a generator cannot approve its own findings.
 
-`run` is the OpenCode execution adapter. A LangGraph runner example demonstrates the runner contract; `grade` accepts prepared evidence without coupling the grader to that runner.
+`run` selects a runner, with OpenCode as the default. The LangGraph example demonstrates the runner contract; `atd-sandbox` executes discovery tests in confined containers. `grade` accepts prepared evidence without coupling grading to the producer.
 
 ## Install
 
@@ -65,7 +66,7 @@ Strict grading exits 13 for malformed or tampered evidence, 14 for `FAIL`, 15 fo
 | `capability` | Verify declared behavior or coverage | Disabled by default |
 | `product` | Report named product measurements | Disabled by default |
 
-Shipped checks: `shell`, `jq_path_contains`, `file_exists`, `output_contains`, `output_not_contains`, `llm_judge`, `metric_score`, `trajectory`, and `human_review`. Required checks determine the typed result; optional checks report evidence without gating. Product dimensions remain separate.
+Available checks: `shell`, `jq_path_contains`, `file_exists`, `output_contains`, `output_not_contains`, `llm_judge`, `metric_score`, `trajectory`, `human_review`, `discovery_finding` and `discovery_test_health`. Required checks determine the typed result; optional checks report evidence without gating. Product dimensions remain separate. The [discovery guide](./docs/AUTONOMOUS_TEST_DISCOVERY.md) documents the new check/evidence trust boundary.
 
 For prose checks, `llm_judge` requires `ANTHROPIC_API_KEY`; abstentions or unresolved votes remain unavailable rather than becoming a synthetic pass. Deterministic checks and `grade` do not require a model.
 
@@ -83,19 +84,20 @@ For prose checks, `llm_judge` requires `ANTHROPIC_API_KEY`; abstentions or unres
 
 ## Tests and reports
 
-Run the deterministic offline suite with:
+Run the local evaluation suites with:
 
 ```bash
 npm test
 ```
 
-The suite uses fixtures and stub runners; it does not measure live model quality, latency, or cost. Optional exports are available with `--report=junit:<path>` and `--report=sarif:<path>`.
+The suites use controlled fixtures and stub runners, with live Docker checks when their prerequisites are present; they do not measure live model quality, latency or cost. The separate discovery pilot does. Optional exports are available with `--report=junit:<path>` and `--report=sarif:<path>`.
 
 ## Documentation
 
 - [Documentation index](./docs/README.md)
 - [v0.5.0 design and migration contract](./docs/EVAL_HARNESS_V2.md)
 - [Runner contract](./docs/runners.md) and [LangGraph example](./examples/langgraph-runner/)
+- [Autonomous Test Discovery](./docs/AUTONOMOUS_TEST_DISCOVERY.md): operation, sandbox, review/replay and QA Factory/Jev gates
 - [Janus benchmark](./docs/JANUS_BENCHMARK.md): offline evidence and the no-go decision on a native adapter
 - [Security policy](./SECURITY.md), [versioning policy](./POLICY.md), and [known issues](./KNOWN_ISSUES.md)
 - [Changelog](./CHANGELOG.md) and [contributing guide](./CONTRIBUTING.md)

SECURITY.md

@@ -35,6 +35,34 @@ Implicit-safe `kind: shell` checks are parsed into a constrained argv pipeline c
 
 Fixture paths and grader artifact paths are checked for workdir confinement. LLM-judge prompts are sent to Anthropic when live judging is enabled; never include secrets in case prompts. The harness cannot redact unknown sensitive content.
 
+### Autonomous Test Discovery
+
+`atd-sandbox` is a narrower, explicit exception to the ordinary host runner:
+generated candidate tests execute only in the policy-approved Docker image,
+with no network, read-only source/dependencies/rootfs, non-root identity,
+dropped capabilities, resource limits and a host deadline. Missing confinement
+fails closed; it never runs the test on the host. Trusted dependency setup is
+separate and requires an explicit network acknowledgement.
+
+This does not make arbitrary case YAML or generic `grade` manifests untrusted
+inputs. The discovery workflow generates its own cases and admits only their
+fixed layout/check kinds before replay, preventing a rewritten discovery
+manifest from inserting host shell commands. Model output cannot provide
+grading checks, signer settings or expectation approvals.
+
+Local signatures provide integrity, not independent control from the same OS
+user. Blocking, automatic repair requests and QA confirmed-defect promotion
+require externally controlled signer trust, stable authenticated differential
+evidence and the operator's original review from a separate trusted directory.
+Neither a JSON trust label nor rewriting a review and its digest manifest grants
+approval. Keep keys outside repositories, case workdirs and all mounted paths.
+
+Residual risks include the shared Docker kernel/VM, same-UID framework/reporting
+processes and a writable output mount without a filesystem-wide quota. See the
+[discovery guide](./docs/AUTONOMOUS_TEST_DISCOVERY.md#limits-and-verification)
+for operational limits. Disk exhaustion, runtime/collection faults and missing
+evidence are inconclusive infrastructure results, never regression proof.
+
 ## Past security advisories
 
 Hardening release **v0.4.2** (2026-05-30) closed 8 audit-surfaced BLOCKERs including: