Skip to content

testing.md

Latest commit

 

History

History
278 lines (201 loc) · 8.32 KB

testing.md

File metadata and controls

278 lines (201 loc) · 8.32 KB
Raw

Testing Quartermaster

Layered playbook for validating the product end-to-end. The npm consumer path (npx quartermaster-mcp) is the priority; source checkout is for development.

Quick commands

Goal Command
Unit + integration tests pnpm -r build && pnpm -r test
Benchmark smoke pnpm bench -- --ci
Full product smoke (CI) pnpm smoke
Local bins (dev) pnpm smoke:local
Published npm (consumer) pnpm smoke:npx
Stress test (full) pnpm stress
Stress test (quick) pnpm stress:quick
Regression (2 rounds) pnpm regression
Regression (local bins) pnpm regression:local
Optional GitHub+Slack eval node examples/smoke/run-gjs-eval.mjs

Automated smoke federates the dev workbench (filesystem, memory, everything, thinking, git) with starter synonymsFile (business-to-dev.json) plus org synonyms. Regression runs smoke + stress twice, audit-loop replay, and checks R@8 stability.

Layer 1: Automated baseline (~5 min)

Floor for every change — same as CI.

pnpm install --frozen-lockfile
pnpm -r build
pnpm -r lint
pnpm -r test
pnpm bench -- --ci

Proves: ranker, config, policy, validation, federation with echo fixtures, MCP protocol over in-memory transport.

Does not prove: published tarball, real downstream servers, host agent behavior.

Layer 2: npm consumer smoke (~10 min)

Validates the published package from a clean environment against the dev workbench.

pnpm smoke:npx   # from repo root
# or:
node examples/smoke/run-smoke.mjs --npx

From CI / pre-publish, use the pack path (installs npm pack tarball):

pnpm smoke

Pass criteria:

  • --version prints a semver
  • --validate --config succeeds on dev workbench config
  • quartermaster doctor reports all downstreams connected
  • quartermaster eval --ci --min-r8 0.5 passes on dev workbench cases
  • MCP protocol checklist + audit CLI loop

Config is generated with synonymsFile: ./business-to-dev.json (starter pack) and small inline org synonyms. Template: examples/smoke/quartermaster-dev-workbench.json.

Fixtures: examples/smoke/, eval cases in examples/regression/eval-cases-dev-workbench.jsonl.

Layer 3: Ranker quality (~15 min)

Static demo (no MCP host)

pnpm -r build
node examples/static-demo/demo.mjs

Dev workbench eval (live MCP)

pnpm -r build
node examples/smoke/run-smoke.mjs --local
# or manually after build-real-config writes a temp config:
pnpm quartermaster eval \
  --ci --min-r8 0.5 \
  --config /path/to/quartermaster-dev.json \
  --cases examples/regression/eval-cases-dev-workbench.jsonl

Blind manifest (untuned floor)

Static tools from bench/cases/real-mcp-blind.json, no synonyms — honest baseline before per-team tuning:

pnpm quartermaster eval \
  --ci --min-r8 0.45 \
  --config <blind-config.json> \
  --cases examples/regression/eval-cases-blind-manifest.jsonl

Regression builds this config automatically each round.

Full benchmarks

pnpm bench

Heritage 171-tool corpus (~91% R@8). Required before ranker/synonym changes — see CONTRIBUTING.md.

Per-team tuning loop

  1. Copy examples/synonyms/business-to-dev.json beside your quartermaster.json and set "synonymsFile": "./business-to-dev.json".
  2. Add org-specific inline synonyms for terms your team uses.
  3. Run real work in Cursor with QM_AUDIT=1.
  4. quartermaster eval --from-audit audit.jsonl --draft-cases cases.jsonl
  5. Fix weak cases (synonyms, tool descriptions), re-run eval --ci.

For GitHub + Slack (requires tokens):

export GITHUB_TOKEN=...
export SLACK_TOKEN=...
node examples/smoke/run-gjs-eval.mjs

Layer 4: MCP protocol E2E without an LLM (~15 min)

Scripted five-step checklist via stdio over the dev workbench:

pnpm smoke:local
# or:
node examples/smoke/mcp-smoke.mjs   # with QM_REAL_CONFIG set by run-smoke.mjs

Or run the full orchestrator: pnpm smoke.

Step Call Expected
1 listTools() retrieve_tools, call_tool, list_servers only
2 retrieve_tools({ query }) Right tool ranked; inputSchema hydrated
3 call_tool({ name, arguments }) Downstream result returned
4 list_servers Connected server; degraded: false
5 Bad tool name isError: true, no crash

Optional: connect an MCP inspector to the stdio process for interactive retrieve_tools tuning.

Layer 5: Policy, validation, reliability (~10 min)

npx -p quartermaster-mcp quartermaster policy test \
  --config quartermaster.json --tool github.create_issue
  • Shadow mode: denials logged, calls still forward
  • Enforce mode: denied tools return isError with policy message
  • Invalid args: validation_error in audit, no downstream crash
  • Bad downstream: list_servers shows degraded: true

Integration tests: packages/proxy/test/policy.integration.test.ts, packages/proxy/test/reliability.test.ts.

Layer 6: Cursor host integration (~30 min)

The user-facing proof. See examples/smoke/CURSOR-E2E.md and docs/recipes/cursor.md.

  1. Copy examples/smoke/cursor-mcp.json.example into ~/.cursor/mcp.json with absolute paths.
  2. Point config at synonymsFile + org synonyms (see dev workbench template).
  3. Set QM_AUDIT=1 and QM_AUDIT_FILE.
  4. Restart Cursor — confirm 3 meta-tools in the MCP panel.
  5. Run the five scenarios in CURSOR-E2E.md.

Layer 7: Audit CLI loop

After a session with audit enabled:

npx -p quartermaster-mcp quartermaster report --audit audit.jsonl --out report.html
npx -p quartermaster-mcp quartermaster savings --audit audit.jsonl --json
npx -p quartermaster-mcp quartermaster inspect --config quartermaster.json --audit audit.jsonl
npx -p quartermaster-mcp quartermaster eval \
  --from-audit audit.jsonl --draft-cases cases.jsonl --config quartermaster.json
npx -p quartermaster-mcp quartermaster dashboard --audit audit.jsonl

Automated: examples/smoke/audit-cli-smoke.mjs (via pnpm smoke) and examples/regression/lib/audit-loop.mjs (via pnpm regression).

Layer 8: Docker (optional)

docker build -t quartermaster .
docker run -v /path/to/quartermaster.json:/config/quartermaster.json quartermaster

Stress test

Hammer ranker scale, MCP federation, concurrency, and chaos. See examples/stress/README.md.

pnpm stress           # full (~7s)
pnpm stress:quick     # dev subset
pnpm stress:ci        # CI gates

Regression (2× smoke + 2× stress)

What a dev team runs before release — full suite twice, compare metrics. See examples/regression/README.md.

pnpm regression           # audit loop + 2× pack smoke + 2× stress + eval (~50s)
pnpm regression:local     # repo bins
pnpm regression:ci        # GitHub Actions

Each round: dev workbench smoke, stress, dev-workbench eval (with synonymsFile), blind-manifest eval (no synonyms). Round 1 vs 2 must agree on R@8 and pass/fail.

Report: examples/regression/results/latest.json

CI coverage

.github/workflows/ci.yml runs:

  1. build, lint, test, bench
  2. pnpm regression:ci (audit loop + 2× smoke + 2× stress + eval stability)

Release checklist

  • pnpm -r build && pnpm -r test && pnpm bench -- --ci green
  • cd packages/proxy && pnpm publish --dry-run
  • pnpm smoke and pnpm smoke:npx pass
  • pnpm regression passes (2 rounds stable)
  • pnpm stress passes
  • quartermaster doctor clean on dev workbench config
  • quartermaster eval --ci --min-r8 0.5 on dev workbench cases
  • Cursor E2E: 3 meta-tools visible, retrieve+call round-trip
  • quartermaster report + savings on sample audit traffic
  • synonymsFile + org synonyms documented for your team
  • Update Dockerfile pinned version if applicable
  • Optional: node examples/smoke/run-gjs-eval.mjs with live tokens