testing.md

Testing toolpass

The Python package lives in toolpass-py/; run these from there, in a virtual environment with Python 3.10 or later:

pip install -e ".[crypto]" pytest pytest-timeout hypothesis ruff mypy types-PyYAML build
ruff format --check src tests examples && ruff check src tests examples
mypy --strict src/toolpass
python -m pytest -q                        # unit, integration, harness and server tests

This is what CI's toolpass-py job runs, on Python 3.10 and 3.13, on Linux and Windows; on Linux it then builds the wheel, installs it into a clean environment and runs the tests again from outside the checkout (TOOLPASS_REQUIRE_INSTALLED=1 makes them fail if they picked up the source tree instead).

Directory	What it tests	Needs
tests/core, tests/authx, tests/net, test_cli.py, test_server.py	the engine, config, caches, signing, the HTTP client, the command and the server	nothing
tests/integrations/<name>	each integration against a fake of its API over TLS, with injected failures (500, 429, 401, timeout), and a check that no secret reaches a log line	nothing; TOOLPASS_SPECS_DIR to validate requests (below)
tests/integrations/test_registry.py	the coverage gate: test_action_<name>_allow and _deny for every action	nothing
tests/integrations/<name>/test_routes.py	the integration's route table (requests mapped to actions): every route names an operation of the vendor's API description, and every write operation under the table's claims is routed or listed as left out	TOOLPASS_SPECS_DIR for the description checks
tests/dashboard	the server collecting the runs agents report (POST /runs, bounds, the key) and serving them to the page; a toolkit reporting to its server, in-process or remote, without ever waiting on it; gateway off putting back entries 0.8 and 0.9 rewrote	nothing
tests/demo	examples/demo: each stand-in system answers the real integration's permission checks for Dana and Sam, and the tour records every run with its requests	nothing
tests/toolkit	the secure-tools toolkit: every check, the hook under the SDKs (test_wire.py), recorded entitlements, the examples	nothing
tests/sdks	real SDKs (PyGithub, slack_sdk) under a toolpass tool, against the fake upstreams	pip install PyGithub slack_sdk; each file skips without its SDK
tests/harness, tests/harness_tests	the fake upstream, spec validation, the stand-in permission engine of the wire tests (wire.py), and their own tests	nothing
tests/frameworks	each framework adapter through the framework's real agent loop, with a scripted model and a real in-process engine; and the wire hook through the same path (wire_scenario.py and the wire_gh fixture in conftest.py: a tool that declares nothing, refused at the wire; for the Claude Agent SDK through its in-process MCP server, which needs no CLI)	the framework's extra, e.g. pip install -e ".[strands]", and pip install mcp for test_mcp.py; each file skips without it. ANTHROPIC_API_KEY for the live tests, which drive the same scenario with Claude
tests/real	the vault integration against a real Vault dev server in Docker, every answer compared with Vault's own sys/capabilities	Docker and the hashicorp/vault:1.17 image, present locally (never pulled); TOOLPASS_REAL=1 turns a skip into a failure
tests/e2e	the kubernetes integration and the server against a real cluster, with groups sent by the caller and named by a groups_connection	a cluster named in TOOLPASS_E2E_KUBERNETES_URL, _TOKEN_FILE and _CA_FILE; test/kind/run.sh creates a kind cluster, seeds it and exports them
tests/contract	integrations against Prism, a mock server that answers from the vendor's OpenAPI description and rejects requests that violate it	TOOLPASS_SPECS_DIR and npx on the PATH (Prism is fetched by npx)
tests/live	your own cases against real systems (below)	TOOLPASS_LIVE_CASES naming a cases file, and the credentials
tests/differential	the Argo CD RBAC evaluator against argocd admin settings rbac can	the argocd binary on the PATH
tests/scenarios	recorded decision scenarios (connections, a scripted upstream, checks with the expected code), each run through the engine	nothing

Every integration's tests run against a fake of its API. With the vendors' published API descriptions present, every request the fakes receive is also validated against the description: path, method, required parameters, body fields, AWS operation members, GraphQL fields and arguments.

../test/specs/fetch.sh ../.specs                         # from toolpass-py/
TOOLPASS_SPECS_DIR=$PWD/../.specs python -m pytest -q tests/integrations

Every resource parser has property tests (tests/integrations/<name>/test_fuzz.py, Hypothesis) that assert an accepted resource is made only of validated pieces before it reaches a URL or query. They run at Hypothesis's default size with the rest of the suite; HYPOTHESIS_PROFILE=nightly and TOOLPASS_FUZZ_SCALE=<n> make them run much longer.

Outside toolpass-py:

test/kind/run.sh                          # kubernetes end to end on a kind cluster
test/kind/helm.sh                         # the Helm chart and the Docker image on a kind cluster
(cd toolpass-ts && npm ci && npm test)    # the Node client
(cd examples/agent-ts && npm ci && npm test)
python3 test/docs/linkcheck.py            # relative links and anchors in every Markdown file

Against your own systems

The live test takes a cases file (see examples/live-cases.yaml) that names a config file and the answers you expect from your own systems; run it once after setting up each connection:

TOOLPASS_LIVE_CASES=$PWD/cases.yaml python -m pytest tests/live      # from toolpass-py/

It probes every connection first and ends with a summary per connection: passed, failed, or not configured, naming the environment variables or files that were missing on this machine. TOOLPASS_LIVE_REQUIRE=1 turns a connection that is not configured into a failure.