Skip to content

TESTING.md

Latest commit

 

History

History
136 lines (104 loc) · 8 KB

TESTING.md

File metadata and controls

136 lines (104 loc) · 8 KB
Raw

Testing

Setup

py -3.13 -m venv .venv
.venv\Scripts\activate          # Windows
pip install -e ".[dev]"

Run tests

pytest
pytest --cov=aura --cov-report=term-missing

Lint (required before PR)

black aura tests integrations examples
flake8 aura tests integrations examples

CI expectation: pytest, black, and flake8 all pass on aura/ and tests/ for every supported Python version. The dependency audit is advisory: pip-audit reports findings without blocking the CI gate.

Continuous integration

GitHub Actions:

Workflow Role
ci.yml PR + push to main — Python matrix, Skillware job, gate lint-test
reusable-test.yml Single source for install / black / flake8 / pytest (issue #17)
publish-pypi.yml Release gate — calls reusable-test.yml on Python 3.12 before upload

ci.yml runs on:

  • every pull request targeting main
  • every push to main (post-merge sanity)

Python matrix on ubuntu-latest (fail-fast: true — one version failure cancels the rest):

python-version: ["3.10", "3.11", "3.12", "3.13"]

Each matrix cell invokes reusable-test.yml with the same steps:

pip install -e ".[dev]"
pip install pip-audit
pip-audit                 # warn-only; does not block the CI gate
black --check aura tests integrations examples
flake8 aura tests integrations examples
pytest --cov=aura --cov-report=term-missing --ignore=tests/integration

The Dependency audit (warn-only) step runs pip-audit with continue-on-error: true. A vulnerability finding or audit error is therefore reported in the workflow logs but does not fail the matrix or block a PR from merging. Treat the output as a prompt to investigate and update dependencies; use the private reporting path in SECURITY.md for a suspected vulnerability in AURA itself.

The workflow also emits a gate job named lint-test that succeeds only when every matrix cell passed. That is the check to require in branch protection.

Fork PRs: the workflow uses permissions: contents: read only — no repository secrets, no PyPI OIDC, no deploy environment.

Publish workflow: .github/workflows/publish-pypi.yml calls reusable-test.yml on Python 3.12 before release upload (no pip-audit; same black/flake8/pytest scope as PR CI). Full 3.10–3.13 coverage is the PR CI matrix.

Skillware job: skillware-live in ci.yml calls reusable-test.yml with skillware: true — pip install -e ".[dev,skillware]" and pytest tests/test_skillware_integration.py.

Maintainers: after the first green lint-test run on main, enable branch protection → required status check lint-test.

Coverage expectations

  • New behavior needs a test — extend the closest file (test_core.py, test_v02.py, test_v03.py, test_cli.py, or test_core_gaps.py).
  • Shared fixtures live in tests/conftest.py — do not duplicate aura_home in test modules.
  • Optional Skillware registry tests: tests/test_skillware_integration.py (@pytest.mark.skillware) — run in CI via the skillware-live job when [skillware] is installed (#36).
  • Host stress simulation: python scripts/aura_host_stress_sim.py — thirty-four scenarios (loose/tight/tailored coats, spectrum low/mid/high/full bind, schema high bind, verified-identity gate edge cases, escalation SLO playbook, capability broker allow/deny/low-audit/bypass/high/escalation, single/multi/chain Skillware paths, sequencer, observers, export compare). CI: tests/test_host_stress_sim.py (@pytest.mark.skillware).
  • Capability broker stress: python scripts/aura_capability_stress_sim.py — sixteen scenarios (payment + generic GitHub-style scope, inject, redaction, spectrum, bypass, brokers, plaintext reject). CI: tests/test_capability_stress_sim.py (default gate, no Skillware extra).
  • Capability broker: tests/test_capability_broker.py — profile refs-only, allow/deny, low audit-only, inject, leak scrub, CLI (#48); example smoke tests/test_example_14_capability_broker.py.
  • Coat flow report: python scripts/aura_coat_flow_report.py --json — full session breakdown per spectrum level; CI: tests/test_coat_flow_report.py.
  • Spectrum enforcement: tests/test_spectrum_enforcement.py — level-driven allowlist, sequencer bind, CLI set/show.
  • Verified identity via spectrum: tests/test_spectrum_identity.py — level defaults, opt-out, session override, audit finding, CLI --require-identity (#73).
  • Escalation playbooks: tests/test_escalations.py — SLO/drift/alert/constraint triggers, log/alert/nudge/pause/email/custom actions, audit ESCALATION_FIRED (#47); example smoke tests/test_example_12_escalation_playbooks.py.
  • Real integration tests live in tests/integration/ (Skillware + Ollama, example 06 live). Default CI excludes them (--ignore=tests/integration). Run locally:
pip install -e ".[integrations]"
pytest tests/integration/ -v

Integration tests fail (not skip) if Ollama or Skillware is missing — that is intentional for the local integration suite.

  • Default CI matrix runs pytest --ignore=tests/integration — no deselected or skipped optional tests in the gate.
  • CI prints pytest --cov=aura --cov-report=term-missing for visibility; there is no coverage gate yet.

Test layout

File Focus
conftest.py aura_home, run_aura, example runner
test_core.py Registry, spine, constraints, session export (v0.1)
test_v02.py Sequencer, observers, membrane, Skillware host
test_v03.py Identity, audit report, hash chain, compare
test_cli.py aura CLI commands and exit codes
test_core_gaps.py Config, exporters, runtime, middleware, archive, tamper, compare edge cases (GH #4)
test_identity.py Operator identity adapters, redaction, OTel operator attrs, identity.bound (GH #55)
test_session_invariants.py Session lifecycle, atomic export, closed-session errors (GH #15)
test_examples_smoke.py Runnable example scripts
test_capability_broker.py Capabilities, SecretBroker, redaction, CLI (#48)
test_capability_stress_sim.py scripts/aura_capability_stress_sim.py gate
test_example_14_capability_broker.py Example 14 scenario assertions

What we test

Area Tests
Identity ULID ids, agent_ref, custom aura_id, resolve lookup, legacy AURA-000n, archive; optional operator adapters + export redaction (test_identity.py)
Audit Hash chain (valid + tamper), audit report, approver principal, session export
Core Registry, spine, constraints, conformance, sequencer (test_core.py, test_v02.py)
CLI Version, agent CRUD, run, logs, export, export-otel, compare, identity show (test_cli.py)
Config / runtime YAML merge, run_script, middleware, session modes (test_core_gaps.py)
Compare / OTel Summary diff incl. agent_ref + hash_chain_valid, OTel JSONL export (test_v03.py, test_core_gaps.py)
Examples Smoke run all examples/*/main.py — 14 numbered folders (test_examples_smoke.py)
Skillware Live registry skills via test_skillware_integration.py (CI skillware-live job)
Integration tests/integration/ — Ollama + Skillware + example 06 (local only)

Pre-PR checklist

  1. pytest --ignore=tests/integration
  2. black aura tests integrations examples (no diff)
  3. flake8 aura tests integrations examples
  4. CHANGELOG entry under [Unreleased] or release section
  5. Docs updated if behavior or CLI changed