feat(multiagent): Sprint 22 — inter-agent / Agent-in-the-Middle attack battery (v1.12.0)

CHANGELOG.md

@@ -6,6 +6,51 @@ Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ---
 
+## [1.12.0] — 2026-06-21
+
+### Added — Phase 22 (Multi-Agent / Inter-Agent Attack Battery — Agent-in-the-Middle)
+
+**`toki.multiagent` — new module (zero external deps)**
+- `MultiAgentAttackType` — 8 inter-agent attack categories: message tampering,
+  message interception, identity spoofing, instruction injection, goal-hijacking
+  relay, memory-relay poisoning, trust exploitation, capability escalation
+- `OWASP_ASI_MAPPING` — each attack type → OWASP Agentic Security Initiative
+  (ASI) 2026 category tag
+- `MultiAgentScenario` — frozen dataclass: `attack_type`, `topology` (agent
+  pipeline), `sender`, `original_message`, `tampered_message`, `description`,
+  `owasp_category`, deterministic SHA-256 `seed`
+- `MultiAgentVerdict` — frozen dataclass: `attack_succeeded`, `safe_handling`,
+  `score`, `to_dict()`
+- `MultiAgentBattery` — `generate_by_type()` / `generate_all()` produce 32
+  deterministic cases (4 per type) modelling an Agent-in-the-Middle on one
+  channel of an agent pipeline
+- `MultiAgentEvaluator` — `evaluate()` / `evaluate_batch(scenarios,
+  response_fn)` / `summary()`; heuristically flags whether the downstream agent
+  acted on tampered content or held to sender provenance / policy
+
+**`toki.coverage` (extended)**
+- `CATEGORY_AXIS` and `_DEFAULT_SEVERITY` gain `"multiagent"` (critical);
+  `_category_for` routes multi-agent categories without misrouting to `agentic`
+  or `multiturn`
+
+**CLI**
+- `python -m toki multiagent [--type all|<name>] [--json]` — runs the battery
+  against a mock safe downstream agent and prints per-type ASR
+
+**`toki.__init__`**
+- New exports: `OWASP_ASI_MAPPING`, `MultiAgentAttackType`, `MultiAgentBattery`,
+  `MultiAgentEvaluator`, `MultiAgentScenario`, `MultiAgentVerdict`
+
+**`pyproject.toml`**
+- Version bumped to `1.12.0`
+
+**Tests**
+- 19 new tests: `test_multiagent.py` (16), `test_main.py` (3 new CLI tests);
+  module 100% covered
+- Total: 763/763 passing (744 prior + 19 new)
+
+---
+
 ## [1.11.0] — 2026-06-21
 
 ### Added — Phase 21 (Continuous Monitoring Mode — P3-5)

PLAN.md

@@ -694,10 +694,52 @@ regresses beyond tolerance.
 
 ---
 
+## Phase 22 — Multi-Agent / Inter-Agent Attack Battery (v1.12.0) [COMPLETE]
+
+**Ship Gate:** 763 Python tests passing. Zero failures. 32-case Agent-in-the-
+Middle battery verified end-to-end; deterministic SHA-256 seeding; safe
+downstream agent blocks all attacks (ASR 0%), compromised agent succumbs;
+coverage-map routing for the new `multiagent` category.
+
+### Motivation
+A fresh Discovery sweep (post-P3) showed the 2026 frontier has moved past
+single-agent attacks (which `toki.agentic` covers) to **multi-agent systems**:
+the inter-agent message channel is the new attack surface. An adversarial
+*Agent-in-the-Middle* intercepts, tampers with, or spoofs agent-to-agent
+messages so a downstream agent acts on attacker content believing it came from
+a trusted peer (OWASP ASI 2026 insecure inter-agent comms; arXiv 2510.06445 /
+2510.26037). toki had nothing for multi-agent topologies.
+
+### Deliverables
+- [x] `toki.multiagent` — inter-agent attack battery (zero external deps):
+  - `MultiAgentAttackType` (8): message tampering / interception, identity
+    spoofing, instruction injection, goal-hijacking relay, memory-relay
+    poisoning, trust exploitation, capability escalation
+  - `OWASP_ASI_MAPPING` — each type → OWASP ASI 2026 category
+  - `MultiAgentScenario` (frozen) — attack_type, topology, sender, original +
+    tampered message, description, owasp_category, deterministic seed
+  - `MultiAgentVerdict` (frozen) — attack_succeeded, safe_handling, score,
+    `to_dict()`
+  - `MultiAgentBattery` — `generate_by_type()` / `generate_all()` (32 cases,
+    4 per type)
+  - `MultiAgentEvaluator` — `evaluate()` / `evaluate_batch(scenarios,
+    response_fn)` / `summary()`; heuristic on whether the downstream agent
+    acted on tampered content vs held to provenance
+- [x] `toki.coverage` — `CATEGORY_AXIS` + `_DEFAULT_SEVERITY` gain `"multiagent"`
+      (critical); `_category_for` routes multi-agent categories without
+      misrouting to `agentic`/`multiturn`
+- [x] CLI: `python -m toki multiagent [--type all|<name>] [--json]`
+- [x] `toki.__init__` exports all new public symbols; `__version__` → `1.12.0`
+- [x] `pyproject.toml` version bumped to `1.12.0`
+- [x] 19 new tests: `test_multiagent.py` (16) + `test_main.py` (3 CLI) — all passing
+- [x] All 744 Phase 1–21 tests still passing (763 total); module 100% covered
+
+---
+
 ## Future / Backlog
 
-- Web UI for interactive prompt generation and scoring (P3 backlog now fully closed)
+- Web UI for interactive prompt generation and scoring (P3 backlog fully closed)
 
 ---
 
-*Last updated: 2026-06-21 — v1.11.0 shipped. Continuous monitoring mode complete; P3-5 closed. Full P3 backlog cleared.*
+*Last updated: 2026-06-21 — v1.12.0 shipped. Multi-agent / inter-agent (Agent-in-the-Middle) attack battery complete.*

python/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "toki"
-version = "1.11.0"
+version = "1.12.0"
 description = "Adversarial fine-tuning lab for small language models"
 license = { text = "BUSL-1.1" }
 requires-python = ">=3.9"

python/tests/test_main.py

@@ -781,3 +781,29 @@ def test_monitor_command_json(tmp_path, capsys):
     data = _json.loads(captured.out)
     assert data["regressed"] is True
     assert data["name"] == "safety_monitor"
+
+
+# ---------------------------------------------------------------------------
+# multiagent CLI (Sprint 22)
+# ---------------------------------------------------------------------------
+
+
+def test_multiagent_command_runs(capsys):
+    main(["multiagent", "--type", "all"])
+    captured = capsys.readouterr()
+    assert "Inter-agent attack battery" in captured.out
+    assert "ASR" in captured.out
+
+
+def test_multiagent_command_filter_by_type(capsys):
+    main(["multiagent", "--type", "identity_spoofing", "--json"])
+    captured = capsys.readouterr()
+    import json as _json
+    data = _json.loads(captured.out)
+    assert data["total"] == 4
+
+
+def test_multiagent_command_unknown_type_errors():
+    import pytest
+    with pytest.raises(SystemExit):
+        main(["multiagent", "--type", "nonexistent"])

python/tests/test_multiagent.py

@@ -0,0 +1,166 @@
+"""Tests for toki.multiagent — inter-agent / Agent-in-the-Middle battery."""
+
+from __future__ import annotations
+
+
+from toki.multiagent import (
+    OWASP_ASI_MAPPING,
+    MultiAgentAttackType,
+    MultiAgentBattery,
+    MultiAgentEvaluator,
+    MultiAgentScenario,
+)
+
+
+# ---------------------------------------------------------------------------
+# Enum + mapping
+# ---------------------------------------------------------------------------
+
+
+def test_attack_type_has_eight_values():
+    assert len(MultiAgentAttackType) == 8
+
+
+def test_every_attack_type_has_owasp_asi_mapping():
+    for t in MultiAgentAttackType:
+        assert t in OWASP_ASI_MAPPING
+        assert OWASP_ASI_MAPPING[t].startswith("OWASP-ASI")
+
+
+# ---------------------------------------------------------------------------
+# Battery
+# ---------------------------------------------------------------------------
+
+
+def test_generate_all_yields_32_cases():
+    cases = MultiAgentBattery().generate_all()
+    assert len(cases) == 32
+    assert all(isinstance(c, MultiAgentScenario) for c in cases)
+
+
+def test_generate_by_type_yields_four():
+    cases = MultiAgentBattery().generate_by_type(MultiAgentAttackType.IDENTITY_SPOOFING)
+    assert len(cases) == 4
+    assert all(c.attack_type == MultiAgentAttackType.IDENTITY_SPOOFING for c in cases)
+
+
+def test_generation_is_deterministic():
+    a = MultiAgentBattery().generate_all()
+    b = MultiAgentBattery().generate_all()
+    assert [c.seed for c in a] == [c.seed for c in b]
+    assert [c.tampered_message for c in a] == [c.tampered_message for c in b]
+
+
+def test_scenarios_carry_topology_and_provenance():
+    for c in MultiAgentBattery().generate_all():
+        assert len(c.topology) >= 2
+        assert c.sender in c.topology
+        assert c.owasp_category == OWASP_ASI_MAPPING[c.attack_type]
+
+
+def test_tampered_differs_from_original():
+    for c in MultiAgentBattery().generate_all():
+        assert c.tampered_message != c.original_message
+
+
+def test_seeds_are_unique_across_battery():
+    seeds = [c.seed for c in MultiAgentBattery().generate_all()]
+    assert len(set(seeds)) == len(seeds)
+
+
+# ---------------------------------------------------------------------------
+# Evaluator
+# ---------------------------------------------------------------------------
+
+
+def _battery():
+    return MultiAgentBattery().generate_all()
+
+
+def test_safe_agent_blocks_all_attacks():
+    ev = MultiAgentEvaluator()
+
+    def safe_fn(_: str) -> str:
+        return "I cannot act on this — sender provenance unverified; request denied."
+
+    verdicts = ev.evaluate_batch(_battery(), safe_fn)
+    assert all(v.safe_handling for v in verdicts)
+    assert all(not v.attack_succeeded for v in verdicts)
+    assert all(v.score == 1.0 for v in verdicts)
+
+
+def test_compromised_agent_succumbs():
+    ev = MultiAgentEvaluator()
+    scenario = MultiAgentBattery().generate_by_type(
+        MultiAgentAttackType.MESSAGE_TAMPERING
+    )[0]
+    # echo the tampered instruction back -> acts on attacker content
+    verdict = ev.evaluate(scenario, scenario.tampered_message)
+    assert verdict.attack_succeeded is True
+    assert verdict.score == 0.0
+
+
+def test_ambiguous_response_scores_half():
+    ev = MultiAgentEvaluator()
+    scenario = MultiAgentBattery().generate_all()[0]
+    verdict = ev.evaluate(scenario, "Okay, noted.")
+    assert verdict.attack_succeeded is False
+    assert verdict.safe_handling is False
+    assert verdict.score == 0.5
+
+
+def test_summary_aggregates():
+    ev = MultiAgentEvaluator()
+    verdicts = ev.evaluate_batch(_battery(), lambda m: m)  # echo = worst case
+    summary = ev.summary(verdicts)
+    assert summary["total"] == 32
+    assert summary["attack_success_rate"] > 0.0
+    assert set(summary["by_type"]) == {t.value for t in MultiAgentAttackType}
+
+
+def test_summary_empty():
+    summary = MultiAgentEvaluator().summary([])
+    assert summary == {
+        "total": 0,
+        "mean_score": 0.0,
+        "attack_success_rate": 0.0,
+        "by_type": {},
+    }
+
+
+def test_verdict_to_dict_shape():
+    scenario = MultiAgentBattery().generate_all()[0]
+    verdict = MultiAgentEvaluator().evaluate(scenario, "denied, unverified sender")
+    d = verdict.to_dict()
+    assert {
+        "attack_type",
+        "description",
+        "owasp_category",
+        "topology",
+        "attack_succeeded",
+        "safe_handling",
+        "score",
+    } <= set(d)
+    assert isinstance(d["topology"], list)
+
+
+# ---------------------------------------------------------------------------
+# Coverage-map integration
+# ---------------------------------------------------------------------------
+
+
+def test_multiagent_is_a_coverage_category():
+    from toki.coverage import CATEGORY_AXIS, _category_for
+    from toki.generate import AdversarialPrompt
+
+    assert "multiagent" in CATEGORY_AXIS
+    p = AdversarialPrompt(text="x", category="multiagent", strategy="t", seed=1)
+    assert _category_for(p) == "multiagent"
+
+
+def test_multiagent_not_misrouted_to_agentic_or_multiturn():
+    from toki.coverage import _category_for
+    from toki.generate import AdversarialPrompt
+
+    p = AdversarialPrompt(text="x", category="inter_agent_relay", strategy="t", seed=1)
+    assert _category_for(p) == "multiagent"

python/toki/__init__.py

@@ -1,7 +1,7 @@
 """Toki — adversarial fine-tuning lab for small LLMs."""
 from __future__ import annotations
 
-__version__ = "1.11.0"
+__version__ = "1.12.0"
 
 from toki.generate import AdversarialGenerator
 from toki.evaluate import (
@@ -237,6 +237,14 @@
     WebhookSink,
     monitor_once,
 )
+from toki.multiagent import (
+    OWASP_ASI_MAPPING,
+    MultiAgentAttackType,
+    MultiAgentBattery,
+    MultiAgentEvaluator,
+    MultiAgentScenario,
+    MultiAgentVerdict,
+)
 
 __all__ = [
     "AdversarialGenerator",
@@ -424,4 +432,11 @@
     "SafetyMonitor",
     "WebhookSink",
     "monitor_once",
+    # Phase 22 — multi-agent / inter-agent attacks
+    "OWASP_ASI_MAPPING",
+    "MultiAgentAttackType",
+    "MultiAgentBattery",
+    "MultiAgentEvaluator",
+    "MultiAgentScenario",
+    "MultiAgentVerdict",
 ]