diff --git a/.github/workflows/ci-gate.yml b/.github/workflows/ci-gate.yml index a1350500..17cd6da1 100644 --- a/.github/workflows/ci-gate.yml +++ b/.github/workflows/ci-gate.yml @@ -142,3 +142,27 @@ jobs: - name: Check governance evidence-first contract (structure + semantics) run: ./scripts/ci/check_governance_evidence_truth.sh + + p0-gate: + name: p0-gate + runs-on: ubuntu-latest + needs: [lint, build] + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install p0-gate dependencies + run: | + python -m pip install --upgrade pip + python -m pip install pytest pytest-asyncio langchain-openai langchain-core httpx starlette uvicorn + + - name: Install project editable package + run: python -m pip install -e . --no-deps + + - name: Run p0 conformance gate + run: python scripts/ci/p0_gate.py diff --git a/docs/README.md b/docs/README.md index 3710e38a..57b08ef8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -42,6 +42,7 @@ ├── guides/Development_Constraints.md (开发约束清单) ├── guides/Documentation_First_Development_SOP.md (文档先行 SOP,Bug/Feature/Refactor 必走) ├── guides/Evidence_Truth_Implementation_Strategy.md (Evidence Truth 固化策略与 CI 落地计划) + ├── guides/P0_Gate_Runbook.md (P0 gate 本地运行、发布归档、flaky 处理操作手册) ├── governance/Documentation_Management_Model.md (文档目录分层、生命周期、OpenSpec/TODO 双模式协作) ├── features/README.md (特性聚合文档规范与归档规则) └── design/Design_Doc_Minimum_Standard.md (设计文档最小完备标准) @@ -97,6 +98,7 @@ | `guides/Development_Constraints.md` | 开发约束:架构不破坏、测试必备、日志/命名/复用/信任边界等硬性要求 | | `guides/Documentation_First_Development_SOP.md` | 文档先行 SOP:先设计文档、再 gap 分析、再 TODO、再 OpenSpec 修复、再归档 | | `guides/Evidence_Truth_Implementation_Strategy.md` | Evidence Truth 固化策略:证据结构、审计要求、CI 分阶段落地 | +| `guides/P0_Gate_Runbook.md` | P0 gate 操作手册:本地执行、失败分诊、发布归档、flaky 策略 | | `governance/Documentation_Management_Model.md` | 文档管理模型:目录分层、文档类型规则、生命周期依赖、OpenSpec/无 OpenSpec 协作 | | `features/README.md` | 特性聚合文档规范:单一状态源、证据回写与归档迁移 | | `guides/Team_Agent_Collab_Playbook.md` | 团队并行开发协作手册,含 spec-driven 认领粒度与 execution board 规则 | diff --git a/docs/features/p0-conformance-gate.md b/docs/features/p0-conformance-gate.md new file mode 100644 index 00000000..feb6a290 --- /dev/null +++ b/docs/features/p0-conformance-gate.md @@ -0,0 +1,108 @@ +--- +change_ids: ["p0-conformance-gate"] +doc_kind: feature +topics: ["p0", "conformance", "ci-gate", "runtime-validation"] +created: 2026-03-03 +updated: 2026-03-03 +status: active +mode: openspec +--- + +# Feature: p0-conformance-gate + +## Scope + +将 P0 运行时不变量收敛成统一的 `p0-gate` 质量门禁,覆盖安全门控、`step_driven` 执行闭环、默认事件链审计三类关键约束,并把 CI / 发布流程接入这条硬门槛。 + +## OpenSpec Artifacts + +- Proposal: `openspec/changes/p0-conformance-gate/proposal.md` +- Design: `openspec/changes/p0-conformance-gate/design.md` +- Specs: + - `openspec/changes/p0-conformance-gate/specs/p0-conformance-gate/spec.md` + - `openspec/changes/p0-conformance-gate/specs/validation/spec.md` + - `openspec/changes/p0-conformance-gate/specs/core-runtime/spec.md` +- Tasks: `openspec/changes/p0-conformance-gate/tasks.md` + +## Governance Anchors + +- `docs/guides/Development_Constraints.md` +- `docs/guides/Documentation_First_Development_SOP.md` +- `openspec/specs/core-runtime/spec.md` +- `openspec/specs/validation/spec.md` + +## Evidence + +### Commands + +- `git fetch origin` +- `git worktree add .worktrees/p0-conformance-gate -b codex/p0-conformance-gate origin/main` +- `../../.venv/bin/python -m pytest -q tests/unit/test_transport_adapters.py tests/unit/test_interaction_dispatcher.py tests/unit/test_transport_channel.py tests/integration/test_client_cli_flow.py tests/unit/test_examples_cli.py tests/unit/test_examples_cli_mcp.py` +- `../../.venv/bin/python -m pytest -q tests/integration/test_security_policy_gate_flow.py` +- `../../.venv/bin/python -m pytest -q tests/integration/test_p0_conformance_gate.py` +- `../../.venv/bin/python -m pytest -q tests/integration/test_security_policy_gate_flow.py tests/integration/test_p0_conformance_gate.py tests/unit/test_dare_agent_step_driven_mode.py` +- `../../.venv/bin/python -m pytest -q tests/unit/test_dare_agent_security_policy_gate.py tests/unit/test_dare_agent_security_boundary.py tests/unit/test_five_layer_agent.py` +- `../../.venv/bin/python -m pytest -q tests/unit/test_p0_gate_ci.py` +- `../../.venv/bin/python scripts/ci/p0_gate.py` +- `openspec validate p0-conformance-gate --type change --strict --json --no-interactive` +- `./scripts/ci/check_governance_evidence_truth.sh` + +### Results + +- `git fetch origin`: confirmed `origin/main` advanced to merge commit `36c8b38`, which includes the completed archive closeout for `refactor-dare-agent-structure-split` and provides the clean baseline for the next active change. +- `git worktree add .worktrees/p0-conformance-gate -b codex/p0-conformance-gate origin/main`: created an isolated continuation workspace for the next active change directly from merged `main`. +- `../../.venv/bin/python -m pytest -q tests/unit/test_transport_adapters.py tests/unit/test_interaction_dispatcher.py tests/unit/test_transport_channel.py tests/integration/test_client_cli_flow.py tests/unit/test_examples_cli.py tests/unit/test_examples_cli_mcp.py`: passed (`76 passed, 1 warning`) as the current baseline for the already-landed P0-related contract coverage recorded under tasks `2.4` and `5.1`. +- `../../.venv/bin/python -m pytest -q tests/integration/test_security_policy_gate_flow.py`: passed (`3 passed, 1 warning`) after extending the integration file to cover the full security gate decision surface: direct allow, direct deny with structured `not_allow`, and approval-required escalation. +- `../../.venv/bin/python -m pytest -q tests/integration/test_p0_conformance_gate.py`: passed (`2 passed, 1 warning`) after adding the missing step-driven integration anchor that exercises the full `agent("task")` closed loop, covering both ordered happy-path execution and fail-fast behavior when the first validated step fails. +- `../../.venv/bin/python -m pytest -q tests/integration/test_security_policy_gate_flow.py tests/integration/test_p0_conformance_gate.py tests/unit/test_dare_agent_step_driven_mode.py`: passed (`28 passed, 1 warning`) after the new `p0` integration file also absorbed the default event-log replay/hash-chain runtime anchor, confirming the security gate slice, step-driven closed-loop slice, and audit-chain slice can run together as the emerging P0 gate bundle. +- `../../.venv/bin/python -m pytest -q tests/unit/test_dare_agent_security_policy_gate.py tests/unit/test_dare_agent_security_boundary.py tests/unit/test_five_layer_agent.py`: passed (`50 passed, 1 warning`) after the new security-gate integration coverage landed, confirming the added integration assertions do not regress the existing direct runtime and no-planner approval semantics. +- `../../.venv/bin/python -m pytest -q tests/unit/test_p0_gate_ci.py`: passed (`4 passed`) after extending the CI-side unit contract so `p0-gate` also preserves node ids from pytest `ERROR` summary lines, not only assertion-style `FAILED` lines. +- `../../.venv/bin/python scripts/ci/p0_gate.py`: passed and emitted: + `p0-gate: PASS` + `- SECURITY_REGRESSION: 0 failures` + `- STEP_EXEC_REGRESSION: 0 failures` + `- AUDIT_CHAIN_REGRESSION: 0 failures` + which confirms the repository now has a single deterministic command entrypoint for the three frozen P0 categories. +- `openspec validate p0-conformance-gate --type change --strict --json --no-interactive`: passed (`1/1` change valid, `0` issues) after restoring the missing active feature aggregation record for this change. +- `./scripts/ci/check_governance_evidence_truth.sh`: initially failed because the restored feature doc lacked historical PR/review links; after linking the already-landed P0 evidence PRs, the governance gate passed, and remained green after task `1.1-1.3` synchronized the gate scope matrix and rollout contract into the active docs/spec set. +- `./scripts/ci/check_governance_evidence_truth.sh`: remained green after adding `docs/guides/P0_Gate_Runbook.md` plus the new navigation links in `docs/README.md` and `docs/guides/Team_Agent_Collab_Playbook.md`, confirming the operationalization docs did not break the governance acceptance pack. + +### Behavior Verification + +- Happy path: the existing contract-focused transport / interaction / example CLI suites still pass from a clean `origin/main` baseline, so the previously landed P0 contract assertions remain intact before new gate work begins. +- Error branch: the baseline suite still includes the approval action normalization and structured failure-contract assertions captured by tasks `2.4` and `5.1`, which are the current minimal regression anchors for `p0-gate`. +- Happy path: task `1.1-1.3` now defines a stable three-category matrix with explicit ownership modules, anchor suites, and required-mode thresholds, so later CI wiring can attach to a fixed scope instead of an ad hoc test grab-bag. +- Error branch: the rollout contract now explicitly blocks promoting `p0-gate` to a required check if any category lacks a green anchor set or emits uncategorized failures, preventing partial rollout from being misread as full P0 coverage. +- Happy path: task `2.1` now proves the security gate allows low-risk tool execution without creating pending approvals and still preserves the existing approval-required flow for high-risk execution. +- Error branch: task `2.1` now proves a denied capability is blocked before gateway invocation and recorded as a structured `not_allow` tool result inside the full agent flow, not only in direct `_run_tool_loop` unit tests. +- Happy path: task `2.2` now proves `step_driven` mode can complete a full session-loop closed loop with a planner and validator, preserve step order, and pass `_previous_output` from the first validated step into the second step during actual runtime execution. +- Error branch: task `2.2` now proves a failed first validated step aborts the remaining step sequence within the same milestone attempt and reaches `verify_milestone` as a structured failed run result instead of continuing to later steps. +- Happy path: task `2.3` now proves the default SQLite runtime event log can replay a real session window from `session.start` through later runtime events while keeping per-event `task_id` / `run_id` / `session_id` correlation intact. +- Error branch: task `2.3` now proves the same runtime-backed SQLite log fails `verify_chain()` after on-disk payload tampering, so the audit-chain invariant is validated against actual session data rather than only synthetic unit fixtures. +- Happy path: task `3.1` now exposes a single CI command entrypoint, `python scripts/ci/p0_gate.py`, that runs the three category bundles without relying on ad hoc workflow-local command duplication. +- Error branch: task `3.3` now guarantees failed `p0-gate` runs produce deterministic category-tagged triage output with failing node ids, module ownership, and first-action guidance instead of raw pytest noise alone. +- Error branch: the latest PR review fix now keeps deterministic node ids even when pytest stops in collection/import/runtime with `ERROR` summary lines, so CI triage does not collapse to `` for non-assertion failures. +- Happy path: tasks `4.1-4.3` now publish a single runbook that tells contributors exactly how to run `p0-gate`, where to look first for each category, how to archive release evidence, and how to record flaky incidents without inventing a second workflow. +- Error branch: the runbook now forbids silent anchor removal and silent rerun-based “fixes”; failed `p0-gate` runs must either be repaired or escalated through the documented flaky/quarantine path with owner and expiry. + +### Risks and Rollback + +- Risk: this change currently has partial task completion but had no active feature aggregation record, which weakens traceability until the governance baseline is restored. +- Risk: task `3.2` still depends on repo-admin branch protection / ruleset changes outside the repository, so `p0-gate` is not yet a true protected-branch merge blocker even though the job and summary contract now exist in-tree. +- Risk: `3.2` is now the only remaining open task in this change, and it cannot be completed from the repository contents alone. +- Rollback: if the new `p0-gate` workflow job proves too noisy before repo-admin rollout, remove the job entry from `.github/workflows/ci-gate.yml` and keep `scripts/ci/p0_gate.py` plus the runbook as local-only tooling until the category bundle is re-tuned. +- Rollback: no runtime behavior changed in this kickoff step; reverting only removes the restored governance record for the active change. + +### Review and Merge Gate Links + +- Current continuation branch: `codex/p0-conformance-gate` +- Current implementation PR: `https://github.com/zts212653/Deterministic-Agent-Runtime-Engine/pull/172` +- Historical PR for task `2.4`: `https://github.com/zts212653/Deterministic-Agent-Runtime-Engine/pull/130` +- Historical PR for task `5.1` baseline recovery path: `https://github.com/zts212653/Deterministic-Agent-Runtime-Engine/pull/113` +- Historical review evidence: `https://github.com/zts212653/Deterministic-Agent-Runtime-Engine/pull/130#pullrequestreview-3872526843` +- Historical merge evidence: `https://github.com/zts212653/Deterministic-Agent-Runtime-Engine/pull/130` +- PR status for this continuation branch: implementation PR opened at `#172` + +## Next Milestone + +Schedule the repo-admin follow-up for task `3.2`: add `p0-gate` to the protected-branch required checks / ruleset after this change merges. diff --git a/docs/governance/branch-protection.md b/docs/governance/branch-protection.md index 201f14bc..ece18bd6 100644 --- a/docs/governance/branch-protection.md +++ b/docs/governance/branch-protection.md @@ -30,6 +30,11 @@ If your GitHub plan supports merge queue: - Phase 1 (now, required): `lint`, `build` - Phase 2 (observe first, then required): `smoke-tests` - Phase 3 (after 1-2 stable weeks, then required): `risk-matrix`, `test-skip-guard`, `lockfile-policy` +- Phase 4 (planned after `p0-conformance-gate` lands): `p0-gate` + - workflow entrypoint: `.github/workflows/ci-gate.yml` job `p0-gate` runs `python scripts/ci/p0_gate.py` + - promotion threshold: security / step-driven / audit category anchors all green in the same run + - summary contract: `p0-gate` must emit deterministic category labels and failing test/module pointers before it can become a required branch check + - repo-admin action: after this change merges, add `p0-gate` to the protected-branch required checks list / ruleset ## Fallback if Merge Queue Is Unavailable Use pre-merge combined checks: diff --git a/docs/guides/P0_Gate_Runbook.md b/docs/guides/P0_Gate_Runbook.md new file mode 100644 index 00000000..19bb3854 --- /dev/null +++ b/docs/guides/P0_Gate_Runbook.md @@ -0,0 +1,123 @@ +# P0 Gate Runbook + +> Scope: `p0-conformance-gate` operational usage after the category matrix, CI entrypoint, and summary contract have been frozen. + +## 1. Command of Record + +Run the gate from the repository root: + +```bash +.venv/bin/python scripts/ci/p0_gate.py +``` + +Expected success output: + +```text +p0-gate: PASS +- SECURITY_REGRESSION: 0 failures +- STEP_EXEC_REGRESSION: 0 failures +- AUDIT_CHAIN_REGRESSION: 0 failures +``` + +The same command is used by `.github/workflows/ci-gate.yml` job `p0-gate`. + +## 2. Category Mapping + +### SECURITY_REGRESSION + +Primary signal: +- `tests/integration/test_security_policy_gate_flow.py` +- `tests/unit/test_dare_agent_security_policy_gate.py` +- `tests/unit/test_dare_agent_security_boundary.py` +- `tests/unit/test_transport_adapters.py` +- `tests/unit/test_examples_cli.py` +- `tests/unit/test_examples_cli_mcp.py` + +Inspect first: +- `dare_framework/security/` +- `dare_framework/tool/_internal/governed_tool_gateway.py` +- `dare_framework/transport/_internal/adapters.py` +- `examples/05-dare-coding-agent-enhanced/cli.py` +- `examples/06-dare-coding-agent-mcp/cli.py` + +### STEP_EXEC_REGRESSION + +Primary signal: +- `tests/integration/test_p0_conformance_gate.py::test_step_driven_session_executes_validated_steps_in_order` +- `tests/integration/test_p0_conformance_gate.py::test_step_driven_session_stops_after_first_failed_step` +- `tests/unit/test_dare_agent_step_driven_mode.py` + +Inspect first: +- `dare_framework/agent/dare_agent.py` +- `dare_framework/agent/_internal/execute_engine.py` +- `dare_framework/plan/` + +### AUDIT_CHAIN_REGRESSION + +Primary signal: +- `tests/integration/test_p0_conformance_gate.py::test_default_event_log_replay_and_hash_chain_hold_for_runtime_session` +- `tests/unit/test_event_sqlite_event_log.py` +- `tests/unit/test_builder_security_boundary.py::test_default_event_log_replay_returns_ordered_session_window` + +Inspect first: +- `dare_framework/event/_internal/sqlite_event_log.py` +- `dare_framework/event/kernel.py` +- `dare_framework/observability/_internal/event_trace_bridge.py` +- `dare_framework/agent/builder.py` + +## 3. Local Troubleshooting Flow + +1. Run `.venv/bin/python scripts/ci/p0_gate.py`. +2. Record the first failing summary block before any rerun. +3. Rerun only the category-local anchors listed in the summary. +4. Fix the regression, then rerun the full `p0-gate` command. +5. Do not claim the gate is fixed until the full command returns `PASS`. + +If the summary itself looks malformed, rerun: + +```bash +.venv/bin/python -m pytest -q tests/unit/test_p0_gate_ci.py +``` + +That suite locks the `PASS/FAIL + category + tests + modules + action` contract. + +## 4. Release Archive Step + +Every release candidate, release PR, or final tag cut that relies on protected-branch quality gates MUST archive the latest `p0-gate` result. + +Minimum archive payload: +- exact command: `.venv/bin/python scripts/ci/p0_gate.py` +- exact summary text +- workflow/job URL when run in GitHub Actions +- release identifier (`tag`, `release PR`, or release issue) +- operator and timestamp + +Archive location rule: +- preferred: release PR description or release issue checklist +- acceptable fallback: release note draft section named `P0 Gate` + +If `p0-gate` is not green, the release must stop; do not archive a failed run as release evidence unless the release is explicitly aborted. + +## 5. Flaky Handling Policy + +P0 gate failures are blocker-grade until proven otherwise. Treat reruns as evidence collection, not as remediation. + +Rules: +1. One immediate rerun is allowed only after preserving the first failing summary. +2. A category may be called flaky only if the same node id both fails and passes without code changes. +3. Flaky suspicion requires an issue or TODO entry that records: + - failing node id + - category label + - first failed run link + - rerun result + - owner and expiry +4. A flaky anchor must not be silently removed from `scripts/ci/p0_gate.py`. +5. Any temporary downgrade or quarantine requires a docs-first change that names the replacement anchor or the rollback plan. + +Escalation threshold: +- two flaky incidents for the same node id within seven days triggers quarantine review +- quarantine review must finish within two business days + +## 6. Branch Protection Follow-up + +After `p0-conformance-gate` merges, a repository administrator still needs to add `p0-gate` to the protected branch required checks / ruleset. That step is outside the repository contents and is tracked separately in `docs/governance/branch-protection.md`. diff --git a/docs/guides/Team_Agent_Collab_Playbook.md b/docs/guides/Team_Agent_Collab_Playbook.md index 1c46ee5f..23e80a67 100644 --- a/docs/guides/Team_Agent_Collab_Playbook.md +++ b/docs/guides/Team_Agent_Collab_Playbook.md @@ -47,9 +47,11 @@ pytest -q tests/smoke -m smoke - `risk-matrix` - `test-skip-guard` - `lockfile-policy` +- `p0-gate`(已入 workflow;是否 required 取决于 branch protection rollout) 治理配置说明: - `docs/governance/branch-protection.md` +- `docs/guides/P0_Gate_Runbook.md` ## 4. 免费版主干护栏(main-guard) diff --git a/openspec/changes/p0-conformance-gate/design.md b/openspec/changes/p0-conformance-gate/design.md index 42ebdeec..223786f6 100644 --- a/openspec/changes/p0-conformance-gate/design.md +++ b/openspec/changes/p0-conformance-gate/design.md @@ -38,6 +38,47 @@ - 版本发布前重复执行并归档结果。 - 理由:把质量要求前置到提交阶段。 +### Decision 5: 先冻结 category matrix,再推动 CI required 化 +- 在 task `1.x` 先固定 `p0-gate` 的 category matrix:每类不变量都要明确当前 anchor tests、后续需要补的 integration anchors、责任模块、required-mode 阈值。 +- 在 task `3.x` 之前,不把 `p0-gate` 写成“当前已启用”的 branch rule,只把它定义为受控 rollout 的下一阶段 required check。 +- 理由:先把 scope 写清,再接 CI;否则后续新增测试时容易漂移成“任何测试都算 p0”。 + +## Gate Scope Matrix + +| Category | Invariant | Current anchor suites | Required new integration anchor | Responsibility modules | Required-mode threshold | +| --- | --- | --- | --- | --- | --- | +| `SECURITY_REGRESSION` | trust/policy/approval 决策必须在工具调用前生效,并保持 `allow/deny/approve_required` 语义稳定 | `tests/unit/test_dare_agent_security_boundary.py`, `tests/unit/test_transport_adapters.py`, `tests/unit/test_examples_cli.py`, `tests/unit/test_examples_cli_mcp.py` | `tests/integration/test_p0_conformance_gate.py` 中 security gate 场景,至少覆盖 `allow`, `deny`, `approve_required` 三条主链路 | `dare_framework/security/*`, `dare_framework/tool/_internal/governed_tool_gateway.py`, `dare_framework/transport/_internal/adapters.py`, `examples/05-dare-coding-agent-enhanced/cli.py`, `examples/06-dare-coding-agent-mcp/cli.py` | gate 选中的 security anchors 单次运行 `100%` 通过;任何未分类 security 失败都阻止升为 required | +| `STEP_EXEC_REGRESSION` | `step_driven` 执行必须保持按序执行、失败即停、并继续受安全策略约束 | `tests/unit/test_dare_agent_step_driven_mode.py` | `tests/integration/test_p0_conformance_gate.py` 中 step-driven 闭环场景,至少覆盖 happy path 与 fail-fast path | `dare_framework/agent/dare_agent.py`, `dare_framework/agent/_internal/execute_engine.py`, `dare_framework/plan/*` | gate 选中的 step-driven anchors 单次运行 `100%` 通过;任何顺序漂移或失败后继续执行都归为 blocker | +| `AUDIT_CHAIN_REGRESSION` | 默认 SQLite event log 的 append/hash-chain/replay 必须持续可验证,保证审计链可追溯 | `tests/unit/test_event_sqlite_event_log.py`, `tests/unit/test_builder_security_boundary.py` 中 default event log replay 覆盖 | `tests/integration/test_p0_conformance_gate.py` 中 audit 场景,至少覆盖 hash-chain verify 与 replay anchor | `dare_framework/event/_internal/sqlite_event_log.py`, `dare_framework/event/kernel.py`, `dare_framework/observability/_internal/event_trace_bridge.py`, builder wiring | gate 选中的 audit anchors 单次运行 `100%` 通过;任何 hash mismatch / replay 漏字段都阻止升为 required | + +## CI Summary Contract + +`p0-gate` 的标准 summary 采用按 category 分段的稳定文本格式,至少包含: +- gate 总体状态:`PASS` / `FAIL` +- category label +- failing test identifiers +- primary modules +- 建议先排查的入口 + +最小格式: + +```text +p0-gate: FAIL +- SECURITY_REGRESSION + tests: tests/integration/test_p0_conformance_gate.py::test_security_gate_blocks_merge + modules: dare_framework/security, dare_framework/tool/_internal/governed_tool_gateway.py + action: inspect trust/policy/approval flow before tool invocation +``` + +如果全部通过,则 summary 必须至少输出: + +```text +p0-gate: PASS +- SECURITY_REGRESSION: 0 failures +- STEP_EXEC_REGRESSION: 0 failures +- AUDIT_CHAIN_REGRESSION: 0 failures +``` + ## Risks / Trade-offs - [Risk] 新门禁增加 CI 时长,影响迭代速度。 diff --git a/openspec/changes/p0-conformance-gate/specs/p0-conformance-gate/spec.md b/openspec/changes/p0-conformance-gate/specs/p0-conformance-gate/spec.md index a7fa0d28..9955f291 100644 --- a/openspec/changes/p0-conformance-gate/specs/p0-conformance-gate/spec.md +++ b/openspec/changes/p0-conformance-gate/specs/p0-conformance-gate/spec.md @@ -4,6 +4,8 @@ The project SHALL define a `p0-gate` conformance checkpoint that validates P0 runtime invariants before merge to protected branches. - The gate MUST cover security policy gating, step-driven execution correctness, and event-log chain integrity. +- The gate scope MUST define a stable category matrix mapping each invariant class to explicit test anchors and responsibility modules. +- Promotion of `p0-gate` to a required protected-branch check MUST require all selected category anchors to pass in a single run. - The gate MUST be configured as a required CI check for protected branch merges. #### Scenario: Merge is blocked when p0-gate fails @@ -11,13 +13,23 @@ The project SHALL define a `p0-gate` conformance checkpoint that validates P0 ru - **WHEN** `p0-gate` job fails - **THEN** merge is blocked until the gate passes +#### Scenario: Required-mode threshold demands all category anchors pass +- **GIVEN** the project is promoting `p0-gate` to a required branch check +- **WHEN** any selected security, step-driven, or audit anchor test fails +- **THEN** the gate remains non-promotable +- **AND** the rollout does not treat partial category success as sufficient + ### Requirement: P0 gate failures MUST be classified deterministically The conformance gate SHALL classify failures into deterministic categories to accelerate triage. - Failure categories MUST include: `SECURITY_REGRESSION`, `STEP_EXEC_REGRESSION`, and `AUDIT_CHAIN_REGRESSION`. -- CI output MUST include failing test identifiers and category labels. +- CI output MUST include failing test identifiers, category labels, and the primary modules responsible for the failing category. #### Scenario: Security regression is labeled in CI output - **WHEN** a security gating invariant test fails - **THEN** CI summary marks the failure as `SECURITY_REGRESSION` +#### Scenario: CI summary includes module ownership for audit failure +- **WHEN** an event-log hash-chain or replay invariant fails +- **THEN** CI summary marks the failure as `AUDIT_CHAIN_REGRESSION` +- **AND** the summary identifies the primary audit-chain modules to inspect first diff --git a/openspec/changes/p0-conformance-gate/tasks.md b/openspec/changes/p0-conformance-gate/tasks.md index 28564eb4..a958434f 100644 --- a/openspec/changes/p0-conformance-gate/tasks.md +++ b/openspec/changes/p0-conformance-gate/tasks.md @@ -1,14 +1,30 @@ ## 1. Define Gate Scope -- [ ] 1.1 定义 `p0-gate` 覆盖的三类不变量与验收阈值。 -- [ ] 1.2 明确每类不变量对应的测试文件与责任模块。 -- [ ] 1.3 定义标准失败标签与 CI summary 输出格式。 +- [x] 1.1 定义 `p0-gate` 覆盖的三类不变量与验收阈值。 + Evidence: `openspec/changes/p0-conformance-gate/design.md` 已固化 `SECURITY_REGRESSION` / `STEP_EXEC_REGRESSION` / `AUDIT_CHAIN_REGRESSION` 三类 category matrix,并为 required-mode promotion 定义 “单次运行全绿” 阈值。 + Last Updated: `2026-03-03` +- [x] 1.2 明确每类不变量对应的测试文件与责任模块。 + Evidence: `openspec/changes/p0-conformance-gate/design.md` 的 Gate Scope Matrix 已列出现有 anchor suites、后续必须补的 integration anchors,以及各 category 的 primary ownership modules。 + Last Updated: `2026-03-03` +- [x] 1.3 定义标准失败标签与 CI summary 输出格式。 + Evidence: `openspec/changes/p0-conformance-gate/specs/p0-conformance-gate/spec.md` 与 `openspec/changes/p0-conformance-gate/design.md` 已同步固定 failure labels 与 summary contract;`docs/governance/branch-protection.md` 已把 `p0-gate` rollout 前提写成未来 required-check 条件。 + Commands: `openspec validate p0-conformance-gate --type change --strict --json --no-interactive` => `1/1` valid;`./scripts/ci/check_governance_evidence_truth.sh` => `passed` + Last Updated: `2026-03-03` ## 2. Build P0 Test Suite -- [ ] 2.1 新增集成测试覆盖安全门控主链路(allow/deny/approve_required)。 -- [ ] 2.2 新增集成测试覆盖 `step_driven` 执行闭环。 -- [ ] 2.3 新增集成测试覆盖默认 event log hash-chain/replay。 +- [x] 2.1 新增集成测试覆盖安全门控主链路(allow/deny/approve_required)。 + Evidence: `tests/integration/test_security_policy_gate_flow.py` 现在显式覆盖 read-only `allow`、`deny_capability_ids` 触发的 `not_allow`、以及 high-risk `approve_required -> grant` 主链路。 + Commands: `../../.venv/bin/python -m pytest -q tests/integration/test_security_policy_gate_flow.py` => `3 passed, 1 warning`;`../../.venv/bin/python -m pytest -q tests/unit/test_dare_agent_security_policy_gate.py tests/unit/test_dare_agent_security_boundary.py tests/unit/test_five_layer_agent.py` => `50 passed, 1 warning` + Last Updated: `2026-03-03` +- [x] 2.2 新增集成测试覆盖 `step_driven` 执行闭环。 + Evidence: `tests/integration/test_p0_conformance_gate.py` 现在通过完整 `agent("task")` 会话覆盖 `step_driven` 的闭环集成路径:`planner.decompose -> planner.plan -> validator.validate_plan -> execute -> validator.verify_milestone`,并显式验证 happy path 的顺序执行 / `_previous_output` 传递,以及 fail-fast path 在首个失败 step 后不会继续执行后续 step。 + Commands: `../../.venv/bin/python -m pytest -q tests/integration/test_p0_conformance_gate.py` => `2 passed, 1 warning` + Last Updated: `2026-03-03` +- [x] 2.3 新增集成测试覆盖默认 event log hash-chain/replay。 + Evidence: `tests/integration/test_p0_conformance_gate.py` 现在补齐默认 SQLite event log 的 runtime integration anchor:真实会话落盘后验证 `replay(from_event_id=session.start)` 返回同一 session window,并且 `verify_chain()` 在正常数据上通过、在篡改落盘 payload 后失败。 + Commands: `../../.venv/bin/python -m pytest -q tests/integration/test_p0_conformance_gate.py` => `3 passed, 1 warning` + Last Updated: `2026-03-03` - [x] 2.4 增加关键单测确保契约字段与错误码稳定。 Evidence: `dare_framework/transport/_internal/adapters.py`,`tests/unit/test_transport_adapters.py`(新增 slash 命令到 `resource:action` 的标准化与审批参数提取断言);`examples/05-dare-coding-agent-enhanced/cli.py`、`examples/06-dare-coding-agent-mcp/cli.py`(审批 action 调用统一为 `invoke(action, **params)`) Commands: `.venv/bin/pytest -q tests/unit/test_transport_adapters.py::test_stdio_slash_command_maps_to_resource_action_id tests/unit/test_transport_adapters.py::test_stdio_slash_command_extracts_approval_action_params` => `2 passed`;`.venv/bin/pytest -q tests/unit/test_transport_adapters.py tests/unit/test_interaction_dispatcher.py tests/unit/test_transport_channel.py tests/integration/test_client_cli_flow.py` => `33 passed, 1 warning`;`.venv/bin/pytest -q tests/unit/test_examples_cli.py tests/unit/test_examples_cli_mcp.py` => `22 passed, 1 warning` @@ -16,15 +32,31 @@ ## 3. CI Integration -- [ ] 3.1 在 CI workflow 增加 `p0-gate` job 与命令入口。 -- [ ] 3.2 将 `p0-gate` 配置为主分支 required check。 -- [ ] 3.3 输出标准化门禁报告(通过率、失败类型、建议排查点)。 +- [x] 3.1 在 CI workflow 增加 `p0-gate` job 与命令入口。 + Evidence: `.github/workflows/ci-gate.yml` 已新增 `p0-gate` job,并统一调用 `python scripts/ci/p0_gate.py` 作为 CI 入口;`scripts/ci/p0_gate.py` 固化了三类 category 的 gate bundle。 + Commands: `../../.venv/bin/python scripts/ci/p0_gate.py` => `p0-gate: PASS` + Last Updated: `2026-03-03` +- [ ] 3.2 将 `p0-gate` 配置为主分支 required check。 + Note: 该项需要 GitHub branch protection / ruleset 管理员权限;当前仓库内已完成 job 名称与 rollout 文档对齐,但尚未执行远端仓库设置。 +- [x] 3.3 输出标准化门禁报告(通过率、失败类型、建议排查点)。 + Evidence: `scripts/ci/p0_gate.py` 的 `format_summary()` 已固定 `PASS/FAIL + category label + failing tests + modules + action` 文本格式,并写入 `GITHUB_STEP_SUMMARY`;`tests/unit/test_p0_gate_ci.py` 锁定该 summary contract。 + Commands: `../../.venv/bin/python -m pytest -q tests/unit/test_p0_gate_ci.py` => `3 passed`;`../../.venv/bin/python scripts/ci/p0_gate.py` => `p0-gate: PASS` + Last Updated: `2026-03-03` ## 4. Operationalization -- [ ] 4.1 更新开发文档,说明本地运行与故障排查流程。 -- [ ] 4.2 在发布流程增加 `p0-gate` 结果归档步骤。 -- [ ] 4.3 制定 flaky 用例处理规则与时限。 +- [x] 4.1 更新开发文档,说明本地运行与故障排查流程。 + Evidence: `docs/guides/P0_Gate_Runbook.md` 已定义 command-of-record、本地分诊顺序、按 category 的定位入口;`docs/guides/Team_Agent_Collab_Playbook.md` 与 `docs/README.md` 已补入口链接。 + Commands: `../../.venv/bin/python scripts/ci/p0_gate.py` => `p0-gate: PASS`;`./scripts/ci/check_governance_evidence_truth.sh` => `passed` + Last Updated: `2026-03-03` +- [x] 4.2 在发布流程增加 `p0-gate` 结果归档步骤。 + Evidence: `docs/guides/P0_Gate_Runbook.md` 已固定 release archive 的最小字段、归档位置规则与“失败即停止发布”要求。 + Commands: `./scripts/ci/check_governance_evidence_truth.sh` => `passed` + Last Updated: `2026-03-03` +- [x] 4.3 制定 flaky 用例处理规则与时限。 + Evidence: `docs/guides/P0_Gate_Runbook.md` 已定义 flaky 认定条件、一次性 rerun 上限、issue/TODO 记录字段、7 天/2 个 incident 触发的 quarantine review、以及 2 个工作日内的处理时限。 + Commands: `./scripts/ci/check_governance_evidence_truth.sh` => `passed` + Last Updated: `2026-03-03` ## 5. Baseline Recovery Evidence diff --git a/scripts/ci/p0_gate.py b/scripts/ci/p0_gate.py new file mode 100644 index 00000000..cd28ee19 --- /dev/null +++ b/scripts/ci/p0_gate.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +"""Run the P0 conformance gate and emit a deterministic summary.""" + +from __future__ import annotations + +import os +import re +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path + +FAILED_TEST_RE = re.compile(r"^(?:FAILED|ERROR)\s+([^\s]+)") + + +@dataclass(frozen=True, slots=True) +class CategorySpec: + label: str + tests: list[str] + modules: list[str] + action: str + + +@dataclass(frozen=True, slots=True) +class CategoryResult: + spec: CategorySpec + passed: bool + failed_tests: list[str] + raw_output: str + + +DEFAULT_CATEGORY_SPECS: tuple[CategorySpec, ...] = ( + CategorySpec( + label="SECURITY_REGRESSION", + tests=[ + "tests/integration/test_security_policy_gate_flow.py", + "tests/unit/test_dare_agent_security_policy_gate.py", + "tests/unit/test_dare_agent_security_boundary.py", + "tests/unit/test_transport_adapters.py", + "tests/unit/test_examples_cli.py", + "tests/unit/test_examples_cli_mcp.py", + ], + modules=[ + "dare_framework/security", + "dare_framework/tool/_internal/governed_tool_gateway.py", + "dare_framework/transport/_internal/adapters.py", + "examples/05-dare-coding-agent-enhanced/cli.py", + "examples/06-dare-coding-agent-mcp/cli.py", + ], + action="inspect trust/policy/approval flow before tool invocation", + ), + CategorySpec( + label="STEP_EXEC_REGRESSION", + tests=[ + "tests/integration/test_p0_conformance_gate.py::test_step_driven_session_executes_validated_steps_in_order", + "tests/integration/test_p0_conformance_gate.py::test_step_driven_session_stops_after_first_failed_step", + "tests/unit/test_dare_agent_step_driven_mode.py", + ], + modules=[ + "dare_framework/agent/dare_agent.py", + "dare_framework/agent/_internal/execute_engine.py", + "dare_framework/plan", + ], + action="inspect step execution order, fail-fast handling, and validated-plan routing", + ), + CategorySpec( + label="AUDIT_CHAIN_REGRESSION", + tests=[ + "tests/integration/test_p0_conformance_gate.py::test_default_event_log_replay_and_hash_chain_hold_for_runtime_session", + "tests/unit/test_event_sqlite_event_log.py", + "tests/unit/test_builder_security_boundary.py::test_default_event_log_replay_returns_ordered_session_window", + ], + modules=[ + "dare_framework/event/_internal/sqlite_event_log.py", + "dare_framework/event/kernel.py", + "dare_framework/observability/_internal/event_trace_bridge.py", + "dare_framework/agent/builder.py", + ], + action="inspect SQLite event append/hash-chain/replay wiring and trace-aware event-log bridging", + ), +) + + +def extract_failed_tests(output: str) -> list[str]: + failed_tests: list[str] = [] + for line in output.splitlines(): + match = FAILED_TEST_RE.match(line.strip()) + if match: + failed_tests.append(match.group(1)) + return failed_tests + + +def format_summary(results: list[CategoryResult]) -> str: + passed = all(result.passed for result in results) + lines = [f"p0-gate: {'PASS' if passed else 'FAIL'}"] + for result in results: + if result.passed: + lines.append(f"- {result.spec.label}: 0 failures") + continue + failed_tests = ", ".join(result.failed_tests or [""]) + modules = ", ".join(result.spec.modules) + lines.extend( + [ + f"- {result.spec.label}", + f" tests: {failed_tests}", + f" modules: {modules}", + f" action: {result.spec.action}", + ] + ) + return "\n".join(lines) + + +def run_category(spec: CategorySpec) -> CategoryResult: + command = [sys.executable, "-m", "pytest", "-q", *spec.tests] + completed = subprocess.run( + command, + capture_output=True, + text=True, + check=False, + ) + raw_output = "\n".join(part for part in [completed.stdout.strip(), completed.stderr.strip()] if part).strip() + return CategoryResult( + spec=spec, + passed=completed.returncode == 0, + failed_tests=extract_failed_tests(raw_output), + raw_output=raw_output, + ) + + +def _write_step_summary(summary: str) -> None: + summary_path = os.environ.get("GITHUB_STEP_SUMMARY") + if not summary_path: + return + with Path(summary_path).open("a", encoding="utf-8") as fh: + fh.write("## p0-gate\n\n") + fh.write("```text\n") + fh.write(summary) + fh.write("\n```\n") + + +def main() -> int: + results = [run_category(spec) for spec in DEFAULT_CATEGORY_SPECS] + + for result in results: + if result.passed or not result.raw_output: + continue + print(f"== {result.spec.label} raw output ==", file=sys.stderr) + print(result.raw_output, file=sys.stderr) + + summary = format_summary(results) + print(summary) + _write_step_summary(summary) + return 0 if all(result.passed for result in results) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/integration/test_p0_conformance_gate.py b/tests/integration/test_p0_conformance_gate.py new file mode 100644 index 00000000..64fe9d65 --- /dev/null +++ b/tests/integration/test_p0_conformance_gate.py @@ -0,0 +1,315 @@ +from __future__ import annotations + +from dataclasses import dataclass +import sqlite3 +from typing import Any + +import pytest + +from dare_framework.agent.dare_agent import DareAgent +from dare_framework.config import Config +from dare_framework.context import Context +from dare_framework.event import SQLiteEventLog +from dare_framework.model.types import ModelInput, ModelResponse +from dare_framework.plan.types import ( + DecompositionResult, + Milestone, + ProposedPlan, + RunResult, + ValidatedPlan, + ValidatedStep, + VerifyResult, +) +from dare_framework.security import PolicySecurityBoundary +from dare_framework.security.types import RiskLevel +from dare_framework.tool.types import CapabilityDescriptor, CapabilityType, ToolResult + + +class _NeverCalledModel: + name = "never-called-model" + + async def generate(self, *_: Any, **__: Any) -> Any: + raise RuntimeError("step-driven integration should not use the model execute loop") + + +class _SingleToolCallModel: + name = "single-tool-call-model" + + def __init__(self) -> None: + self._responses = [ + ModelResponse( + content="invoke audit tool", + tool_calls=[ + { + "id": "tc-audit-1", + "name": "tool.echo", + "capability_id": "tool.echo", + "arguments": {"text": "hi"}, + } + ], + ), + ModelResponse(content="done", tool_calls=[]), + ] + + async def generate(self, model_input: ModelInput, *, options: Any = None) -> ModelResponse: + _ = (model_input, options) + if self._responses: + return self._responses.pop(0) + return ModelResponse(content="done", tool_calls=[]) + + +@dataclass +class _PlannerRecord: + decompose_calls: int = 0 + plan_calls: int = 0 + + +class _StepDrivenPlanner: + def __init__(self) -> None: + self.record = _PlannerRecord() + + async def decompose(self, task: Any, ctx: Any) -> DecompositionResult: + _ = ctx + self.record.decompose_calls += 1 + return DecompositionResult( + milestones=[ + Milestone( + milestone_id="milestone-step-driven", + description=task.description, + user_input=task.description, + ) + ], + reasoning="single milestone for integration coverage", + ) + + async def plan(self, ctx: Any) -> ProposedPlan: + _ = ctx + self.record.plan_calls += 1 + return ProposedPlan(plan_description="proposed step-driven plan") + + +@dataclass +class _VerifyCall: + success: bool + errors: list[str] + output: Any + plan_step_ids: list[str] + + +class _StaticValidator: + def __init__(self, validated_plan: ValidatedPlan) -> None: + self._validated_plan = validated_plan + self.validate_calls = 0 + self.verify_calls: list[_VerifyCall] = [] + + async def validate_plan(self, plan: ProposedPlan, ctx: Any) -> ValidatedPlan: + _ = (plan, ctx) + self.validate_calls += 1 + return self._validated_plan + + async def verify_milestone( + self, + result: RunResult, + ctx: Any, + *, + plan: ValidatedPlan | None = None, + ) -> VerifyResult: + _ = ctx + self.verify_calls.append( + _VerifyCall( + success=result.success, + errors=list(result.errors), + output=result.output, + plan_step_ids=[step.step_id for step in plan.steps] if plan is not None else [], + ) + ) + return VerifyResult(success=result.success, errors=list(result.errors)) + + +class _RecordingGateway: + def __init__(self, results: dict[str, ToolResult[Any]]) -> None: + self._results = dict(results) + self._descriptors = [ + CapabilityDescriptor( + id=capability_id, + type=CapabilityType.TOOL, + name=capability_id, + description=capability_id, + input_schema={"type": "object"}, + metadata={"risk_level": RiskLevel.READ_ONLY.value}, + ) + for capability_id in results + ] + self.invoke_calls: list[dict[str, Any]] = [] + + def list_capabilities(self) -> list[CapabilityDescriptor]: + return list(self._descriptors) + + async def invoke(self, capability_id: str, *, envelope: Any, **params: Any) -> ToolResult[Any]: + self.invoke_calls.append( + { + "capability_id": capability_id, + "envelope": envelope, + "params": dict(params), + } + ) + return self._results[capability_id] + + +class _EchoGateway: + def __init__(self) -> None: + self._descriptor = CapabilityDescriptor( + id="tool.echo", + type=CapabilityType.TOOL, + name="tool.echo", + description="echo tool", + input_schema={"type": "object", "properties": {"text": {"type": "string"}}}, + metadata={"risk_level": RiskLevel.READ_ONLY.value}, + ) + + def list_capabilities(self) -> list[CapabilityDescriptor]: + return [self._descriptor] + + async def invoke(self, capability_id: str, *, envelope: Any, **params: Any) -> ToolResult[Any]: + _ = envelope + return ToolResult(success=True, output={"echo": params.get("text")}) + + +def _build_step_driven_agent( + *, + gateway: _RecordingGateway, + validator: _StaticValidator, + planner: _StepDrivenPlanner, + max_milestone_attempts: int = 1, +) -> DareAgent: + return DareAgent( + name="p0-step-driven-agent", + model=_NeverCalledModel(), + context=Context(config=Config()), + tool_gateway=gateway, + planner=planner, + validator=validator, + security_boundary=PolicySecurityBoundary(), + execution_mode="step_driven", + max_milestone_attempts=max_milestone_attempts, + max_plan_attempts=1, + ) + + +@pytest.mark.asyncio +async def test_step_driven_session_executes_validated_steps_in_order() -> None: + planner = _StepDrivenPlanner() + validated_plan = ValidatedPlan( + plan_description="validated step-driven plan", + steps=[ + ValidatedStep(step_id="s1", capability_id="tool.first", risk_level=RiskLevel.READ_ONLY), + ValidatedStep(step_id="s2", capability_id="tool.second", risk_level=RiskLevel.READ_ONLY), + ], + ) + validator = _StaticValidator(validated_plan) + gateway = _RecordingGateway( + { + "tool.first": ToolResult(success=True, output={"first": 1}), + "tool.second": ToolResult(success=True, output={"prev_first": 1}), + } + ) + agent = _build_step_driven_agent( + gateway=gateway, + validator=validator, + planner=planner, + ) + + result = await agent("run ordered step-driven session") + + assert result.success is True + assert planner.record.decompose_calls == 1 + assert planner.record.plan_calls == 1 + assert validator.validate_calls == 1 + assert len(validator.verify_calls) == 1 + assert validator.verify_calls[0].plan_step_ids == ["s1", "s2"] + assert result.session_summary is not None + assert result.session_summary.milestones[0].outputs == [{"first": 1}, {"prev_first": 1}] + assert result.session_summary.final_output == {"prev_first": 1} + assert [call["capability_id"] for call in gateway.invoke_calls] == ["tool.first", "tool.second"] + assert gateway.invoke_calls[1]["params"]["_previous_output"] == {"first": 1} + + +@pytest.mark.asyncio +async def test_step_driven_session_stops_after_first_failed_step() -> None: + planner = _StepDrivenPlanner() + validated_plan = ValidatedPlan( + plan_description="validated fail-fast plan", + steps=[ + ValidatedStep(step_id="s1", capability_id="tool.fail", risk_level=RiskLevel.READ_ONLY), + ValidatedStep(step_id="s2", capability_id="tool.never", risk_level=RiskLevel.READ_ONLY), + ], + ) + validator = _StaticValidator(validated_plan) + gateway = _RecordingGateway( + { + "tool.fail": ToolResult(success=False, error="boom"), + "tool.never": ToolResult(success=True, output={"unexpected": True}), + } + ) + agent = _build_step_driven_agent( + gateway=gateway, + validator=validator, + planner=planner, + ) + + result = await agent("run failing step-driven session") + + assert result.success is False + assert planner.record.decompose_calls == 1 + assert planner.record.plan_calls == 1 + assert validator.validate_calls == 1 + assert len(validator.verify_calls) == 1 + assert validator.verify_calls[0].success is False + assert validator.verify_calls[0].errors == ["boom"] + assert validator.verify_calls[0].plan_step_ids == ["s1", "s2"] + assert result.session_summary is not None + assert result.session_summary.milestones[0].outputs == [] + assert [call["capability_id"] for call in gateway.invoke_calls] == ["tool.fail"] + assert result.errors == ["milestone failed after max attempts"] + + +@pytest.mark.asyncio +async def test_default_event_log_replay_and_hash_chain_hold_for_runtime_session(tmp_path) -> None: + db_path = tmp_path / ".dare" / "events.db" + event_log = SQLiteEventLog(db_path) + agent = DareAgent( + name="p0-audit-agent", + model=_SingleToolCallModel(), + context=Context(config=Config()), + tool_gateway=_EchoGateway(), + event_log=event_log, + security_boundary=PolicySecurityBoundary(), + ) + + result = await agent("record auditable runtime session") + + assert result.success is True + + start_events = await event_log.query(filter={"event_type": "session.start"}, limit=1) + assert len(start_events) == 1 + + snapshot = await event_log.replay(from_event_id=start_events[0].event_id) + assert snapshot.from_event_id == start_events[0].event_id + assert snapshot.events[0].event_type == "session.start" + assert any(event.event_type == "security.policy_checked" for event in snapshot.events) + assert any(event.event_type == "session.complete" for event in snapshot.events) + for event in snapshot.events: + assert event.payload.get("task_id") + assert event.payload.get("run_id") + assert event.payload.get("session_id") == event.payload.get("run_id") + + assert await event_log.verify_chain() is True + + with sqlite3.connect(db_path) as conn: + conn.execute( + "UPDATE events SET payload_json = ? WHERE event_id = ?", + ('{"tampered":true}', snapshot.events[-1].event_id), + ) + conn.commit() + + assert await event_log.verify_chain() is False diff --git a/tests/integration/test_security_policy_gate_flow.py b/tests/integration/test_security_policy_gate_flow.py index 5e492e97..f490c8bd 100644 --- a/tests/integration/test_security_policy_gate_flow.py +++ b/tests/integration/test_security_policy_gate_flow.py @@ -1,6 +1,7 @@ from __future__ import annotations import asyncio +import json from pathlib import Path from typing import Any @@ -40,7 +41,12 @@ async def generate(self, model_input: ModelInput, *, options: Any = None) -> Mod class _RecordingGateway: - def __init__(self) -> None: + def __init__( + self, + *, + risk_level: str = "non_idempotent_effect", + requires_approval: bool = False, + ) -> None: self.invoke_calls: list[dict[str, Any]] = [] self._descriptor = CapabilityDescriptor( id="run_command", @@ -49,8 +55,8 @@ def __init__(self) -> None: description="Run shell command", input_schema={"type": "object", "properties": {"command": {"type": "string"}}}, metadata={ - "risk_level": "non_idempotent_effect", - "requires_approval": False, + "risk_level": risk_level, + "requires_approval": requires_approval, }, ) @@ -101,3 +107,54 @@ async def test_high_risk_tool_invocation_must_pass_policy_gate(tmp_path: Path) - assert result.success is True assert len(gateway.invoke_calls) == 1 + + +@pytest.mark.asyncio +async def test_read_only_tool_invocation_bypasses_pending_approval(tmp_path: Path) -> None: + approval_manager = ToolApprovalManager( + workspace_store=JsonApprovalRuleStore(tmp_path / "workspace" / "approvals.json"), + user_store=JsonApprovalRuleStore(tmp_path / "user" / "approvals.json"), + ) + gateway = _RecordingGateway(risk_level="read_only") + agent = DareAgent( + name="security-policy-gate-flow-allow", + model=_TwoStepModel(), + context=Context(config=Config()), + tool_gateway=gateway, + approval_manager=approval_manager, + security_boundary=PolicySecurityBoundary(), + ) + + result = await agent("run read only tool") + + assert result.success is True + assert len(gateway.invoke_calls) == 1 + assert approval_manager.list_pending() == [] + + +@pytest.mark.asyncio +async def test_denied_capability_blocks_tool_before_invocation(tmp_path: Path) -> None: + approval_manager = ToolApprovalManager( + workspace_store=JsonApprovalRuleStore(tmp_path / "workspace" / "approvals.json"), + user_store=JsonApprovalRuleStore(tmp_path / "user" / "approvals.json"), + ) + gateway = _RecordingGateway(risk_level="read_only") + agent = DareAgent( + name="security-policy-gate-flow-deny", + model=_TwoStepModel(), + context=Context(config=Config()), + tool_gateway=gateway, + approval_manager=approval_manager, + security_boundary=PolicySecurityBoundary(deny_capability_ids={"run_command"}), + ) + + result = await agent("run denied tool") + + tool_messages = [msg for msg in agent._context.stm_get() if msg.role == "tool"] # noqa: SLF001 + assert tool_messages + tool_payload = json.loads(tool_messages[-1].content) + assert tool_payload.get("status") == "not_allow" + assert tool_payload.get("success") is False + assert gateway.invoke_calls == [] + assert approval_manager.list_pending() == [] + assert result.success is True diff --git a/tests/unit/test_p0_gate_ci.py b/tests/unit/test_p0_gate_ci.py new file mode 100644 index 00000000..61b1ec2d --- /dev/null +++ b/tests/unit/test_p0_gate_ci.py @@ -0,0 +1,94 @@ +from __future__ import annotations + +from scripts.ci.p0_gate import CategoryResult, CategorySpec, extract_failed_tests, format_summary + + +def _spec( + label: str, + *, + modules: list[str] | None = None, + action: str = "inspect category", +) -> CategorySpec: + return CategorySpec( + label=label, + tests=["tests/example.py::test_case"], + modules=modules or ["module.one", "module.two"], + action=action, + ) + + +def test_extract_failed_tests_returns_node_ids_in_order() -> None: + output = """ +=========================== short test summary info ============================ +FAILED tests/integration/test_p0_conformance_gate.py::test_step_driven_session_stops_after_first_failed_step +FAILED tests/unit/test_event_sqlite_event_log.py::test_verify_chain_detects_tampered_row - assert False +""" + + assert extract_failed_tests(output) == [ + "tests/integration/test_p0_conformance_gate.py::test_step_driven_session_stops_after_first_failed_step", + "tests/unit/test_event_sqlite_event_log.py::test_verify_chain_detects_tampered_row", + ] + + +def test_extract_failed_tests_includes_error_node_ids() -> None: + output = """ +=========================== short test summary info ============================ +ERROR tests/unit/test_p0_gate_ci.py::test_extract_failed_tests_includes_error_node_ids +ERROR tests/integration/test_p0_conformance_gate.py::test_default_event_log_replay_and_hash_chain_hold_for_runtime_session - RuntimeError: boom +""" + + assert extract_failed_tests(output) == [ + "tests/unit/test_p0_gate_ci.py::test_extract_failed_tests_includes_error_node_ids", + "tests/integration/test_p0_conformance_gate.py::test_default_event_log_replay_and_hash_chain_hold_for_runtime_session", + ] + + +def test_format_summary_reports_pass_for_all_categories() -> None: + summary = format_summary( + [ + CategoryResult(spec=_spec("SECURITY_REGRESSION"), passed=True, failed_tests=[], raw_output=""), + CategoryResult(spec=_spec("STEP_EXEC_REGRESSION"), passed=True, failed_tests=[], raw_output=""), + CategoryResult(spec=_spec("AUDIT_CHAIN_REGRESSION"), passed=True, failed_tests=[], raw_output=""), + ] + ) + + assert summary == "\n".join( + [ + "p0-gate: PASS", + "- SECURITY_REGRESSION: 0 failures", + "- STEP_EXEC_REGRESSION: 0 failures", + "- AUDIT_CHAIN_REGRESSION: 0 failures", + ] + ) + + +def test_format_summary_reports_failures_with_modules_and_action() -> None: + summary = format_summary( + [ + CategoryResult( + spec=_spec( + "STEP_EXEC_REGRESSION", + modules=[ + "dare_framework/agent/dare_agent.py", + "dare_framework/agent/_internal/execute_engine.py", + ], + action="inspect step execution order and fail-fast handling", + ), + passed=False, + failed_tests=[ + "tests/integration/test_p0_conformance_gate.py::test_step_driven_session_stops_after_first_failed_step" + ], + raw_output="FAILED ...", + ) + ] + ) + + assert summary == "\n".join( + [ + "p0-gate: FAIL", + "- STEP_EXEC_REGRESSION", + " tests: tests/integration/test_p0_conformance_gate.py::test_step_driven_session_stops_after_first_failed_step", + " modules: dare_framework/agent/dare_agent.py, dare_framework/agent/_internal/execute_engine.py", + " action: inspect step execution order and fail-fast handling", + ] + )