Generated from
reliability-scorecard.json. Edit the structured source and runpython3 scripts/check-reliability-scorecard.py --write.
This is the current decision view for #807. It distinguishes observed release or field evidence from historical observations, targets, benchmark conditions, and work that is not yet measured. A green test count is not an observed product reliability result.
Rows dependent on installed-artifact certification or exact-commit release governance remain Not yet measured until #779 or #805 produces the named receipt. No target or SLO is implied by an empty target field.
| Category | Metric | Status | Current result | Definition | Cohort / window | Source / privacy | Owner / confidence | Target / action |
|---|---|---|---|---|---|---|---|---|
| Core journey | Install and readiness success | Not yet measured | — | A supported packaged install verifies its binary and reaches an actionable doctor or health result. Numerator: Qualifying installs that verify and reach actionable readiness. Denominator: All qualifying install attempts in the cohort. | Supported release and platform install channels. Window: No retained installed-artifact cohort yet. | scripts/user-journey-e2e.mjs and setup --verify-only reports; #779 must retain the release/platform receipt. Evidence: none retained. Privacy: Version, platform, stage, duration, and normalized exit class only. | Release maintainers. Confidence: Not measured; current hermetic journey coverage is binary fixture evidence, not a shipped cohort. | Target: none adopted. Action: Keep unmeasured until #779 retains per-platform installed-artifact outcomes. |
| Core journey | First successful session and output | Not yet measured | — | A ready install starts a project-scoped session, accepts first input, and emits first output. Numerator: Qualifying journeys reaching first output. Denominator: Qualifying journeys that passed readiness. | Supported packaged installs using supported harnesses. Window: No retained release cohort yet. | scripts/user-journey-e2e.mjs and scripts/certify-release.sh. Evidence: none retained. Privacy: Stage names, durations, harness id, platform, commit, and exit class; no prompt or output content. | Runtime maintainers. Confidence: Not measured; deterministic fake-harness coverage cannot establish field success or latency. | Target: none adopted. Action: Add stage receipts to installed-artifact certification before publishing a baseline. |
| Core journey | Inspect, control, reconnect, and restart recovery | Not yet measured | — | A live session can be inspected, controlled, reattached, and recovered after a supported daemon restart. Numerator: Journeys completing every required inspect, input, kill, terminal, reconnect, and restart stage. Denominator: Journeys entering the control and recovery stage. | Supported release/platform combinations in #779. Window: No retained installed-artifact recovery cohort yet. | user journey, daemon lifecycle, handoff/cursor, and orphan-recovery acceptance families. Evidence: none retained. Privacy: Opaque synthetic session ids and normalized outcomes only. | Runtime maintainers. Confidence: Not measured; lower-level acceptance exists without one retained end-to-end receipt. | Target: none adopted. Action: Keep unmeasured until #779 records stage-level recovery outcomes. |
| Operation reliability | Launch and input outcome classification | Not yet measured | — | Launch and input attempts resolve to success, explicit refusal, capability mismatch, or normalized failure without silent ambiguity. Numerator: Attempts in each normalized outcome class. Denominator: All qualifying launch or input attempts. | Supported API clients and harnesses by release/platform. Window: No retained outcome cohort. | Daemon API contract tests and future privacy-preserving operation receipts. Evidence: none retained. Privacy: Operation kind, capability/version, stage, and normalized error only. | Runtime maintainers. Confidence: Not measured; contract coverage proves behavior classes, not observed rates. | Target: none adopted. Action: Define a retained release-study cohort before adopting rates or thresholds. |
| Operation reliability | Attach, reconnect, and peer-replacement outcomes | Not yet measured | — | Attach and reconnect attempts distinguish success, stale endpoint, peer replacement, and explicit recovery-required outcomes. Numerator: Attempts in each attach or peer outcome class. Denominator: All qualifying attach or reconnect attempts. | Bundled clients against the supported local API. Window: No retained compatibility cohort. | coven-client health negotiation and transport tests. Evidence: none retained. Privacy: Transport class and opaque peer fingerprint result only. | Client contract maintainers. Confidence: Not measured; deterministic negotiation tests do not establish incidence. | Target: none adopted. Action: Populate from #779 client/platform receipts without retaining endpoint paths. |
| Operation reliability | Ordered event consumption and cursor recovery | Not yet measured | — | Consumers retain event order and recover cursors without silently skipping required events. Numerator: Qualifying cursor journeys preserving order and recovery. Denominator: All qualifying cursor journeys. | Supported clients and release/platform combinations. Window: No retained release cohort. | Event cursor, handoff, and API contract acceptance tests. Evidence: none retained. Privacy: Sequence numbers and normalized recovery outcomes only. | Client contract maintainers. Confidence: Not measured; acceptance coverage is not an observed shipped rate. | Target: none adopted. Action: Add a cursor stage to the installed-artifact receipt before publishing a baseline. |
| Operation reliability | Kill, cancel, terminal, and explicit-unknown resolution | Not yet measured | — | Control attempts resolve to a named terminal or recovery-required state; unknown is never counted as success. Numerator: Attempts in each terminal, refusal, failure, or explicit-unknown class. Denominator: All qualifying kill, cancel, and terminal-resolution attempts. | Supported harness and platform combinations. Window: No retained outcome cohort. | Daemon lifecycle, PTY, cancellation, and session-state acceptance tests. Evidence: none retained. Privacy: Terminal class, duration, and normalized error only. | Runtime maintainers. Confidence: Not measured; tests prove state semantics without incidence. | Target: none adopted. Action: Preserve unknown as its own class when a release receipt is introduced. |
| Operation reliability | Output loss, cleanup, and crash recovery | Not yet measured | — | Output truncation, backpressure, drop, orphan, cleanup, crash, and restart outcomes are classified with recoverability. Numerator: Incidents in each loss, cleanup, orphan, and recovery class. Denominator: Qualifying sessions or injected recovery trials. | Supported release/platform combinations and declared deterministic trials. Window: No retained release cohort. | Output-marker, orphan-recovery, daemon lifecycle, and chaos acceptance families. Evidence: none retained. Privacy: Counts, byte totals, state classes, and recovery result; no output content. | Runtime maintainers. Confidence: Not measured; deterministic failure injection is supporting evidence only. | Target: none adopted. Action: Keep fixture outcomes separate from shipped incidence in future receipts. |
| Release quality | Exact-commit required-check compliance | Not yet measured | — | Every published version is bound to one accepted source commit with every required check successful for that exact commit. Numerator: Published versions with a retained exact-commit governance receipt. Denominator: All published versions in the window. | GitHub Releases and package channels. Window: Begins when #805 ships the receipt. | #805 exact-commit release governance. Evidence: none retained. Privacy: Commit, tag, check names/conclusions, signer result, and artifact digests only. | Release maintainers. Confidence: Not measured; #805 is open and no qualifying receipt exists. | Target: none adopted. Action: Do not claim compliance until #805 retains the immutable receipt. |
| Release quality | Release-channel version and support consistency | Not yet measured | — | Release, tag, registries, installers, checksums, provenance, notes, and support state identify the same public version. Numerator: Published versions with coherent channel evidence. Denominator: All published versions in the window. | Supported public release channels. Window: Begins when #805 ships reconciliation evidence. | #805 release receipt and reconciliation path. Evidence: none retained. Privacy: Public version and artifact metadata only. | Release maintainers. Confidence: Not measured; current release metadata is not retained as one decision receipt. | Target: none adopted. Action: Keep unmeasured until one receipt covers every supported channel. |
| Release quality | Installed-artifact E2E coverage | Not yet measured | — | Published artifacts pass the named user journey on each supported release/platform/install channel. Numerator: Certified release/platform/install-channel cells. Denominator: Required cells in the support matrix. | Published artifacts for the current supported release family. Window: Begins when #779 retains its matrix receipt. | #779 installed-artifact certification. Evidence: none retained. Privacy: Platform, package, commit, stage outcome, duration, and normalized error only. | Release maintainers. Confidence: Not measured; #779 is open and no qualifying matrix receipt exists. | Target: none adopted. Action: Do not infer artifact coverage from source-tree CI. |
| Release quality | Escaped defects, discovery, remediation, and rollback | Not yet measured | — | Supported-release defects are classified by severity, surface, discovery source, time to safe state, and rollback or disable action. Numerator: Defects in each declared classification. Denominator: Supported releases and reporting window; rates require an approved denominator. | Supported releases only. Window: No owner-approved reporting window yet. | Release notes, advisories, issues, and future release-quality receipts. Evidence: none retained. Privacy: Public issue/advisory metadata and normalized classifications; no private report content. | Release maintainers. Confidence: Not measured; no normalized retained ledger exists. | Target: none adopted. Action: Define the window and private-to-public aggregation rules before publishing counts. |
| Performance/resource | CLI deterministic timing collector | Benchmark condition/input | Default input is 3 iterations; report includes per-run min, median, p95, and max. | Per-run fixture timing for command startup, daemon health, listing, tailing, and first output. Numerator: Recorded fixture durations by operation. Denominator: Configured iterations in one benchmark run. | Disposable COVEN_HOME with the fake Codex fixture on one CI host. Window: One artifact per run; no retained multi-run product window. | scripts/benchmark-cli.mjs. Evidence: scripts/benchmark-cli.mjs. Privacy: Synthetic fixture, environment redaction, no user repository or provider content. | Runtime maintainers. Confidence: Valid deterministic benchmark input; insufficient for field SLOs or p95 claims. | Target: none adopted. Action: Do not gate or publish as an observed product result until sample and regression budgets are approved. |
| Performance/resource | Concurrent chaos/resource collector | Benchmark condition/input | Inputs cover 1, 8, and 32 concurrent sessions plus declared injected recovery trials. | Per-run deterministic load and failure-injection measurements for concurrency, cancellation, storage, backlog, memory, and recovery. Numerator: Recorded durations, rates, sizes, and resource samples by fixture trial. Denominator: Configured 1, 8, and 32-session fixture trials in one run. | Disposable local fixture environment on one CI host. Window: One report-schema-v3 artifact per run; no retained product window. | scripts/benchmark-chaos.mjs and its deterministic acceptance tests. Evidence: scripts/benchmark-chaos.mjs. Privacy: Synthetic sessions; resource totals omit process names and command lines. | Runtime maintainers. Confidence: Valid benchmark condition; not field incidence or an adopted regression budget. | Target: none adopted. Action: Keep deterministic acceptance separate from trend data and observed outcomes. |
| AgentFS/security boundary | AgentFS storage and mount boundary certification | Not yet measured | — | Representative platform trials classify storage, path, materialization, concurrency, crash cleanup, credential observation, case handling, handle reuse, access control, and safe disable. Numerator: Passing certified AgentFS matrix cells. Denominator: Required cells for the declared support state. | Feature-gated AgentFS storage and experimental mount surfaces. Window: No retained certification window. | AgentFS conformance/mount tests and #779 certification matrix. Evidence: none retained. Privacy: Synthetic paths and payloads; no credentials or repository content. | AgentFS maintainers. Confidence: Not measured; mount remains experimental and must not inherit generic CI success. | Target: none adopted. Action: Keep mount disabled/experimental until its threat-derived matrix is retained. |
| API/client compatibility | API negotiation and client compatibility | Not yet measured | — | Supported client families negotiate versions/capabilities and classify malformed, unsupported, stale-peer, and non-replay outcomes. Numerator: Compatible and explicitly denied cells by client/version/capability. Denominator: Required public contract fixture cells. | Published packages and supported client families. Window: No retained release compatibility matrix. | API contract fixtures, coven-client tests, and future #779 receipts. Evidence: none retained. Privacy: Version, capability, route class, and normalized error only. | Client contract maintainers. Confidence: Not measured; source compatibility tests are not a published-package matrix. | Target: none adopted. Action: Populate from installed published packages before reporting compatibility coverage. |
| Usefulness | Reviewed or accepted task outcome | Not yet measured | — | An opt-in controlled study reaches a named reviewed or accepted outcome and records rework, rollback, elapsed time, repeat use, and a lightweight rating. Numerator: Study tasks reaching the predeclared accepted outcome. Denominator: Opt-in study tasks in the declared cohort. | Not defined; requires an accountable product decision and consent model. Window: Not defined. | Future opt-in beta or controlled study; never inferred from session completion. Evidence: none retained. Privacy: Aggregate outcome classes and ratings only; no prompts, repository content, credentials, or full output. | OpenCoven product owner. Confidence: Not measured; no cohort, consent mechanism, or accepted-outcome contract exists. | Target: none adopted. Action: Keep unmeasured until the product owner approves cohort, consent, definition, and breach action. |
- Observed current and Historical observation require a dated, retained evidence reference. This initial scorecard intentionally has no such rows because no qualifying receipt or field cohort is retained.
- Benchmark condition/input describes a deterministic fixture or collector configuration only. It must never be quoted as achieved product reliability or a field SLO.
- Not yet measured is an explicit decision state with an owner and next action; it is neither success nor failure.
- Raw prompts, credentials, repository content, and full command output are outside this scorecard contract.