diff --git a/docs/adrs/0023-evidence-mapping.md b/docs/adrs/0023-evidence-mapping.md new file mode 100644 index 00000000..550e7ee5 --- /dev/null +++ b/docs/adrs/0023-evidence-mapping.md @@ -0,0 +1,214 @@ +--- +layout: page +title: Evidence Mapping +--- + +- **ADR:** 0023 +- **Proposal Author(s):** @jpower432 +- **Status:** Proposed +- **Supersedes:** ADR-0022 + +## Context + +ADR-0022 introduced `#Evidence` on `#AssessmentLog` so that evaluation tools can record what they consulted during an assessment. + +The v1.3.0 implementation added the field and a reshaped `#Evidence` type, but shipped without the `address` and `digest` fields described in the accepted ADR. The type as released can only carry inline data via `payload`, which means tools cannot record *where* evidence was observed without embedding the content itself. During review, the question of whether `address` should be a plain string or a structured reference was left unresolved. + +## Action + +A new mapping type specific to evidence references. It extends the `mapping-reference` system with fields for sub-origin, entry-level references, and integrity verification. + +```cue +// EvidenceMapping references a source consulted during evidence collection. +// Properties describe the specific observation, not the artifact itself. +#EvidenceMapping: { + // reference-id ties this evidence to a mapping-reference in the artifact's metadata + "reference-id": string + + // coordinate identifies a specific origin within the referenced source + // (line number, JSON path, API endpoint path, file fragment) + // Used for opaque sources where Gemara cannot address internal structure. + coordinate?: string + + // entry-id identifies a specific entry within a structured Gemara artifact + // Used when the referenced source is a Gemara artifact with addressable entries. + "entry-id"?: string @go(EntryId) + + // digest is a hash of the evidence content at collection time + // Verifies that mutable-address evidence has not changed since the tool observed it. + // Not needed for content-addressable systems (OCI, git) or inline payloads. + digest?: string + + // remarks is prose regarding this evidence reference + remarks?: string +} +``` + +`coordinate` and `entry-id` serve different layers but are not mutually exclusive. An audit-layer evidence entry may reference a specific assessment log entry (`entry-id`) within an evaluation log, while `coordinate` could identify a sub-origin within that entry's output. For opaque sources, only `coordinate` applies. For structured sources, `entry-id` may be sufficient. + +`digest` is a mapping-level property because it captures the content hash *at collection time*. The same artifact referenced by two different evidence entries may have different digests if observed at different times. + +Replace the v1.3.0 `#Evidence` definition with one that uses `#EvidenceMapping` for source references. + +```cue +#Evidence: { + // id uniquely identifies this evidence entry + id: string + + // type categorizes the kind of evidence + type: #EvidenceType + + // collected-at is the timestamp when the evidence was gathered + "collected-at": #Datetime @go(CollectedAt) + + // payload is the raw evidence data collected inline + payload?: _ @go(Payload,type=any) + + // origin references the source consulted during evidence collection + origin?: #EvidenceMapping @go(Origin) + + // description explains what this evidence represents + description?: string +} +``` + +`payload` and `origin` serve complementary roles: + +- `payload` carries *what was observed* — the raw data, a snippet, or a summary. +- `origin` records *where it came from* — the source, the specific position within it, and its integrity at collection time. + +Both may be present. A tool may record the specific line it observed (`payload`) alongside where that line lives (`origin`). Or a tool may provide only one: `payload` for inline-only evidence, `origin` for reference-only evidence. + +The `evidence` field on `#AssessmentLog` in stable `evaluationlog.cue` remains `[#Evidence, ...#Evidence]`. No change to the stable schema's field type. When the same source is consulted by multiple steps or assessments, it appears once in `mapping-references` and is referenced by multiple evidence entries — each with its own `coordinate`, `digest`, and `payload` capturing that specific observation. + +## Examples + +### Multiple observations of the same source + +Two assessment steps consult the same file. The artifact appears once in `mapping-references`; each observation gets its own evidence entry with a specific `coordinate` and `payload`: + +```yaml +metadata: + mapping-references: + - id: "gomod" + title: "Go module definition" + version: "v0.5.0" + url: "file://go.mod" + +evidence: + - id: "e-001" + type: "config-file" + collected-at: "2026-06-13T14:30:00Z" + origin: + reference-id: "gomod" + coordinate: "go.mod:3" + digest: "sha256:abc123..." + payload: "go 1.23" + description: "Go version declaration" + + - id: "e-002" + type: "config-file" + collected-at: "2026-06-13T14:30:02Z" + origin: + reference-id: "gomod" + coordinate: "go.mod:12" + digest: "sha256:abc123..." + payload: "require github.com/sigstore/cosign/v2 v2.4.1" + description: "Cosign dependency version" +``` + +### Audit layer — structured Gemara artifact reference + +An audit-layer entry references a specific assessment log within an evaluation log using `entry-id`: + +```yaml +metadata: + mapping-references: + - id: "eval-run-42" + title: "Quarterly evaluation — Q2 2026" + version: "1.0.0" + url: "oci://registry.example.com/evaluations/run-42:v1.0.0" + +evidence: + - id: "e-005" + type: "EvaluationLog" + collected-at: "2026-06-13T15:00:00Z" + origin: + reference-id: "eval-run-42" + entry-id: "assessment-mfa-check" + description: "MFA enforcement assessment result from Q2 evaluation" +``` + +### Conflicting sources + +Two sources disagree about the same control. Each evidence entry traces to its origin, enabling consumers to weigh by source trust: + +```yaml +metadata: + mapping-references: + - id: "github-api" + title: "GitHub REST API" + version: "2026" + url: "https://api.github.com" + - id: "security-insights" + title: "Security Insights YAML" + version: "1.0.0" + url: "file://SECURITY-INSIGHTS.yml" + +evidence: + - id: "e-007" + type: "api-response" + collected-at: "2026-06-13T14:30:00Z" + origin: + reference-id: "github-api" + coordinate: "/repos/org/repo/vulnerability-alerts" + payload: + vulnerability_reporting_enabled: true + description: "GitHub API reports vulnerability reporting is enabled" + + - id: "e-008" + type: "config-file" + collected-at: "2026-06-13T14:30:01Z" + origin: + reference-id: "security-insights" + coordinate: "SECURITY-INSIGHTS.yml:12" + digest: "sha256:e9f0a1..." + payload: + vulnerability-reporting: + accepts-vulnerability-reports: false + description: "Security Insights contradicts GitHub API" +``` + +## Consequences + +### Positive + +- **No breaking changes.** All additions are optional fields on the experimental `#Evidence` type and a new experimental `#EvidenceMapping` type. The `evidence` field type on `#AssessmentLog` in stable `evaluationlog.cue` is unchanged. +- **Consistent with Gemara's mapping system.** Evidence references tie into `mapping-references` via `reference-id`, following the same pattern as every other Gemara cross-reference. +- **Mapping vs artifact separation.** Observation-level properties (`coordinate`, `digest`, `entry-id`) live on the mapping where they belong. Artifact-level properties (name, version, URL) stay in `mapping-references`. Two observations of the same source produce two evidence entries referencing the same mapping-reference. +- **Leverages existing infrastructure.** `mapping-references` provides discoverability for evidence sources. No new artifact type is needed. +- **Supports evidence trust differentiation.** When multiple sources provide conflicting evidence, `origin` enables consumers to trace each piece of evidence to its source and weigh findings based on source trust and collection method. Combined with the existing `confidence-level` on `#AssessmentLog`, this provides the full chain from raw observation to assessed confidence. + +### Negative + +`#EvidenceMapping` adds to the schema vocabulary. This is justified by evidence-specific concerns (integrity, sub-origin, temporal coupling) that do not exist on other mapping types. + +## Alternatives Considered + +### Plain `address` and `digest` fields on `#Evidence` (no mapping type) + +Add `address` and `digest` directly to `#Evidence` without introducing `#EvidenceMapping`. + +**Rejected because:** Evidence references have concerns that justify a dedicated mapping type: sub-origin within opaque sources, entry-level references for structured sources, integrity verification, and the principle that mapping-level properties should not be mixed with artifact-level properties. A flat structure also cannot distinguish between opaque (`address`) and structured (`entry-id`) reference patterns. + +### Reuse `#ArtifactMapping` or `#EntryMapping` + +Use an existing mapping type on `#Evidence` instead of creating a new one. + +**Rejected because:** Existing Gemara mapping types (`#ArtifactMapping`, `#EntryMapping`, `#MultiEntryMapping`) do not fit evidence references: + +1. **Evidence-specific concerns.** Integrity verification (`digest`), sub-origin within opaque sources (`coordinate`), and temporal coupling (`collected-at`) are specific to evidence collection and do not exist on other mappings. + +2. **Opaque and structured sources need different fields.** Evaluation-layer evidence uses `coordinate` for opaque sources (files, APIs). Audit-layer evidence uses `entry-id` for structured Gemara artifacts. No existing mapping supports both — `#ArtifactMapping` has no `entry-id`, and `#EntryMapping` requires it. + +3. **Observations are contextual.** Multiple steps may consult the same artifact but observe different parts. The observation's properties (`coordinate`, `digest`, `payload`) describe the specific collection event, not the artifact. Multiple evidence entries reference one `mapping-reference`, each with its own context. diff --git a/docs/adrs/0024-evidence-type-definitions.md b/docs/adrs/0024-evidence-type-definitions.md new file mode 100644 index 00000000..c2c51c53 --- /dev/null +++ b/docs/adrs/0024-evidence-type-definitions.md @@ -0,0 +1,218 @@ +--- +layout: page +title: Evidence Type Definitions +--- + +- **ADR:** 0024 +- **Proposal Author(s):** @jpower432 +- **Status:** Proposed (**Companion to:** ADR-0023) + +## Context + +ADR-0023 adds `#EvidenceMapping` to `#Evidence` so evaluation tools can record where evidence was observed, with sub-origin coordinates and integrity digests. That addresses the Layer 5 question: *what did we observe and where did it come from?* + +This ADR addresses the Layer 3 counterpart: *what do evidence types mean?* + +`#AssessmentPlan` currently has `frequency` (a bare string) and `evidence-requirements` (a bare string). These fields are loosely typed and disconnected. `frequency` conflates two distinct concepts: how often the control is reviewed, and how long a specific kind of evidence remains valid. An API scan goes stale after a day even though the control review may not be due for months. A red team report stays current across multiple review cycles. There is no way for tooling to compute either dimension. + +## Action + +Split the bare-string `frequency` into two typed fields at their natural levels and replace `evidence-requirements` with structured `#EvidenceTypeDefinition`. + +```cue +#EvidenceTypeDefinition: { + // id matches the type field on #Evidence instances + id: string + + // description explains what this evidence type represents + description?: string + + // valid-for is the evidence validity period in days. + // Evidence older than this many days is considered stale. + "valid-for": int @go(ValidFor) +} +``` + +On `#AssessmentPlan`, type `frequency` as an integer (control review frequency in days) and add the evidence type registry: + +```cue +#AssessmentPlan: { + id: string + "requirement-id": string @go(RequirementId) + // frequency is the control review frequency in days + frequency: int + "evidence-types": [#EvidenceTypeDefinition, ...#EvidenceTypeDefinition] @go(EvidenceTypes) + "evaluation-methods": [#AcceptedMethod & {type: #EvaluationMethodType}, ...#AcceptedMethod & {type: #EvaluationMethodType}] @go(EvaluationMethods) + parameters?: [#Parameter, ...#Parameter] +} +``` + +`frequency` and `valid-for` answer different questions: + +- `frequency` (plan-level): "How often is this control reviewed?" — drives the audit schedule. +- `valid-for` (evidence-type-level): "How long does this evidence stay current?" — drives staleness alerts. + +## End-to-end example + +This traces a single requirement (OSPS-AC-01: MFA enforcement) through all layers. + +### Layer 3 — Policy defines evidence types + +```yaml +# Policy artifact (type: Policy) +metadata: + id: "acme-security-policy" + type: Policy + gemara-version: "1.3.0" + mapping-references: + - id: "OSPS-B" + title: "OpenSSF Baseline for Open Source Projects" + version: "2025.02.25" + url: "https://baseline.openssf.org" + +imports: + catalogs: + - reference-id: "OSPS-B" + +adherence: + evaluation-methods: + - id: "EV-AUTO-01" + type: "Behavioral" + mode: "Automated" + required: true + description: "Automated API and configuration scanning" + - id: "EV-MANUAL-01" + type: "Behavioral" + mode: "Manual" + description: "Quarterly human review of access controls" + assessment-plans: + - id: "plan-ac01" + requirement-id: "OSPS-AC-01.01" + frequency: 90 + evidence-types: + - id: "api-response" + description: "Live API query of platform settings" + valid-for: 1 + - id: "config-file" + description: "Security Insights or configuration file check" + valid-for: 1 + - id: "quarterly-access-review" + description: "Human review of MFA enforcement status" + valid-for: 90 + evaluation-methods: + - id: "EV-AUTO-01" + type: "Behavioral" + mode: "Automated" + required: true +``` + +### Layer 4 — EvaluationLog produces evidence (ADR-0023 schema) + +An evaluator (e.g., PVTR) produces an EvaluationLog. The assessment log references the plan and attaches evidence. Only the relevant assessment-log fragment is shown here (see ADR-0023 for the full `#Evidence` + `#EvidenceMapping` structure): + +```yaml +# Within an EvaluationLog's assessment-logs: +- requirement: + entry-id: "OSPS-AC-01.01" + plan: + reference-id: "acme-policy" + entry-id: "plan-ac01" + result: Passed + message: "Two-factor authentication is enforced by the organization" + confidence-level: High + start: "2026-06-25T14:00:00Z" + end: "2026-06-25T14:00:01Z" + evidence: + - id: "EV-AC-01-01" + type: "api-response" + collected-at: "2026-06-25T14:00:00Z" + origin: + reference-id: "github-api" + coordinate: "/orgs/acme/settings" + payload: + login: acme + two_factor_requirement_enabled: true + description: "GitHub API confirms org-wide MFA enforcement" + - id: "EV-AC-01-02" + type: "config-file" + collected-at: "2026-06-25T14:00:01Z" + origin: + reference-id: "security-insights" + coordinate: "SECURITY-INSIGHTS.yml:8" + digest: "sha256:e9f0a1b2c3..." + payload: + mfa: + required: true + enforcement: "organization" + description: "Security Insights confirms MFA policy" +``` + +### Consumer — walking the evidence chain + +A consumer resolving posture for a requirement uses both ADRs together. `#EvidenceTypeDefinition` (this ADR) tells the consumer what evidence is expected and whether it is fresh. `#EvidenceMapping` (ADR-0023) tells the consumer where each piece of evidence came from. + +``` +Given an assessment log for requirement OSPS-AC-01.01: + +1. Control review due? + plan.frequency → 90 days + Last review: 15 days ago → NOT DUE + +2. For each evidence entry: + a. Is it fresh? + evidence.type → "api-response" + Look up evidence-type definition by id → valid-for: 1 day + evidence.collected-at → 2h ago → CURRENT + + b. Where did it come from? + evidence.origin → #EvidenceMapping + origin.reference-id → "github-api" → resolve in mapping-references + → url: "https://api.github.com", version: "2026" + origin.coordinate → "/orgs/acme/settings" + origin.digest → (none — live API, not content-addressable) + + Result: current api-response evidence from GitHub API at /orgs/acme/settings +``` + +| Evidence | Type | Valid for | Collected | Source | Coordinate | Status | +|:---|:---|:---|:---|:---|:---|:---| +| EV-AC-01-01 | api-response | 1 day | 2h ago | GitHub API | /orgs/acme/settings | Current | +| EV-AC-01-02 | config-file | 1 day | 2h ago | Security Insights | SECURITY-INSIGHTS.yml:8 | Current | + +### Posture reporting + +Consumers can generate reports that combine both dimensions — freshness from the type definition, provenance from the mapping: + +``` +Requirement: OSPS-AC-01.01 (MFA Enforcement) + Control review: not due (last reviewed 15d ago, frequency 90d) + Evidence: + - api-response — STALE (collected 36h ago, valid-for 1d) + source: GitHub API → /orgs/acme/settings + - config-file — STALE (collected 36h ago, valid-for 1d) + source: Security Insights → SECURITY-INSIGHTS.yml:8 + integrity: sha256:e9f0a1b2c3... + - quarterly-access-review — current (collected 15d ago, valid-for 90d) + Alert: api-response and config-file overdue — re-scan recommended +``` + +## Consequences + +### Positive + +- **No changes to `#Evidence`.** ADR-0023's schema is unchanged. The `type` field already exists. +- **Two dimensions, typed separately.** `frequency` (plan-level) drives the audit schedule; `valid-for` (evidence-type-level) drives staleness alerts. Both are integers in days — computable, not prose. + +### Negative + +- Adds `#EvidenceTypeDefinition` to the schema vocabulary. +- Retypes `frequency` from `string` to `int` and replaces `evidence-requirements` on `#AssessmentPlan`, which is a breaking change to the experimental schema. + +## Alternatives Considered + +### Keep `frequency` and `evidence-requirements` as bare strings + +Leave `#AssessmentPlan` as-is and let consumers interpret the fields. + +**Rejected because:** Every consumer would independently invent freshness semantics. The bare strings carry no structure for tooling to act on. +