CI Causal Memory: spacetime graph that learns from every run

[safal207]

Purpose

Add an append-only causal spacetime memory for CI so every run records what changed, where a signal appeared, what preceded it, what fixed it, and whether the fix remained effective.

This is a follow-up to #230 / PR #231 and complements #232. It must not make CI self-modifying or grant merge authority.

Core idea

The current Lotus system graph models one audit trajectory through repository, workflow, job, step, shell, test, evidence, verdict, and governance space.

The new layer connects multiple exact-head runs over time:

commit/change
  → workflow/job/step
  → observation/failure signature
  → causal hypothesis
  → fix commit
  → validation run
  → regression guard
  → later recurrence or durable prevention

CI “learning” means accumulating deterministic evidence and producing advisory rule or regression-test proposals. It must never silently rewrite gates, tests, workflows, or policies.

Spatial model

Every observation must be located in a stable hierarchy:

1. repository
2. ref and exact commit
3. workflow
4. job
5. step
6. shell and working directory
7. command/test target
8. changed file/module/path
9. artifact/log span
10. detector or policy rule

Required spatial edges include:

• contains
• executes
• reads
• writes_artifact
• tests
• depends_on
• observed_at
• affects_path

Temporal model

Each run is immutable and linked through explicit time relations:

• preceded_by
• supersedes
• first_seen_in
• last_seen_in
• recurred_after
• fixed_by
• validated_by
• stale_after_head_change

Evidence from run N may not be reused for run N+1 unless the relevant commit, workflow, policy, environment, and tested bytes are unchanged and the reuse rule is explicit.

Causal model

Nodes:

• run
• change_set
• environment_state
• workflow_topology
• observation
• failure_signature
• hypothesis
• confirmed_cause
• fix
• validation
• regression_guard
• recurrence
• learning_proposal
• human_decision

Edges:

• may_cause
• correlates_with
• reproduces
• confirms
• contradicts
• mitigated_by
• prevented_by
• recurs_despite
• proposed_from
• approved_by_human

A cause is never marked confirmed from correlation alone.

Per-run observation artifact

Each CI run emits a deterministic artifact such as:

ci-causal-observation-v0.1.json

Minimum fields:

• schema version
• repository identity
• ref
• exact commit SHA
• tree SHA when trusted materialization exists
• workflow run ID and attempt
• workflow/job/step identity
• command/test target
• start/end timestamps and duration
• conclusion
• normalized failure signature
• changed paths
• evidence hashes
• detector/policy versions
• environment fingerprint with secrets removed
• predecessor run reference
• authority boundary

Logs must be reduced to stable fingerprints and bounded excerpts; secrets and personal data must not enter memory.

Durable learning pipeline

1. Observe

The PR workflow emits only the immutable observation artifact. It cannot write repository memory.

2. Aggregate

A separate trusted workflow_run job on the default branch downloads completed-run artifacts without executing PR code.

3. Link

The aggregator links the observation to prior exact-head runs and deduplicates normalized signatures.

4. Score

Deterministic confidence levels:

• observed_once
• repeated
• reproduced
• fix_correlated
• fix_validated
• regression_protected

No opaque ML score is allowed in the gate path.

5. Propose

Repeated or validated patterns create advisory proposals:

• add a regression fixture;
• expand an existing graph route;
• improve a detector;
• narrow noisy CI;
• mark an obsolete rule;
• investigate a flaky environment dependency.

6. Human acceptance

A proposal may open a draft issue or draft PR. It cannot directly modify protected CI, approve, merge, deploy, or close findings.

Storage

Do not commit observations from untrusted PR workflows directly to main.

Initial options, in preferred order:

1. immutable Actions artifacts per run;
2. a trusted aggregator-generated artifact/index;
3. a dedicated ci-memory branch updated only by the default-branch workflow;
4. draft PRs for durable accepted fixtures and graph changes.

The repository source of truth remains reviewed code, not ephemeral learned state.

Suggested files

standards/lotus-family/ci-memory/
  SCHEMA.md
  ci-causal-observation-v0.1.schema.json
  ci-causal-memory-v0.1.schema.json
  example-observation.json
  example-memory.json

standards/lotus-family/conformance/
  lotus_family_ci_observation.py
  lotus_family_ci_memory.py
  test_ci_observation.py
  test_ci_memory.py

.github/workflows/
  lotus-ci-observation.yml
  lotus-ci-memory-aggregate.yml

Graph integration

Extend the Lotus graph with at least these nodes:

• time.run_observed
• time.prior_run_linked
• evidence.failure_signature
• evidence.change_set
• cause.hypothesis
• cause.confirmed
• fix.applied
• fix.validated
• guard.regression_added
• memory.pattern_repeated
• memory.proposal_created
• authority.human_acceptance_required

Required trajectories:

New failure

change_set → observation → failure_signature → hypothesis → UNKNOWN/advisory investigation

Confirmed fix

failure_signature → confirmed_cause → fix → validation_run → regression_guard → durable prevention

Recurrence

prior_signature → later_run → same_signature → recurrence → proposal to strengthen detector

Stale learning

policy/workflow/test semantics changed → prior inference stale → no automatic reuse

Acceptance criteria

• Every completed CI run can emit one schema-valid immutable observation.
• Two identical normalized failures are linked without duplicating the signature node.
• A changed exact head invalidates run-specific approval/review evidence.
• Correlation alone cannot produce confirmed_cause.
• A fix becomes fix_validated only after a later exact-head run passes the previously failing path.
• A regression guard records the exact fixture/test/policy that protects the cause.
• Unknown or conflicting evidence remains visible as UNKNOWN.
• PR code cannot modify durable memory or the aggregator.
• Learning output is advisory and requires human acceptance.
• PASS != APPROVED != MERGED remains explicit.

First vertical slice

Pilot only on Lotus Family conformance:

1. emit observation JSON from the conformance workflow;
2. normalize the failing unittest/test identity and reason code;
3. link consecutive exact-head runs;
4. recognize a repeated signature;
5. generate a draft issue body suggesting a regression fixture;
6. verify that no repository or CI mutation occurs automatically.

Authority boundary

The graph remembers evidence and proposes improvements. It grants no ownership, approval, execution, delivery, deployment, or merge authority.