docs - establish Prism relational reasoning north star (RFC 065, RFC 066)

[dannymeijer]

Summary

This documentation-only PR adds the Polyglot SQL AST boundary RFC, a Prism relational-reasoning research whitepaper, and RFC 066: the normative North Star for bounded memo optimization, workload-level shared-work reasoning, heterogeneous placement, provider-neutral planning evidence, and adaptive Prism re-entry across local, cluster, bounded, continuous, and explicitly modeled graph or recursive work.

RFC 065 makes Polyglot an optional typed SQL ingress and egress boundary while retaining Prism as semantic owner. RFC 066 defines Prism as one standalone reasoning engine operating in pre-execution and bounded adaptive re-entry modes over the same authored graph, semantic rules, memo, property model, and explanation contract. Local execution is the one-placement case of distributed planning; streaming adds boundedness, update, time, progress, and state legality; graph work adds explicit identity, path, recursion, fixpoint, and termination semantics. Target lowerers, Session, adapters, Catalyst/AQE, DataFusion, PostgreSQL, graph targets, and other target optimizers retain representation-specific lowering, physical planning, runtime adaptation, and execution ownership.

The PR also amends RFC 008 and RFC 048 so externally retained cross-execution evidence, distributed target facts, continuous frontiers, and state-compatibility evidence enter the architecture without giving Prism execution or storage responsibilities. The whitepaper treats Spark Catalyst/AQE, GRainDB, DuckPGQ, and differential dataflow as specific prior art, connects Prism evidence to local plan-diff and blast-radius inputs, and defines reproducible research without making a performance or operational-cause claim.

Type of change

• Bug fix
• New feature
• Refactor / maintenance
• Documentation
• CI / tooling
• RFC (adds/updates docs/rfcs/*)

Area(s)

Select the primary areas touched (labels sync from checked lines when the triage workflow runs):

• Package & tests
• Specification (RFCs)
• Documentation
• Automation & repo config
• Other

Key details

• User-facing behavior: No runtime behavior changes. The PR proposes the contracts future Prism, SQL-ingress, target-egress, evidence-provider, statistics, and adaptive-planning work must satisfy.
• One engine, two temporal modes: Pre-execution planning and bounded adaptive re-entry use one Prism semantic and explanation model. Adaptive re-entry is an explicit planning event over unfinished bounded work or future continuous work, not a second optimizer.
• Execution modes: Local execution is the one-placement case of distributed planning. Cluster topology, scheduling, shuffle realization, retry, and recovery stay target- or coordinator-owned.
• Streaming semantics: Boundedness, update mode, time, watermark or progress, bounded state, replay, and sink requirements constrain rewrite legality. Continuous re-entry may change only future work at a compatible committed frontier.
• Graph semantics: Prism plan, lineage, and evidence graphs remain distinct from user graph-query semantics. Relational and graph-native realizations may be compared only under explicit node/edge identity, path, recursion, fixpoint, termination, and result contracts.
• Open evidence ingress: Any conforming caller may supply immutable evidence. Prism remains useful without it, treats evidence by declared authority and provenance rather than provider identity, and never owns the provider's storage or service lifecycle.
• Runtime evidence: Declared or inferred facts, pre-execution estimates, and observations remain distinct. Historical estimates and observations may rank legal alternatives but cannot establish rewrite legality or become authored truth.
• Execution ownership: Session or another coordinator owns observation collection, safe checkpoints, completed work, committed progress, state lifecycle, fallback policy, and the decision to request replanning. Target engines retain target-local physical adaptation.
• Operational boundary: Prism may expose estimate-versus-observation divergence and plan regressions, but operational-cause interpretation, alerting, organization-wide blast radius, incident classification, and automated response remain outside Prism.
• Research programme: Adds independent evidence-provider tests, native Catalyst/CBO/AQE baselines, optimizer-complexity stress graphs, local-versus-cluster studies, streaming trace and state-migration studies, property-graph and recursive-query studies, development-to-production impact studies, unchanged target-native baselines, explicit ablations, planning-cost measurements, and selected-versus-realized plan comparison.
• Research evidence: Retains the exact exploratory versions, fixtures, query and graph shapes, PostgreSQL results and EXPLAIN differences, the separation between Prism and Polyglot prototypes, and the incomplete DataFusion lane.
• Risks: RFCs 048, 065, and 066 remain ambitious Drafts. Stable planning-context transport, materialization lifecycle, continuous-frontier and state-migration contracts, graph and fixpoint profiles, safe checkpoints, public optimizer APIs, and calibrated change-impact forecasting still require focused follow-on contracts and reproducible evidence.

Testing / verification

• make ci (or make fmt-check, make build, make test)
• Manual verification described below

Manual verification notes:

• git diff --cached --check and git diff --check passed.
• RFC review passed: required header fields, canonical section order, Draft and Planned lifecycle tails, plain RFC references, normative language, balanced non-goals, and no forbidden Draft implementation plan.
• RFC 065 and RFC 066 numbering remains unique.
• Terminology scans found no stale InQL, adaptive-reentry, initial-spike, session-only evidence, or internal-review wording in the reviewed documents.
• Per maintainer instruction, package CI was not rerun for this Markdown-only amendment.

Docs impact

• No docs changes needed
• Docs updated
• Docs follow Divio intent (tutorial/how-to/reference/explanation) where applicable

If docs updated:

• Link(s): docs/rfcs/008_optimizer_boundary_stats_cbo_aqe.md, docs/rfcs/048_cluster_execution_backend_mode.md, docs/rfcs/065_polyglot_sql_ast_ingress_and_egress.md, docs/rfcs/066_prism_relational_reasoning_and_shared_work_optimization.md, docs/whitepapers/prism_relational_reasoning.md
• RFC lifecycle: RFC 008 remains Planned. RFCs 048, 065, and 066 remain Draft. This PR specifies direction and evidence requirements; it does not implement the RFCs.

Checklist

• I kept public docs user-focused and moved implementation mechanics out of the normative contracts.
• I avoided duplicating canonical install/run instructions in multiple places.
• I added/updated tests where it materially reduces regressions.
• I separated semantic correctness, target-plan shape, optimizer tractability, measured performance, and operational interpretation.
• I preserved target-optimizer ownership while allowing independently supplied evidence to inform Prism planning.
• I kept scheduler, streaming-runtime, graph-engine, state, recovery, and sink-commit mechanics outside Prism while preserving their legality-relevant properties.
• I kept longitudinal storage and organization-wide operational intelligence outside Prism.
• I credit Polyglot and its creator explicitly without presenting upstream parser breadth as IncQL compatibility.