AGENTS.md

Project guidance

• The architecture source of truth is docs/kafka-successor-architecture.md; the product problem catalogue is docs/kafka-pain-points.md; product motivation and Kafka equivalents are in docs/why-whitewater.md; operational experience requirements are in docs/operational-experience.md.
• docs/tasks.md is the living execution tracker. Keep one current-focus milestone, mark work in progress before implementation, and check items only after linked tests/evidence pass.
• Preserve the public model riverbed -> domain -> feed -> key -> cursor -> subscription; do not expose physical partitions. Domain replaces Space in new WCL, typed commands, docs, and client helpers; keep older Space commands and the persisted space_id JSON/catalog field as compatibility aliases until a separately versioned migration, without changing resource IDs or existing Feeds.
• WCL commands, HTTP control requests, CLI commands, SDKs, and future MCP tools must converge on the typed ControlController; do not duplicate resource semantics in interface layers.
• Whitewater Streams must offer the same Writer, Reader, Subscription, StateStore, Pipe, Cursor, retry, and atomic-effect semantics in Rust, Python, Java, C#, Node.js/TypeScript, and Go. Java is not the privileged processing runtime; gate published features and runnable guides on one cross-language three-Node conformance suite, with explicit unsupported capabilities. Node.js record nanoseconds use bigint/decimal strings, never imprecise JavaScript numbers.
• Standard three-Node development uses the OpenRaft-backed Control Plane and reports control_plane authority only after majority commit. local_prototype is reserved for Nodes without Control Plane configuration.
• Public terminology is Control Plane; Raft, voters, and majority quorum are internal correctness mechanisms rather than user-managed resources.
• Feed names use lowercase dotted segments matching ^[a-z][a-z0-9]*(\.[a-z][a-z0-9]*)*$ with a 512-byte maximum, while immutable FeedId values carry identity across renames.
• Supported Riverbeds require at least three Nodes and three active replicas per internal range or progress shard, including the standard development topology. A 12/24-Node candidate-pool placement test does not prove a live 12/24-Node Riverbed or unlimited capacity: current executable configuration derives eligible storage Nodes from static Control Plane Nodes, and Feed creation initially takes the first three. Separate storage roles/voters and prove larger live Riverbeds before scaling claims.
• Treat Whitewater as both a distributed database and a streaming platform: Feeds provide immutable temporal history, while persisted replicated Indexes provide current and queryable state.
• Record-attached arbitrary key/value bytes are called Metadata, never Headers; reserve header terminology for transport protocols such as HTTP.
• Every record stores signed 64-bit Unix epoch nanoseconds in event_time_ns and ingest_time_ns; do not reduce record time to milliseconds. JSON represents these as decimal strings to avoid JavaScript precision loss.
• Standalone Readers own independent current Cursors; a declared Subscription is metadata only until shared RF3 member leases and progress are implemented. The local SubscriptionLeaseTracker state machine is not persisted or wired to a join endpoint and must not be treated as authoritative after failover. Each eventual named Subscription owns independent durable acknowledged progress shared only by Readers cooperating in it. No Kafka-style offset/transaction internal topic may appear as a user Feed: keep progress, lease, retry, and effect-coordinator state in separate internal RF3 state/journals with bounded Fjall-local rows/checkpoints; a Fjall transaction does not atomically commit the Feed log. Preserve existing Reader progress via explicit one-to-one migration, never silently share it. A named multi-range Reader can return an opaque delivered_cursor progress token distinct from records[].cursor; acknowledge the former, never assume the last record Cursor covers other ranges. Keep the internal acknowledged/delivered frontier out of public Reader definitions; translate the acknowledged frontier only from bounded, corroborated frozen source history in the same consensus cutover that changes routing, reset delivered work to acknowledged progress and fence old sessions. Refuse cutover on missing/stale lineage or a Writer dedup conflict instead of guessing a Cursor; a merge may replay already acknowledged work but must never skip unacknowledged history. The current consensus-catalog frontier is a correctness prototype only: do not claim many-Reader production scale until ReaderId-sharded RF3 progress, bounded exact fetch retries, and load/fault evidence replace per-fetch Control Plane writes.
• New Feed Cursors are opaque hashes scoped to immutable FeedId plus request identity, not encoded physical positions; already persisted legacy tokens remain valid. The single-range owner rebuilds an indexed committed Cursor-to-position lookup from its checked log; never infer placement from a Cursor or claim scalable multi-range continuation until a durable logical ordering/index contract is implemented.
• On Control Plane Riverbeds, a current owner must obtain matching committed-prefix/digest evidence from another RF3 replica before serving a range read; fail closed when a follower is ahead, checksums differ, or corroboration is unavailable. This per-read check does not replace a durable quorum committed watermark when multiple copies have been lost.
• Use C:\Code\Stackapps\kafka-trunk as a read-only implementation reference and record retained lessons in docs/kafka-source-lessons.md; preserve proven correctness behavior without copying Kafka's public partition coupling.
• Feed history has one immutable semantic model; use explicit persisted replicated Indexes rather than cleanup-policy or compacted-Feed modes. A Domain is an administrative namespace, not a Feed cleanup-policy variant.
• Application-defined secondary Indexes are part of Whitewater's core storage contract. Use stable Feed/application-Key or MessageId references, a small shared set of bounded keyspaces rather than one LSM tree per Index, serializable primary/posting/uniqueness updates, and applied-Cursor fencing. Never claim a Fjall-local transaction atomically commits the separate Active Range log; do not expose Index queries until replicated correctness is proven.
• StateStores belong to Domains: Feed-derived stores name their source FeedId, while manual stores require their own replicated mutation journal and checkpoint. A Domain is not a log. Enrichment must support Reader → logical StateStore lookup by application Key → Writer without client co-partitioning; commit input progress and output effect idempotently or fail closed. A declared StateStore is metadata only, never a live data service.
• Managed Pipes should provide rolling-window aggregation as a built-in StateStore view with identical semantics in all six SDKs: users declare business Key, duration, and aggregation, not sliding step, panes, partitions, local Fjall, or a required fourth Node. Event time, watermark/lateness, retention, revision, cost bounds, RF3 state, and epoch-fenced movable compute leases must be explicit and tested before claiming window support.
• The pinned Fjall dependency currently backs isolated local Index, Reader-progress, and Subscription-progress prepare/commit prototypes only; it has not been selected as a replicated production engine. New Subscription declarations now persist private Control Plane-authoritative initial progress placement, but existing declarations lacking it must fail closed; placement movement and drain remain unimplemented. The in-process coordinator can load this assignment; bounded internal prepare/commit/committed-read endpoints fence receiver/owner/epoch against current placement, while local Fjall IO runs off Tokio. Optional Subscription-only native mTLS uses a separate listener, CA validation and per-Node certificate fingerprints to bind the sender to the assigned owner; when configured it removes these endpoints from the shared-key public HTTP listener and refuses incomplete certificate/pin configuration. The development shared-key mode remains a prototype. The internal coordinator refuses any observed contradictory read or missing majority; an unavailable replica prevents treating an empty read as authoritative. Prepare/commit/read replies include the claimed responder NodeId, SubscriptionId and epoch and mismatches must fail closed before counting quorum, but these are not cryptographic identity proofs. The shared development key does not verify Node identity or encrypt traffic; optional mTLS authenticates pinned request/response peers but forwarded commit-vote evidence is not independently signed, other internal protocols still use the development transport, and quorum-backed recovery is absent. A matching digest from two local prepare-vote objects is not authenticated RF3 commit evidence and cannot justify serving acknowledged progress. Reader-progress rows and pacing policy are not wired to live Reader APIs or RF3 yet. Benchmark and fault-test Fjall against alternatives before a production storage decision, and retain an engine abstraction until then. Keep blocking Fjall operations off Tokio workers when integrating the prototype.
• Client traffic uses TLS. Inter-Node traffic requires authenticated encryption and verified Node identity: native mTLS is the default, while a trusted service mesh or equivalent orchestrator transport is an allowed mechanism when identity, rotation, audit, and downgrade prevention are verified. API keys authorize access but must never be used directly as encryption keys.
• Prioritize correctness contracts before throughput optimization.
• The host may not have Rust installed. Verification can run with rust:1.90-bookworm in Docker.
• Acceptance helper regression tests: python -m unittest discover -s tests -p "test_*acceptance.py". Failure diagnostics are captured in acceptance-logs/ before test-container cleanup and uploaded by CI.
• Ad-hoc performance drivers live in scripts/bench-*.py (never inside the acceptance glob): bench-feed.py (Whitewater, project whitewater-bench on 7771-7773), bench-kafka.py (compose.kafka.yml, 19092-19094), and examples/BenchFeed.java in whitewater-java share one BENCH_RESULT JSON contract; methodology and measured baselines are in docs/performance-testing.md.
• scripts/run-acceptance.py runs serially by default (the regression gate); --lane NAME/WHITEWATER_ACCEPTANCE_LANE runs one lane as a CI matrix leg, and --lanes N/WHITEWATER_ACCEPTANCE_LANES runs the lane catalog concurrently on one Docker host — see docs/parallel-acceptance-testing.md. Lane fixtures key off WHITEWATER_VERIFY_PORT_1..3, WHITEWATER_VERIFY_SUBNET, and COMPOSE_PROJECT_NAME.
• SDK conformance uses sdk/spec/conformance; set SDK endpoints explicitly to the isolated Riverbed, not inherited development endpoints. The Rust compression live check is cargo test --lib optional_compression_conforms_against_live_nodes -- --ignored with WHITEWATER_ENDPOINTS naming three isolated Nodes.
• Final verification is mandatory after the last edit, before marking work complete or committing: run cargo fmt --check, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-targets (which includes Rust unit and integration tests). Targeted tests during development do not replace the full final run; rerun affected and full checks after a subsequent code edit.
• Also run every repository-maintained Python/Docker integration, fault, and live acceptance script (scripts/test-*.py) as the final end-to-end gate for code changes, sequentially in an isolated disposable test Riverbed; check each script's Compose/project/volume effects first. Use compose.verify.yml and set COMPOSE_FILE=compose.verify.yml plus WHITEWATER_TEST_BASE_PORT=7371 for the default three-Node scripts (including M1.11 and M4 auto-split); follower movement and cross-Node read scripts share the isolated four-Node project on ports 7171–7174, while the owner movement script uses ports 7271–7273. Never disrupt an existing Riverbed, alter its persistent volumes, or perform destructive cleanup without specific user confirmation. If Docker's default address pools are exhausted, set WHITEWATER_VERIFY_SUBNET to an unused subnet for the disposable compose.verify.yml project and keep it set for all scripts; inspect existing network ranges first and do not prune the user's networks. If safe isolation, Docker, credentials, or a required service is unavailable, report each skipped/failed suite and the blocker; never claim "all tests passed" or mark live acceptance done.
• Standard development and membership testing use compose.cluster.yml with at least three Nodes via docker compose up --scale node=N.
• The current gossip-style membership is discovery only, not consensus or record replication.
• Autoscaling must use sustained thresholds, hysteresis, cooldowns, and one-node steps.
• Never scale in a node unless ownership/data draining has completed and the node reports safe-to-remove. Do not use the legacy metadata-only owner transfer as an operator shortcut; owner moves require a verified frozen committed boundary and a compare-and-set Control Plane cutover.
• Infrastructure scaling runs outside data nodes; do not mount Docker or orchestrator credentials into a data node.