Release simple-volume v0.2.0

Author: seslly

README.md

@@ -1,6 +1,6 @@
 # simple-volume
 
-`simple-volume` is a Kubernetes-native async replicated local-volume prototype.
+`simple-volume` is a Kubernetes-native async replicated local-volume system.
 Applications use normal PVCs backed by the `simple-volume.shipstuff.io` CSI
 driver. The controller and node agents handle the local replica lifecycle,
 freshness policy, and promotion state.
@@ -26,9 +26,10 @@ acceptable fit for many single-writer workloads.
 See [docs/why-simple-volume.md](docs/why-simple-volume.md) for the full design
 rationale, replication model, and storage options we evaluated.
 
-## V0 Scope
+## Initial Release Scope
 
-The V0 scope is intentionally narrow and complete enough for initial release:
+The `v0.2.0` release is intentionally narrow and ready for initial real
+workload use with understood async-replication tradeoffs:
 
 - dynamic PVC provisioning into logical SimpleVolumes
 - chart-configured local storage pools
@@ -40,38 +41,56 @@ The V0 scope is intentionally narrow and complete enough for initial release:
 
 Replication logic does not run inside CSI. CSI is the Kubernetes mount boundary;
 the controller owns policy and the node agent owns local filesystem operations.
-Like `local-path`, v0 does not enforce PV capacity at the filesystem layer. The
-PVC request is used for Kubernetes API shape, placement decisions, and operator
-visibility; workloads can still consume available space in the backing local
-pool unless the host filesystem enforces its own limits.
+Like `local-path`, `simple-volume` does not enforce PV capacity at the
+filesystem layer. The PVC request is used for Kubernetes API shape, placement
+decisions, and operator visibility; workloads can still consume available space
+in the backing local pool unless the host filesystem enforces its own limits.
 
 ## Replication Model
 
-The intended V0 replication path is watch-driven:
+The intended replication path is watch-driven and, by default, shadow-backed:
 
 - the active node-agent watches configured durable paths inside the active
   volume
 - file events are debounced into batches
-- replica agents receive batch sync requests
-- replica agents pull changed files from the active agent's read-only rclone
-  WebDAV endpoint
+- in `shadow` consistency mode, the source agent first stages those changes into
+  a local shadow tree
+- replica agents receive batch sync requests and pull from the completed shadow
+  tree through the active agent's read-only rclone WebDAV endpoint
 - the WebDAV server uses a short directory cache so newly written files are
   visible to watch-triggered pulls
+- a generation is considered successfully delivered only after the configured
+  number of replicas confirms it
 - an off-hours full resync schedule, such as `0 4 * * *`, provides a safety net
   for missed events or agent restarts
 
 Volumes can replicate only selected folders/files. For game servers, this keeps
 large reconstructable game downloads out of the hot replication path while still
 protecting save/config state.
 
+The default chart resources are sized for a low-memory steady state. The node
+agent uses conservative rclone settings for replication work: single-transfer
+copying, one checker, no explicit transfer buffer, and no multi-threaded stream
+copying. That keeps memory predictable on small clusters, including Pi-class
+nodes, but it also means large initial seeds can be slow. Operators that want
+faster whole-volume seeding should raise the agent memory limit and the rclone
+concurrency/buffer settings together once those knobs are exposed. Raising
+memory alone only increases headroom; it does not make a single-transfer sync
+faster.
+
+Treat 1:1 full-volume seeding as an operational bootstrap path, not the normal
+replication profile. It walks many more files, includes reconstructable game
+runtime data, and can consume noticeably more accounted memory from kernel page
+cache and rclone directory/check state than scoped steady-state replication.
+
 The operator chooses the storage-capable node set when installing the chart by
 scheduling the node-agent DaemonSet with normal Kubernetes selectors, affinity,
 and tolerations. Workloads that consume `simple-volume` storage should also
 declare matching scheduling intent in their own manifests. `simple-volume` owns
 the per-volume active/fresh labels, but it should not depend on a mutating
 admission webhook to make unrelated workload charts schedulable.
 
-The node agent exposes the V0 watch control surface used by the controller and
+The node agent exposes the watch control surface used by the controller and
 manual E2E validation:
 
 - `POST /replication/watch/start` starts or replaces an active watch for a
@@ -104,7 +123,7 @@ Example start request:
 }
 ```
 
-The controller reconciles this same behavior for annotated PVCs. For V0 it
+The controller reconciles this same behavior for annotated PVCs. It
 discovers the active/source node from the running pod that mounts the claim,
 uses all other ready node-agent pods as replicas, starts the source watch, and
 optionally runs a scoped full sync against each replica.
@@ -115,15 +134,33 @@ metadata:
     simple-volume.shipstuff.io/replication-enabled: "true"
     simple-volume.shipstuff.io/replication-include-paths: "writes.log,saves/**"
     simple-volume.shipstuff.io/replication-exclude-paths: "downloads/**"
+    simple-volume.shipstuff.io/replication-prune-excluded: "false"
     simple-volume.shipstuff.io/replication-debounce: "2s"
+    simple-volume.shipstuff.io/replication-consistency-mode: "shadow"
+    simple-volume.shipstuff.io/replication-confirmed-replicas: "1"
     simple-volume.shipstuff.io/replication-full-sync-on-start: "true"
     simple-volume.shipstuff.io/replication-full-sync-schedule: "0 4 * * *"
 ```
 
+`shadow` is the safe default for DB-like file trees because replicas pull from a
+source-side staged copy instead of the live directory while the app is mutating
+it. Workloads that do not need this can opt out with
+`replication-consistency-mode: "live"`. Shadow mode writes an additional local
+copy of the included paths on the active node, so narrow include paths are the
+main mitigation for disk growth and SSD wear.
+
+Excluded paths are preserved on replicas by default. This is important for
+large reconstructable caches such as Steam game installs: they can be excluded
+from replication without being deleted during a full sync. Set
+`simple-volume.shipstuff.io/replication-prune-excluded: "true"` only when the
+replica must be an exact filtered mirror and deleting excluded files is
+acceptable.
+
 Replication uses rclone metadata preservation and the controller infers replica
 file ownership from the active pod's `runAsUser`, `runAsGroup`, or `fsGroup`
 when available. PVC annotations can override this for workloads without a pod
-security context:
+security context. File-mode overrides preserve existing executable bits so
+launch scripts and runtimes are not made non-executable by normalization:
 
 ```yaml
 metadata:
@@ -138,7 +175,7 @@ The current schedule parser intentionally supports exact minute/hour cron
 windows such as `0 4 * * *`; ranges and step expressions are left for the
 controller-runtime implementation.
 
-Opt-in V0 failover is PVC annotation driven:
+Opt-in automatic failover is PVC annotation driven:
 
 ```yaml
 metadata:

cmd/simple-volume/main.go

@@ -48,6 +48,7 @@ func runController(args []string) {
 	tokenSecretName := fs.String("sync-token-secret-name", getenv("SIMPLE_VOLUME_TOKEN_SECRET_NAME", "simple-volume-sync-token"), "secret containing the agent sync token")
 	tokenSecretKey := fs.String("sync-token-secret-key", getenv("SIMPLE_VOLUME_TOKEN_SECRET_KEY", "token"), "secret key containing the agent sync token")
 	reconcileInterval := fs.Duration("reconcile-interval", 30*time.Second, "replication reconcile interval")
+	failoverTimeout := fs.Duration("failover-timeout", 10*time.Second, "maximum time to spend on failover reconciliation before continuing replication")
 	_ = fs.Parse(args)
 	log.Printf("starting simple-volume controller driver=%s addr=%s replication=%t", v1alpha1.DriverName, *addr, *enableReplication)
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
@@ -63,6 +64,7 @@ func runController(args []string) {
 		TokenSecretName:   *tokenSecretName,
 		TokenSecretKey:    *tokenSecretKey,
 		ReconcileInterval: *reconcileInterval,
+		FailoverTimeout:   *failoverTimeout,
 	})
 	if err != nil && err != context.Canceled {
 		log.Fatalf("replication controller: %v", err)
@@ -131,6 +133,8 @@ func runAgent(args []string) {
 	})
 	mux.HandleFunc("/replication/sync-batch", agent.SyncBatchHandler(pool, auth, agent.ExecRunner{}, 10*time.Minute, replicationLocks))
 	mux.HandleFunc("/replication/full-sync", agent.FullSyncHandler(pool, auth, agent.ExecRunner{}, time.Hour, replicationLocks))
+	mux.HandleFunc("/replication/shadow/prepare", agent.ShadowPrepareHandler(pool, auth, agent.ExecRunner{}, time.Hour, replicationLocks))
+	mux.HandleFunc("/replication/archive", agent.ArchiveHandler(pool, auth))
 	mux.HandleFunc("/replication/watch/start", watchManager.StartHandler(auth))
 	mux.HandleFunc("/replication/watch/stop", watchManager.StopHandler(auth))
 	mux.HandleFunc("/replication/watch/status", watchManager.StatusHandler(auth))

docs/e2e-deployment-tasks.md

@@ -1,15 +1,16 @@
 # Simple Volume E2E Deployment Tasks
 
-This is the implementation checklist to get `simple-volume` from the current
-V0 scaffold to a cluster-tested E2E flow.
+This is the implementation checklist that moved `simple-volume` from scaffold
+to a cluster-tested E2E flow. The current release gate is `v0.2.0`.
 
 ## 1. Publish The Image
 
 - Commit the current scaffold and push `main`.
 - Confirm `.github/workflows/publish-images.yml` publishes
   `ghcr.io/shipstuff/simple-volume:sha-<sha>` on the `main` push.
 - Use the `sha-<sha>` tag for first cluster installs.
-- Cut `v0.1.0` only after the CSI smoke test passes.
+- Cut `v0.2.0` only after local checks, image/chart publish workflows, and the
+  Windrose canary two-leg fire drill pass.
 
 ## 2. Define A Safe Test Pool
 
@@ -74,7 +75,7 @@ kubectl exec deploy/simple-volume-demo -- sh -c 'echo ok >> /data/e2e.txt && cat
   source agent's read-only WebDAV endpoint.
 - Add cron-style off-hours full resync from the controller as a safety net,
   using `sync.fullResyncSchedule` such as `0 4 * * *`; do not create one
-  Kubernetes CronJob per volume in V0.
+  Kubernetes CronJob per volume.
 - Make CSI node authorization read the controller/volume status instead of the
   current local static authorization scaffold.
 - Add per-volume node labels for active/healthy scheduling.
@@ -90,7 +91,7 @@ kubectl exec deploy/simple-volume-demo -- sh -c 'echo ok >> /data/e2e.txt && cat
   local copy under `.simple-volume-backups/`, restores from the promoted leader,
   and remains a replica unless an explicit move-back is requested.
 
-Recommended V0 replication policy shape:
+Recommended replication policy shape:
 
 ```yaml
 sync:
@@ -110,7 +111,7 @@ save/config state and reconstructable game files.
 
 ## 6. Release Gate
 
-Before tagging `v0.1.0`, all of these should pass:
+Before tagging `v0.2.0`, all of these should pass:
 
 ```bash
 go test ./...
@@ -126,3 +127,5 @@ And in-cluster:
 - demo PVC binds.
 - demo pod mounts and writes data.
 - non-empty uninitialized pool adoption fails by default.
+- Windrose canary two-leg failover completes with confirmed post-promotion full
+  sync and watch restart on each promoted active node.

docs/failover-selection-tasks.md

@@ -1,7 +1,7 @@
 # Failover Selection Tasks
 
-This tracks the post-`v0.1.0` failover selection work discussed after the
-initial release tag.
+This tracks failover selection work included in the `v0.2.0` initial usable
+release.
 
 ## Implemented
 
@@ -20,6 +20,10 @@ initial release tag.
   requests and node allocatable capacity.
 - Keep PV/PVC annotations as the status source of truth and keep CSI as the
   final mount authorization guard.
+- Do not delete unbound Pending consumer pods during failover. After promotion,
+  a Pending pod can be scheduled naturally once the active node label moves.
+  Only delete/evict consumer pods that are already bound to a non-promoted node
+  and need fencing or owner-controller recreation.
 
 ## Still Future
 

docs/replication-pruning-tasks.md

@@ -0,0 +1,39 @@
+# Replication Pruning Tasks
+
+## Decision
+
+Excluded paths are preserved on replicas by default. Scoped replication treats
+included paths as authoritative durable data and excluded paths as node-local
+cache unless an operator explicitly opts into pruning.
+
+Use:
+
+```yaml
+metadata:
+  annotations:
+    simple-volume.shipstuff.io/replication-prune-excluded: "true"
+```
+
+only when the replica should be an exact filtered mirror and deleting excluded
+files is acceptable.
+
+## Rationale
+
+For game servers, excluded paths often contain large reconstructable runtime
+data such as Steam libraries, game binaries, shader caches, or logs. Deleting
+those directories during full sync saves disk but can turn failover into a slow
+redownload. Preserving them gives each node a warm local cache while still
+replicating the durable save/config paths.
+
+## Tasks
+
+- [x] Default rclone full sync to preserve excluded paths.
+- [x] Add opt-in `replication-prune-excluded` PVC annotation.
+- [x] Pass prune policy from controller to source shadow prepare and target
+  full sync requests.
+- [x] Cover default preservation and opt-in pruning with unit tests.
+- [x] Run a Windrose canary failover after a confirmed full sync and verify the
+  already-warm replica does not redownload the excluded game install on
+  promotion.
+- [ ] Document any workloads that require exact filtered mirrors and should set
+  `replication-prune-excluded: "true"`.

docs/shadow-consistency-tasks.md

@@ -0,0 +1,78 @@
+# Shadow Consistency Tasks
+
+This tracks the next replication reliability step after the Windrose canary
+fire drills showed that live file replication can still expose an app-level
+inconsistent RocksDB state.
+
+## Implemented In This Pass
+
+- Add `simple-volume.shipstuff.io/replication-consistency-mode`.
+- Default consistency mode to `shadow`; allow opt-out with `live`.
+- Add global `simple-volume.shipstuff.io/replication-confirmed-replicas`.
+- Default confirmed replicas to `1`.
+- Let the source agent prepare a local shadow copy under
+  `.simple-volume-shadows/<namespace>/<volume>/current/data`.
+- Keep shadow metadata beside the payload so internal marker files are not
+  restored into the user volume.
+- Let full sync replicas pull from the prepared shadow path instead of the live
+  volume path when shadow mode is active.
+- Let watch batches update the source shadow before replicas pull changed files.
+- Treat deletes from shadow batches as authoritative so replicas can prune files
+  without waiting for an off-hours full sync.
+- Mark a generation successful only after the configured number of replicas
+  confirms delivery.
+
+## Validated
+
+- Windrose canary two-leg fire drills pass with shadow mode enabled.
+- Startup full sync completes before the source watch starts.
+- Each promotion runs a post-promotion startup full sync and restarts the watch
+  from the promoted active node.
+- Returning stale nodes back up local state before restoring from the promoted
+  leader.
+
+## Still To Measure
+
+- Source-node disk growth during active gameplay.
+- Shadow bytes copied per batch for RocksDB-heavy workloads.
+- Scheduled maintenance-window full sync behavior over longer runs.
+
+## Operational Caveats
+
+Shadow mode increases source-node writes because durable data is written once by
+the app and again into the local shadow. The expected max active-node footprint
+is approximately:
+
+```text
+live included data + shadow included data
+```
+
+This is why scoped `replication-include-paths` matter. Avoid `**` for workloads
+that constantly rewrite large binary trees unless the extra disk use and SSD
+wear are acceptable.
+
+Mitigations:
+
+- replicate only durable paths, not reconstructable runtime downloads
+- debounce hot file changes into batches
+- keep full sync as startup, repair, and maintenance-window work
+- expose metrics for shadow bytes copied, generation duration, confirmation
+  lag, and failed confirmations
+- later prefer reflink copies where the filesystem supports them
+- later add a snapshot backend for ZFS or Btrfs hosts
+
+## Backend Boundary
+
+The default backend remains file-shadow plus rclone/WebDAV because it keeps
+`simple-volume` portable and lightweight.
+
+ZFS would be useful as an optional future backend:
+
+- atomic source snapshots
+- cheap local generation creation
+- block-level incremental send/receive
+- stronger checksums and rollback tools
+
+It should stay optional. Making ZFS mandatory would turn this project into a
+Kubernetes operator for replicated ZFS datasets instead of a simple local-volume
+controller with async recovery points.