[BUG] Persistent storage has no single-writer guard: a second process silently destroys acknowledged writes

[tdewitt]

Every persistent StorageBackend serializes its entire in-memory map to a temp file and renames it over the shared file. Nothing locks the data directory, so a second emulator process starts without complaint and each write is a whole-map overwrite of the other process's state. Writes that returned 200 are silently and permanently lost.

This is not specific to one service. The write lives in the three generic backends — PersistentStorage.java:139, HybridStorage.java:152, WalStorage.java:194, each doing writeValue(tempFile, store) — so it affects all 22 services that register a store (bigquery, cloudfunctions, cloudkms, cloudlogging, cloudmonitoring, cloudrun, cloudsql, credentials, datastore, eventarc, firebaseauth, firestore, gcs, gke, iam, kafka, operations, pubsub, scheduler, secretmanager, serviceusage, tasks).

Reproduction

Two emulator processes sharing one data directory:

java -Dfloci-gcp.port=4611 -Dfloci-gcp.storage.mode=persistent \
     -Dfloci-gcp.storage.persistent-path=/tmp/shared -jar quarkus-run.jar &
java -Dfloci-gcp.port=4612 -Dfloci-gcp.storage.mode=persistent \
     -Dfloci-gcp.storage.persistent-path=/tmp/shared -jar quarkus-run.jar &

GCS — acknowledged upload destroyed

pA create bucket-a                -> 200
pB create bucket-b                -> 200
pA upload bucket-a/payroll.csv    -> 200
pB upload bucket-b/other.csv      -> 200

gcs-objects.json on disk holds only bucket-b/other.csv

after stopping both and restarting a single process:
  bucket-a    = 404
  payroll.csv = 404   <-- uploaded with 200
  bucket-b    = 200
  other.csv   = 200

GCS object payload bytes are also held in a whole-map store (gcs-object-data.json, StorageBackend<String, byte[]>), so this destroys object contents, not just metadata.

Secret Manager — completed deletion resurrected

Worse than plain loss, because it fails toward existence:

pA DELETE doomed                  -> 200, doomed correctly absent from disk
pB create bystander (unrelated)   -> 200
doomed REAPPEARS in secretmanager-secrets.json

after restart, single process:
  GET doomed = 200
  secretmanager-deletions.json = {}

The empty marker file means startup recovery can never repair it. A secret the API said was deleted is permanently back.

How users reach this

This is unlikely to affect most users, and it surfaced through deliberate testing rather than through a real incident.

Persistence is opt-in. The default mode is memory, which writes nothing and is entirely unaffected. The documented configurations are safe:

• The documented Compose example in docs/configuration/docker-compose.md uses a named volume, and Compose namespaces volumes per project, so parallel instances get separate storage.
• Sidecar containers use host-persistent-path, a separate config key from persistent-path, and only ever touch nested subdirectories such as cloudsql/ and kafka/ — never the data-directory root. FLOCI_GCP_DOCKER_RESOURCE_NAMESPACE already isolates sidecar container and volume names for parallel instances.

Reaching the collision requires pointing two live processes at one directory with a non-default storage mode enabled. The most plausible accidental route is contributor-local rather than end-user: the repo-root docker-compose.yml bind-mounts host ./data, and make run (quarkus:dev from the repo root) also resolves persistent-path to ./data, so enabling persistence with both running would collide.

How this surfaced: adversarial probing while validating unrelated Secret Manager deletion-durability work. "Multi-process writers" was a known untested area, so the scenario was constructed by hand — two JVMs on separate ports, pointed at one directory, persistence forced on.

Probably a low priority, given how unlikely it is to hit — though the failure is severe when it does occur, being silent, permanent, and unrecoverable.

For whoever picks this up: StorageFactory.java:110-119 already guards this exact failure mode in-process, commented "the stale duplicate flushes after the active instance and clobbers persisted state" — so the cross-process gap looks like an oversight rather than an accepted constraint.

Suggested fix

Acquire an advisory FileChannel.tryLock on a lockfile in the persistent path during storage factory init, and refuse startup with an actionable message naming the directory when the lock is held. The OS releases the lock on process death, so SIGKILL leaves no stale lock, and memory mode is unaffected.

Failing that, the constraint should at least be documented — today there is no lock, no warning, no log line, and nothing in the docs saying not to do this.

Environment

floci-gcp 0.7.0, packaged JVM build (target/quarkus-app/quarkus-run.jar), macOS (Darwin 27.0.0), Temurin 25.0.2. Reproduced in persistent mode; hybrid and wal share the same whole-map write path.