Harden JSON-backed persistence: atomic writes + per-path serialization

[ttpears]

Problem

Every JSON-backed service persisted state with a read-modify-write followed by a direct fs.writeFile, with no locking and no atomic write. Two concrete risks:

1. Torn/corrupt files. fs.writeFile truncates the target and streams bytes in place. A crash (or container kill) mid-write leaves a truncated file that fails JSON.parse on the next startup — losing users, TSIG keys, tokens, or backups.
2. Lost updates. Services do read-modify-write against in-memory maps and then persist the whole set. Overlapping writers to the same file interleave and clobber each other. The clearest case: apiTokenService.verifyToken() fires a throttled lastUsedAt "touch" fire-and-forget, which can race a concurrent createToken/revokeToken and drop that change.

Affected services: tsigKeyService, apiTokenService, userService, backupService, plus webhookConfigService and ssoConfigService (same pattern).

Fix

New shared helper backend/src/utils/atomicJson.ts:

• writeFileAtomic / writeJsonAtomic — write to a uniquely-named temp file in the same directory, fsync, then fs.rename over the target. Rename is atomic on the same filesystem, so a reader only ever observes the old complete file or the new complete file, never a partial one. The temp file is cleaned up on failure.
• withFileLock — serializes all operations for a given absolute path through a per-path promise chain, so concurrent callers apply one at a time in call order. A rejecting task never wedges the queue for later callers, and the map entry is dropped once the chain drains (no unbounded growth).
• removeFileLocked — deletes under the same lock (used by backup eviction, so a delete cannot interleave with a write to the same zone file).
• flushFileLock — drains in-flight work for a path (graceful shutdown / test teardown).

Each service's write path now goes through the helper. backupService still writes/deletes multiple zone files per operation — each is now individually atomic and serialized, while distinct files stay concurrent.

Behavior is unchanged: same JSON.stringify(value, null, 2) on-disk shape; only how the bytes hit disk changed. Read paths and data shapes are untouched.

Tests

• New backend/src/utils/__tests__/atomicJson.test.ts: asserts the final write is a rename onto the target (not in-place); a failed write leaves the existing target intact and complete JSON with no stray .tmp files; 50 concurrent writes to one path all serialize with the last write winning (no lost update); same-path tasks run strictly ordered and never overlap; a rejecting task does not wedge the queue; removeFileLocked serializes against writes.
• Existing service test teardown for apiTokenService now drains the fire-and-forget write via flushFileLock before removing its temp dir.

Verification:

• cd backend && npm run build — clean.
• cd backend && npx jest — 39 suites, 369 tests, all green.

🤖 Generated with Claude Code