Request Deduplication Middleware for Idempotency

[phertyameen]

Labels: middleware, reliability, data-integrity, medium-priority

Description:

Implement request deduplication to prevent duplicate submissions and ensure idempotent operations, especially for critical actions like puzzle submissions and point claims.

Requirements:

• Detect duplicate requests using idempotency keys

• Support client-provided idempotency keys (X-Idempotency-Key header)

• Generate automatic idempotency keys for unsafe operations (POST, PUT, DELETE)

• Store request fingerprints in cache (Redis) with TTL

• Return cached response for duplicate requests

• Configure TTL per endpoint type (short for queries, long for mutations)

• Handle race conditions when multiple identical requests arrive simultaneously

• Support idempotency for critical operations:

• • Puzzle answer submissions
• • Point claiming
• • Payment transactions (future)
• • Friend requests

• Clean up expired idempotency records

• Log duplicate request attempts for monitoring

Acceptance Criteria:

• Duplicate puzzle submissions within 5 minutes return same result
• Point claiming cannot be triggered multiple times
• Client-provided idempotency keys honored
• Automatic key generation for requests without explicit keys
• Race conditions handled with distributed locks
• Cached responses include all original headers and status codes
• Idempotency records expire after appropriate TTL
• GET requests not subject to deduplication
• Performance impact minimal (<10ms overhead)
• Duplicate detection works across multiple server instances

Idempotency Key Strategies:

• Client-provided: X-Idempotency-Key header (UUID)
• Auto-generated: Hash of userId + endpoint + request body
• Combination: userId + timestamp + action type

TTL by Operation Type:

• Puzzle submission: 300 seconds (5 minutes)
• Point claiming: 600 seconds (10 minutes)
• Friend request: 3600 seconds (1 hour)
• Profile update: 60 seconds (1 minute)
• Read operations: No idempotency needed

Implementation Approach:

• Store in Redis: key = idempotencyKey, value = {statusCode, body, headers}
• Use Redis SETNX for atomic duplicate check
• Distributed lock for race condition handling

Error Handling:

• Invalid idempotency key format: 400 Bad Request
• Redis connection failure: Allow request through (fail open)
• Expired idempotency record: Treat as new request