-
Notifications
You must be signed in to change notification settings - Fork 0
Deduplicate ingestion cursor and state management #7
Copy link
Copy link
Open
Labels
architectureStructural and design improvementsStructural and design improvementsenhancementNew feature or requestNew feature or request
Description
Problem
Cursor tracking logic for ingestion is repeated across import-openclaw.ts, import-cron.ts, and ingestion-state.ts. Each ingestion source implements its own cursor read/write pattern with no shared abstraction. There is also no deduplication logic for events that may be ingested more than once.
Proposed Solution
- Shared
IngestionCursorclass — Encapsulates cursor read, advance, and persist operations - Event deduplication — Check for existing event IDs before insert (idempotent ingestion)
- Consistent error handling — Shared retry/skip logic for malformed records
- Ingestion registry — Central registry of ingestion sources with their cursor states
Acceptance Criteria
- A shared
IngestionCursorutility handles all cursor state operations -
import-openclaw.tsandimport-cron.tsuse the shared utility - Duplicate events are detected and skipped (idempotent)
- Unit tests cover cursor advancement and deduplication
- No behavioral changes to existing ingestion flows
🤖 Generated with Claude Code
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
architectureStructural and design improvementsStructural and design improvementsenhancementNew feature or requestNew feature or request