Skip to content

Deduplicate ingestion cursor and state management #7

@BunsDev

Description

@BunsDev

Problem

Cursor tracking logic for ingestion is repeated across import-openclaw.ts, import-cron.ts, and ingestion-state.ts. Each ingestion source implements its own cursor read/write pattern with no shared abstraction. There is also no deduplication logic for events that may be ingested more than once.

Proposed Solution

  1. Shared IngestionCursor class — Encapsulates cursor read, advance, and persist operations
  2. Event deduplication — Check for existing event IDs before insert (idempotent ingestion)
  3. Consistent error handling — Shared retry/skip logic for malformed records
  4. Ingestion registry — Central registry of ingestion sources with their cursor states

Acceptance Criteria

  • A shared IngestionCursor utility handles all cursor state operations
  • import-openclaw.ts and import-cron.ts use the shared utility
  • Duplicate events are detected and skipped (idempotent)
  • Unit tests cover cursor advancement and deduplication
  • No behavioral changes to existing ingestion flows

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureStructural and design improvementsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions