Skip to content

RepinWorkerService reconciles pins sequentially with no concurrency bound #237

Description

@meshackyaro

Description

RepinWorkerService.runOnce() (src/ipfs-pinning/repin-worker.service.ts) iterates every degraded/failed pin record and awaits pinningService.reconcile(record.cid) one at a time in a for...of loop. IpfsPinningService.reconcile() (src/ipfs-pinning/ipfs-pinning.service.ts) itself makes real network calls per provider (provider.verify(), and potentially replicate()pinWithProvider() for topping up), so a sweep covering many simultaneously-degraded pins runs entirely serially, with total sweep time scaling linearly with the number of unhealthy pins times per-provider network latency.

Component

Backend

Difficulty

🟡 Medium

Tasks

  • Process the sweep's target pin records with bounded concurrency (e.g. process N CIDs in parallel via a small worker pool) instead of a fully sequential loop
  • Ensure error handling per-CID (already present via the existing try/catch around reconcile) is preserved when running concurrently
  • Add a test with multiple degraded pins and a slow mock provider, asserting the sweep completes faster than the sequential baseline

Acceptance Criteria

  • A sweep covering many degraded/failed pins no longer takes time linear in the number of pins times provider latency
  • A test demonstrates the improved throughput while preserving per-CID error isolation

Estimated Time

1 day

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions