Batch Processing for Portfolio Snapshots

[Cedarich]

Labels: Backend, Performance, Infrastructure
Complexity: High (200 points) ⚙️

Description

Refactor the portfolio snapshot generation to use worker threads or batch processing to handle thousands of users without blocking the main event loop.

Requirements / Context

• Current implementation might be sequential.
• Use BullMQ or Job queues.

Acceptance Criteria

• Parallel snapshot generation for large user sets.
• Progress tracking for the snapshot job.
• Resilience: retry logic for failed individual snapshots.

---

[Chinonso-Peter]

@Chinonso-Peter has applied to work on this issue as part of the Stellar Wave Program's 3rd wave.

Can i work on this issue

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Chinonso-Peter to this issue.

[Sage-senpai]

@Sage-senpai has applied to work on this issue as part of the Stellar Wave Program's 3rd wave.

I'd like to implement this. I have hands-on experience building BullMQ job queue systems in NestJS, including queue producers/consumers, worker concurrency configuration, progress tracking via job.updateProgress(), and per-job retry logic with exponential backoff.
My plan: create a SnapshotQueue BullMQ module that accepts a batch of user IDs, fans them out as individual jobs processed in parallel across a configurable worker pool. Each worker generates one user's portfolio snapshot independently, reports progress back to the parent job, and retries up to 3 times on failure without affecting other users in the batch. The main event loop stays unblocked throughout. I'll wire it into the existing snapshot service with minimal disruption to the current API surface.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Sage-senpai to this issue.

[oscarj007]

@oscarj007 has applied to work on this issue as part of the Stellar Wave Program's 3rd wave.

Hey, am a full stack developer and I have experience to back it up please let me solve the issHey, am a full stack developer and I have experience to back it up please let me solve the issue

ℹ️ Repo Maintainers: To accept this application, review their application or assign @oscarj007 to this issue.

[drips-wave]

Congratulations, @Chinonso-Peter! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on March 30, 2026.

🧑‍💻 @Chinonso-Peter: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊

[drips-wave]

This issue has been marked as completed by a Drips Wave moderator for @Chinonso-Peter as part of the Stellar Wave Program's 3rd Wave 🥳

Moderator's reason: [Auto-assessed] The contributor successfully implemented batch processing for portfolio snapshots using BullMQ, including progress tracking via Redis and robust retry logic with exponential backoff. The PR addresses all acceptance criteria and was merged by the maintainers.

😎 @Chinonso-Peter: You earned 200 Points for completing this issue! After the current Wave ends, you'll be eligible for a percentage of the Wave's reward pool based on the percentage of total points you've earned. Learn more here.

🧑‍💻 Repo maintainers: How'd the contributor do? Leave a review to share your experience working with them.

ℹ️ Note: This issue was marked complete via moderation. Points were issued even though the GitHub issue remains open.