feat(backend): implement scalable async task queue & webhook system (#71)#97
Merged
ayomideadeniran merged 2 commits intoSoroLabs:mainfrom Mar 24, 2026
Merged
feat(backend): implement scalable async task queue & webhook system (#71)#97ayomideadeniran merged 2 commits intoSoroLabs:mainfrom
ayomideadeniran merged 2 commits intoSoroLabs:mainfrom
Conversation
added 2 commits
March 24, 2026 11:29
Remove the async task queue and webhook system implementation to re-implement with a different approach as requested. Removed: - core/src/jobs.rs (entire module) - Job-related configuration from AppConfig - Job endpoints (/jobs/submit, /jobs/:id, /jobs/:id/cancel) - Job dependencies (dashmap, uuid, chrono) - JobQueue and JobWorker from AppState This prepares the codebase for a fresh implementation.
…oroLabs#71) Re-implement the async task queue using SQLx for database persistence instead of in-memory DashMap. This provides durability and allows jobs to survive server restarts. Key Changes: - Use SQLx with support for both PostgreSQL and SQLite - Jobs stored in database with full persistence - Database migrations for jobs table schema - Improved job worker with semaphore-based concurrency control - Webhook system with retry logic and exponential backoff New Files: - core/src/jobs.rs - SQLx-based job queue implementation - core/migrations/001_create_jobs_table.sql - Database schema Modified: - core/Cargo.toml - Added sqlx, uuid, chrono dependencies - core/src/main.rs - Added job endpoints and integration Features: - POST /jobs/submit - Submit async job - GET /jobs/:id - Check job status - POST /jobs/:id/cancel - Cancel job - Configurable concurrency (max_concurrent_jobs) - Automatic cleanup of old completed jobs - Progress tracking stored in database
|
@knightsdev001 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implement a background task queue system to handle deep-profiling jobs and batch analyses without blocking the main API thread.
New API Endpoints
/jobs/submitjob_idimmediately/jobs/:id/jobs/:id/cancelKey Features
DashMap<JobId, Job>Job Types
Analyze- Single contract analysisCompare- Compare two contractsOptimizeLimits- Optimize resource limitsConfiguration
New environment variables:
JOB_TIMEOUT_SECS(default: 300)JOB_CLEANUP_INTERVAL_SECS(default: 3600)JOB_RETENTION_SECS(default: 3600)WEBHOOK_TIMEOUT_SECS(default: 10)WEBHOOK_MAX_RETRIES(default: 3)Files Added
core/src/jobs.rs(925 lines)Tests
Closes #71