feat: Database Query Optimization & API Latency Reduction

[emarc99]

PR Description

📌 Context & Problem Statement

The backend API was experiencing high latency on read-heavy endpoints (/api/courses, /api/books, /api/users), impacting the user experience on the frontend dashboard and content pages.

Root cause analysis identified the following bottlenecks:

1. Missing Database Indexes: Frequently filtered and sorted query paths (status, createdBy, author, categoryRef, recipient) caused full collection scans in MongoDB.
2. Over-Fetching & Embedded Document Bloat: List queries fetched entire document trees, including large embedded arrays like course sections (lessons) and book reviews.
3. Mongoose Overhead: Queries returned full hydrated Mongoose Document instances with change-tracking metadata instead of lean plain JavaScript objects.
4. Lack of List Pagination: List endpoints loaded all matching database records in a single query payload.
5. Restrictive Connection Pool: Default MongoDB max pool size of 10 caused connection queuing under concurrent request load.

---

🚀 Key Improvements

1. Database Schema Indexing Strategy

Added targeted single-field and compound indexes to optimize filter and sort paths:

• Course: Added compound index { status: 1, createdAt: -1 } (published listing), { categoryRef: 1, status: 1 } (category filtering), { createdBy: 1 } (creator dashboard), and { enrolledUsers: 1 } (user enrollment count).
• Book: Added indexes on { author: 1 }, { category: 1 }, and { createdAt: -1 }.
• User: Added indexes on { role: 1 } and { verifiedEducator: 1 }.
• Space: Added indexes on { host: 1 } and compound index { status: 1, eventDate: 1 }.
• Notification: Added compound index { recipient: 1, isDeleted: 1, createdAt: -1 }.

2. Selective Field Projections & .lean() Query Execution

• Selective Field Selection: Course and book list endpoints (getCourses, getBooks) now select only light summary fields (_id, title, description, category, price, currency, thumbnail/image, rating, numReviews, createdBy/author, createdAt). Heavy nested arrays (sections, reviews, enrolledUsers) are excluded from list views and fetched only on detail endpoints.
• Lean Queries: Applied .lean() across all read controllers (courseController, bookController, userController) to bypass Mongoose document hydration overhead.
• Selective Populate: Refined populate calls to retrieve only required fields from referenced documents (e.g., .populate("createdBy", "name email avatar")).

3. Pagination Support

• Implemented page and limit query parameters on getCourses and getBooks.
• Paginated responses return structured metadata:

  {
    "success": true,
    "page": 1,
    "limit": 20,
    "total": 500,
    "hasMore": true,
    "courses": [...]
  }

4. Connection Pooling & Compression Middleware

• Connection Pool Tuning: Configured maxPoolSize: 50 (configurable via process.env.MAX_POOL_SIZE), minPoolSize: 10, and maxIdleTimeMS: 30000 in src/config/db.js.
• Response Compression: Configured express compression middleware with threshold: 1024 and compression level 6 in app.js.
• Cache Key Synchronization: Updated route cache key generators to incorporate query strings so paginated requests remain isolated in Redis cache.

---

📊 Empirical Benchmark Results

Benchmarked against 500 course and 500 book documents over 50 test iterations (node scripts/benchmark.js):

Endpoint / Query	Unoptimized Latency	Optimized Latency	Speedup	Latency Reduction
Course List Query (GET /api/courses)	3,726.98 ms	7.32 ms	509.0x faster	99.8% reduction
Book List Query (GET /api/books)	782.20 ms	5.51 ms	141.9x faster	99.3% reduction

All target endpoints now respond well below the 200ms acceptance criteria under normal load.

---

🧪 Verification & Test Results

• Automated Test Suite: Ran npm test across all suites:

  Test Suites: 29 passed, 29 total
  Tests:       172 passed, 172 total
  Time:        29.83 s
  

• Regression Protection: Confirmed zero breaking changes to existing API contracts.

---

📦 Commit History

1. fc381c0 perf(db): add database indexes to Course, Book, User, Space, and Notification schemas
2. 3e1f872 perf(api): optimize list and detail queries with selective projections, lean execution, and pagination
3. 6f3681b perf(config): tune MongoDB connection pool settings and configure response compression
4. 43eed18 test(perf): update Jest configuration, test setup, and add query latency benchmark script

---

🔗 Related Issue

Closes #1

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 6ff656e2-b7d3-46d5-a06f-8b887fdb232e

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[zeemscript]

Strict review blocker: this branch conflicts with the base branch. Please rebase and resolve conflicts before requesting merge.

[zeemscript]

Strict review blocker: this branch conflicts with the base branch. Please rebase and resolve conflicts before requesting merge.

[zeemscript]

Strict review blocker: this branch conflicts with the base branch. Please rebase and resolve conflicts before requesting merge.

[emarc99]

@zeemscript Gimme a few, lemme fix it and request for a proper review.

[zeemscript]

@emarc99 this PR has merge conflicts with the main branch. Please resolve the conflicts (merge main in or rebase) and push the fix so it can be merged. Thanks!

[emarc99]

@emarc99 this PR has merge conflicts with the main branch. Please resolve the conflicts (merge main in or rebase) and push the fix so it can be merged. Thanks!

On it!