Skip to content

perf: paginate wakatime/sync users query (#3161)#3203

Open
vedant7007 wants to merge 1 commit into
Priyanshu-byte-coder:mainfrom
vedant7007:fix/wakatime-sync-pagination-3161
Open

perf: paginate wakatime/sync users query (#3161)#3203
vedant7007 wants to merge 1 commit into
Priyanshu-byte-coder:mainfrom
vedant7007:fix/wakatime-sync-pagination-3161

Conversation

@vedant7007

Copy link
Copy Markdown

Summary

`/api/wakatime/sync` did one unbounded `.select()` and pulled every user with a wakatime API key (plus their encrypted keys) into serverless memory in a single response. At a few thousand users that would either:

  • hit Supabase's ~4.5 MB response cap,
  • blow the Vercel function timeout, or
  • get killed by the runtime memory limit,

and the nightly sync would silently fail.

What changed

Adopted the same pagination pattern already in use by `/api/cron/sync`:

  • `PAGE_SIZE = 50`
  • deterministic `.order("id")` so pagination doesn't drift
  • `.range(page * PAGE_SIZE, (page + 1) * PAGE_SIZE - 1)`
  • `while (hasMore)` loop that breaks out when a short page is returned

Kept the inner `Promise.allSettled` `CHUNK_SIZE = 5` for parallelism across the Wakatime API within each page — that part was already sensible, just needed to be nested inside the pagination.

What's unchanged

Decrypt-then-fetch-then-upsert logic per user is identical. No behavior change for the actual sync — only the fetch/scan strategy differs.

Test plan

  • `tsc --noEmit` clean
  • verified fix mirrors the sibling `/api/cron/sync` pagination line-for-line (same PAGE_SIZE, same `.order("id")` + `.range()` shape, same `if (users.length < PAGE_SIZE) hasMore = false` tail check)
  • once approved: manual smoke — confirm the cron completes across a few paginated batches without truncating

Fixes #3161

/api/wakatime/sync did one unbounded `.select()` and pulled every user
with a wakatime API key (plus their encrypted keys) into serverless
memory in a single response. At a few thousand users that would either
hit Supabase's ~4.5 MB response cap, blow the function timeout, or get
killed by the memory limit — and the nightly sync would silently fail.

Adopt the same pagination pattern already in use by /api/cron/sync:
PAGE_SIZE=50, deterministic .order("id"), .range(page*50, page*50+49),
break out when a short page is returned. Kept the inner Promise.allSettled
CHUNK_SIZE=5 for API-call parallelism inside each page.

Fixes Priyanshu-byte-coder#3161
@github-actions github-actions Bot added type:bug GSSoC type bonus: bug fix type:feature GSSoC type bonus: new feature type:performance GSSoC type bonus: performance (+15 pts) gssoc26 GSSoC 2026 contribution labels Jul 13, 2026
@github-actions

Copy link
Copy Markdown

GSSoC Label Checklist 🏷️

@Priyanshu-byte-coder — please apply the appropriate labels before merging:

Difficulty (pick one):

  • level:beginner — 20 pts
  • level:intermediate — 35 pts
  • level:advanced — 55 pts
  • level:critical — 80 pts

Quality (optional):

  • quality:clean — ×1.2 multiplier
  • quality:exceptional — ×1.5 multiplier

Validation (required to score):

  • gssoc:approved — counts for points
  • gssoc:invalid / gssoc:spam / gssoc:ai-slop — does not score

Type labels (type:*) are auto-detected from files and title. Review and adjust if needed.
Points formula: (difficulty × quality_multiplier) + type_bonus

@vedant7007

Copy link
Copy Markdown
Author

starred the repo — should turn the star-required gate green on the next re-run 🌟

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gssoc26 GSSoC 2026 contribution type:bug GSSoC type bonus: bug fix type:feature GSSoC type bonus: new feature type:performance GSSoC type bonus: performance (+15 pts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] /api/wakatime/sync loads every user with a wakatime key in one unbounded query

1 participant