[codex] Fix quadratic saved timeline scans#63
Conversation
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: [codex] Fix quadratic saved timeline scans This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0bf6247b67
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| with timeline_edges as ( | ||
| select account_id, tweet_id, 'home' as kind, raw_json | ||
| from tweet_collections | ||
| from tweet_collections indexed by idx_tweet_collections_tweet |
There was a problem hiding this comment.
Preserve account-scoped saved timeline index lookups
useTimelineRouteData sends the selected account id for the saved timeline routes, so normal Likes/Bookmarks requests hit this branch with account set and no all-account dedupe subquery. For that case, forcing idx_tweet_collections_tweet makes SQLite scan the whole collection table before applying e.account_id (SCAN tweet_collections USING INDEX idx_tweet_collections_tweet) instead of using the existing idx_tweet_collections_kind_account (kind=? AND account_id=?) lookup, so a multi-account or large like/bookmark database now pays for unrelated rows on account-scoped saved views. Keep the tweet-id hint only for the all-account/dedupe path, or use a separate account-scoped CTE that lets the kind/account index remain usable.
Useful? React with 👍 / 👎.
What changed
Root cause
The all-account dedupe subquery reused a Likes/Bookmarks CTE. SQLite selected the collection-kind index for both passes, rescanning every saved row for every candidate. With 63,200 likes this produced roughly four billion comparisons inside synchronous
StatementSync.all(), blocking the Node event loop and every HTTP request.Impact
Today, Likes, and Bookmarks now use date-ordered tweet scans with indexed collection lookups. The exact production-shaped Likes query improved from timing out after 12 seconds to 157 ms against the live local database.
Validation
pnpm test— 141 files, 1,172 testspnpm exec vitest run src/lib/queries.test.ts --reporter=verbose— 51 tests; 10,000-row regression completes in 57 mspnpm checkpnpm build