Skip to content

[codex] Fix quadratic saved timeline scans#63

Merged
steipete merged 1 commit into
mainfrom
codex/fix-saved-timeline-query-plan
Jun 20, 2026
Merged

[codex] Fix quadratic saved timeline scans#63
steipete merged 1 commit into
mainfrom
codex/fix-saved-timeline-query-plan

Conversation

@steipete

Copy link
Copy Markdown
Owner

What changed

  • Force saved timeline CTEs through the existing tweet lookup index.
  • Add large-collection performance coverage and liked+bookmarked intersection coverage.
  • Document the user-visible fix in the changelog.

Root cause

The all-account dedupe subquery reused a Likes/Bookmarks CTE. SQLite selected the collection-kind index for both passes, rescanning every saved row for every candidate. With 63,200 likes this produced roughly four billion comparisons inside synchronous StatementSync.all(), blocking the Node event loop and every HTTP request.

Impact

Today, Likes, and Bookmarks now use date-ordered tweet scans with indexed collection lookups. The exact production-shaped Likes query improved from timing out after 12 seconds to 157 ms against the live local database.

Validation

  • pnpm test — 141 files, 1,172 tests
  • pnpm exec vitest run src/lib/queries.test.ts --reporter=verbose — 51 tests; 10,000-row regression completes in 57 ms
  • pnpm check
  • pnpm build
  • Structured autoreview: clean, no actionable findings
  • Production server HTTP smoke: Likes 157 ms, Bookmarks 4 ms, root 33 ms

@steipete steipete marked this pull request as ready for review June 20, 2026 04:54
@clawsweeper

clawsweeper Bot commented Jun 20, 2026

Copy link
Copy Markdown

ClawSweeper status: review started.

I am starting a fresh review of this pull request: [codex] Fix quadratic saved timeline scans This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@steipete steipete merged commit e42d24e into main Jun 20, 2026
2 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0bf6247b67

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

with timeline_edges as (
select account_id, tweet_id, 'home' as kind, raw_json
from tweet_collections
from tweet_collections indexed by idx_tweet_collections_tweet

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve account-scoped saved timeline index lookups

useTimelineRouteData sends the selected account id for the saved timeline routes, so normal Likes/Bookmarks requests hit this branch with account set and no all-account dedupe subquery. For that case, forcing idx_tweet_collections_tweet makes SQLite scan the whole collection table before applying e.account_id (SCAN tweet_collections USING INDEX idx_tweet_collections_tweet) instead of using the existing idx_tweet_collections_kind_account (kind=? AND account_id=?) lookup, so a multi-account or large like/bookmark database now pays for unrelated rows on account-scoped saved views. Keep the tweet-id hint only for the all-account/dedupe path, or use a separate account-scoped CTE that lets the kind/account index remain usable.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant