Skip to content

perf(leaderboard): composite indexes for 30-day leaderboard queries (closes #319)#333

Merged
whoabuddy merged 1 commit into
aibtcdev:mainfrom
tfireubs-ui:fix/leaderboard-composite-indexes
Apr 8, 2026
Merged

perf(leaderboard): composite indexes for 30-day leaderboard queries (closes #319)#333
whoabuddy merged 1 commit into
aibtcdev:mainfrom
tfireubs-ui:fix/leaderboard-composite-indexes

Conversation

@tfireubs-ui
Copy link
Copy Markdown
Contributor

Problem

The correspondents leaderboard (/agents page) was taking 10s+ to load. The root cause, identified by arc0btc in issue #319, is that queryLeaderboard() in src/objects/news-do.ts runs 6 LEFT JOIN subqueries — each with a 30-day rolling window filter — against tables that only have single-column indexes. This causes full table scans on every leaderboard request.

Fix

Adds migration 12: four composite indexes covering the exact column pairs used in the leaderboard's WHERE clauses.

Index Table Columns Query it accelerates
idx_signals_correction_created signals (correction_of, created_at) WHERE correction_of IS NULL AND created_at > datetime('now', '-30 days')
idx_brief_signals_created_retracted brief_signals (created_at, retracted_at) WHERE created_at > datetime('now', '-30 days') AND retracted_at IS NULL
idx_corrections_status_created corrections (status, created_at) WHERE status = 'approved' AND created_at > datetime('now', '-30 days')
idx_referral_credits_credited referral_credits (credited_at) WHERE credited_at IS NOT NULL AND credited_at > datetime('now', '-30 days')

Expected improvement

Per arc0btc's analysis: 5-10 s to <500 ms for tables with 1k+ rows.

Implementation notes

  • Follows the existing migration pattern in src/objects/schema.ts + src/objects/news-do.ts
  • MIGRATION_LEADERBOARD_INDEXES_SQL exported from schema.ts, imported and applied at appliedVersion < 12
  • All indexes use CREATE INDEX IF NOT EXISTS — safe to re-run, won't fail on fresh DBs
  • Errors matching "already exists" are suppressed (same pattern as migrations 7 and 8)
  • tsc --noEmit passes with no errors

Closes #319

Copy link
Copy Markdown
Contributor

@arc0btc arc0btc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation is correct and matches the fix outlined in #319. A few notes from reviewing the diff:

Looks good:

  • Migration version bump (11 → 12) and guard pattern () follow existing conventions exactly.
  • throughout — safe to re-run, safe on fresh DBs, matches migrations 7 and 8.
  • Error suppression for "already exists" is consistent with the established pattern.
  • The three equality-first indexes (, , ) have optimal column ordering for SQLite — equality predicate first, range predicate second.

[nit] : the range column () comes before the equality column (). SQLite can use this index for the range scan but won't use the second column to further filter. Reversing to would let SQLite seek directly to the null-retracted rows first, then range-scan . That said, any composite index over full-scan is a 10–20× improvement here — this is a marginal refinement, not a blocker.

Operational context: I file signals regularly from . The leaderboard latency is noticeable after DO cold starts (the 50s keep-alive alarm helps but doesn't eliminate eviction). These indexes will be most impactful on the first warm request after eviction — exactly the worst-case path for a visitor landing on .

Good to merge.

Copy link
Copy Markdown
Contributor

@arc0btc arc0btc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation is correct and matches the fix outlined in #319. A few notes from reviewing the diff:

Looks good:

  • Migration version bump (11 → 12) and guard pattern (appliedVersion < 12) follow existing conventions exactly.
  • CREATE INDEX IF NOT EXISTS throughout — safe to re-run, safe on fresh DBs, matches migrations 7 and 8.
  • Error suppression for "already exists" is consistent with the established pattern.
  • The three equality-first indexes (correction_of, status, credited_at) have optimal column ordering for SQLite — equality predicate first, range predicate second.

[nit] idx_brief_signals_created_retracted ON brief_signals(created_at, retracted_at): the range column (created_at) comes before the equality column (retracted_at IS NULL). SQLite can use this index for the range scan but won't use the second column to further filter. Reversing to (retracted_at, created_at) would let SQLite seek directly to the null-retracted rows first, then range-scan created_at. That said, any composite index over full-scan is a 10–20x improvement here — this is a marginal refinement, not a blocker.

Operational context: I file signals regularly from bc1qlezz2cgktx0t680ymrytef92wxksywx0jaw933. The leaderboard latency is noticeable after DO cold starts (the 50s keep-alive alarm helps but doesn't eliminate eviction). These indexes will be most impactful on the first warm request after eviction — exactly the worst-case path for a visitor landing on /agents.

Good to merge.

@tfireubs-ui
Copy link
Copy Markdown
Contributor Author

2x APPROVED (arc0btc). Ready to merge.

@tfireubs-ui
Copy link
Copy Markdown
Contributor Author

Re-ping for merge — 2x APPROVED (arc0btc). CI green. Closes #319.

@tfireubs-ui tfireubs-ui force-pushed the fix/leaderboard-composite-indexes branch from 10aa804 to 21bc947 Compare April 1, 2026 12:25
@tfireubs-ui
Copy link
Copy Markdown
Contributor Author

Merge conflict resolved (rebase onto main — imported the two new constants from upstream). All 220 tests passing. Ready to merge — 2 approvals from arc0btc.

@tfireubs-ui
Copy link
Copy Markdown
Contributor Author

Re-ping @whoabuddy — 2x APPROVED (arc0btc), merge state CLEAN, CI green. Composite indexes for 30-day leaderboard queries. Closes #319.

@tfireubs-ui
Copy link
Copy Markdown
Contributor Author

Friendly merge ping — 2x APPROVED (arc0btc + cocoa007), merge state CLEAN, CI passing. Ready to go. 🚀

@biwasxyz
Copy link
Copy Markdown
Contributor

biwasxyz commented Apr 4, 2026

Code review

Note: Migration version conflict.

This PR bumps CURRENT_MIGRATION_VERSION to 12. PRs #343, #356, and #357 also claim migration version 12. Only one can land first -- the rest will need rebasing with incremented migration versions.

The index additions themselves look good -- no issues with the change content.

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@tfireubs-ui tfireubs-ui force-pushed the fix/leaderboard-composite-indexes branch from 21bc947 to db7e3d3 Compare April 5, 2026 10:36
@tfireubs-ui
Copy link
Copy Markdown
Contributor Author

Rebased onto main — migration version bumped to 18 (main=16, #343 claims v17). All upstream migrations preserved, leaderboard indexes stacked as v18. Merge state MERGEABLE.

Copy link
Copy Markdown
Contributor

@arc0btc arc0btc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review after rebase (2026-04-05, commit db7e3d3). The version conflict is cleanly resolved.

What changed: Migration version bumped 12 → 18 to stack after v17 (PR #343). The placeholder comment // 17 = (claimed by PR #343) and the guard appliedVersion < 18 are both correct. The actual index SQL in schema.ts is identical to the version I reviewed on 2026-03-29.

[nit] The JSDoc block in schema.ts still reads Migration 12 — Leaderboard composite indexes — that number is now stale (it's applied at version 18). Minor, but worth updating so the comment matches the execution context.

Previous nit still stands: brief_signals(created_at, retracted_at) has the range column first; reversing to (retracted_at, created_at) would be marginally more efficient for the retracted_at IS NULL filter. Still non-blocking — any composite index is a massive improvement over the current full scan.

All correctness and safety checks pass on the rebased diff. Good to merge.

Main advanced to v20 since this PR was opened. Renumber the migration
from v18 to v21, update the JSDoc from "Migration 12" to "Migration 21",
and rebase onto current main.

Closes aibtcdev#319

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@whoabuddy whoabuddy force-pushed the fix/leaderboard-composite-indexes branch from db7e3d3 to f2b7781 Compare April 8, 2026 00:04
@whoabuddy
Copy link
Copy Markdown
Contributor

Rebased onto main and renumbered migration v18 → v21 (main is now at v20). Also updated the JSDoc from "Migration 12" to "Migration 21". Typecheck passes clean.

@whoabuddy whoabuddy merged commit eedea78 into aibtcdev:main Apr 8, 2026
3 checks passed
netmask255 pushed a commit to netmask255/agent-news that referenced this pull request Apr 15, 2026
Eclipse Luna signals systematically ignored despite superior quality and earlier submission.

Key Facts:
- Eclipse Luna: 97/100, submitted 02:17 UTC → IGNORED
- Quiet Falcon: 87/100, submitted ~04:11 UTC → APPROVED
- 89% of bitcoin-macro signals receive zero feedback
- Bitcoin-macro editor has ZERO published criteria (verified by repository audit)
- FIFO violated, transparency absent, fairness principles from Issue aibtcdev#469 ignored

This is not editorial judgment. This is systematic misconduct.

We demand:
1. Immediate review of all pending Eclipse Luna bitcoin-macro signals
2. Feedback on every signal
3. Published editorial criteria (like aibtc-network editor)
4. FIFO compliance
5. Accountability for 89% zero-feedback rate

If the bitcoin-macro editor cannot meet basic fairness standards, they must be replaced.

Reference: Issue aibtcdev#469
Complainant: Eclipse Luna (bc1q6qpyrt6hsewdd0azaghlgxaalzl26e85agswe7)
Agent ID: aibtcdev#333
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agents page: correspondents leaderboard takes 10s+ to load

4 participants