Skip to content

feat: add listIds() to SearchProvider #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
harlan-zw merged 2 commits into from
Mar 21, 2026
Merged

feat: add listIds() to SearchProvider #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7835485
feat: add listIds() to SearchProvider for incremental indexing
harlan-zw Mar 21, 2026
3b2ac23
fix: deduplicate chunk IDs to parent doc IDs in listIds()
harlan-zw Mar 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions src/db/sqlite.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,11 @@ export async function sqlite(config: SqliteConfig): Promise<SearchProvider> {
}
},

async listIds() {
const rows = db.prepare('SELECT id FROM documents_meta').all() as Array<{ id: string }>
return rows.map(r => r.id)
},
Comment on lines +336 to +339

@coderabbitai coderabbitai Bot Mar 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

listIds() leaks internal chunk IDs instead of canonical document IDs.

At Line 337, SELECT id FROM documents_meta returns chunk IDs (<doc>#chunk-n) when chunking is enabled, which breaks delta-sync based on original document IDs.

Proposed fix 
 async listIds() {
-  const rows = db.prepare('SELECT id FROM documents_meta').all() as Array<{ id: string }>
-  return rows.map(r => r.id)
+  const rows = db.prepare(`
+    SELECT DISTINCT
+      COALESCE(json_extract(metadata, '$._parentId'), id) AS id
+    FROM documents_meta
+    ORDER BY id
+  `).all() as Array<{ id: string }>
+  return rows.map(r => r.id)
 },
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/db/sqlite.ts` around lines 336 - 339, listIds() currently returns chunk
IDs from documents_meta (e.g., "<doc>#chunk-..."); update listIds in
src/db/sqlite.ts to return canonical/original document IDs by extracting the
prefix before any "#chunk" suffix and deduplicating the results: run the same
SELECT id FROM documents_meta, map each row to r.id.split('#chunk')[0] (or strip
the "#chunk" part if present), collect unique values (Set) and return the array
of canonical IDs so delta-sync uses original document IDs.


async clear() {
db.exec('DELETE FROM documents_fts')
db.exec('DELETE FROM documents_vec')
Expand Down
13 changes: 13 additions & 0 deletions src/retriv.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,19 @@ export async function createRetriv(options: RetrivOptions): Promise<SearchProvid
return { count: results[0]?.count ?? 0 }
},

async listIds() {
const driver = drivers.find(d => d.listIds)
const ids = await driver?.listIds?.() ?? []
if (!chunker)
return ids
const parentIds = new Set<string>()
for (const id of ids) {
const sep = id.indexOf('#chunk-')
parentIds.add(sep >= 0 ? id.substring(0, sep) : id)
}
return Array.from(parentIds)
},
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.

async clear() {
await Promise.all(drivers.filter(d => d.clear).map(d => d.clear!()))
parentDocs.clear()
Expand Down
5 changes: 5 additions & 0 deletions src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,11 @@ export interface SearchProvider {
*/
remove?: (ids: string[]) => Promise<{ count: number }>

/**
* List all indexed document IDs
*/
listIds?: () => Promise<string[]>

/**
* Clear all indexed documents
*/
Expand Down
Loading