Repository navigation
Conversation
Source provider sync ran inline on the daemon's request-serving event loop: runSourceIndexJob called provider.sync directly, so Notion JSON parsing, unknown-tag rewriting and hashing, and Web linkedom/Defuddle extraction all blocked request handling for corpus-sized work. Add a dedicated source sync worker thread, one per job, following the native source worker and Pi agent worker patterns. The worker hosts the Notion and Web sync implementations. It never opens the workspace database or the secret store: DB owner requests are relayed through the daemon thread to the existing DB owner (bounded allowlist, at most 8 in flight), and secret lookups are limited to the source's configured tokenRef. Progress and log entries are relayed back. Cancellation is polled from the job state, delivered to the worker, rejects further owner requests, and terminates the worker after a 5s grace. A worker crash or exit fails the job explicitly with no in-process fallback, and daemon shutdown terminates and reaps live workers. The Notion and Web adapters no longer expose an inline sync, so the job runner can only reach them through the worker. Provider sync contexts now carry getSecret so secret resolution stays with the daemon. GitHub and Discord still sync inline; their DB writes use legacy accessor closures that must move to owner operations before they can run in the worker. Refs #2051 Assisted-by: Claude-Code:claude-opus-5-5
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (24)
Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 0 remain after this review. 📝 WalkthroughWalkthroughNotion and Web source synchronization can run in dedicated worker threads. The daemon relays database-owner operations, configured secrets, progress, and logs. The changes add worker lifecycle handling, provider integration, tests, build support, and documentation. ChangesWorker-backed source synchronization
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~60 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant SourceIndexJob
participant SourceSyncHost
participant SourceSyncWorker
participant DbOwnerRelay
SourceIndexJob->>SourceSyncHost: Start Notion or Web sync
SourceSyncHost->>SourceSyncWorker: Send sync job
SourceSyncWorker->>SourceSyncHost: Request database-owner operation
SourceSyncHost->>DbOwnerRelay: Submit permitted request
DbOwnerRelay->>SourceSyncHost: Return result or error
SourceSyncHost->>SourceSyncWorker: Return relay response
SourceSyncWorker->>SourceSyncHost: Send sync result or error
SourceSyncHost->>SourceIndexJob: Complete or fail indexing job
Suggested reviewers: Merge Risk: ⚪ Minimal · up to Notion and Web syncs now run in worker threads, so provider parsing no longer blocks the daemon's event loop. No concrete merge-blocking issue was found. The author notes that the full daemon lifecycle on an installed build was not tested. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 42 functions across 21 files. (3 skipped: 3 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
comments:check rejects the block comment on runSourceSyncInWorker. Assisted-by: Claude-Code:claude-opus-5-5
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Assisted-by: Claude-Code:claude-opus-5-5
Assisted-by: Claude-Code:claude-opus-5-5
A Notion sync with thousands of failures could produce a worker result frame over the 4 MiB cap, so the job reported the IPC limit error instead of the failure count. Trim per-failure details in the result frame only, keeping the count and the first failure intact. Failure artifacts are written in the worker and are unaffected. Assisted-by: Claude-Code:claude-opus-5-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Source provider sync ran on the daemon's request-serving event loop.
runSourceIndexJobinroutes/sources-routes.tscalledprovider.syncdirectly, so Notion response parsing, unknown-tag rewriting, and hashing, and Web linkedom plus Defuddle extraction, all ran inline with request handling. Network I/O was async and DB work already went to the DB owner, but the CPU work did not.This PR adds the worker boundary from #2051 and moves the Notion and Web providers behind it. GitHub and Discord still sync inline (see Notes), so this uses
Refs #2051, notFixes.Changes
source-sync-worker.ts/source-sync-worker-handle.ts/source-sync-worker-protocol.ts: a dedicated worker thread per sync job. The layout follows the embedding worker (entry, handle, protocol). The lifecycle followsnative-memory-source-worker.tsand the Pi agent worker: versioned frames,readyhandshake with a 15s deadline, embedded or bundled path resolution, and terminate on settle.DbOwnerClient(registerDbOwnerRelayindb-owner-runtime.ts), so the providers' existingdbOwnerQuery,dbOwnerTransaction, and source artifact upsert/index/purge calls are forwarded to the daemon's existing DB owner. No second owner process is started. The relay accepts onlyquery,batch,transaction, and the foursource_artifact_*kinds, with at most 8 in flight. A full relay returnsDB_OWNER_QUEUE_FULL, so the existing admission retry insubmitWithAdmissionstill applies. Owner errors keep their codes across the boundary.SourceProviderSyncContextnow carriesgetSecret. Inside the worker, that call goes back to the daemon, which resolves only the source's configuredtokenRefand rejects any other name.shouldContinueand sendscancelto the worker, which flips the provider'sshouldContinue. Notion already aborts in-flight requests on that signal. Once a cancel is sent, in-flight owner jobs are cancelled and new owner requests are rejected, so a cancelled sync cannot keep writing during deletion. If the worker has not settled 5s after the cancel, it is terminated. That covers Web's in-flight fetch.Logger.forwardTo, so worker warnings reach the daemon log and the live stream. The job runner keeps its existing outcome handling (complete/errorwith a failure count, telemetry, failure artifacts written by the provider).source sync worker exited with code N) and the job fails. Nothing falls back to in-process execution.stopSourceIndexJobsnow callscloseSourceSyncWorkers(), which terminates live workers and waits for them to be reaped before waiting on runs.notion-source-provider.ts,web-source-provider.ts: the adapters dropsyncand setsyncInWorker: true. The sync functions are exported for the worker and for tests. The job runner dispatchessyncInWorkerproviders only to the worker, so the inline path for these two providers no longer exists.github-source-provider.ts,discord-source-provider.ts: resolve tokens throughcontext.getSecretinstead of importingsecrets. Behavior is unchanged. Only the shiftedsiteTokens were regenerated.source-sync-workeris added toplatform/daemon/build.ts,scripts/build-native-bun.tsworker entries, andDAEMON_JS_WORKER_FILES.worker-process-boundary.test.tsnow lists the new in-process helpers.existsSync, the same pattern as the native source worker. The legacy sync DB count is unchanged at 144.web/docs/src/content/docs/sources.md: new "Sync execution" section.Type
feat— new user-facing feature (bumps minor)fix— bug fixrefactor— restructure without behavior changechore— build, deps, config, docsperf— performance improvementtest— test coveragePackages affected
@signet/core@signet/daemon@signet/cli/ dashboard@signet/sdk@signet/connector-*@signet/webpredictorscripts/build-native-bun.tsworker entriesScreenshots
N/A, no UI changes.
Migration Notes (if applicable)
No DB migration.
Testing
bun testpasses (focused:source-sync-worker.test.ts(12),routes/(all), provider tests for Notion/Web/GitHub/Discord (87),worker-process-boundary,logger,scripts/audit-event-loop-contract.test.ts,surfaces/cli/src/lib/runtime.test.ts)bun run typecheckpasses (pre-commit hook, all workspaces)bun run lintpasses (biome checkon changed files,comments:check)source-sync-worker.test.tsruns the real worker module in a real worker thread against a real DB owner process:fetchstubbed inside the worker, writes its artifact through the relayed owner and authenticates with the relayedtokenRef. Thereadythread ID differs from the test thread.agent_remove) are rejected; a workerprocess.exit(17)fails the run; cooperative cancel resolves before the grace period; an owner write the worker submits after cancel is rejected and never commits; a 10,000-failure result that is over the cap is trimmed to fit while keeping all 10,000 entries and the first failure intact; a worker that ignores cancel is terminated;closeSourceSyncWorkers()terminates live workers.I also bundled
source-sync-worker.tswith the daemon'sBun.buildsettings and ran the Notion case against the bundled output. It passed. The fullplatform/daemonbuild could not run locally because the connector workspaces are not built in this checkout.The new route tests in
sources-routes.test.ts:errorwith the exit message.notion source sync completed with 100 failure(s). Telemetry reportsfailed: 100, and all 100 failure artifacts keep their full text. On the previous commit this job instead failed with the IPC-limit error.Before this change the inline
provider.syncran instead, so these tests fail against main.Not run locally: a full daemon lifecycle on an installed build. Three
db-owner-client.test.tscases (sqlite-vec / SIGABRT) and three harness tests fail here identically on main because of the local SQLite and unbuilt-connector setup.Live verification. I ran the daemon from source at this PR's head (
1550699) and frommain(ea8e8b5). Each daemon had its own isolated workspace, HOME, and port. A local stub site served a 4.6 MB HTML page. It was bound only to the machine's own global IPv6 address, because the Web provider rejects loopback and private targets, and it refused every other client. Each source was added throughPOST /api/sources/web. A separate client polled/health/liveevery 20 ms during the sync.source_web_pageartifact. Max/health/livelatency during the sync was 8.9 ms (3.8 ms on a second run), with p99 ≤ 2 ms, and nothing failed. Onmainthe same sync blocked one/health/liverequest for 33.3 s. Each heavy sync onmainalso loggedEvent loop critically blocked (~30 s)andEvent-loop stall latched, and DB-owner jobs hit their deadline (Vector backfill probe failed … exceeded its deadline). On this PR, none of these appeared across 10 heavy-page sync runs (7 completed, 3 interrupted). No second DB owner process was started: the daemon kept the same 2 owner children during a sync.DELETE /api/sources/:id5 s into the heavy extraction. On this PR it returned 200 in 48 ms. The worker's threads were torn down about 5 s later (cancel grace), and the thread count was back to idle about 15 s after the delete. No rows with thatsource_idremained, and re-adding the same URL immediately ran a new job tocomplete. For a deletion during a slow, streaming fetch, the stub saw the client drop the connection 5.2 s after the DELETE. Onmainthe DELETE response took about 26 s to arrive, because the event loop was busy with the extraction.exitCode: 0. Both DB owner children were gone, and no process from the scratch root was left. After a restart, the source was still configured with no stale job and zero partial rows. The only warnings,synthesis worker exited with code 1and aVector backfill probe failed … drainingrace at shutdown, also appear onmain.NOTION_API_BASE), so I didn't run a live Notion sync against a stub. I added a Notion source whosetokenRefhad no stored secret. The worker asked the daemon for the secret, and the job ended inerrorwithin 170 ms withFailed to resolve Notion token ref '…': Secret '…' not found. No network request was made. Rejection of a different token ref is covered only by the unit test.phys_footprintstayed flat at 253 MB, and threads settled back to the idle count.AI disclosure
Assisted-bytags in commits)Notes
Design decisions the issue left open:
tokenRef.What remains for #2051:
getDbAccessor().withReadDbAsync/withWriteTxAsyncclosures (3 sites in GitHub; 6 in Discord plus 2 indiscord-desktop-cache-source.ts), which cannot cross a thread. They must become owner operations first. GitHub's stale-artifact read can approach the 1 MiB owner result cap atmaxItemsPerRepo10,000, so it needs paging. Discord also needs its gateway-tail socket lifecycle and the synchronous Desktop-cache file reads moved. Once both providers move,SourceProviderAdapter.sync(the inline path) can be deleted.document-import-service.ts,import-normalizer.ts,routes/import-routes.ts) has not been audited or moved.transcript-capture-worker.ts) is unchanged.Summary by CodeRabbit