fix: correct git-repo incremental diff and isolate connector ingestion failures#414
Open
Greg Land (bikeusaland) wants to merge 8 commits into
Open
Conversation
The git-repo connector declared previousHead and persisted each repo's head in state.latestIds, but createRepoManifest never read the prior head back. changedFiles was always `git diff --name-status HEAD` (uncommitted working- tree changes only), so the manifest never described commits landed since the last ingestion run — contradicting the recent-changes window the agent is told about. Pass the recorded head into createRepoManifest and, when it is still a reachable commit and differs from the current head, compute `git diff <previousHead> HEAD` and `git log <previousHead>..HEAD`. Fall back to the working-tree diff on the first run, when nothing new was committed, or when the recorded head is unreachable (force-push / rewritten history), so a diff against a missing commit can never throw. Fixes langchain-ai#409 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ingestAllConnectors ran connectors serially with no error handling, so the first connector that threw (e.g. an un-refreshable token) aborted the whole "all" run and discarded results from every connector that already succeeded. Run them concurrently with Promise.allSettled and turn each rejection into a well-formed `error` result so the agent still sees everything that ran. The X connector had the same shape internally: any single stream error (e.g. a 429 on one endpoint) threw out of ingest, orphaning already-written dumps and skipping writeConnectorState so latestIds never advanced — forcing a full re-fetch next run. Wrap each stream (and each list within list_posts) in try/catch, record a warning, and always write state at the end, mirroring the resilient pattern already used by the hackernews and git-repo connectors. A run that fetched nothing but hit an error now reports `error` instead of a benign `skipped`. Fixes langchain-ai#412 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two correctness bugs in the connector ingestion path.
1. git-repo connector diffed the working tree, not changes since the last run (#409)
GitRepoManifestdeclaredpreviousHeadand connector state already persisted each repo's head inlatestIds, butcreateRepoManifestnever read the prior head back.changedFileswas alwaysgit diff --name-status HEAD(uncommitted working-tree changes only), so the manifest never described commits landed since the last ingestion — contradicting the recent-changes window the agent is told about.Now the recorded head is passed in and, when it is still a reachable commit and differs from the current head, the manifest is built from
git diff <previousHead> HEADandgit log <previousHead>..HEAD. It falls back to the working-tree diff on the first run, when nothing new was committed, or when the recorded head is unreachable (force-push / rewritten history) — arev-parse --verifyguard ensures a diff against a missing commit can never throw.2. One connector/stream failure aborted the whole ingestion run (#412)
ingestAllConnectorsran connectors serially with no error handling, so the first connector that threw (e.g. an un-refreshable token) discarded results from every connector that already succeeded. It now runs them concurrently withPromise.allSettled, turning each rejection into a well-formederrorresult.The X connector had the same shape internally: any single stream error (e.g. a 429 on one endpoint) threw out of
ingest, orphaning already-written dumps and skipping the state write solatestIdsnever advanced — forcing a full re-fetch next run. Each stream (and each list withinlist_posts) is now wrapped in try/catch, records a warning, and always writes state at the end, mirroring the resilient pattern already used by the hackernews and git-repo connectors. A run that fetched nothing but hit an error now reportserrorinstead of a benignskipped.Tests
Three new test files (7 new tests):
test/git-repo-connector.test.ts— drives real temp repos across two runs: first run has nopreviousHead, second run diffs since the recorded head, and an unreachable recorded head falls back safely.test/x-connector-stream-isolation.test.ts— one failing stream keeps the succeeding dump and records a warning; all-failing yieldserror.test/ingest-all-connectors.test.ts— a throwing connector does not discard a succeeding connector's result.Full suite green: typecheck,
eslint .,prettier --check, and 420 tests pass.Closes #409
Closes #412
🤖 Generated with Claude Code