Skip to content

[MongoDB Replication] Raw Change Streams #309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Conversation

rkistner
Copy link
Contributor

This switches to using aggregate/getMore commands directly, instead of the built-in watch methods.

Motivations:

  1. This completely avoids the resumeToken time travel issues worked around in [MongoDB Replication] Fix resumeTokens going back in time on busy change streams #301.
  2. By getting data back in batches, we can avoid the overhead from repeatedly calling tryNext() for every individual change (impact not measured yet).

As a follow-up to this, we can start fetching the next batch of data as soon as we've received the current one, allow us to read and write concurrently, further increasing throughput (same as we do for initial snapshots).

However, this approach is much more low-level than the previous approach, so we need to test that we're handling all the edge cases correctly:

  1. Primary step-down and other similar error conditions.
  2. Are we using sessions correctly?
  3. Any considerations for older MongoDB versions (6.0)?
  4. Should we check the driver specs?
  5. Are we using resume tokens correctly?

Copy link

changeset-bot bot commented Jul 22, 2025

⚠️ No Changeset found

Latest commit: a841596

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant