Skip to content

feat: Implement ReadN #276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat: Implement ReadN #276

wants to merge 6 commits into from

Conversation

raulb
Copy link
Member

@raulb raulb commented Apr 25, 2025

Description

Since the introduction of end-to-end batching on the Connector SDK, (part of the Connector SDK 0.12.0), source connectors can add support to batching by implementing the ReadN method. This wasn't a breaking change since the SDK will fallback to the traditional Read method as it can be spotted in the logs when starting Conduit.

Example:

2025-04-24T19:21:21+00:00 INF source does not support batch reads, falling back to single reads component=plugin connector_id=postgres-to-kafka:postgres-source plugin_name=builtin:postgres plugin_type=source

This pull-request implements ReadN while still maintaining support to Read method for those who wish to use this connector in an older version of Conduit. This is probably not necessary considering Postgres is a builtin connector.

To really utilize the advantage of this, it is recommended to run Conduit with the flag that enables a new upcoming architecture via --preview.pipeline-arch-v2. More information on this blog post.

Note

Still need to:

  • Remove Read and Next leaving only ReadN and NextN

Running benchmarks

Inserting 20M records using Benchi using CDC:

  • WIth Read: 142824.47 msg/s
  • With ReadN: 153107.18 msg/s (previously )

This is a 7,2% better on CDC.

Quick checks:

  • I have followed the Code Guidelines.
  • There is no other pull request for the same update/change.
  • I have written unit tests.
  • I have made sure that the PR is of reasonable size and can be easily reviewed.

@raulb raulb self-assigned this Apr 25, 2025
@raulb raulb marked this pull request as ready for review April 25, 2025 13:26
@raulb raulb requested a review from a team as a code owner April 25, 2025 13:26
@raulb raulb moved this from Triage to Ready for review in Conduit Main Apr 25, 2025
@raulb raulb requested a review from lyuboxa April 25, 2025 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready for review
Development

Successfully merging this pull request may close these issues.

1 participant