Run a connector on a Specific Connect broker

[eazhilan-nagarajan]

Hello all,

We have been using Kafka connect to run multiple connectors and there are couple of connectors which have very high throughput. At time we see these high throughput connectors run into problems, so just wondering if it's possible to tell the connect cluster (by a config parameter) to run a specific only a specific connect broker.

This might sound silly but why I ask is we don't want the other connectors to be disturbed by the problematic ones.

Hoping for some help from the community! Thanks!

Eazhilan

[scholzj]

AFAIK there is no option for this in Kafka Connect. The usual approach to address this issue is to have multiple independent Kafka Connect instances (https://strimzi.io/docs/operators/latest/full/deploying.html#con-kafka-connect-multiple-instances-str) and deploy the problematic connectors on a separate instance.

[eazhilan-nagarajan]

AFAIK there is no option for this in Kafka Connect. The usual approach to address this issue is to have multiple independent Kafka Connect instances (https://strimzi.io/docs/operators/latest/full/deploying.html#con-kafka-connect-multiple-instances-str) and deploy the problematic connectors on a separate instance.

Thanks for your reply. I understand and your suggestion was another thought we had in mind as well. So from the document I understand, though we run multiple instances of connect when we use the same default values for the group ID and other internal topics, all the instances will work as a cluster.

Is my understanding correct?

Thanks!

[scholzj]

Yeah, if you use the same default values, the Connects will not be independent - they will be connected through the topics. So you have to configure different values for each of them for these fields.

[eazhilan-nagarajan]

Perfect! Thanks @scholzj 😀