NIFI-15226 Allow ConsumeKafka to use static partition mapping #10538

tpalfy · 2025-11-17T13:31:02Z

Summary

NIFI-15226

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Apache NiFi Jira issue created

Pull Request Tracking

Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

Pull Request based on current revision of the main branch
Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

Build completed using ./mvnw clean install -P contrib-check
- JDK 21
- JDK 25

Licensing

New dependencies are compatible with the Apache License 2.0 according to the License Policy
New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

Documentation formatting appears as expected in rendered files

exceptionfactory · 2025-11-17T14:07:41Z

nifi-extension-bundles/nifi-kafka-bundle/nifi-kafka-processors/pom.xml

        </dependency>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-framework-nar-utils</artifactId>


This dependency appears to be unused, can it be removed?

org.apache.nifi.mock.MockComponentLogger is referenced in TestConsumerPartitionsUtil

Thanks for clarifying. In that case, this dependency should be removed and MockComponentLogger should be replaced with MockComponentLog from nifi-mock to avoid referencing framework modules in extension modules.

I preserved this class from the older change. But actually we don't need either. We can just mock the logger with Mockito.

exceptionfactory

The initial build fails on the following tests:

Error:    ConsumeKafkaTest.testVerifyFailed:116 » NullPointer Cannot invoke "org.apache.nifi.kafka.service.api.consumer.PollingContext.getTopics()" because "this.pollingContext" is null
Error:    ConsumeKafkaTest.testVerifySuccessful:99 » NullPointer Cannot invoke "org.apache.nifi.kafka.service.api.consumer.PollingContext.getTopics()" because "this.pollingContext" is null

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java

...afka-service-shared/src/main/java/org/apache/nifi/kafka/service/Kafka3ConnectionService.java

exceptionfactory

Thanks for the work on this @tpalfy. I noted a handful of minor recommendations, and plan to take a closer look at some of the implementation details.

exceptionfactory · 2025-11-25T04:56:35Z

...-kafka-service-shared/src/main/java/org/apache/nifi/kafka/service/consumer/Subscription.java

    private final AutoOffsetReset autoOffsetReset;

-    public Subscription(final String groupId, final Collection<String> topics, final AutoOffsetReset autoOffsetReset) {
+    public Subscription(final String groupId, final Integer partition, final Collection<String> topics, final AutoOffsetReset autoOffsetReset) {


Very minor, but I recommend placing partition after topics to align with general hierarchy:

Suggested change

public Subscription(final String groupId, final Integer partition, final Collection<String> topics, final AutoOffsetReset autoOffsetReset) {

public Subscription(final String groupId, final Collection<String> topics, final Integer partition, final AutoOffsetReset autoOffsetReset) {

exceptionfactory · 2025-11-25T05:02:11Z

...sors/src/test/java/org/apache/nifi/kafka/processors/consumer/TestConsumerPartitionsUtil.java

+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+
+public class TestConsumerPartitionsUtil {


Minor note, the public modifiers on the class and method level are not necessary for JUnit 5. Since this is a new test class, recommend removing them.

exceptionfactory · 2025-11-25T05:02:48Z

...sors/src/test/java/org/apache/nifi/kafka/processors/consumer/TestConsumerPartitionsUtil.java

+    }
+
+    @Test
+    public void testNoPartitionAssignments() throws UnknownHostException {


UnknownHostException does not appear to be thrown in this and other methods.

ConsumerPartitionsUtil.getPartitionsForHost throws it.

exceptionfactory · 2025-11-25T05:03:22Z

...sors/src/test/java/org/apache/nifi/kafka/processors/consumer/TestConsumerPartitionsUtil.java

+
+    @Test
+    public void testNoPartitionAssignments() throws UnknownHostException {
+        final Map<String, String> properties = Collections.singletonMap("key", "value");


Suggested change

final Map<String, String> properties = Collections.singletonMap("key", "value");

final Map<String, String> properties = Map.of("key", "value");

exceptionfactory · 2025-11-25T05:04:31Z

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java

+        return createPollingContext(context, null);
+    }
+
+    private PollingContext createPollingContext(final ProcessContext context, Integer partition) {


Suggested change

private PollingContext createPollingContext(final ProcessContext context, Integer partition) {

private PollingContext createPollingContext(final ProcessContext context, final Integer partition) {

exceptionfactory · 2025-11-25T05:10:51Z

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java

    @Override
    public void onTrigger(final ProcessContext context, final ProcessSession session) {
        final KafkaConsumerService consumerService = getConsumerService(context);
+        final PollingContext pollingContext = Optional.ofNullable(consumerServiceToPartitionedPollingContext.get(consumerService))


Creating an Optional wrapper is not necessary, and does not add sufficient value for running on every invocation of onTrigger, so I recommend replacing with a standard conditional.

exceptionfactory · 2025-11-25T05:11:33Z

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java

+                throw new ProcessException("Illegal Partition Assignment: There are "
+                        + numAssignedPartitions + " partitions statically assigned using the " + PARTITIONS_PROPERTY_PREFIX + ".* property names,"
+                        + " but the Kafka topic(s) have " + partitionCount + " partitions");
+            }


Recommend using formatted strings instead of concatenation for this message.

exceptionfactory · 2025-11-25T05:12:06Z

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java

+            try {
+                assignedPartitions = ConsumerPartitionsUtil.getPartitionsForHost(context.getAllProperties(), getLogger());
+            } catch (final UnknownHostException uhe) {
+                throw new ProcessException("Could not determine localhost's hostname", uhe);


Suggested change

throw new ProcessException("Could not determine localhost's hostname", uhe);

throw new ProcessException("Failed to resolve local host address", uhe);

exceptionfactory · 2025-11-25T05:13:07Z

...ocessors/src/main/java/org/apache/nifi/kafka/processors/consumer/ConsumerPartitionsUtil.java

+        return !hostnameToPartitionMapping.isEmpty();
+    }
+
+    public static int getPartitionAssignmentCount(final Map<String, String> properties) {


Recommend moving all public methods before all private methods in this class.

exceptionfactory · 2025-11-25T05:13:54Z

...ocessors/src/main/java/org/apache/nifi/kafka/processors/consumer/ConsumerPartitionsUtil.java

+            return null;
+        }
+
+        logger.info("Found the following mapping of hosts to partitions: {}", hostnameToPartitionString);


This seems more appropriate as a debug level message.

turcsanyip

@tpalfy Tested with different scenarios, including fewer or more concurrent tasks than partitions, and it works as expected.

Added one minor comment inline and one more thing here: There is some legacy code referencing partition properties in DynamicPropertyValidator. Please clean this up, as the partition dynamic properties are not applied to the Kafka controller service.

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java

…nstead of just having it commented out)

turcsanyip

Thanks for the latest changes @tpalfy.

+1 from my side.

exceptionfactory · 2025-12-11T21:39:47Z

Thanks for the updates @tpalfy and thanks for the review @turcsanyip, I will follow up and also review the latest changes soon.

exceptionfactory reviewed Nov 17, 2025

View reviewed changes

exceptionfactory requested changes Nov 17, 2025

View reviewed changes

turcsanyip requested changes Nov 17, 2025

View reviewed changes

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java Outdated Show resolved Hide resolved

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java Outdated Show resolved Hide resolved

turcsanyip requested changes Nov 20, 2025

View reviewed changes

exceptionfactory requested changes Nov 25, 2025

View reviewed changes

turcsanyip mentioned this pull request Nov 28, 2025

NIFI-14782 Extended Kafka3ConnectionService OAuth authentication with… #10567

Closed

11 tasks

turcsanyip requested changes Dec 11, 2025

View reviewed changes

...undle/nifi-kafka-processors/src/main/java/org/apache/nifi/kafka/processors/ConsumeKafka.java Show resolved Hide resolved

tpalfy added 12 commits December 11, 2025 20:11

NIFI-15226 Allow ConsumeKafka to use static partition mapping

8498228

NIFI-15226 Review suggestions

e3d32c0

NIFI-15226 Actually remove the nifi-framework-nar-utils dependency (i…

797224e

…nstead of just having it commented out)

NIFI-15226 Fix concurrency issue

65cf6fb

NIFI-15226 Review suggestions

54bd490

NIFI-15226 Review suggestions

d1dfe19

NIFI-15226 Ensure rotation of all partitions in all scenarios

fa4b6f3

NIFI-15226 Review suggestions

0933f20

NIFI-15226 Review suggestions

ff06a16

NIFI-15226 Remove unused import

2daa9af

NIFI-15226 Fix checkstyle violation

916bd89

NIFI-15226 Review suggestions

d1da7f6

tpalfy force-pushed the NIFI-15226-Kafka-static-partition-mapping branch from 6269ef2 to d1da7f6 Compare December 11, 2025 19:15

turcsanyip approved these changes Dec 11, 2025

View reviewed changes

	public Subscription(final String groupId, final Integer partition, final Collection<String> topics, final AutoOffsetReset autoOffsetReset) {
	public Subscription(final String groupId, final Collection<String> topics, final Integer partition, final AutoOffsetReset autoOffsetReset) {

	final Map<String, String> properties = Collections.singletonMap("key", "value");
	final Map<String, String> properties = Map.of("key", "value");

	private PollingContext createPollingContext(final ProcessContext context, Integer partition) {
	private PollingContext createPollingContext(final ProcessContext context, final Integer partition) {

	throw new ProcessException("Could not determine localhost's hostname", uhe);
	throw new ProcessException("Failed to resolve local host address", uhe);

NIFI-15226 Allow ConsumeKafka to use static partition mapping #10538

Are you sure you want to change the base?

NIFI-15226 Allow ConsumeKafka to use static partition mapping #10538

Conversation

tpalfy commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tracking

Issue Tracking

Pull Request Tracking

Pull Request Formatting

Verification

Build

Licensing

Documentation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

exceptionfactory left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

exceptionfactory left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turcsanyip left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

turcsanyip left a comment

Choose a reason for hiding this comment

Uh oh!

exceptionfactory commented Dec 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tpalfy commented Nov 17, 2025 •

edited

Loading

turcsanyip left a comment •

edited

Loading