Fix: rd_kafka_consume_batch incorrectly advances offset by 2 on EOF messages

[Ankith-Confluent]

Summary

This PR fixes a bug in rd_kafka_consume_batch() where EOF messages incorrectly advance the consumer position by 2 instead of 1 when enable.partition.eof=true. The bug causes rd_kafka_position() to return last_offset + 2 instead of the correct last_offset + 1 after consuming an EOF message in batch mode.

Motivation

• Addresses this bug Offset increased for eof pseudo messages #4844
• Prevents potential message loss or duplicate consumption due to incorrect offset positioning
• Bug has existed since v2.2.0 (introduced in commit 51397194a63a4bbf07caa757c84f8a86edf7af5b)

Root Cause

In src/rdkafka_queue.c, the rd_kafka_q_serve_rkmessages() function updates the consumer position for all messages in a batch, including EOF messages. EOF messages have error code RD_KAFKA_RESP_ERR__PARTITION_EOF but are not control messages, so they were incorrectly incrementing the offset.

Changes

1. File: src/rdkafka_queue.c
   Added a check to prevent EOF messages from updating the consumer position.

2. Regression subtest: tests/0137-barrier_batch_consume.c
   New Subtest do_test_consume_batch_eof_position() is added

Test Flow

1. Produces 5 messages to a test topic
2. Consumes messages using rd_kafka_consume_batch_queue() with
   enable.partition.eof=true
3. Receives EOF message after consuming all real messages
4. Verifies rd_kafka_position() returns last_offset + 1 (not last_offset + 2)

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Copilot]

Pull Request Overview

This PR fixes a bug in rd_kafka_consume_batch() where EOF messages incorrectly advance the consumer position by 2 instead of 1 when enable.partition.eof=true, causing rd_kafka_position() to return an incorrect offset.

• Added a check to prevent EOF messages from updating consumer position in rd_kafka_q_serve_rkmessages()
• Added comprehensive regression test to verify correct consumer position after EOF messages

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/rdkafka_queue.c	Fixed offset advancement logic to exclude EOF messages from position updates
tests/0137-barrier_batch_consume.c	Added regression test to verify EOF message position handling

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}