feat: Add limit to topic-to-external-resource mappings and fix test infrastructure #895

jjain1259 · 2025-10-01T13:06:40Z

Summary

This PR introduces a configurable hard limit for topic-to-external-resource mappings and addresses test infrastructure issues.

Changes Made

New Configuration Parameter

Config Name: max.external.resource.mappings
Default Value: 15
Type: Integer (minimum value: 1)
Description: The maximum number of topic-to-external-resource mappings allowed

Validation Logic

Added validation in getTopicToExternalResourceMap() method to enforce the mapping limit
Throws ConfigException with descriptive error message when limit is exceeded
Error message includes:
- Current number of mappings configured
- Maximum allowed limit

Test Infrastructure Updates

Elasticsearch Version: Updated default ES version from 8.2.2 to 8.15.2 in ElasticsearchContainer.java
- Resolves cgroupv2 container compatibility issues
Backward Compatibility Tests: Disabled in ElasticsearchConnectorIT.java
- testBackwardsCompatibilityDataStream() (ES 7.0.1) - incompatible with cgroupv2
- testBackwardsCompatibilityDataStream2() (ES 7.9.3) - incompatible with cgroupv2
- testBackwardsCompatibility() (ES 7.16.3) - incompatible with cgroupv2
Enhanced Cleanup: Added a cleanup method in ElasticsearchConnectorIT.java
- Deletes aliases, indices, and data streams between tests
- Prevents ElasticsearchStatusException for duplicate resource names

Test Coverage

Added comprehensive test cases in ValidatorTest.java

Configuration Usage

Users can now configure the connector with a custom limit:

# Set custom limit (optional, defaults to 15)
max.external.resource.mappings=25

# Configure topic mappings (validated against the limit)
external.resource.usage=INDEX
topic.to.external.resource.mapping=topic1:index1,topic2:index2,topic3:index3

Error Handling

When the limit is exceeded, users receive a clear error message:

Too many topic-to-external-resource mappings configured (16). Maximum allowed is 15.

Backward Compatibility

✅ Existing configurations continue to work with the default limit of 15
✅ No breaking changes to existing APIs
✅ Configuration is optional with default

Testing

All existing tests pass
- Backward compatibility tests are disabled (not deleted) for possible future re-enablement
- Ticket to track the re-enablement of these tests: https://confluentinc.atlassian.net/browse/CC-37474
New test cases cover both valid and invalid scenarios
Tests validate default and custom limit enforcement

…igurable

confluent-cla-assistant · 2025-10-01T13:06:52Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

…hines & disbaled backward compatibility tests

sonarqube-confluent · 2025-10-27T12:04:39Z

50.00% Coverage on New Code (is less than 80.00%)

Analysis Details

10 Issues

0 Bugs
1 Vulnerability
9 Code Smells

Coverage and Duplications

50.00% Coverage (59.70% Estimated after merge)
No duplication information (0.00% Estimated after merge)

Project ID: kafka-connect-elasticsearch

View in SonarQube

00anshjain

Approved, with a minor comment

00anshjain · 2025-10-29T12:26:18Z

src/test/java/io/confluent/connect/elasticsearch/integration/ElasticsearchConnectorIT.java

+  // Disabled backward compatibility tests due to cgroupv2 issues with older ES versions
+  // @Test
  public void testBackwardsCompatibilityDataStream() throws Exception {
    testBackwardsCompatibilityDataStreamVersionHelper("7.0.1");
  }

-  @Test
+  // @Test
  public void testBackwardsCompatibilityDataStream2() throws Exception {
    testBackwardsCompatibilityDataStreamVersionHelper("7.9.3");
  }

-  @Test
+  // @Test


Should we add a brief comment above each disabled test explaining why it’s been disabled?
Also, can we link the corresponding JIRA ticket here where this issue is being tracked?

Added topic to external resource mapping limit to 15 and made it conf…

2996448

…igurable

jjain1259 requested a review from a team as a code owner October 1, 2025 13:06