Skip to content

Commit 0ddbba7

Browse files
robooorobooo
authored
2.0.2 (#34)
* Update confluent-kafka to 2.0.2 * Update list_groups to list_consumer_groups * Add protobuf dep * Debug failing list group * remove trailing whitespace * Add List Groups keyword * Fix results handling at Create Topic keyword * Add Consumer group state global vars * Fix List Groups * Add Describe Consumer Groups keyword * Add Delete Groups keyword * Handle exception better at describe_config * Fix test teardowns with closing consumer group * Fix Purge Test * Fix usage of threaded consumer Threaded consumer missed proper closing and it was not possible to access default params like server, port and group_id. Now it's possible to access these data and close threaded consumer with removal of consumer group too. * Close consumers correctly at test_avro suite * Update actions/checkout to v3 * Update actions/checkout to v3 * Generate doc during release publish * Update test env services to 7.3.3 * Fix list consumer groups test * Handle better responses at admin_client * Update documentation
1 parent 1b535ba commit 0ddbba7

File tree

13 files changed

+350
-89
lines changed

13 files changed

+350
-89
lines changed

‎.github/workflows/main.yml

Lines changed: 6 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ jobs:
2323
# Steps represent a sequence of tasks that will be executed as part of the job
2424
steps:
2525
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
26-
- uses: actions/checkout@v2
26+
- uses: actions/checkout@v3
2727
with:
2828
ref: ${{ github.head_ref }}
2929

@@ -37,16 +37,9 @@ jobs:
3737
run: docker inspect --format "{{json .State.Health.Status }}" $(docker-compose -f examples/docker-compose.yml ps -q)
3838
- name: Execute tests
3939
run: python3 -m robot -d ./docs examples/
40-
- name: Generate keyword documentation
41-
run: python3 -m robot.libdoc -f html src/ConfluentKafkaLibrary docs/index.html
42-
- uses: stefanzweifel/git-auto-commit-action@v4
40+
continue-on-error: true
41+
- name: Archive test log
42+
uses: actions/upload-artifact@v3
4343
with:
44-
file_pattern: docs/index.html
45-
commit_message: Add keyword documentation
46-
push_options: '--force'
47-
- name: Deploy 🚀
48-
uses: JamesIves/[email protected]
49-
with:
50-
branch: gh-pages # The branch the action should deploy to.
51-
folder: docs # The folder the action should deploy.
52-
44+
name: log.html
45+
path: docs/log.html

‎.github/workflows/python-publish.yml

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,9 @@ jobs:
1313
runs-on: ubuntu-latest
1414

1515
steps:
16-
- uses: actions/checkout@v2
16+
- uses: actions/checkout@v3
1717
- name: Set up Python
18-
uses: actions/setup-python@v2
18+
uses: actions/setup-python@v4
1919
with:
2020
python-version: '3.x'
2121
- name: Install pypa/build
@@ -37,4 +37,15 @@ jobs:
3737
uses: pypa/gh-action-pypi-publish@master
3838
with:
3939
password: ${{ secrets.PYPI_API_TOKEN }}
40-
40+
- name: Generate keyword documentation
41+
run: python3 -m robot.libdoc -f html src/ConfluentKafkaLibrary docs/index.html
42+
- uses: stefanzweifel/git-auto-commit-action@v4
43+
with:
44+
file_pattern: docs/index.html
45+
commit_message: Add keyword documentation
46+
push_options: '--force'
47+
- name: Deploy 🚀
48+
uses: JamesIves/[email protected]
49+
with:
50+
branch: gh-pages # The branch the action should deploy to.
51+
folder: docs # The folder the action should deploy.

‎docs/index.html

Lines changed: 31 additions & 18 deletions
Large diffs are not rendered by default.

‎examples/docker-compose.yml

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
version: '3.6'
22
services:
33
zookeeper:
4-
image: confluentinc/cp-zookeeper:5.3.1
4+
image: confluentinc/cp-zookeeper:7.3.3
55
hostname: zookeeper
66
container_name: zookeeper
77
ports:
@@ -12,13 +12,14 @@ services:
1212
test: ['CMD-SHELL', 'nc -zv localhost 2181 && exit 0 || exit 1']
1313

1414
broker:
15-
image: confluentinc/cp-enterprise-kafka:5.3.1
15+
image: confluentinc/cp-enterprise-kafka:7.3.3
1616
hostname: broker
1717
container_name: broker
1818
depends_on:
1919
- zookeeper
2020
ports:
2121
- '9092:9092'
22+
- '29092:29092'
2223
healthcheck:
2324
test: ['CMD-SHELL', 'nc -zv localhost 9092 && exit 0 || exit 1']
2425
environment:
@@ -30,7 +31,7 @@ services:
3031
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
3132

3233
schema-registry:
33-
image: confluentinc/cp-schema-registry:5.3.1
34+
image: confluentinc/cp-schema-registry:7.3.3
3435
hostname: schema-registry
3536
container_name: schema-registry
3637
depends_on:
@@ -42,10 +43,12 @@ services:
4243
test: ['CMD-SHELL', 'nc -zv localhost 8081 && exit 0 || exit 1']
4344
environment:
4445
SCHEMA_REGISTRY_HOST_NAME: schema-registry
46+
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
47+
SCHEMA_REGISTRY_LISTENERS: 'http://0.0.0.0:8081'
4548
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'
4649

4750
rest-proxy:
48-
image: confluentinc/cp-kafka-rest:5.3.1
51+
image: confluentinc/cp-kafka-rest:7.3.3
4952
depends_on:
5053
- zookeeper
5154
- broker

‎examples/test.robot

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ Library ConfluentKafkaLibrary
33
Library Collections
44

55
Suite Setup Starting Test
6-
6+
Suite Teardown Stop Thread
77

88
*** Test Cases ***
99
Verify Topics
@@ -28,6 +28,7 @@ Produce Without Value
2828
Subscribe Topic group_id=${group_id} topics=${topic_name}
2929
${messages}= Poll group_id=${group_id} max_records=1
3030
Should Be Equal As Strings ${messages} [None]
31+
[Teardown] Basic Teardown ${group_id}
3132

3233
Verify Position
3334
${group_id}= Create Consumer
@@ -123,7 +124,7 @@ Remove And Publish New Messages From Threaded Consumer
123124
[Teardown] Clear Messages From Thread ${MAIN_THREAD}
124125

125126
Purge Test
126-
${producer_id}= Create Producer
127+
${producer_id}= Create Producer message.timeout.ms=${30000}
127128
Produce group_id=${producer_id} topic=${TEST_TOPIC} value=After partition=${P_ID}
128129
Produce group_id=${producer_id} topic=${TEST_TOPIC} value=Clear partition=${P_ID}
129130

@@ -143,12 +144,14 @@ Offsets Test
143144
Assign To Topic Partition ${group_id} ${tp}
144145
Sleep 5sec
145146
Store Offsets group_id=${group_id} offsets=${offsets}
146-
147+
[Teardown] Unassign Teardown ${group_id}
147148

148149
*** Keywords ***
149150
Starting Test
150151
Set Suite Variable ${TEST_TOPIC} test
151152
${thread}= Start Consumer Threaded topics=${TEST_TOPIC}
153+
${gid}= Get Thread Group Id ${thread}
154+
Log ${gid}
152155
Set Suite Variable ${MAIN_THREAD} ${thread}
153156
${producer_group_id}= Create Producer
154157
Set Suite Variable ${PRODUCER_ID} ${producer_group_id}
@@ -186,8 +189,20 @@ Basic Teardown
186189
[Arguments] ${group_id}
187190
Unsubscribe ${group_id}
188191
Close Consumer ${group_id}
192+
${groups}= Create List ${group_id}
193+
${admin_client_id}= Create Admin Client
194+
${resp}= Delete Groups ${admin_client_id} group_ids=${groups}
195+
Log ${resp}
189196

190197
Unassign Teardown
191198
[Arguments] ${group_id}
192199
Unassign ${group_id}
193200
Close Consumer ${group_id}
201+
${groups}= Create List ${group_id}
202+
${admin_client_id}= Create Admin Client
203+
${resp}= Delete Groups ${admin_client_id} group_ids=${groups}
204+
Log ${resp}
205+
206+
Stop Thread
207+
${resp}= Stop Consumer Threaded ${MAIN_THREAD}
208+
Log ${resp}

‎examples/test_adminclient.robot

Lines changed: 115 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -13,18 +13,103 @@ AdminClient Topic Creation
1313
END
1414

1515
${admin_client_id}= Create Admin Client
16-
Create Topics group_id=${admin_client_id} new_topics=${topics}
16+
${results}= Create Topics group_id=${admin_client_id} new_topics=${topics}
17+
Log ${results}
1718
${topics}= List Topics ${admin_client_id}
1819
FOR ${topic} IN @{topic_names}
1920
List Should Contain Value ${topics} ${topic}
2021
END
2122
[Teardown] Delete Topics ${admin_client_id} ${topic_names}
2223

23-
AdminClient List Groups
24-
[Documentation] If you run this test as first switch to Should Be Empty keyword.
24+
AdminClient List Consumer Groups
25+
${producer_group_id}= Create Producer
26+
Produce ${producer_group_id} topic=adminlisttest value=Hello partition=${0}
27+
Wait Until Keyword Succeeds 10x 0.5s All Messages Are Delivered ${producer_group_id}
28+
29+
${group_id}= Create Consumer auto_offset_reset=earliest
30+
Subscribe Topic ${group_id} topics=adminlisttest
31+
Sleep 2s # Wait for subscription
32+
2533
${admin_client_id}= Create Admin Client
26-
${groups}= List Groups ${admin_client_id}
27-
Should Not Be Empty ${groups}
34+
${states}= Create List ${CONSUMER_GROUP_STATE_STABLE}
35+
${groups}= List Groups ${admin_client_id} states=${states}
36+
Log ${groups}
37+
Log ${groups.valid}
38+
FOR ${group} IN @{groups.valid}
39+
Log ${group.group_id}
40+
IF "${group_id}" == "${group.group_id}"
41+
Log ${group.group_id}
42+
Log ${group.state}
43+
Pass Execution "Consumer found in list"
44+
END
45+
END
46+
Fail
47+
[Teardown] Basic Teardown ${group_id}
48+
49+
AdminClient Describe Consumer Groups
50+
[Documentation] Finish the test with memebers + verification
51+
${producer_group_id}= Create Producer
52+
Produce ${producer_group_id} topic=admindescribetest value=Hello partition=${0}
53+
Wait Until Keyword Succeeds 10x 0.5s All Messages Are Delivered ${producer_group_id}
54+
55+
${group_id}= Create Consumer auto_offset_reset=earliest
56+
Subscribe Topic ${group_id} topics=admindescribetest
57+
Sleep 2s # Wait for subscription
58+
${group2_id}= Create Consumer auto_offset_reset=earliest
59+
Subscribe Topic ${group2_id} topics=admindescribetest
60+
Sleep 2s # Wait for subscription
61+
${groups}= Create List ${group_id} ${group2_id}
62+
63+
${admin_client_id}= Create Admin Client
64+
${described_groups}= Describe Groups ${admin_client_id} group_ids=${groups}
65+
Log ${described_groups}
66+
67+
FOR ${member} IN @{described_groups["${group_id}"].members}
68+
Log ${member}
69+
END
70+
Log ${described_groups["${group_id}"].state}
71+
Log ${described_groups["${group2_id}"].state}
72+
73+
[Teardown] Run Keywords Basic Teardown ${group_id} AND
74+
... Basic Teardown ${group2_id}
75+
76+
AdminClient Delete Consumer Groups
77+
${producer_group_id}= Create Producer
78+
Produce ${producer_group_id} topic=admindeltest value=Hello partition=${0}
79+
Wait Until Keyword Succeeds 10x 0.5s All Messages Are Delivered ${producer_group_id}
80+
81+
${group_id}= Create Consumer auto_offset_reset=earliest
82+
Subscribe Topic ${group_id} topics=admindeltest
83+
Sleep 2s # Wait for subscription
84+
${group2_id}= Create Consumer auto_offset_reset=earliest
85+
Subscribe Topic ${group2_id} topics=admindeltest
86+
Sleep 2s # Wait for subscription
87+
${groups}= Create List ${group2_id}
88+
${messages}= Poll group_id=${group2_id} max_records=5
89+
Sleep 1s
90+
Unsubscribe ${group2_id}
91+
Close Consumer ${group2_id}
92+
93+
${admin_client_id}= Create Admin Client
94+
${deletion}= Delete Groups ${admin_client_id} group_ids=${groups}
95+
Should Be Equal ${deletion["${group2_id}"]} ${None}
96+
97+
${current_groups}= List Groups ${admin_client_id}
98+
Log ${current_groups.valid}
99+
FOR ${group} IN @{current_groups.valid}
100+
Log ${group.group_id}
101+
IF "${group_id}" == "${group.group_id}"
102+
Log ${group.group_id}
103+
Log ${group.state}
104+
Log "Consumer found in list"
105+
END
106+
IF "${group2_id}" == "${group.group_id}"
107+
Log ${group.group_id}
108+
Log ${group.state}
109+
Fail "Group 1 consumer was not removed!"
110+
END
111+
END
112+
[Teardown] Basic Teardown ${group_id}
28113

29114
AdminClient New Partitions
30115
${topic_name}= Set Variable admin_testing_partition
@@ -33,17 +118,20 @@ AdminClient New Partitions
33118
Create Topics group_id=${admin_client_id} new_topics=${topic}
34119

35120
${new_parts}= New Partitions ${topic_name} new_total_count=${2}
36-
Create Partitions group_id=${admin_client_id} new_partitions=${new_parts}
121+
${resp}= Create Partitions group_id=${admin_client_id} new_partitions=${new_parts}
122+
Log ${resp}
37123
[Teardown] Delete Topics ${admin_client_id} ${topic_name}
38124

39125
AdminClient Describe Configs
40126
${resource}= Config Resource ${ADMIN_RESOURCE_BROKER} 1
127+
Log ${resource.name}
41128
${admin_client_id}= Create Admin Client
42129
${config}= Describe Configs ${admin_client_id} ${resource}
130+
Log ${config}
43131

44132
Should Not Be Empty ${config}
45-
${name}= Set Variable ${config['offsets.commit.timeout.ms'].name}
46-
${value}= Set Variable ${config['offsets.commit.timeout.ms'].value}
133+
${name}= Set Variable ${config["${resource.name}"]['offsets.commit.timeout.ms'].name}
134+
${value}= Set Variable ${config["${resource.name}"]['offsets.commit.timeout.ms'].value}
47135
Should Be Equal As Strings ${name} offsets.commit.timeout.ms
48136
Should Be Equal As Integers ${value} ${5000}
49137

@@ -53,7 +141,24 @@ AdminClient Alter Configs
53141
${resource}= Config Resource ${ADMIN_RESOURCE_BROKER} 1 set_config=${data}
54142
${admin_client_id}= Create Admin Client
55143

56-
Alter Configs ${admin_client_id} ${resource}
144+
${resp}= Alter Configs ${admin_client_id} ${resource}
145+
Log ${resp}
57146
Sleep 1s
58147
${config}= Describe Configs ${admin_client_id} ${resource}
59-
Should Be Equal As Integers ${54321} ${config['log.retention.ms'].value}
148+
Should Be Equal As Integers ${54321} ${config["${resource.name}"]['log.retention.ms'].value}
149+
150+
151+
*** Keywords ***
152+
All Messages Are Delivered
153+
[Arguments] ${producer_id}
154+
${count}= Flush ${producer_id}
155+
Log Reaming messages to be delivered: ${count}
156+
Should Be Equal As Integers ${count} 0
157+
158+
Basic Teardown
159+
[Arguments] ${group_id}
160+
Unsubscribe ${group_id}
161+
Close Consumer ${group_id}
162+
${groups}= Create List ${group_id}
163+
${admin_client_id}= Create Admin Client
164+
Delete Groups ${admin_client_id} group_ids=${groups}

‎examples/test_avro.robot

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ Library Collections
44
Library String
55

66
Suite Setup Starting Test
7+
Suite Teardown Stop Thread
78

89

910
*** Test Cases ***
@@ -105,8 +106,11 @@ Basic Teardown
105106
[Arguments] ${group_id}
106107
Unsubscribe ${group_id}
107108
Close Consumer ${group_id}
108-
109-
Unassign Teardown
110-
[Arguments] ${group_id}
111-
Unassign ${group_id}
112-
Close Consumer ${group_id}
109+
${groups}= Create List ${group_id}
110+
${admin_client_id}= Create Admin Client
111+
${resp}= Delete Groups ${admin_client_id} group_ids=${groups}
112+
Log ${resp}
113+
114+
Stop Thread
115+
${resp}= Stop Consumer Threaded ${MAIN_THREAD}
116+
Log ${resp}

‎examples/test_oauth.robot

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ ${SEEN_RF_OAUTH_CB} ${False}
1212
*** Test Cases ***
1313
Example Oauth
1414
[Documentation] Example of how to use OAUTH with library and call functools
15-
... via get_token function. For better handling there could be
15+
... via get_token function. For better handling there could be
1616
... some global variable which can be set inside of python lib.
1717
... Not executable right now, needs update env (issue #21).
1818

‎setup.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,12 @@
2828
],
2929
install_requires = [
3030
'robotframework >= 3.2.1',
31-
'confluent-kafka == 1.9.0',
31+
'confluent-kafka == 2.0.2',
3232
'requests >= 2.25.1',
3333
'avro-python3 >= 1.10.1',
3434
'fastavro >= 1.3.2',
35-
'jsonschema >= 3.2.0'
35+
'jsonschema >= 3.2.0',
36+
'protobuf >= 4.22.0'
3637
],
3738
package_dir = {'' : 'src'},
3839
packages = ['ConfluentKafkaLibrary'],

0 commit comments

Comments
 (0)