-
Notifications
You must be signed in to change notification settings - Fork 7
/
Copy pathKafka Cluster Installation setup in KRAFT mode(no Zookeeper) on Multi Nodes
84 lines (76 loc) · 4.39 KB
/
Kafka Cluster Installation setup in KRAFT mode(no Zookeeper) on Multi Nodes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
********Kafka Cluster multi different node install/setup (in Kraft mode, no zookeeper)*********Kafka V3.4.0======
1. Login (ssh) to each box update & install yum & wget
login to jump box or connect to machines that you have access directly
yum update -y
yum install -y wget curl java
2. AWS EC2 Instances(RHEL): add into /etc/hosts file and ping the hosts from one box to other, it should communicate each other.
10.20.2.217 kafka1
10.20.3.33 kafka2
10.20.4.46 kafka3
3. Create a user "kafka"
useradd kafka
4. Create couple of directories and update & install yum & wget
mkdir -p /data/kafka/kafka1.logs /data/kafka/kafka2.logs /data/kafka/kafka3.logs
mkdir -p /opt/kafka
chown -R kafka:kafka data opt
4. Download kafka (3.4.0, kraft) and untar - reference https://kafka.apache.org/quickstart
cd /opt/kafka
sudo wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz or
curl -O https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
tar -xzf kafka_2.13-3.4.0.tgz
5. Actual fun begins here,...at "server.properties" file ** key is kraft folder
cd /opt/kafka/config/kraft
do below changes in "server.properties" on each box
5.1 process.roles=broker,controller --> its optional change by default both roles
5.2 node.id=1 (node.id=2,node.id=3,...node.id=n based on N no.of nodes on respective node)
5.3 controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093 (list of voters to elect leader)
5.4 listeners=PLAINTEXT://kafka1:9092,CONTROLLER://kafka1:9093 -->kafka1 box
listeners=PLAINTEXT://kafka2:9092,CONTROLLER://kafka2:9093 -->kafka2 box
listeners=PLAINTEXT://kafka3:9092,CONTROLLER://kafka2:9093 -->kafka3 box
5.5 advertised.listeners=PLAINTEXT://kafka1:9092 -->kafka1 box
advertised.listeners=PLAINTEXT://kafka2:9092 -->kafka2 box
advertised.listeners=PLAINTEXT://kafka2:9092 -->kafka3 box
5.6 log.dirs=/data/kafka/kafka1.logs -->Kafka1 box
log.dirs=/data/kafka/kafka2.logs -->Kafka2 box
log.dirs=/data/kafka/kafka3.logs -->Kafka3 box
6. Generate a Cluster UUID in anyone of the box, and make sure rest of other boxes must have same cluster ID by export
KAFKA_CLUSTER_ID="$(/opt/kafka/bin/kafka-storage.sh random-uuid)" --> any box
and set/export same echo $KAFKA_CLUSTER_ID in other boxes
export KAFKA_CLUSTER_ID=`echo $KAFKA_CLUSTER_ID` (backquote `, not single ' or double ") or copy & past xport KAFKA_CLUSTER_ID="......."
7. Format Log Directories
/opt/kafka/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c /opt/kafka/config/kraft/server.properties
8. Start the Kafka Server
1. /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties -->interactive mode
2. /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/kraft/server.properties -->backend mode
3. Adding service mode, to start & stop on reboot or failure/down automatically - no manual intervention
9. (8.3) Adding Kafka service daemon process, Add below content to the kafka.service file
cd /etc/systemd/system
sudo vi kafka.service
[subbuj@ip-10-20-2-216 system]$ cat kafka.service
[Unit]
Description=kafka Service
After=network-online.target
Requires=network-online.target
[Service]
Type=simple
Restart=on-failure
User=kafka
Group=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh /opt/kafka/config/kraft/server.properties
WorkingDirectory=/opt/kafka
[Install]
WantedBy=multi-user.target
10. make sure file permissions and load "sudo systemctl daemon-reload" and try "sudo systemctl start kafka" and "sudo systemctl stop kafka"
-rw-r--r--. 1 root root 403 Apr 20 18:11 kafka.service
sudo systemctl daemon-reload
sudo systemctl start kafka
systemctl status kafka
sudo systemctl stop kafka
systemctl status kafka
systemctl enable kafka.service --> To enbale service to pick-up on reboot [op/:Created symlink /etc/systemd/system/multi-user.target.wants/kafka.service → /etc/systemd/system/kafka.service.]
11. Few Samples:
sudo /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --version
sudo /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --list
sudo /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --create --topic testsj
sudo /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --create --topic testsj-r3p10 --partitions 10 --replication-factor 3