strimzi kafka operator manages existing kafka

[lanzhiwang]

I directly deployed a kafka cluster using the kafka installation file, kafka_2.12-2.5.0.tgz. I specified the path on the physical machine /kafka-zookeeper-data/kafka-0 as the log directory of kafka, and /kafka-zookeeper-data/zookeeper-0 as the data directory of zookeeper, and then I created topics to produce and consume data. In other words, I wrote some data in /kafka-zookeeper-data/kafka-0 and /kafka-zookeeper-data/zookeeper-0.

In the second step, I created pv and pvc, and pv uses hostpath, /kafka-zookeeper-data/kafka-0 and /kafka-zookeeper-data/zookeeper-0

$ cat pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-data-my-cluster-kafka-0
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
    path: /kafka-zookeeper-data/kafka-0
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - 192.168.34.46
  volumeMode: Filesystem
  storageClassName: local-path

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-data-my-cluster-zookeeper-0
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
    path: /kafka-zookeeper-data/zookeeper-0
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - 192.168.34.46
  volumeMode: Filesystem
  storageClassName: local-path

$ cat pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-my-cluster-kafka-0
  namespace: kafka
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: local-path
  volumeMode: Filesystem
  volumeName: pv-data-my-cluster-kafka-0

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-my-cluster-zookeeper-0
  namespace: kafka
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: local-path
  volumeMode: Filesystem
  volumeName: pv-data-my-cluster-zookeeper-0

Finally, I deployed the strimzi kafka operator and also created the kafka cluster.

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      log.message.format.version: "2.5"
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    jmxOptions:
      "-Xms": 8192m
      "-Xmx": 8192m
    listeners:
      external:
        tls: false
        type: nodeport
      plain: {}
      tls: {}
    replicas: 1
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 500m
        memory: 500Mi
    storage:
      class: local-path
      deleteClaim: false
      size: 1Gi
      type: persistent-claim
    template:
      pod:
        securityContext:
          fsGroup: 0
          runAsUser: 0
    version: 2.5.0
  zookeeper:
    jmxOptions:
      "-Xms": 4096m
      "-Xmx": 4096m
    replicas: 1
    resources:
      limits:
        cpu: "1"
        memory: 1000Mi
      requests:
        cpu: 500m
        memory: 500Mi
    storage:
      class: local-path
      deleteClaim: false
      size: 1Gi
      type: persistent-claim
    template:
      pod:
        securityContext:
          fsGroup: 0
          runAsUser: 0

The kafka cluster can be successfully created, but I found that the operator cannot automatically discover the existing topic, and will not create a kafkatopic because of the topic in kafka itself. The data that has been written to the cluster cannot be consumed again.

I would like to ask if the operator does not support this scenario or the steps I have taken are wrong.

Can the operator manage an existing Kafka cluster ？

[scholzj]

I'm not sure what exactly are you trying to achieve. HostPath volumes are not a good choice because they don't bind to a particular machines. If you wanna use local disks, you should use Local Persistent Volumes. If you want to migrate the data from other Kafka cluster by copying them, I think that should work - but you need to make sure to place them into the right place in the PV => the same one as where Strimzi would put them.

[lanzhiwang]

We first deployed Kafka in k8s using deployment and hostpath methods. Now we find that your company’s Kafka operator is very good. We want to try to use them to manage Kafka clusters, including the Kafka clusters that have been deployed using deployment and hostpath methods.

[scholzj]

Strimzi is using following paths ...

For ZooKeeper:

• The data should be in /var/lib/zookeeper/data. The mount point here is /var/lib/zookeeper. So the data needs to be on your volume.
• For Kafka:
  • When you use JBOD storage, the Kafka data are in a path like this: /var/lib/kafka/data-0/kafka-log0 where the 0 in data-0 is the ID of the JBOD volume and the 0 in the kafka-log0 is the pod ID (the sequence number at the end of the pod). The mount path is /var/lib/kafka/data-0 - so the kafka-log0 has to be on your disk.
  • When you use non-JBOD storage, the Kafka data are in a path like this: /var/lib/kafka/data/kafka-log0 where the 0 in the kafka-log0 is the pod ID (the sequence number at the end of the pod). The mount path is /var/lib/kafka/data, so the kafka-log0 again needs to be on your volume.