Merge pull request #270 from Yolean/storagclass-kustomize

Use standard storage class by default, demonstrate kustomization per hosting provider

Author: solsson

README.md

@@ -12,14 +12,15 @@ To quote [@arthurk](https://github.com/Yolean/kubernetes-kafka/issues/82#issueco
 ## Getting started
 
 We suggest you `apply -f` manifests in the following order:
- * Your choice of storage classes from [./configure](./configure/)
  * [namespace](./00-namespace.yml)
  * [./rbac-namespace-default](./rbac-namespace-default/)
  * [./zookeeper](./zookeeper/)
  * [./kafka](./kafka/)
 
 That'll give you client "bootstrap" `bootstrap.kafka.svc.cluster.local:9092`.
 
+With the introduction of [-k](#kustomize) support, apply `-f` will report `error: unable to decode "zookeeper/kustomization.yaml": Object 'Kind' is missing in ...`. With kubectl v1.14+ you can avoid that using: `kubectl apply -k variants/scale-3-5/`.
+
 ## Fork
 
 Our only dependency is `kubectl`. Not because we dislike Helm or Operators, but because we think plain manifests make it easier to collaborate.
@@ -30,21 +31,35 @@ If you begin to rely on this kafka setup we recommend you fork, for example to e
 With the introduction of [app customization](https://kubectl.docs.kubernetes.io/pages/app_customization/introduction.html) in `kubectl` 1.14 there's an alternative to forks. We as a community can maintain a set of overlays.
 
 See the [variants](./variants) folder for different overlays. For example to scale to 1 kafka broker try `kubectl apply -k variants/scale-1/`.
+Variants also include examples of how to configure volumes for GKE, AWS and AKS with different storage classes.
+
+### Quickstart
 
-Currently `apply -k` replaces `apply -f ./zookeeper; apply -f ./kafka`.
-The original commands now result in `error: unable to decode "zookeeper/kustomization.yaml": Object 'Kind' is missing in ...`
-and though they still seem to work you can get around that with a v1.14+ kubectl using: `kubectl apply -k variants/as-is/`.
+```
+kubectl create namespace kafka && \
+kubectl apply -k github.com/Yolean/kubernetes-kafka/variants/dev-small/?ref=storagclass-kustomize
+```
+
+When all pods are Ready, test with for example `kafkacat -b localhost:9094 -L` over `kubectl -n kafka port-forward kafka-0 9094`.
 
 ### Maintaining your own kustomization
 
-`kubectl apply -k` takes a single overlay, meaning that you can't compose different overlays from this repo.
-You'll probably want to maintain your own variant.
-One option is to keep kubernets-kafka as a git submodule and edit the relative path from an example variant.
+Start your variant as a new folder in your choice of version control, with a base `kustomization.yaml` pointing to a tag or revision in this repository:
+
+```
+bases:
+- github.com/Yolean/kubernetes-kafka/rbac-namespace-default/?ref=553f327
+- github.com/Yolean/kubernetes-kafka/kafka/?ref=553f327
+- github.com/Yolean/kubernetes-kafka/zookeeper/?ref=553f327
+```
+
+Then pick and chose from patches our [example variants](https://github.com/Yolean/kubernetes-kafka/tree/master/variants) to tailor your Kafka setup.
 
 ## Version history
 
 | tag    | k8s ≥ | highlights  |
 | ------ | ----- | ----------- |
+| v6.0.0 | 1.11+ | Kafka 2.2.0 + `apply -k` (kubectl 1.14+) |
 | v5.1.0 | 1.11+ | Kafka 2.1.1 |
 | v5.0.3 | 1.11+ | Zookeeper fix [#227](https://github.com/Yolean/kubernetes-kafka/pull/227) + [maxClientCnxns=1](https://github.com/Yolean/kubernetes-kafka/pull/230#issuecomment-445953857) |
 | v5.0  | 1.11+  | Destabilize because in Docker we want Java 11 [#197](https://github.com/Yolean/kubernetes-kafka/pull/197) [#191](https://github.com/Yolean/kubernetes-kafka/pull/191) |
@@ -62,21 +77,14 @@ Have a look at:
  * [./consumers-prometheus](./consumers-prometheus/)
  * [or plain JMX](https://github.com/Yolean/kubernetes-kafka/pull/96)
  * what's happening in the [monitoring](https://github.com/Yolean/kubernetes-kafka/labels/monitoring) label.
- * Note that this repo is intentionally light on [automation](https://github.com/Yolean/kubernetes-kafka/labels/automation). We think every SRE team must build the operational knowledge first.
+ * Note that this repo is intentionally light on [automation](https://github.com/Yolean/kubernetes-kafka/labels/automation). We think every SRE team must build the operational knowledge first. But there is an example of a [Cruise Control](./cruise-control/) setup.
 
 ## Outside (out-of-cluster) access
 
 Available for:
 
  * [Brokers](./outside-services/)
 
-## Fewer than three nodes?
-
-For [minikube](https://github.com/kubernetes/minikube/), [youkube](https://github.com/Yolean/youkube) etc:
-
- * [Scale 1](https://github.com/Yolean/kubernetes-kafka/pull/44)
- * [Scale 2](https://github.com/Yolean/kubernetes-kafka/pull/118)
-
 ## Stream...
 
  * [Kubernetes events to Kafka](./events-kube/)

configure/README.md

@@ -100,7 +100,7 @@ spec:
       name: data
     spec:
       accessModes: [ "ReadWriteOnce" ]
-      storageClassName: kafka-broker
+      storageClassName: standard
       resources:
         requests:
           storage: 10Gi

configure/docker-storageclass-broker.yml

@@ -0,0 +1,3 @@
+resources:
+- node-reader.yml
+- pod-labler.yml

configure/docker-storageclass-zookeeper.yml

@@ -0,0 +1,7 @@
+bases:
+- ../scale-3-5
+resources:
+- aks-storageclass-broker-managed.yaml
+- aks-storageclass-zookeeper-managed.yaml
+patchesStrategicMerge:
+- volume-claims.yaml

configure/minikube-storageclasses.yml

@@ -0,0 +1,32 @@
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: kafka
+  namespace: kafka
+spec:
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes: [ "ReadWriteOnce" ]
+      storageClassName: kafka-broker
+      resources:
+        requests:
+          storage: 10Gi
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: pzoo
+  namespace: kafka
+spec:
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes: [ "ReadWriteOnce" ]
+      storageClassName: kafka-zookeeper
+      resources:
+        requests:
+          storage: 1Gi

kafka/50kafka.yml

@@ -0,0 +1,7 @@
+bases:
+- ../scale-3-5
+resources:
+- aws-storageclass-broker-gp2.yaml
+- aws-storageclass-zookeeper-gp2.yaml 
+patchesStrategicMerge:
+- volume-claims.yaml

rbac-namespace-default/kustomization.yaml

@@ -0,0 +1,32 @@
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: kafka
+  namespace: kafka
+spec:
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes: [ "ReadWriteOnce" ]
+      storageClassName: kafka-broker
+      resources:
+        requests:
+          storage: 10Gi
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: pzoo
+  namespace: kafka
+spec:
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes: [ "ReadWriteOnce" ]
+      storageClassName: kafka-zookeeper
+      resources:
+        requests:
+          storage: 1Gi

configure/aks-storageclass-broker-managed.yml renamed to variants/aks-managed/aks-storageclass-broker-managed.yaml

@@ -0,0 +1,8 @@
+[
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "advertised.listeners=PLAINTEXT://:9092,OUTSIDE://localhost:9094"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "auto.create.topics.enable=true"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "num.partitions=1"}
+]

configure/aks-storageclass-zookeeper-managed.yml renamed to variants/aks-managed/aks-storageclass-zookeeper-managed.yaml

@@ -0,0 +1,13 @@
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: kafka
+  namespace: kafka
+spec:
+  template:
+    spec:
+      containers:
+      - name: broker
+        env:
+        - name: JMX_PORT
+          value: ""

variants/aks-managed/kustomization.yaml

@@ -0,0 +1,19 @@
+# With kubectl -n kafka port-forward kafka-0 9094
+bases:
+- ../scale-1-ephemeral
+patchesStrategicMerge:
+- jmx-disable.yaml
+patchesJson6902:
+# Currently these duplicate each others' directives to avoid an apply -k error: found conflict between different patches
+- target:
+    group: apps
+    version: v1
+    kind: StatefulSet
+    name: kafka
+  path: listener-localhost.json
+- target:
+    group: apps
+    version: v1
+    kind: StatefulSet
+    name: kafka
+  path: auto-create-topics.json

variants/aks-managed/volume-claims.yaml

@@ -0,0 +1,8 @@
+[
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "advertised.listeners=PLAINTEXT://:9092,OUTSIDE://localhost:9094"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "auto.create.topics.enable=true"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"},
+  {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "num.partitions=1"}
+]

configure/aws-storageclass-broker-gp2.yml renamed to variants/aws-gp2/aws-storageclass-broker-gp2.yaml

@@ -0,0 +1,8 @@
+bases:
+- ../scale-3-5
+resources:
+- gke-storageclass-broker-pd.yaml
+- gke-storageclass-zookeeper-ssd.yaml
+- gke-storageclass-zookeeper-regional-euwest1cd-ssh.yaml
+patchesStrategicMerge:
+- volume-claims.yaml

configure/aws-storageclass-zookeeper-gp2.yml renamed to variants/aws-gp2/aws-storageclass-zookeeper-gp2.yaml

@@ -0,0 +1,48 @@
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: kafka
+  namespace: kafka
+spec:
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes: [ "ReadWriteOnce" ]
+      storageClassName: kafka-broker
+      resources:
+        requests:
+          storage: 10Gi
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: pzoo
+  namespace: kafka
+spec:
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes: [ "ReadWriteOnce" ]
+      storageClassName: kafka-zookeeper
+      resources:
+        requests:
+          storage: 1Gi
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: zoo
+  namespace: kafka
+spec:
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes: [ "ReadWriteOnce" ]
+      storageClassName: kafka-zookeeper-regional
+      resources:
+        requests:
+          storage: 1Gi

variants/aws-gp2/kustomization.yaml

@@ -4,7 +4,6 @@ metadata:
   name: kafka
   namespace: kafka
 spec:
-  replicas: 1
   template:
     spec:
       volumes:

variants/aws-gp2/volume-claims.yaml

@@ -1,13 +1,5 @@
 bases:
-- ../../kafka
-- ../../zookeeper
+- ../scale-1
 patchesStrategicMerge:
 - kafka.yaml
 - zookeeper.yaml
-patchesJson6902:
-- target:
-    group: apps
-    version: v1
-    kind: StatefulSet
-    name: kafka
-  path: kafka-scale1-overrides.json

variants/dev-small/auto-create-topics.json

@@ -11,16 +11,8 @@ metadata:
   name: pzoo
   namespace: kafka
 spec:
-  replicas: 1
   template:
     spec:
-      initContainers:
-      - name: init-config
-        env:
-        - name: PZOO_REPLICAS
-          value: '1'
-        - name: ZOO_REPLICAS
-          value: '0'
       volumes:
       - name: data
         emptyDir: {}
@@ -32,16 +24,9 @@ metadata:
   name: zoo
   namespace: kafka
 spec:
-  replicas: 0
   template:
     spec:
-      initContainers:
-      - name: init-config
-        env:
-        # There's no validation on these numbers adding up to a coherent zk config, so watch out
-        - name: PZOO_REPLICAS
-          value: '1'
-        - name: ZOO_REPLICAS
-          value: '0'
-        - name: ID_OFFSET
-          value: '2'
+      volumes:
+      - name: data
+        emptyDir: {}
+  volumeClaimTemplates: []

variants/dev-small/jmx-disable.yaml

@@ -1,4 +1,5 @@
 bases:
+- ../../rbac-namespace-default
 - ../../kafka
 - ../../zookeeper
 patchesStrategicMerge: