[DENA-415] Shared elasticsearch manifest (#16)

* change the repository structure- one generated manifest + kustomize overlays

* improve example, regenerate charts

* update paths, change secret name

* remove unnecessary fiels

* applying comments

* regenerate manifests

* update example

* applying comments

* remove namespace by default

* groundwork for elasticsearch

* split elasticsearch into two manifests

* remove the multi node role cluster

* update README

* set pvc size

* single node working deployment

* kibana resources, remove unnecessary modifications

* kibana annotations

* latest es release

* replace env variables dependent on namespace

* test variable index before replacing

* remove unused secret

* add comment

* move hosts patch to separate file

Makefile

@@ -1,4 +1,5 @@
 BITNAMI_REDIS_RELEASE=18.1.6
+BITNAMI_ES_RELEASE=19.16.1
 
 gen-redis-manifest:
 	docker run -ti --rm \
@@ -7,3 +8,11 @@ gen-redis-manifest:
 		--workdir=/opt/manifests \
 		--entrypoint=./gen-yaml/gen.sh \
 		alpine/helm
+
+gen-es-manifest:
+	docker run -ti --rm \
+		--volume $${PWD}/elasticsearch:/opt/manifests \
+		--env BITNAMI_ES_RELEASE=${BITNAMI_ES_RELEASE} \
+		--workdir=/opt/manifests \
+		--entrypoint=./gen-yaml/gen.sh \
+		alpine/helm

README.md

@@ -20,6 +20,7 @@ templating chart with UW specific values and are using them as Kustomize bases.
 | Directory/ File    | Purpose                                                                               |
 |--------------------|---------------------------------------------------------------------------------------|
 | `Makefile`         | Targets with parameters to generate customized manifests                              |
+| `components`       | Reusable kustomize components                                                         |
 | `{name}/example`   | Example of everything needed to deploy the generated chart                            |
 | `{name}/gen-yaml`  | Everything needed to generate manifest from helm chart                                |
 | `{name}/manifests` | Manifest- generated and ready to be referenced. Together with hand- crafted overlays. |

components/bitnami-common/kustomization.yaml

@@ -17,9 +17,7 @@ patches:
       - op: remove
         path: /spec/template/metadata/labels/app.kubernetes.io~1managed-by
     target:
-      group: apps
-      version: v1
-      kind: StatefulSet
+      group: apps # Deployments, StatefulSets, DaemonSets
   # remove these empty affinity nodes, as in Kustomize 5.0.x kustomize generates them as strings with value "null" instead of just null, and they can not be applied.
   # added this issue in kustomize: https://github.com/kubernetes-sigs/kustomize/issues/5171
   - patch: |-
@@ -28,6 +26,4 @@ patches:
       - op: remove
         path: /spec/template/spec/affinity/nodeAffinity
     target:
-      group: apps
-      version: v1
-      kind: StatefulSet
+      group: apps # Deployments, StatefulSets, DaemonSets

elasticsearch/README.md

@@ -0,0 +1,41 @@
+# Elasticsearch
+
+## TL;DR
+- [Is this Elasticsearch right for you?](#cluster-topology)
+- [Example kustomization file deploying Elasticsearch](example)
+- [Operations](#usage)
+- [Grafana dashboards](#metrics)
+
+## Source
+
+This manifest is build on the base of [Bitnami Elasticsearch Helm chart](https://github.com/bitnami/charts/tree/main/bitnami/elasticsearch).
+
+## Cluster topology
+### Multipurpose nodes
+This Elasticsearch manifest deploys cluster with multipurpose nodes.
+This is the simplest topology, which is suitable for small clusters.
+
+This means, all the nodes can be elected as master,
+used for data storage and for operations.
+
+
+### Nodes with dedicated roles
+For bigger clusters, it is recommended to use dedicated master and data nodes,
+optionally also ingest and coordinating nodes. Read more about node roles in [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html).
+
+Manifest with dedicated node roles can be generated the same way as multipurpose one, by providing
+appropriate values to parameters like `data.replicaCount` in [gen.sh script](gen-yaml/gen.sh).
+
+If you need cluster with dedicated node roles, please contact us at #help-dev-enablement,
+we will be happy to help you add its manifest to the shared-kustomize-bases.
+
+## Usage
+### Scaling
+In order to add nodes to cluster, first scale down to 0 replicas
+in order to avoid inconsistencies in the configuration.
+
+## Metrics
+TODO
+
+## Backup
+TODO

elasticsearch/example/kustomization.yaml

@@ -0,0 +1,38 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  #- ssh://github.com/utilitywarehouse/shared-kustomize-bases//elasticsearch/manifests?ref=v0.0.0
+  - ../manifests
+
+# OpsLevel annotations
+commonAnnotations:
+  "app.uw.systems/description": "some es instance"
+  "app.uw.systems/tier": "tier_4"
+
+patches:
+  # PVC size
+  - target:
+      kind: StatefulSet
+      name: elastisearch-master
+    patch: |-
+      - op: replace
+        path: /spec/volumeClaimTemplates/0/spec/resources/requests/storage
+        value: 10Gi
+  # Resource requests and limits
+  - target:
+      kind: StatefulSet
+      name: elasticsearch-master
+    patch: |-
+      - op: replace
+        path: /spec/template/spec/containers/0/resources/requests/cpu
+        value: "500m"
+      - op: replace
+        path: /spec/template/spec/containers/0/resources/limits/cpu
+        value: "1000m"
+      - op: replace
+        path: /spec/template/spec/containers/0/resources/requests/memory
+        value: "4Gi"
+      - op: replace
+        path: /spec/template/spec/containers/0/resources/limits/memory
+        value: "8Gi"

elasticsearch/gen-yaml/gen.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+set -o errexit -o pipefail -o nounset
+
+# Fetch from helm repo: https://github.com/bitnami/charts/tree/main/bitnami/elasticsearch
+helm repo add bitnami https://charts.bitnami.com/bitnami
+
+helm template "elasticsearch" bitnami/elasticsearch --version "${BITNAMI_ES_RELEASE}" \
+  --set commonAnnotations."app\.uw\.systems\/repos"="https://github.com/utilitywarehouse/shared-kustomize-bases/tree/main/elasticsearch" \
+  --set global.kibanaEnabled="true" \
+  --set metrics.enabled="true" \
+  --set master.replicaCount="3" \
+  --set master.masterOnly="false" \
+  --set data.replicaCount="0" \
+  --set coordinating.replicaCount="0" \
+  --set ingest.replicaCount="0" \
+  --set master.resources.requests.cpu="250m" \
+  --set master.resources.limits.cpu="2000m" \
+  --set master.resources.requests.memory="4Gi" \
+  --set master.resources.limits.memory="8Gi" \
+  --set master.persistence.size="100Gi" \
+  --set sysctlImage.enabled="false" \
+  --set kibana.resources.requests.cpu="250m" \
+  --set kibana.resources.limits.cpu="1000m" \
+  --set kibana.resources.requests.memory="200Mi" \
+  --set kibana.resources.limits.memory="1Gi" \
+  --set kibana.commonAnnotations."app\.uw\.systems\/repos"="https://github.com/utilitywarehouse/shared-kustomize-bases/tree/main/elasticsearch" \
+  --set kibana.metrics.enabled="true" \
+  >"manifests/elasticsearch.yaml"