Update Istio configuration to use CNI node agents instead of istio-init containers (#474)

# Summary

We are using `Istio` as a service mesh provider for our Multi Cluster
tests. The way it works by default is `Istio` adds privileged
`init-istio` container to every Pod that configures network accordingly.

>By default Istio injects an init container, istio-init, in pods
deployed in the mesh. The istio-init container sets up the pod network
traffic redirection to/from the Istio sidecar proxy. This requires the
user or service-account deploying pods to the mesh to have sufficient
Kubernetes RBAC permissions to deploy [containers with the NET_ADMIN and
NET_RAW
capabilities](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-capabilities-for-a-container).

While this works fine it is not meeting the
[PSS](https://v1-32.docs.kubernetes.io/docs/concepts/security/pod-security-standards/)
restricted level, thus making it less secure. Related
[HELP-81729](https://jira.mongodb.org/browse/HELP-81729) and
#473 that enables
`restricted` level in `warn` mode. Additionally we provide Istio sidecar
configuration as an example in our code snippets thus not following the
best practice.

There is another way to configure Istio mesh that does not require
`istio-init` init-container - using [Istio CNI node
agent](https://istio.io/latest/docs/setup/additional-setup/cni/#using-the-istio-cni-node-agent).
This PR configures our e2e tests and code snippets that way. Great blog
entry about difference between `istio-init` and Istio CNI node agent
architecture ->
https://www.solo.io/blog/traffic-ambient-mesh-istio-cni-node-configuration.

With `istio-init`:
<img width="810" height="820" alt="image"
src="https://github.com/user-attachments/assets/026350af-3b51-4fe9-9cb8-c8911e661eca"
/>

With `Istio CNI node agent`:
<img width="942" height="1084" alt="image"
src="https://github.com/user-attachments/assets/37733169-7737-4063-90a0-de3d116402a9"
/>

⚠️ Init containers execute before the sidecar proxy starts, which can
result in traffic loss during their execution. This can be avoided by
setting `runAsUser: 1337`. More info ->
https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers

## Proof of Work

Passing CI is enough. Since `private_gke_code_snippets` are not run
automatically in CI I've triggered manual patch to test this ->
https://spruce.mongodb.com/version/68d50e694baed3000742566d/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC

## Checklist

- [ ] Have you linked a jira ticket and/or is the ticket in the title?
- [x] Have you checked whether your jira ticket required DOCSP changes?
- [x] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details

Author: MaciejKaras

docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/om_https_enabled.yaml

@@ -22,7 +22,7 @@ spec:
         spec:
           volumes:
             - name: mongodb-versions
-              emptyDir: {}
+              emptyDir: { }
           containers:
             - name: mongodb-ops-manager
               volumeMounts:
@@ -37,6 +37,8 @@ spec:
           initContainers:
             - name: setting-up-rhel-mongodb
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -48,6 +50,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-4-4
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -59,6 +63,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-5-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -70,6 +76,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -81,6 +89,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0-sig
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -92,6 +102,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0-21
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -103,6 +115,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0-21-sig
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -112,9 +126,10 @@ spec:
               volumeMounts:
                 - name: mongodb-versions
                   mountPath: /mongodb-ops-manager/mongodb-releases
-
             - name: setting-up-rhel-mongodb-7-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -126,6 +141,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-7-0-sig
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -137,6 +154,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-8-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -148,6 +167,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-8-0-sig
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L

docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/om_localmode-single-pv.yaml

@@ -35,6 +35,8 @@ spec:
           initContainers:
             - name: setting-up-rhel-mongodb-4-2-8
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -46,6 +48,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-6-0-21
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -57,6 +61,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-7-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L
@@ -68,6 +74,8 @@ spec:
                   mountPath: /mongodb-ops-manager/mongodb-releases
             - name: setting-up-rhel-mongodb-8-0
               image: curlimages/curl:latest
+              securityContext:
+                runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
               command:
                 - curl
                 - -L

docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/remote_fixtures/nginx.yaml

@@ -29,6 +29,8 @@ spec:
       initContainers:
         - name: setting-up-mongosh-1-4-1
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -38,6 +40,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-1-9-1
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -47,6 +51,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-1-10-4
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -56,6 +62,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-0-0
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -65,6 +73,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-0-2
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -74,6 +84,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-0-2-om7
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -83,6 +95,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-1-5-om7
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -92,6 +106,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-2-3-om7
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -101,6 +117,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-2-4-om7
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -110,6 +128,8 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-4-0
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
@@ -119,22 +139,23 @@ spec:
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
         - name: setting-up-mongosh-2-5-6
           image: curlimages/curl:latest
+          securityContext:
+            runAsUser: 1337 # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
           command:
             - sh
             - -c
             - curl -LO https://downloads.mongodb.com/compass/mongosh-2.5.6-linux-x64-openssl11.tgz --output-dir /mongodb-ops-manager/mongodb-releases/compass && true
           volumeMounts:
             - name: mongosh-versions
               mountPath: /mongodb-ops-manager/mongodb-releases/compass
-
       restartPolicy: Always
-      securityContext: {}
+      securityContext: { }
       terminationGracePeriodSeconds: 30
       volumes:
         - name: mongodb-versions
-          emptyDir: {}
+          emptyDir: { }
         - name: mongosh-versions
-          emptyDir: {}
+          emptyDir: { }
         - configMap:
             name: nginx-conf
           name: nginx-conf

docker/mongodb-kubernetes-tests/tests/opsmanager/om_remotemode.py

@@ -14,6 +14,7 @@
 
 VERSION_NOT_IN_WEB_SERVER = "4.2.1"
 
+
 # If this test is failing after an OM Bump, ensure that the nginx deployment fixture contains the associated mongosh
 # version. More details in this ticket: https://jira.mongodb.org/browse/CLOUDP-332640
 
@@ -47,6 +48,10 @@ def add_mdb_version_to_deployment(deployment: Dict[str, Any], version: str):
             "name": KubernetesTester.random_k8s_name(prefix="mdb-download"),
             "image": "curlimages/curl:latest",
             "command": ["sh", "-c", f"{curl_command} && true"],
+            "securityContext": {
+                # workaround for init-container istio issue -> https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
+                "runAsUser": 1337,
+            },
             "volumeMounts": [
                 {
                     "name": "mongodb-versions",

multi_cluster/tools/install_istio.sh

@@ -38,6 +38,7 @@ make -f ../tools/certs/Makefile.selfsigned.mk "${CTX_CLUSTER3}-cacerts" || make
 # create cluster secret objects with the certs and keys
 kubectl --context="${CTX_CLUSTER1}" delete ns istio-system || true
 kubectl --context="${CTX_CLUSTER1}" create ns istio-system
+kubectl --context="${CTX_CLUSTER1}" label --overwrite ns istio-system pod-security.kubernetes.io/enforce=privileged
 kubectl --context="${CTX_CLUSTER1}" create secret generic cacerts -n istio-system \
   --from-file=${CTX_CLUSTER1}/ca-cert.pem \
   --from-file=${CTX_CLUSTER1}/ca-key.pem \
@@ -46,6 +47,7 @@ kubectl --context="${CTX_CLUSTER1}" create secret generic cacerts -n istio-syste
 
 kubectl --context="${CTX_CLUSTER2}" delete ns istio-system || true
 kubectl --context="${CTX_CLUSTER2}" create ns istio-system
+kubectl --context="${CTX_CLUSTER2}" label --overwrite ns istio-system pod-security.kubernetes.io/enforce=privileged
 kubectl --context="${CTX_CLUSTER2}" create secret generic cacerts -n istio-system \
   --from-file=${CTX_CLUSTER2}/ca-cert.pem \
   --from-file=${CTX_CLUSTER2}/ca-key.pem \
@@ -54,6 +56,7 @@ kubectl --context="${CTX_CLUSTER2}" create secret generic cacerts -n istio-syste
 
 kubectl --context="${CTX_CLUSTER3}" delete ns istio-system || true
 kubectl --context="${CTX_CLUSTER3}" create ns istio-system
+kubectl --context="${CTX_CLUSTER3}" label --overwrite ns istio-system pod-security.kubernetes.io/enforce=privileged
 kubectl --context="${CTX_CLUSTER3}" create secret generic cacerts -n istio-system \
   --from-file=${CTX_CLUSTER3}/ca-cert.pem \
   --from-file=${CTX_CLUSTER3}/ca-key.pem \
@@ -67,6 +70,10 @@ apiVersion: install.istio.io/v1alpha1
 kind: IstioOperator
 spec:
   tag: ${VERSION}
+  components:
+    cni:
+      namespace: istio-system
+      enabled: true
   meshConfig:
     defaultConfig:
       terminationDrainDuration: 30s
@@ -81,13 +88,17 @@ spec:
       network: network1
 EOF
 
-bin/istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml -y &
+bin/istioctl install --context="${CTX_CLUSTER1}" --set components.cni.enabled=true -f cluster1.yaml -y &
 
 cat <<EOF >cluster2.yaml
 apiVersion: install.istio.io/v1alpha1
 kind: IstioOperator
 spec:
   tag: ${VERSION}
+  components:
+    cni:
+      namespace: istio-system
+      enabled: true
   meshConfig:
     defaultConfig:
       terminationDrainDuration: 30s
@@ -102,13 +113,17 @@ spec:
       network: network1
 EOF
 
-bin/istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml -y &
+bin/istioctl install --context="${CTX_CLUSTER2}" --set components.cni.enabled=true -f cluster2.yaml -y &
 
 cat <<EOF >cluster3.yaml
 apiVersion: install.istio.io/v1alpha1
 kind: IstioOperator
 spec:
   tag: ${VERSION}
+  components:
+    cni:
+      namespace: istio-system
+      enabled: true
   meshConfig:
     defaultConfig:
       terminationDrainDuration: 30s
@@ -123,7 +138,7 @@ spec:
       network: network1
 EOF
 
-bin/istioctl install --context="${CTX_CLUSTER3}" -f cluster3.yaml -y &
+bin/istioctl install --context="${CTX_CLUSTER3}" --set components.cni.enabled=true -f cluster3.yaml -y &
 
 wait
 

multi_cluster/tools/install_istio_central.sh

@@ -10,4 +10,4 @@ source multi_cluster/tools/download_istio.sh
 cd istio-${VERSION}
 
 bin/istioctl x uninstall --context="${CTX_CLUSTER}" --purge --skip-confirmation
-bin/istioctl install --context="${CTX_CLUSTER}" --set profile=default --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY --skip-confirmation
+bin/istioctl install --context="${CTX_CLUSTER}" --set components.cni.enabled=true --set profile=default --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY --skip-confirmation

public/architectures/setup-multi-cluster/ra-03-setup-istio/install_istio_separate_network.sh

@@ -94,6 +94,7 @@ spec:
       network: network1
 EOF
 bin/istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml -y
+
 samples/multicluster/gen-eastwest-gateway.sh \
     --mesh mesh1 --cluster cluster1 --network network1 | \
     bin/istioctl --context="${CTX_CLUSTER1}" install -y -f -

public/samples/ops-manager/ops-manager-remote-mode.yaml

@@ -73,7 +73,6 @@ spec:
           volumeMounts:
             - name: mongodb-versions
               mountPath: /mongodb-ops-manager/mongodb-releases/linux
-
         - name: setting-up-rhel-mongodb-4-4-ent
           image: curlimages/curl:latest
           command: