k8s-security

• Kubernetes (kind)
  • Create cluster with Istio
  • Deploy demo app (sentences generator)
  • Test the app
  • Test Kyverno policies
    • Admission policies
    • Mutation policies
  • Test Falco
    • Trigger a rule
    • Access Falcosidekick UI
• How to dev - demo app
  • Docker compose run
  • Local run
  • Configuration
    • Common module
    • Aggregator
    • Provider

Kubernetes (kind)

Prerequisites:

• kubectl (obviously)
• just: alternative to make (cargo install just)
• kind
• cloud-provider-kind
• helmfile
• kyverno CLI

Create cluster with Istio

Create kind cluster:

just kube-cluster-create

Install Istio (ambient mode) with Kyverno and Falco:

just kube-cluster-init

Note

You will need to start a new shell where you will run cloud-provider-kind so that the LoadBalancer, the one created by the Gateway, gets an external IP.

Deploy demo app (sentences generator)

just helm-demo-app-install

Test the app

To get the external IP of our service:

export INGRESS_HOST=$(kubectl get -n my-sentences-demo-app gateway sentences-demo-app-gw -o jsonpath='{.status.addresses[0].value}')
echo $INGRESS_HOST

And then you try the app with:

curl http://$INGRESS_HOST/

You can check in Kiali, that the mTLS is enabled in our namespace:

kubectl port-forward -n istio-system svc/kiali 20001:20001

Go to http://localhost:20001/, and see the app in Kiali:

And the traffic graph (you need to send some requests to the app to see it):

Test Kyverno policies

If you want to test all the Kyverno policies more easily, you can use:

just kyverno-valid-tests kyverno-invalid-tests kyverno-mutate-tests

You can access the Kyverno Policy Reporter with:

kubectl port-forward -n kyverno svc/policy-reporter-ui 8082:8080

And go to http://localhost:8082/.

Admission policies

1. ClusterPolicy check-image: check that images, coming from ghcr.io/mfernd/k8s-security*, has a signature (by sigstore/cosign)

# invalid pod image (does not have a signature)
kyverno apply charts/kubernetes_yaml/kyverno/check-images.cpol.yaml --resource charts/kubernetes_yaml/kyverno/tests/invalid-pod-image.yaml
# valid pod image (latest has a signature)
kyverno apply charts/kubernetes_yaml/kyverno/check-images.cpol.yaml --resource charts/kubernetes_yaml/kyverno/tests/valid-pod.yaml

2. ClusterPolicy require-requests-limits: check that all pods, in namespaces beginning by my* or app*, have requests and limits resources defined

# invalid pod (does not have requests and limits)
kyverno apply charts/kubernetes_yaml/kyverno/require-requests-limits.cpol.yaml --resource charts/kubernetes_yaml/kyverno/tests/invalid-pod-requests-limits.yaml
# valid pod (has requests and limits)
kyverno apply charts/kubernetes_yaml/kyverno/require-requests-limits.cpol.yaml --resource charts/kubernetes_yaml/kyverno/tests/valid-pod.yaml

3. ClusterPolicy restrict-nodeport: prevent the use of NodePort services

# invalid service (has NodePort)
kyverno apply charts/kubernetes_yaml/kyverno/restrict-nodeport.cpol.yaml --resource charts/kubernetes_yaml/kyverno/tests/invalid-svc-nodeport.yaml
# valid service (to test that ClusterIP, at least, works)
kyverno apply charts/kubernetes_yaml/kyverno/restrict-nodeport.cpol.yaml --resource charts/kubernetes_yaml/kyverno/tests/valid-svc-clusterip.yaml

Mutation policies

1. ClusterPolicy add-default-securitycontext: add a default securityContext to all pods

kyverno apply charts/kubernetes_yaml/kyverno/add-default-securitycontext.cpol.yaml --resource charts/kubernetes_yaml/kyverno/tests/mutate-pod-security-context.yaml

We see that the pod has been patched with a securityContext.

Applying 3 policy rule(s) to 1 resource(s)...

mutate policy add-default-securitycontext applied to default/Pod/nginx:
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
  - image: nginx:1.27.3
    name: nginx
    ports:
    - containerPort: 80
  securityContext:
    fsGroup: 2000
    runAsGroup: 3000
    runAsNonRoot: true
    runAsUser: 1000

---

pass: 1, fail: 0, warn: 0, error: 0, skip: 0

2. ClusterPolicy add-istio-mesh-namespace: add the istio.io/dataplane-mode: ambient label to all namespace beginning by my* or app*

kyverno apply charts/kubernetes_yaml/kyverno/add-istio-mesh-ns.cpol.yaml --resource charts/kubernetes_yaml/kyverno/tests/mutate-ns-istio-mesh.yaml

We see that the namespace has our new istio label istio.io/dataplane-mode: ambient.

Applying 1 policy rule(s) to 1 resource(s)...

mutate policy add-istio-mesh-namespace applied to default/Namespace/my-namespace:
apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio.io/dataplane-mode: ambient
  name: my-namespace
  namespace: default

---

pass: 1, fail: 0, warn: 0, error: 0, skip: 0

Test Falco

Trigger a rule

We will try to trigger the rule Read sensitive file untrusted:

just falco-trigger-rule

Expected result:

{
  "hostname":"mfernd-k8s-security-control-plane",
  "output":"17:55:52.489810605: Warning Sensitive file opened for reading by non-trusted program (file=/etc/shadow gparent=<NA> ggparent=<NA> gggparent=<NA> evt_type=openat user=root user_uid=0 user_loginuid=-1 process=cat proc_exepath=/usr/bin/cat parent=containerd-shim command=cat /etc/shadow terminal=34816 container_id=edabb60361fc container_image=docker.io/library/nginx container_image_tag=1.27 container_name=bayrou k8s_ns=trigger-falco-rule k8s_pod_name=bayrou)",
  "output_fields":{
    "container.id":"edabb60361fc",
    "container.image.repository":"docker.io/library/nginx",
    "container.image.tag":"1.27",
    "container.name":"bayrou",
    "evt.time":1734890152489810605,
    "evt.type":"openat",
    "fd.name":"/etc/shadow",
    "k8s.ns.name":"trigger-falco-rule",
    "k8s.pod.name":"bayrou",
    "proc.aname[2]":"<NA>",
    "proc.aname[3]":null,
    "proc.aname[4]":null,
    "proc.cmdline":"cat /etc/shadow",
    "proc.exepath":"/usr/bin/cat",
    "proc.name":"cat",
    "proc.pname":"containerd-shim",
    "proc.tty":34816,
    "user.loginuid":-1,
    "user.name":"root",
    "user.uid":0
  },
  "priority":"Warning",
  "rule":"Read sensitive file untrusted",
  "source":"syscall",
  "tags":[
    "T1555",
    "container",
    "filesystem",
    "host",
    "maturity_stable",
    "mitre_credential_access"
  ],
  "time":"2024-12-22T17:55:52.489810605Z"
}

Tip

You can see the rule in the default falco_rules.yaml.

Access Falcosidekick UI

kubectl port-forward -n falco svc/falco-falcosidekick-ui 2802:2802

Go to http://localhost:2802/ (credentials are admin/admin).

And you can see the triggered rule:

How to dev - demo app

Docker compose run

just up

Local run

just dev

With mprocs to execute services in parallel.

Configuration

Common module

Source: crates/common/

Env Var	Description	Default
APP_HOST	Application host	"0.0.0.0"
APP_PORT	Application port	3000 (or 80 in Docker)

Used by all other modules.

Aggregator

Source: crates/aggregator-svc/

Env Var	Description	Default
APP_WORKERS_CONFIG	Used to know where to get words (see workers_config.example.toml)	N/A

Provider

Source: crates/provider-svc/

Env Var	Description	Default
APP_PROVIDER_KIND	Define the provider type of the instance (see struct WordKind for possible values)	N/A