This document provides a summary of the tests included in the CNF Test Suite. Each test lists a general overview of what the test does, a link to the test code for that test, and links to additional information when relevant/available.
To learn how to run these tests, see the USAGE.md
To learn why these tests were written, see the RATIONALE.md
The increase and decrease capacity tests: HPA (horizonal pod autoscale) will autoscale replicas to accommodate when there is an increase of CPU, memory or other configured metrics to prevent disruption by allowing more requests by balancing out the utilisation across all of the pods.
Decreasing replicas works the same as increase but rather scale down the number of replicas when the traffic decreases to the number of pods that can handle the requests.
You can read more about horizonal pod autoscaling to create replicas here and in the K8s scaling cheatsheet.
- Expectation: The number of replicas for a Pod increases
What's tested: The pod is increased and replicated to 3 for the CNF image or release being tested.
- Expectation: The number of replicas for a Pod decreases
What's tested: After increase_capacity increases the replicas to 3, it decreases back to 1.
- Expectation: The Helm chart is published in a Helm Repsitory.
What's tested: Checks if the helm chart is found in a remote repository when running helm search.
- Expectation: No syntax or validation problems are found in the chart.
What's tested: Checks the syntax & validity of the chart using helm lint
- Expectation: The CNF was installed using Helm.
What's tested: Checks if the CNF is installed by using a Helm Chart.
- Expectation: The CNF Software version can be successfully incremented, then rolled back.
What's tested: Checks if the Pod can be upgraded to a new software version, then restored back to the orginal software version by using the Kubectl Set Image & Kubectl Rollout Undo commands.
- Expectation: The CNF Software version can be successfully incremented.
What's tested: Checks if the Pod can be upgraded to a new software version by using the Kubectl Set Image
- Expectation: The CNF Software version is successfully rolled back to its original version.
What's tested: Checks if the Pod can be rolled back to the original software version by using the Kubectl Set Image to perform a rollback.
- Expectation: The CNF Software version is successfully downgraded to a software version older than the orginal installation version.
What's tested: Checks if the Pod can be rolled back older software version(Older than the original software version) by using the Kubectl Set Image to perform a downgrade.
- Expectation: CNF should be compatible with multiple and different CNIs
What's tested: This installs temporary kind clusters and will test the CNF against both Calico and Cilium CNIs.
- Expectation: CNF should not use Kubernetes alpha APIs
What's tested: This checks if a CNF uses alpha or unstable versions of Kubernetes APIs
- Expectation: CNF image size is under 5 gigs
What's tested: Checks the size of the image used.
- Expectation: CNF starts up under one minute
What's tested: Checks how long the it takes for the CNF to pass a Readiness Probe and reach a ready/running state.
- Expectation: CNF container has one process type
What's tested: This verifies that there is only one process type within one container. This does not count against child processes. Example would be nginx or httpd could have a parent process and then 10 child processes but if both nginx and httpd were running, this test would fail.
- Expectation: CNFs accessible to other applications should be exposed via a Service.
What's tested: This tests and checks if the containers within a CNF have services exposed via a Kubernetes Service resource. Application access for microservices within a cluster should be exposed via a Service. Read more about K8s Service here.
- Expectation: Multiple microservices should not share the same database.
What's tested: This tests if multiple CNFs are using the same database.
- Expectation: Container images should use specialized init systems for containers.
What's tested: This tests if containers in pods have dumb-init, tini or s6-overlay as init processes.
- Expectation: All workload resources are successfully rescheduled onto other available node(s).
What's tested: A node is drained and workload resources rescheduled to another node, passing with a liveness and readiness check. This will skip when the cluster only has a single node.
- Expectation: Volume host path configurations should not be used.
What's tested: This tests if volume host paths are configured and used by the CNF.
- Expectation: Local storage should not be used or configured.
What's tested: This tests if local volumes are being used for the CNF.
- Expectation: Elastic persistent volumes should be configured for statefulness.
What's tested: This checks for elastic persistent volumes in use by the CNF.
- Expectation: Elastic volumes and or statefulsets should be used for databases to maintain a minimum resilience level in K8s clusters.
What's tested: This checks if elastic volumes and stateful sets are used for MySQL databases. If no MySQL database is found, the test is skipped.
- Expectation: The CNF should continue to function when network latency occurs
What's tested: This experiment causes network degradation without the pod being marked unhealthy/unworthy of traffic by kube-proxy (unless you have a liveness probe of sorts that measures latency and restarts/crashes the container). The idea of this experiment is to simulate issues within your pod network OR microservice communication across services in different availability zones/regions etc.
The applications may stall or get corrupted while they wait endlessly for a packet. The experiment limits the impact (blast radius) to only the traffic you want to test by specifying IP addresses or application information. This experiment will help to improve the resilience of your services over time.
- Expectation: The CNF should continue to function when disk fill occurs and pods should not be evicted to another node.
What's tested: This experiment stresses the disk with continuous and heavy IO to cause degradation in the shared disk. This experiment also reduces the amount of scratch space available on a node which can lead to a lack of space for newer containers to get scheduled. This can cause (Kubernetes gives up by applying an "eviction" taint like "disk-pressure") a wholesale movement of all pods to other nodes.
- Expectation: The CNF should continue to function when pod delete occurs
What's tested: This experiment helps to simulate such a scenario with forced/graceful pod failure on specific or random replicas of an application resource and checks the deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.
- Expectation: The CNF should continue to function when pod memory hog occurs
What's tested: The pod-memory hog experiment launches a stress process within the target container - which can cause either the primary process in the container to be resource constrained in cases where the limits are enforced OR eat up available system memory on the node in cases where the limits are not specified.
- Expectation: The CNF should continue to function when pod io stress occurs
What's tested: The pod-io stress experiment the disk with continuous and heavy IO to cause degradation in reads/writes by other microservices that use this shared disk.
- Expectation: The CNF should be resilient to a lossy/flaky network and should continue to provide some level of availability.
What's tested: The pod-network corruption experiment injects packet corruption on the CNF by starting a traffic control (tc) process with netem rules to add egress packet corruption.
- Expectation: The CNF should continue to function and be resilient to a duplicate network.
What's tested: The pod-network duplication experiment injects network duplication into the CNF by starting a traffic control (tc) process with netem rules to add egress delays.
- ✔️ Added to CNF Test Suite in release v0.26.0
- Expectation: That the CNF dosen't crash is resilient to DNS resolution failures.
What's tested: The pod-dns error experiment injects chaos to disrupt DNS resolution in kubernetes pods and causes loss of access to services by blocking DNS resolution of hostnames/domains.
- Expectation: The Helm chart should have a liveness probe configured.
What's tested: This test scans all of the CNFs workload resources and check if a Liveness Probe has been configuered for each container.
- Expectation: The Helm chart should have a readiness probe configured.
What's tested: This test scans all of the CNFs workload resources and check if a Readiness Probe has been configuered for each container.
- Expectation: Resource output logs should be sent to STDOUT/STDERR
What's tested: This checks and verifies that STDOUT/STDERR logging is configured for the CNF.
- Expectation: The CNF is configured and sending metrics to a Prometheus server.
What's tested: Tests for the presence of Prometheus and if the CNF configured to sent metrics to the prometheus server.
- Expectation: Fluentd or FluentBit is installed and capturing logs for the CNF.
What's tested: Checks for presence of a Unified Logging Layer and if the CNFs logs are being captured by the Unified Logging Layer. fluentd and fluentbit are currently supported.
- Expectation: CNF should emit OpenMetrics compatible traffic.
What's tested: Checks if the CNFs metrics are OpenMetrics compliant.
- Expectation: The CNF is sending traces to Jaeger.
What's tested: Checks if Jaeger is installed and the CNF is configured to send traces to the Jaeger Server.
- ✔️ Added to CNF Test Suite in release v0.27.0
- Expectation: Container runtime sockets should not be mounted as volumes
What's tested This test checks all of the CNFs containers and looks to see if any of them have access a container runtime socket from the host.
- Expectation: Containers should not run in privileged mode
What's tested: Checks if any containers are running in privileged mode (using Kubescape)
- ✔️ Added to CNF Test Suite in release v0.27.0
- Expectation: A CNF should not run services with external IPs
What's tested: Checks if the CNF has services with external IPs configured
- Expectation: A CNF should not have any 'seLinuxOptions' configured that allow privilege escalation.
What's tested: Checks if the CNF has escalatory seLinuxOptions configured.
- Expectation: The CNF should only have "safe" sysctls mechanisms configured, that are isolated from other Pods.
What's tested: Checks the CNF for usage of non-namespaced sysctls mechanisms that can affect the entire host.
- Expectation: Containers should not allow privilege escalation
What's tested: Check that the allowPrivilegeEscalation field in the securityContext of each container is set to false.
- Expectation: No vulnerable K8s version being used in conjunction with the subPath feature.
What's tested: This test checks for vulnerable K8s versions and the actual usage of the subPath feature for all Pods in the CNF.
- Exepectation: Application credentials should not be found in the CNFs configuration files
What's tested: Checks the CNF for sensitive information in environment variables, by using list of known sensitive key names. Also checks for configmaps with sensitive information.
- Expectation: The CNF should not have access to the host systems network.
What's tested: Checks if there is a host network attached to any of the Pods in the CNF.
- Expectation: The automatic mapping of service account tokens should be disabled.
What's tested: Check if the CNF is using service accounts that are automatically mapped.
- Expectation: Ingress and Egress traffic should be blocked on Pods.
What's tested: Checks each Pod in the CNF for a defined ingress and egress policy.
- Expectation: Containers should not have insecure capabilities enabled.
What's tested: Checks the CNF for any usage of insecure capabilities using the following deny list
- Expectation: Containers should run with non-root user and allowPrivilegeEscalation should be set to false.
What's tested: Checks if the CNF has runAsUser and runAsGroup set to a user id greater than 999. Also checks that the allowPrivilegeEscalation field is set to false for the CNF. Read more at ARMO-C0013
- Expectation: Containers should not have hostPID and hostIPC privileges
What's tested: Checks if containers are running with hostPID or hostIPC privileges. Read more at ARMO-C0038
- Expectation: Security services are being used to harden application.
What's tested: Check if there are AppArmor, Seccomp, SELinux or Capabilities defined in the securityContext of the CNF's containers and pods. Read more at ARMO-C0055
- Expectation: Containers should have resource limits defined
What's tested: Check if there is a ‘limits’ field defined for the CNF. Check for each limitrange/resourcequota if there is a max/hard field defined, respectively. Read more at ARMO-C0009.
- Expectation: Containers should use an immutable file system when possible.
What's tested: Checks whether the readOnlyRootFilesystem field in the SecurityContext is set to true. Read more at ARMO-C0017
- Expectation: Containers should not have hostPath mounts
What's tested: Checks the CNF's POD spec for any hostPath volumes, if found it checks the volume for the field mount.readOnly == false (or if it doesn’t exist). Read more at ARMO-C0045
- Expectation: Resources should not be deployed in the default namespace.
What's tested: Checks if any of the CNF's resources are deployed in the default namespace.
- Expectation: The CNF should use an immutable tag that maps to a symantic version of the application.
What's tested: Checks if the CNF is using a 'latest' tag instead of a semantic version.
- ✔️ Added to CNF Test Suite in release v0.27.0
- Expectation: Checks if pods are using the 'app.kubernetes.io/name' label
What's tested: Checks if the CNF validates that the label app.kubernetes.io/name is specified with some value.
- Expectation: The CNF should use an immutable tag that maps to a symantic version of the application.
What's tested: Checks if the CNF is using a 'latest' tag instead of a semantic version using OPA Gatekeeper.
- Expectation: The nodePort configuration field is not found in any of the CNF's services.
What's tested: Checks the CNF for any associated K8s Services that configured to expose the CNF by using a nodePort.
- Expectation: The hostPort configuration field is not found in any of the defined containers.
What's tested: Checks the CNF's workload resources for any containers using the hostPort configuration field to expose the application.
- Expectation: That no hardcoded IP addresses or subnet masks are found in the Kubernetes workload resources for the CNF.
What's tested: The hardcoded ip address test will scan all of the CNF's workload resources and check for any static, hardcoded ip addresses being used in the configuration.
- Expectation: The CNF is using K8s secrets for the management of sensitive data.
What's tested: The secrets used test will scan all the Kubernetes workload resources to see if K8s secrets are being used.
- Expectation: Immutable configmaps are being used for non-mutable data.
What's tested: The immutable configmap test will scan the CNF's workload resources and see if immutable configmaps are being used.
- Expectation: 5g core should continue to function during various CNF tests.
What's tested: Checks the pfcp heartbeat between the smf and upf to make sure it remains close to baseline.
- Expectation: 5g core should use suci concealment.
What's tested: Checks to see if the 5g core supports suci concealment.
- Expectation: An ORAN RIC should use an e2 connection.
What's tested: Checks if a RIC uses a oran compatible e2 connection.
- Expectation: The K8s cluster passes the K8s conformance tests
What's tested: Check if your platform passes the K8s conformance test. See https://github.com/cncf/k8s-conformance for details on what is tested.
- Expectation: The cluster has Cluster API enabled which manages at least one Node.
What's tested: Checks the platforms Kubernetes Nodes to see if they were instansiated by ClusterAPI.
- Expectation: All worker nodes are using an OCI compliant run-time.
What's tested: Inspects all worker nodes and checks if the run-time being used for scheduling is OCI compliant.
(PoC) Worker reboot recovery
- Expectation: Pods should reschedule after a node failure.
- WARNING: this is a destructive test and will reboot your host node! Do not run this unless you have completely separate cluster, e.g. development or test cluster.
What's tested: Run node failure test which forces a reboot of the Node ("host system"). The Pods on that node should be rescheduled to a new Node.
- Expectation: The cluster admin role should not be bound to a Pod
What's tested: Check which subjects have cluster-admin RBAC permissions – either by being bound to the cluster-admin clusterrole, or by having equivalent high privileges.
- Expectation: That the the k8s control plane is secure and not hosted on an insecure port
What's tested: Checks if the insecure-port flag is set for the K8s API Server.
- Expectation: The K8s dashboard should not exposed to the public internet when the software version is older than v2.0.1
What's tested: Checks if Kubernetes dashboard exists and exposed externally as a service (nodeport/loadbalancer) and if the software version of the container image is older than v2.0.1.
- Added in release v0.27.0
- Expectation: The platform should be using Helm v3+ without Tiller.
What's tested: Checks if a Helm v2 / Tiller image is deployed and used on the platform.