Workload Tests
Service providers have historically had issues with the installability of vendor network functions. This category tests the installabilityand lifecycle management (the create, update, and delete of network applications) against widely used K8s installation solutions such as Helm.
To test the increasing and decreasing of capacity: increase_decrease_capacity
A CNF should be able to increase and decrease its capacity without running into errors.
Test if the Helm chart is published: helm_chart_published
If a helm chart is published, it is significantly easier to install for the end user.
The management and versioning of the helm chart are handled by the helm registry and client tools rather than manually as directly referencing the helm chart source.
Test if the Helm chart is valid: helm_chart_valid
A chart should pass the lint specification
Test if the Helm deploys: helm_deploy
A helm chart should be deployable to a cluster
To check if a CNF version can be rolled back: rollback
K8s best practice is to allow K8s to manage the rolling back of an application resource instead of having operators manually rolling back the resource by using something like blue/green deploys.
To test if the CNF can perform a rolling update: rolling_update
See rolling downgrade
To check if a CNF version can be downgraded through a rolling_version_change: rolling_version_change
See rolling downgrade
To check if a CNF version can be downgraded through a rolling_downgrade: rolling_downgrade
(update, version change, downgrade): K8s best practice for version/installation management (lifecycle management) of applications is to have K8s track the version of the manifest information for the resource (deployment, pod, etc) internally. Whenever a rollback is needed the resource will have the exact manifest information that was tied to the application when it was deployed. This adheres the principles driving immutable infrastructure and declarative specifications.
To check if the CNF is compatible with different CNIs: cni_compatibility
A CNF should be runnable by any CNI that adheres to the CNI specification
[POC] To check if a CNF uses Kubernetes alpha APIs 'alpha_k8s_apis': alpha_k8s_apis
If a CNF uses alpha or undocumented APIs, the CNF is tightly coupled to an unstable platform
Good microservice practices promote agility which means less time will occur between deployments. One benefit of more agility is it allows for different organizations and teams to deploy at the rate of change that they build out features, instead of deploying in lock step with other teams. This is very important when it comes to changes that are time sensitive like security patches.
To check if the CNF has a reasonable image size: reasonable_image_size
A CNF with a large image size of 5 gig or more tends to indicate a monolithic application
To check if the CNF have a reasonable startup time: reasonable_startup_time
A CNF that starts up with a time (adjusted for server resources) that is approaching a minute is indicative of a monolithic application. The liveness probe's initialDelaySeconds and failureThreshhold determine the startup time and retry amount of the CNF. Specifically, if the initiaDelay is too long it is indicative of a monolithic application. If the failureThreshold is too high it is indicative of a CNF or a component of the CNF that has too many intermittent failures.
To check if the CNF has multiple process types within one container: single_process_type
A microservice should have only one process (or set of parent/child processes) that is managed by a non home grown supervisor or orchestrator. The microservice should not spawn other process types (e.g. executables) as a way to contributeto the workload but rather should interact with other processes through a microservice API.
To check if the CNF exposes any of its containers as a service 'service_discovery': service_discovery
A K8s microservice should expose it's API though a K8s service resource. K8s services handle service discovery and load balancing for the cluster.
To check if the CNF uses a shared database: shared_database
A K8s microservice should not share a database with another K8s database because it forces the two services to upgrade in lock step
To check if the CNF uses container images with specialized init systems: specialized_init_systems
There are proper init systems and sophisticated supervisors that can be run inside of a container. Both of these systems properly reap and pass signals. Sophisticated supervisors are considered overkill because they take up too many resources and are sometimes too complicated. Some examples of sophisticated supervisors are: supervisord, monit, and runit. Proper init systems are smaller than sophisticated supervisors and therefore suitable for containers. Some of the proper container init systems are tini, dumb-init, and s6-overlay.
If infrastructure is immutable, it is easily reproduced, consistent, disposable, will have a repeatable deployment process, and will not have configuration or artifacts that are modifiable in place. This ensures that all configuration is stateless. Any data that is persistent should be managed by K8s statefulsets.
Test if the CNF crashes when node drain occurs: node_drain
No CNF should fail because of stateful configuration. A CNF should function properly if it is rescheduled on other nodes. This test will remove resources which are running on a target node and reschedule them on the another node.
To test if the CNF uses a volume host path: volume_hostpath_not_found
When a cnf uses a volume host path or local storage it makes the application tightly coupled to the node that it is on.
To test if the CNF uses local storage: no_local_volume_configuration
A CNF should refrain from using the local storage class
To test if the CNF uses elastic volumes: elastic_volumes
A cnf that uses elastic volumes can be rescheduled to other nodes by the orchestrator easily
To test if the CNF uses a database with either statefulsets, elastic volumes, or both: database_persistence
When a traditional database such as mysql is configured to use statefulsets, it allows the database to use a persistent identifier that it maintains across any rescheduling. Persistent Pod identifiers make it easier to match existing volumes to the new Pods that have been rescheduled. https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
Cloud native systems promote resilience by putting a high priority on testing individual components (chaos testing) as they are running (possibly in production).Reliability in traditional telecommunications is handled differently than in Cloud Native systems. Cloud native systems try to address reliability (MTBF) by having the subcomponents have higher availability through higher serviceability (MTTR) and redundancy. For example, having ten redundant subcomponents where seven components are available and three have failed will produce a top level component that is more reliable (MTBF) than a single component that "never fails" in the cloud native world.
Test if the CNF crashes when network latency occurs: pod_network_latency
Network latency can have a significant impact on the overall performance of the application. Network outages that result from low latency can cause a range of failures for applications and can severely impact user/customers with downtime. This chaos experiment allows you to see the impact of latency traffic on the CNF.
Test if the CNF crashes when disk fill occurs: disk_fill
Disk Pressure is a scenario we find in Kubernetes applications that can result in the eviction of the application replica and impact its delivery. Such scenarios can still occur despite whatever availability aids K8s provides. These problems are generally referred to as "Noisy Neighbour" problems.
Test if the CNF crashes when pod delete occurs: pod_delete
In a distributed system like Kubernetes, application replicas may not be sufficient to manage the traffic (indicated by SLIs) when some replicas are unavailable due to any failure (can be system or application). The application needs to meet the SLO (service level objectives) for this. It's imperative that the application has defenses against this sort of failure to ensure that the application always has a minimum number of available replicas.
Test if the CNF crashes when pod memory hog occurs: pod_memory_hog
If the memory policies for a CNF are not set and granular, containers on the node can be killed based on their oom_score and the QoS class a given pod belongs to (best-effort ones are first to be targeted). This eval is extended to all pods running on the node, thereby causing a bigger blast radius.
Test if the CNF crashes when pod io stress occurs: pod_io_stress
Stressing the disk with continuous and heavy IO can cause degradation in reads/ writes by other microservices that use this shared disk. Scratch space can be used up on a node which leads to the lack of space for newer containers to get scheduled which causes a movement of all pods to other nodes. This test determines the limits of how a CNF uses its storage device.
Test if the CNF crashes when pod network corruption occurs: pod_network_corruption
A higher quality CNF should be resilient to a lossy/flaky network. This test injects packet corruption on the specified CNF's container by starting a traffic control (tc) process with netem rules to add egress packet corruption.
Test if the CNF crashes when pod network duplication occurs: pod_network_duplication
A higher quality CNF should be resilient to erroneously duplicated packets. This test injects network duplication on the specified container by starting a traffic control (tc) process with netem rules to add egress delays.
Test if the CNF crashes when DNS errors occur: pod_dns_errors
A CNF should be resilient to name resolution (DNS) disruptions within the kubernetes pod. This ensures that at least some application availability will be maintained if DNS resolution fails.
To test if there is a liveness entry in the Helm chart: liveness
A cloud native principle is that application developers understand their own resilience requirements better than operators[1]. This is exemplified in the Kubernetes best practice of pods declaring how they should be managed through the liveness and readiness entries in the pod's configuration.
[1] "No one knows more about what an application needs to run in a healthy state than the developer. For a long time, infrastructure administrators have tried to figure out what “healthy” means for applications they are responsible for running. Without knowledge of what actually makes an application healthy, their attempts to monitor and alert when applications are unhealthy are often fragile and incomplete. To increase the operability of cloud native applications, applications should expose a health check."" Garrison, Justin; Nova, Kris. Cloud Native Infrastructure: Patterns for Scalable Infrastructure and Applications in a Dynamic Environment . O'Reilly Media. Kindle Edition.
To test if there is a readiness entry in the Helm chart: readiness
A CNF should tell Kubernetes when it is ready to serve traffic.
In order to maintain, debug, and have insight into a production environment that is protected (versioned, kept in source control, and changed only by using a deployment pipeline), its infrastructure elements must have the property of being observable. This means these elements must externalize their internal states in some way that lends itself to metrics, tracing, and logging.
To check if logs are being sent to stdout/stderr (standard out, standard error) instead of a log file: log_output
By sending logs to standard out/standard error "logs will be treated like event streams" as recommended by 12 factor apps principles.
To check if prometheus is installed and configured for the cnf: prometheus_traffic
Recording metrics within a cloud native deployment is important because it gives the maintainer of a cluster of hundreds or thousands of services the ability to pinpoint small anomalies, such as those that will eventually cause a failure.
To check if logs and data are being routed through a Unified Logging Layer: routed_logs
A CNF should have logs managed by a unified logging layer It's considered a best-practice for CNFs to route logs and data through programs like fluentd to analyze and better understand data.
To check if OpenMetrics is being used and or compatible.: open_metrics
OpenMetrics is the de facto standard for transmitting cloud native metrics at scale, with support for both text representation and Protocol Buffers and brings it into an Internet Engineering Task Force (IETF) standard. A CNF should expose metrics that are OpenMetrics compatible
To check if tracing is being used with Jaeger.: tracing
A CNF should provide tracing that conforms to the open telemetry tracing specification
"Cloud native security is a [...] mutifaceted topic [...] with multiple, diverse components that need to be secured. The cloud platform, the underlying host operating system, the container runtime, the container orchestrator,and then the applications themselves each require specialist security attention" -- Chris Binne, Rory Mccune. Cloud Native Security. (Wiley, 2021)(pp. xix)*
To check if the cnf performs a CRI socket mount: container_sock_mounts
Container daemon socket bind mounts allows access to the container engine on the node. This access can be used for privilege escalation and to manage containers outside of Kubernetes, and hence should not be allowed..
To check if there are any privileged containers: privileged_containers
... docs describe Privileged mode as essentially enabling “…access to all devices on the host as well as [having the ability to] set some configuration in AppArmor or SElinux to allow the container nearly all the same access to the host as processes running outside containers on the host.” In other words, you should rarely, if ever, use this switch on your container command line. Binnie, Chris; McCune, Rory (2021-06-17T23:58:59). Cloud Native Security . Wiley. Kindle Edition.
To check if External IPs are used for services: external_ips
Service externalIPs can be used for a MITM attack (CVE-2020-8554). Restrict externalIPs or limit to a known set of addresses. See: kyverno/kyverno#1367
To check if any containers are running as a root user (checks the user outside the container that is running dockerd): non_root_user
Even with other security controls used within a Linux system running containers, such as namespaces that segregate access between pods in Kubernetes and OpenShift or containers within a runtime, it is highly advisable never to run a container as the root user." Binnie, Chris; McCune, Rory (2021-06-17T23:58:59). Cloud Native Security . Wiley. Kindle Edition.
To check if any containers allow for privilege escalation: privilege_escalation
When privilege escalation is enabled for a container, it will allow setuid binaries to change the effective user ID, allowing processes to turn on extra capabilities. In order to prevent illegitimate escalation by processes and restrict a processes to a NonRoot user mode, escalation must be disabled.
To check if an attacker can use a symlink for arbitrary host file system access (CVE-2021-25741): symlink_file_system
Due to CVE-2021-25741, subPath or subPathExpr volume mounts can be used to gain unauthorised access to files and directories anywhere on the host filesystem. In order to follow a best-practice security standard and prevent unauthorised data access, there should be no active CVEs affecting either the container or underlying platform.
To check if selinux has been configured properly: selinux_options
If SELinux options is configured improperly it can be used to escalate privileges and should not be allowed.
To check if any pods in the CNF use sysctls with restricted values: sysctls
Sysctls can disable security mechanisms or affect all containers on a host, and should be disallowed except for an allowed "safe" subset. A sysctl is considered safe if it is namespaced in the container or the Pod, and it is isolated from other Pods or processes on the same Node. This test ensures that only those "safe" subsets are specified in a Pod.
To check if there are applications credentials in configuration files: application_credentials
Developers store secrets in the Kubernetes configuration files, such as environment variables in the pod configuration. Such behavior is commonly seen in clusters that are monitored by Azure Security Center. Attackers who have access to those configurations, by querying the API server or by accessing those files on the developer’s endpoint, can steal the stored secrets and use them.
To check if there is a host network attached to a pod: host_network
When a container has the hostNetwork feature turned on, the container has direct access to the underlying hostNetwork. Hackers frequently exploit this feature to facilitate a container breakout and gain access to the underlying host network, data and other integral resources.
To check if there is automatic mapping of service accounts: service_account_mapping
When a pod gets created and a service account wasn't specified, then the default service account will be used. Service accounts assigned in this way can unintentionally give third-party applications root access to the K8s APIs and other applicaton services. In order to follow a zero-trust / fine-grained security methodology, this functionality will need to be explicitly disabled by using the automountServiceAccountToken: false flag. In addition, if RBAC is not enabled, the SA has unlimited permissions in the cluster.
To check if there is an ingress and egress policy defined.: ingress_egress_blocked
By default, no network policies are applied to Pods or namespaces, resulting in unrestricted ingress and egress traffic within the Pod network. In order to prevent lateral movement or escalation on a compromised cluster, administrators should implement a default policy to deny all ingress and egress traffic. This will ensure that all Pods are isolated by default and further policies could then be used to specifically relax these restrictions on a case-by-case basis.
To check for insecure capabilities: insecure_capabilities
Giving insecure and unnecessary capabilities for a container can increase the impact of a container compromise.
To check if containers are running with non-root user with non-root membership: non_root_containers
Container engines allow containers to run applications as a non-root user with non-root group membership. Typically, this non-default setting is configured when the container image is built. . Alternatively, Kubernetes can load containers into a Pod with SecurityContext:runAsUser specifying a non-zero user. While the runAsUser directive effectively forces non-root execution at deployment, NSA and CISA encourage developers to build container applications to execute as a non-root user. Having non-root execution integrated at build time provides better assurance that applications will function correctly without root privileges.
To check if containers are running with hostPID or hostIPC privileges: host_pid_ipc_privileges
Containers should be isolated from the host machine as much as possible. The hostPID and hostIPC fields in deployment yaml may allow cross-container influence and may expose the host itself to potentially malicious or destructive actions. This control identifies all PODs using hostPID or hostIPC privileges.
To check if security services are being used to harden containers: linux_hardening
In order to reduce the attack surface, it is recommend, when it is possible, to harden your application using security services such as SELinux®, AppArmor®, and seccomp. Starting from Kubernetes version 1.22, SELinux is enabled by default.
To check if containers have resource limits defined: resource_policies
CPU and memory resources should have a limit set for every container or a namespace to prevent resource exhaustion. This control identifies all the Pods without resource limit definitions by checking thier yaml definition file as well as their namespace LimitRange objects. It is also recommended to use ResourceQuota object to restrict overall namespace resources, but this is not verified by this control.
To check if containers have immutable file systems: immutable_file_systems
Mutable container filesystem can be abused to gain malicious code and data injection into containers. By default, containers are permitted unrestricted execution within their own context. An attacker who has access to a container, can create files and download scripts as they wish, and modify the underlying application running on the container.
To check if containers have hostPath mounts (check: is this a duplicate of state test - ./cnf-testsuite volume_hostpath_not_found): hostpath_mounts
hostPath mount can be used by attackers to get access to the underlying host and thus break from the container to the host. (See “3: Writable hostPath mount” for details).
Declarative APIs for an immutable infrastructure are anything that configures the infrastructure element. This declaration can come in the form of a YAML file or a script, as long as the configuration designates the desired outcome, not how to achieve said outcome. "Because it describes the state of the world, declarative configuration does not have to be executed to be understood. Its impact is concretely declared. Since the effects of declarative configuration can be understood before they are executed, declarative configuration is far less error-prone. " --Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 183-186). Kindle Edition
To check if a CNF is using the default namespace: default_namespace
*Namespaces provide a way to segment and isolate cluster resources across multiple applications and users. As a best practice, workloads should be isolated with Namespaces and not use the default namespace.
To test if mutable tags being used for image versioning(Using Kyverno): latest_tag: latest_tag
"You should avoid using the :latest tag when deploying containers in production as it is harder to track which version of the image is running and more difficult to roll back properly."
To test if the recommended labels are being used to describe resources: required_labels
Defining and using labels help identify semantic attributes of your application or Deployment. A common set of labels allows tools to work collaboratively, while describing objects in a common manner that all tools can understand. You should use recommended labels to describe applications in a way that can be queried.
To test if there are versioned tags on all images (using OPA Gatekeeper): versioned_tag
"You should avoid using the :latest tag when deploying containers in production as it is harder to track which version of the image is running and more difficult to roll back properly."
To test if there are node ports used in the service configuration: nodeport_not_used
Using node ports ties the CNF to a specific node and therefore makes the CNF less portable and scalable
To test if there are host ports used in the service configuration: hostport_not_used
Using host ports ties the CNF to a specific node and therefore makes the CNF less portable and scalable
To test if there are any (non-declarative) hardcoded IP addresses or subnet masks in the K8s runtime configuration: hardcoded_ip_addresses_in_k8s_runtime_configuration
Using a hard coded IP in a CNF's configuration designates how (imperative) a CNF should achieve a goal, not what (declarative) goal the CNF should achieve
To check if a CNF uses K8s secrets: secrets_used
If a CNF uses kubernetes K8s secrets instead of unencrypted environment variables or configmaps, there is less risk of the Secret (and its data) being exposed during the workflow of creating, viewing, and editing Pods
To check if a CNF version uses immutable configmaps: immutable_configmap
"For clusters that extensively use ConfigMaps (at least tens of thousands of unique ConfigMap to Pod mounts), preventing changes to their data has the following advantages:
- protects you from accidental (or unwanted) updates that could cause applications outages
- improves performance of your cluster by significantly reducing load on kube-apiserver, by closing watches for ConfigMaps marked as immutable."
A 5g core is an important part of the service provider's telecommuncations offering. A cloud native 5g architecture uses immutable infrastructure, declarative configuration, and microservices when creating and hosting 5g cloud native network functions.
To check if the 5g core is resistant to chaos: smf_upf_core_validator
*A 5g core's SMF and UPF CNFs have a hearbeat, implemented use the PFCP protocol standard, which measures if the connection between the two CNFs is active. After measure a baseline of the heartbeat a comparison between the baseline and the performance of the heartbeat while running test functions will expose the cloud native resilience of the cloud native 5g core.
To check if the 5g core is using 5g authentication: suci_enabled
*In order to protect identifying information from being sent over the network as clear text, 5g cloud native cores should implement SUPI and SUCI concealment
A cloud native radio access network's (RAN) cloud native functions should use immutable infrastructure, declarative configuration, and microservices. ORAN cloud native functions should adhere to cloud native principles while also complying with the ORAN alliance's standards.
To check if an ORAN compliant RAN is using the e2 3gpp standard: oran_e2_connection
*A near real-time RAN intelligent controler (RIC) uses the E2 standard as an open, interoperable, interface to connect to RAN-optimizated applications, onboarded as xApps. The xApps use platform services available in the near-RT RIC to communicate with the downstream network functions through the E2 interface.
To check if the plateform passes K8s Conformance tests: k8s-conformance
- A Vendor's Kubernetes Platform should pass Kubernetes Conformance. This ensures that the platform offering meets the same required APIs, features & interoperability expectations as in open source community versions of K8s. Applications that can operate on a Certified Kubernetes should be cross-compatible with any other Certified Kubernetes platform.
To check if the plateform is being managed by ClusterAPI: clusterapi-enabled
- A Kubernetes Platform should leverage Cluster API to ensure that best-practices are followed for both bootstrapping & cluster lifecycle management. Kubernetes is a complex system that relies on several components being configured correctly, maintaining an in-house lifecycle management system for kubernetes is unlikey to meet best practice guideline unless significant resources are deticated to it.
To check if the plateform is using an OCI compliant runtime: oci-compliant
*The OCI Initiative was created to ensure that runtimes conform to both the runtime-spec and image-spec. These two specifications outline how a “filesystem bundle” is unpacked on disk and that the image itself contains sufficient information to launch the application on the target platform. As a best practice, your platform must use an OCI compliant runtime, this ensures that the runtime used is cross-compatible and supports interoperability with other runtimes. This means that workloads can be freely moved to other runtimes and prevents vendor lock in.
To check if workloads are rescheduled on node failure: worker-reboot-recovery
*Cloud native systems should be self-healing. To follow cloud-native best practices your platform should be resiliant and reschedule all workloads when such node failures occur.
To check if the plateform has a default Cluster admin role: cluster-admin
*Role-based access control (RBAC) is a key security feature in Kubernetes. RBAC can restrict the allowed actions of the various identities in the cluster. Cluster-admin is a built-in high privileged role in Kubernetes. Attackers who have permissions to create bindings and cluster-bindings in the cluster can create a binding to the cluster-admin ClusterRole or to other high privileges roles. As a best practice, a principle of least privilege should be followed and cluster-admin privilege should only be used on an as-needed basis.
Check if the plateform is using insecure ports for the API server: Control_plane_hardening
*The control plane is the core of Kubernetes and gives users the ability to view containers, schedule new Pods, read Secrets, and execute commands in the cluster. Therefore, it should be protected. It is recommended to avoid control plane exposure to the Internet or to an untrusted network and require TLS encryption.
Check if the Dashboard is exposed externally: Dashboard exposed
- If Kubernetes dashboard is exposed externally in Dashboard versions before 2.01, it will allow unauthenticated remote management of the cluster. It's best practice to not expose the K8s Dashboard or any management planes if they're unsecured.
Check if Tiller is being used on the plaform: Tiller images
*Tiller, found in Helm v2, has known security challenges. It requires administrative privileges and acts as a shared resource accessible to any authenticated user. Tiller can lead to privilege escalation as restricted users can impact other users. It is recommend to use Helm v3+ which does not contain Tiller for these reasons