Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add perf testing #86

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add perf testing #86

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions container/ccnp-example/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ RUN chown $USER:$GROUP /run/ccnp

COPY test ./
COPY sdk/python3/example/py_sdk_example.py ./
COPY test/perf/py_perf.py ./
COPY --from=python-builder cc-trusted-api/common/python/dist/cctrusted_base*.whl ./
COPY --from=python-builder ccnp-sdk/dist/ccnp*.whl ./

Expand Down
26 changes: 26 additions & 0 deletions deployment/kubernetes/manifests/ccnp-example-perf-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: ccnp-perf-example-PLACEHOLDER
namespace: perf
spec:
selector:
matchLabels:
app: ccnp-perf-example-PLACEHOLDER
template:
metadata:
labels:
app: ccnp-perf-example-PLACEHOLDER
annotations:
"ccnp.cc-api/require": "true"
spec:
containers:
- name: ccnp-perf-example-PLACEHOLDER
image: "docker.io/library/ccnp-example:latest"
imagePullPolicy: Always
resources:
limits:
memory: "128Mi"
cpu: "100m"
nodeSelector:
"feature.node.kubernetes.io/cpu-security.tdx.protected": "true"
105 changes: 26 additions & 79 deletions test/perf/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,102 +1,49 @@
# Performance Test
# CCNP Performance Test

We have these KPIs for performance test.

| KPI​ | HIB/LIB​ | Unit​ | Comment​ |
| ------------------------------------------- | -------- | ----- | ----------------------------------------------------- |
| CCNP service get measurement throughput​ | HIB​ | ops​ | Service Throughput​ |
| CCNP service get measurement response time​ | LIB​ | ms​ | Service Response time​ |
| CCNP service get eventlog throughput​ | HIB​ | ops​ | Service Throughput​ |
| CCNP service get eventlog response time​ | LIB​ | ms​ | Service Response time​ |
| CCNP service get quote throughput​ | HIB​ | ops​ | Service Throughput​ |
| CCNP service get quote response time​ | LIB​ | ms​ | Service Response time​ |
| CCNP initialization time​ | LIB​ | s​ | CCNP device plugin, DaemonSet and service readiness.​ |

*Note: we use the CCNP SDK to access the CCNP service because it's convenient to prepare the request data (e.g. container ID, etc.)​
CCNP performance tests focus on the latency of calling CCNP SDK key APIs: `get_cc_eventlog`, `get_cc_report` and `get_cc_measurement`.
It will simulate requests from multiple pods in parallel and calculate average time of all the requests.

Below are the steps for you to build and run the performance test.

## Prerequisites

To run the test, you need a K8S cluster with CCNP enabled (CCNP Device Plugin and CCNP Service deployed and ready).

## Build

```bash
# Make sure you are on the repo's top dir
cd <the-dir-of-confidential-cloud-native-primitives>
Please make sure you have CCNP deployed in a K8S cluster, and ccnp-example image has been built.
Please refer to [here](../../deployment/README.md) for image building and CCNP deployment.

# Run doker build
docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --build-arg no_proxy=$no_proxy -t ccnp-perf:latest -f container/ccnp-perf/Dockerfile .
## Run Tests

# View build result
docker image ls | grep ccnp-perf
### Deploy pods for performance testing

# Save the docker image for later use
docker save ccnp-perf:latest > ccnp-perf_latest.tar
```bash
# Deploy ccnp-example pods
$ sudo ./deploy-perf.sh -r <ccnp-example image registry> -g <ccnp-example image tag> -n <number of pods>
```

## Deploy
### Run Tests

The script will run tests in parallel. The log will be saved in files with prefix `perf_output` under current directory.

```bash
# Load the docker image for K8S using containerd.
# You need to run this on the node where you want to deploy the ccnp-perf test
ctr -n=k8s.io image import ccnp-perf_latest.tar
# Test for get event log
$ sudo ./perf-para.sh -n <number of pods> -e

# Make sure you are on the repo's top dir
cd <the-dir-of-confidential-cloud-native-primitives>
# Test for get measurement
$ sudo ./perf-para.sh -n <number of pods> -m

# Deploy ccnp-perf test
kubectl apply -f deployment/kubernetes/manifests/ccnp-perf-deployment.yaml
# Test for get quote
$ sudo ./perf-para.sh -n <number of pods> -r
```

## Test
Run below script to calculate average time of a request.

```bash
# Get the pod name of ccnp-perf
kubectl get pod | grep ccnp-perf

# Run all perf test on the specified pod name got from above command
kubectl exec -ti <ccnp-perf-pod-name> -- python3 -m pytest --log-cli-level=INFO --verbose ccnp_perf.py
$ sudo ./average.sh -f perf_output_quote
```

Sample test output looks like this:
### Clear

Run below command to delete the pods for performance testing.

```bash
root@ccnp-perf-0:~/ccnp/confidential-cloud-native-primitives# kubectl exec -ti ccnp-perf-7f8798bf85-8s6zg -- python3 -m pytest --log-cli-level=INFO --verbose
ccnp_perf.py
==================================================================== test session starts ====================================================================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0 -- /usr/local/bin/python3
cachedir: .pytest_cache
rootdir: /run/ccnp
collected 7 items

ccnp_perf.py::test_svc_get_cc_measurement_throughput
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 70.75 ops (operations per second)
PASSED [ 14%]
ccnp_perf.py::test_svc_get_cc_measurement_response
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 25.89662575 ms (milliseconds)
PASSED [ 28%]
ccnp_perf.py::test_svc_get_cc_eventlog_throughput
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 57.8 ops (operations per second)
PASSED [ 42%]
ccnp_perf.py::test_svc_get_cc_eventlog_response
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 76.130223 ms (milliseconds)
PASSED [ 57%]
ccnp_perf.py::test_svc_get_cc_report_throughput
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 54.9 ops (operations per second)
PASSED [ 71%]
ccnp_perf.py::test_svc_get_cc_report_response
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 29.38618825 ms (milliseconds)
PASSED [ 85%]
ccnp_perf.py::test_ccnp_init PASSED [100%]

=============================================================== 7 passed in 66.95s (0:01:06) ================================================================
root@ccnp-perf-0:~/ccnp/confidential-cloud-native-primitives#
```
$ sudo ./deploy-per.sh -n <number of pods> -d
```
73 changes: 73 additions & 0 deletions test/perf/average.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/bin/bash
# Input filename keyword and keyword for grep. Find the files in current dir, search the numbers needed, calculate sum and average

grep_keyword="total command "

usage() {
cat << EOM
usage: $(basename "$0") [OPTION]...
-f Input file name keyword
-k Keyword to be searched in the input files
EOM
exit 1
}

process_args() {
while getopts ":f:k:h" option; do
case "${option}" in
f) filename_keyword=${OPTARG};;
k) grep_keyword=${OPTARG};;
h) usage;;
*) echo "Invalid option: -${OPTARG}" >&2
usage
;;
esac
done
}


# Find all files containing the filename keyword
files=$(find . -type f -name "*${filename_keyword}*"| cut -d/ -f2)

total_sum=0
total_count=0
total_aver=0

# Loop through the found files
for file in $files; do
sum=0
count=0

tag=$(echo "$file" | cut -d- -f2)
tmpfile="tmp_output_"$tag
touch "$tmpfile"

# Grep the keyword and extract the number. The number is the 2nd last word in the line.
grep -i "${grep_keyword}" "$file" | awk '{print $(NF-1)}' > "$tmpfile"

while IFS= read -r line; do
# Add the number to sum
sum=$(echo "$sum + $line" | bc)
# Increment the count
count=$((count + 1))
done < "$tmpfile"

# Calculate the average
if [ $count -ne 0 ]; then
average=$(echo "scale=6; $sum / $count" | bc)
echo "Output file: $file"
echo "Number file: $tmpfile"
echo "Count: $count"
echo "Sum: $sum"
echo "Average: $average"
else
echo "The file is empty"
fi
total_sum=$(echo "$sum + $total_sum" | bc)
total_count=$(echo "$count + $total_count" | bc)
done

echo "Total Sum: $total_sum"
echo "Total count: $total_count"
total_aver=$(echo "scale=6; $total_sum / $total_count" | bc)
echo "Total average: $total_aver"
102 changes: 102 additions & 0 deletions test/perf/ccnp-pytest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Performance Test

We have these KPIs for performance test.

| KPI​ | HIB/LIB​ | Unit​ | Comment​ |
| ------------------------------------------- | -------- | ----- | ----------------------------------------------------- |
| CCNP service get measurement throughput​ | HIB​ | ops​ | Service Throughput​ |
| CCNP service get measurement response time​ | LIB​ | ms​ | Service Response time​ |
| CCNP service get eventlog throughput​ | HIB​ | ops​ | Service Throughput​ |
| CCNP service get eventlog response time​ | LIB​ | ms​ | Service Response time​ |
| CCNP service get quote throughput​ | HIB​ | ops​ | Service Throughput​ |
| CCNP service get quote response time​ | LIB​ | ms​ | Service Response time​ |
| CCNP initialization time​ | LIB​ | s​ | CCNP device plugin, DaemonSet and service readiness.​ |

*Note: we use the CCNP SDK to access the CCNP service because it's convenient to prepare the request data (e.g. container ID, etc.)​

Below are the steps for you to build and run the performance test.

## Prerequisites

To run the test, you need a K8S cluster with CCNP enabled (CCNP Device Plugin and CCNP Service deployed and ready).

## Build

```bash
# Make sure you are on the repo's top dir
cd <the-dir-of-confidential-cloud-native-primitives>

# Run doker build
docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --build-arg no_proxy=$no_proxy -t ccnp-perf:latest -f container/ccnp-perf/Dockerfile .

# View build result
docker image ls | grep ccnp-perf

# Save the docker image for later use
docker save ccnp-perf:latest > ccnp-perf_latest.tar
```

## Deploy

```bash
# Load the docker image for K8S using containerd.
# You need to run this on the node where you want to deploy the ccnp-perf test
ctr -n=k8s.io image import ccnp-perf_latest.tar

# Make sure you are on the repo's top dir
cd <the-dir-of-confidential-cloud-native-primitives>

# Deploy ccnp-perf test
kubectl apply -f deployment/kubernetes/manifests/ccnp-perf-deployment.yaml
```

## Test

```bash
# Get the pod name of ccnp-perf
kubectl get pod | grep ccnp-perf

# Run all perf test on the specified pod name got from above command
kubectl exec -ti <ccnp-perf-pod-name> -- python3 -m pytest --log-cli-level=INFO --verbose ccnp_perf.py
```

Sample test output looks like this:

```bash
root@ccnp-perf-0:~/ccnp/confidential-cloud-native-primitives# kubectl exec -ti ccnp-perf-7f8798bf85-8s6zg -- python3 -m pytest --log-cli-level=INFO --verbose
ccnp_perf.py
==================================================================== test session starts ====================================================================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0 -- /usr/local/bin/python3
cachedir: .pytest_cache
rootdir: /run/ccnp
collected 7 items

ccnp_perf.py::test_svc_get_cc_measurement_throughput
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 70.75 ops (operations per second)
PASSED [ 14%]
ccnp_perf.py::test_svc_get_cc_measurement_response
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 25.89662575 ms (milliseconds)
PASSED [ 28%]
ccnp_perf.py::test_svc_get_cc_eventlog_throughput
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 57.8 ops (operations per second)
PASSED [ 42%]
ccnp_perf.py::test_svc_get_cc_eventlog_response
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 76.130223 ms (milliseconds)
PASSED [ 57%]
ccnp_perf.py::test_svc_get_cc_report_throughput
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 54.9 ops (operations per second)
PASSED [ 71%]
ccnp_perf.py::test_svc_get_cc_report_response
----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 29.38618825 ms (milliseconds)
PASSED [ 85%]
ccnp_perf.py::test_ccnp_init PASSED [100%]

=============================================================== 7 passed in 66.95s (0:01:06) ================================================================
root@ccnp-perf-0:~/ccnp/confidential-cloud-native-primitives#
```
Loading
Loading