mmTLS

mmTLS is a highly scalable TLS middlebox for monitoring encrypted traffic.

mmTLS achieves high throughput and low latency leveraging the techniques below.

1. Single connection architecture which removes redundant TLS/TCP handshakes, flow management, memory copies, and re-encryption
2. Scalable key distribution via SmartNIC (or a dedicated CPU core)
3. Minimal overhead on private tag generation/verification
4. Support for event-driven API for programming TLS connections

Accessing machines remotely for AE

mmTLS currently leverages the Bluefield-2 SmartNIC, so we recommend accessing our test machine with the SmartNIC from remotely and running test scripts on them. We have set up a simple testbed to evaluate the functionality of our work, used for ATC’24 artifact evaluation. Please let us know ([email protected] or HotCRP) and we will provide the credential to login to the test machine. Our testbed consists of 7 machines: 4 clients, 1 middlebox, and 2 backend servers (see the picture below) and all of them are accessible by ssh. Please log into the access server (i.e., box3.kaist.ac.kr) first, and you can log into other machines from it. The figure below depicts the topology of our testbed.

This page assumes that you have access to box3.kaist.ac.kr via ssh.

The first login to the access machine should be done as "atc-ae", which is an account we provide for your first access.

The second login to the other machines from the access machine should be done as "junghan", which has all the binaries and scripts.

# on your local machine
ssh [email protected] -p [port]

Now you can log into the middlebox machine (box1.kaist.ac.kr) which has the AE scripts for figure 8, 9, 10, 13, 14, and 16.

# on [email protected]
ssh [email protected]

Or, you can log into the client machine (wood1.kaist.ac.kr) which has the AE scripts for figure 11, 12, and 17.

# on [email protected]
ssh [email protected]

For figure 15, which requires compiled chromium, you should log into box2.kaist.ac.kr with -X option to use X window.

# on your local machine
ssh -X [email protected] -p [port]

# on [email protected]
ssh -X [email protected]

Figure 8 - Persistent Connection Test (25 minutes)

We have prepared an automated script to generate results for figure 8. Log in to box1.kaist.ac.kr first.

# on [email protected]
ssh [email protected]

Then, run the script, 'run-persistent.sh'

# on [email protected]
cd ~/mmTLS/proxies/mOS/mmTLS
./run-persistent.sh

It will take about 25 minutes. The script will print the throughput of mmTLS (1), mmTLS (2), splitTLS (1), splitTLS (2), and mcTLS (2) as below.

The reason why mmTLS (2) achieves much higher throughput for larger files ( > 64KB) is our mmTLS application decrypts only first 64KB of HTTP response.

Figure 9 - Ephemeral Connection Test (5 minutes)

- For AE of Figure 9, we have found a mis-configuration in our last evaluation.
- With a correct configuration, mmTLS produces 40K/s, 20% lower than figure 9.
- We also note that mcTLS achieves a higher performance.
- Despite the change, we still observe that mmTLS achieves 40x to 60x performance improvement.

- We will update the final result for the camera-ready version.
- Sorry for the confusion, but we will fix this graph for the camera-ready version.

Login to the middlebox machine (box1.kaist.ac.kr).

# on [email protected]
ssh [email protected]

Then, run the script, 'run-ephemeral.sh' as below.

# on [email protected]
cd ~/mmTLS/proxies/mOS/mmTLS
./run-ephemeral.sh

This script will take about 5 minutes and print the throughput as below.

Figure 11 - Response Time - LAN (6 minutes)

We measure the average of 100 LAN response times and 100 WAN response times. To simplify your evaluation, we have prepared a simple all-in-one script on the client side. Log into wood1.kaist.ac.kr first.

# on [email protected]
ssh [email protected]

Move to the fig11 directory and run 'all-in-one.sh' script. It takes about 6 minutes.

# on [email protected]
cd ~/fig11
./all-in-one.sh

The result will be equivalent to figure 11 for both 11(a) -GCM- and 11(b) -CBC-.

- For AE of Figure 11 (a), we found that there were disk IO delays in E2E response time.
- On this evaluation, we removed the disk IO by warming up the web servers.
- For AE of Figure 11 (b), we found that the key exchange protocols were not consistent among mmTLS and the baselines.
- mmTLS and Split-TLS were using RSA for key exchange while mcTLS was using DHE with 1024-bit DH parameter.
- We make them use a common key exchange protocol, DHE with 2048-bit DH parameter, which is supported by mcTLS.

- Sorry for the confusion, but we will fix this graph for the camera-ready version.

Figure 12 - Response Time - WAN (15 minutes)

For figure 12, we have prepared a simple 'all-in-one.sh' script on the client side as well as fig 11. Log into wood1.kaist.ac.kr first.

# on [email protected]
ssh [email protected]

Then, run below.

# on [email protected]
cd ~/fig12
./all-in-one.sh

It will take about 15 minutes, and print the result as below.

Note that this script modifies the default routing table entry to make the WAN traffic come and go via LAN (private network) interface instead of the default WAN (public network) interface. (Unless, WAN traffic will not go to the middlebox which is connected via the LAN interface.) If you are accessing via ssh to the WAN interface of the client (wood1), your ssh session will be lost. Please do ssh from the access machine (box3), and run the script. Since the news site is updated every hour, the result will be different with our evaluation. We recommend you to check the gap between mmTLS and split-TLS, rather than the absolute response time of mmTLS. Also, www.washingtonpost.com no longer supports http/1.1. It currently accepts only http/2. However, the nginx proxy (split-TLS) currently supports up to http/1.1 as a backend upstream connection, so we could not fully reproduce the result of split-TLS. So, the result for washingtonpost will appear empty. If you think the absolute response time is necessary, please let us know. We will prepare other popular web sites to test split-TLS to WAN.

Figure 13a - Scalability (7 minutes)

Log into the middlebox machine, and just run the script, 'run-scalability.sh', which automatically tests mmTLS middlebox and nginx TLS proxy with various numbers of cores.

# on [email protected]
ssh [email protected]

# on [email protected]
cd ~/mmTLS/proxies/mOS/mmTLS
./run-scalability.sh

The script will take about 7 minutes and print the result as below.

Figure 13b - Comparison with E2E TLS Server (7 minutes)

Login to the middlebox machine (box1.kaist.ac.kr).

# on [email protected]
ssh [email protected]

Run the script below.

# on [email protected]
cd ~/mmTLS/proxies/mOS/mmTLS
./run-compare-with-e2e.sh

It will take about 7 minutes and finally print the throughput of mmTLS and E2E-TLS as below.

Figure 14 - Overhead on Tag Generation (2 minutes)

Go to the openssl-modified directory on the mmTLS root directory, and run the 'run-tag.sh' script. It generates private tags in various methods for record size of 1KB, 2KB, 4KB, 8KB, and 16KB, then measures the relative overhead using average time spent.

# on [email protected]
ssh [email protected]

# on [email protected]
cd ~/mmTLS/proxies/mOS/mmTLS
./run-tag.sh

It will take about 2 minutes. Each column means Original (no private tag), mmTLS (optimal), Reusing ciphertext, and Double tags (naive), respectively.

Figure 15 - Web Browser Test

Note

This evaluation needs chromium GUI application and the extension program, since we manually measure the page load time shown by the extension program. For the same reason, it is difficult to automatically reproduce the figure, so we only provide the manual test in this section.

To execute GUI chromium, X window server should be installed on your local machine. Please make sure your terminal can open a GUI application via X window. There are many X window server for various platform. If you are not familiar with X window, use MobaXterm, which is a terminal with an embedded X window server.

Here, we note that the absolute result may not be exactly same as the figure above, since the chromium client machine is changed from the last evaluation. Again, we recommend you to check the gap between E2E-TLS/mmTLS and split-TLS in terms of the page load time.

Reproducing

First log in to the client machine, box2.kaist.ac.kr with -X option to enable X window. You should have been connected to the first access server (box3.kaist.ac.kr) with -X option before ssh into the client machine.

# on your local machine
ssh -X [email protected]

Then, log in to the machine that has the pre-built chromium with -X option.

# on [email protected]
ssh -X [email protected]

You can run the split-TLS chromium using the script below.

# on [email protected]
./split-chromium.sh 200 # choose among 10, 20, 50, 100, and 200

It will load the page consisting of 200 embedding resources as below.

After loading, click the first extension and check the total loading timing. (952ms in the screenshot) You can repeat by typing "Ctrl + F5". (No refresh button or "only F5", since they do not establish a new TLS connection.) Since it's a LAN connection, the result will be pretty stable, even though you do not repeat it 100 times fully to measure the average.

If you want to change the number of embedding resources, run the script with an argument among 10, 20, 50, 100, and 200.

After checking the page load time with Split-TLS chromium, close the browser using "Alt + F4" and open E2E-TLS chromium using the script below.

# on [email protected]
./e2e-chromium.sh 200 # choose among 10, 20, 50, 100, and 200

You can repeat the page loading by typing "Ctrl + F5", and change the number of embedding resources as well.

After checking the page load time with E2E-TLS chromium, close the browser and open mmTLS chromium using the script below.

# on [email protected]
./mm-chromium.sh 200 # choose among 10, 20, 50, 100, and 200

This script will setup the mmTLS configuration and remotely runs the mmTLS middlebox. Again, you can repeat the test and change the number of embedding resources.

Figure 16 - DPI Application on the TLS Middlebox (6 minutes)

First, login to the middlebox machine (box1.kaist.ac.kr).

# on [email protected]
ssh [email protected]

Then, run the script below.

# on [email protected]
cd ~/mmTLS/proxies/mOS/mmTLS
./run-dpi.sh

It will take about 6 minutes, and print the result as below.

Figure 17 - Another Sample Application - Cipherstats (30 minutes)

We provide an all-in-one script that runs the mmTLS middlebox application, 'my_cipherstat' on the middlebox machine (box1.kaist.ac.kr) and 'key-server' on the SmartNIC. Login to the client machine (wood1.kaist.ac.kr).

# on [email protected]
ssh [email protected]

Then, go to the fig17 directory, and run 'all-in-one.sh' as below.

# on [email protected]
cd ~/fig17
./all-in-one.sh alexa-test

It will print whether the site is accessible from our testbed as below.

It will take about 30 minutes. At the end of the script, it will stop the middlebox and print the summarized result by bringing it from the middlebox machine (box1.kaist.ac.kr).

Since the web sites on WAN are updated every hour, the result might not be exactly the same as the figure.

Functionality Check

In this section, we provide how to manually run the mmTLS middlebox for functionality check.

mmTLS middlebox application

Login to box1.kaist.ac.kr first.

# on [email protected]
ssh [email protected]

Then, go to the directory with our mmTLS application, "my_ips", which decrypts first 64KB of HTTP response.

# on [email protected]
cd ~/mmTLS/proxies/mOS/mmTLS

Run the script, "setup-mmtls-endpoints.sh" for configuring ARP tables of endpoints.

./setup-mmtls-endpoints.sh

Run the "my_ips" application.

sudo ./my_ips -c 16

The "-c" option means the number of cores used by "my-ips". You will see the logs of real-time throughput after initializing DPDK EAL. They are used for measuring the throughput of persistent connections.

mmTLS key-server on the SmartNIC

To run "key-server" on the SmartNIC, open one more terminal and login to the SmartNIC on box1.kaist.ac.kr.

# on [email protected]
ssh [email protected]

# on [email protected]
ssh 192.168.100.2

Then, you can run "key-server" on the SmartNIC as below.

# on [email protected]
cd bf2_key_server
sudo ./key-server -c 8 -i p1

The "-c" option means the number of cores used by "key-server" and the "-i" option means the interface for sending raw UDP packet. You will see the logs of number of keys processed by the "key-server". They are used for measuring the throughput of ephemeral connections.

h2load as an mmTLS client

Finally, open one more terminal and run the "h2load" as a client on one of our clients, wood1.kaist.ac.kr.

# on [email protected]
ssh [email protected]

# on [email protected]
~/nghttp2/src/h2load https://10.11.90.3:443/1k/test0 --tls13-ciphers=TLS_AES_256_GCM_SHA384 --key-send

The "--key-send" option means that the client sends session keys to the "key-server".

Quit

After receiving the response, quit "my-ips" and "key-server". Close the third terminal which runned h2load by typing Ctrl + D. Then, go to the second terminal which runs "key-server", quit it by typing Ctrl + C, and close the second terminal by typing Ctrl + D. Similarly, go to the first terminal which runs "my-ips", quit it by typing Ctrl + C, and close the first termnal by typing Ctrl + D.

Configuring nginx for a baseline middlebox (Split-TLS) and endpoints

The configuration of nginx as a Split-TLS middlebox is shown below. If you want to test without the script we prepared, make h2load or ab on the client machines send requests to the responsible port. Note that svr0 and svr1 refer to box3.kaist.ac.kr and box4.kaist.ac.kr, respectively.

# LAN

# endpoint:	0xxxx,	proxy to svr0: 	1xxxx,	proxy to svr1:	2xxxx
# persistent: 	x0xxx, 	ephemeral: 	x1xxx,
# TCP:		xx080, 	TLS12: 		xx442, 	TLS13:		xx443

# 00080: endpoint persistent		TCP
# 01080: endpoint ephemeral		TCP

# 00442: endpoint persistent		TLS12
# 01442: endpoint ephemeral		TLS12

# 00443: endpoint persistent		TLS13
# 01443: endpoint ephemeral		TLS13

# 10080: proxy to svr0 persistent	TCP 
# 11080: proxy to svr0 ephemeral	TCP

# 10442: proxy to svr0 persistent	TLS12
# 11442: proxy to svr0 ephemeral	TLS12

# 10443: proxy to svr0 persistent	TLS13
# 11443: proxy to svr0 ephemeral	TLS13

# 20080: proxy to svr1 persistent	TCP
# 21080: proxy to svr1 ephemeral	TCP

# 20442: proxy to svr1 persistent	TLS12
# 21442: proxy to svr1 ephemeral	TLS12

# 20443: proxy to svr1 persistent	TLS13
# 21443: proxy to svr1 ephemeral	TLS13


# WAN

# proxy to WAN:		3xxxx

# 30443: proxy to usatoday		ephemeral	TLS13
# 31443: proxy to bbc			ephemeral	TLS13
# 32443: proxy to nytimes		ephemeral	TLS13
# 33443: proxy to cnn			ephemeral	TLS13
# 34443: proxy to washingtonpost	ephemeral	TLS13

The configuration of nginx as endpoints is a subset of above. Endpoints (box3, box4) use only upper 6 ports; 80, 1080, 442, 1442, 443, 1443. You don't need to modify the configuration files. It's just for reference.