Skip to content

Commit 48509d5

Browse files
SanjoDeundiakSanjoDeundiak
committed
docs: add MPTCP example for aws
1 parent 6fd6391 commit 48509d5

2 files changed

Lines changed: 332 additions & 0 deletions

File tree

‎examples/command/portals/mptcp/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
## MPTCP
2+
3+
This example shows how to create a TCP Portal that leverages MPTCP to achieve up to 25Gbits/s throughput between different AWS accounts.
4+
5+
## Usage
6+
7+
- Create a working directory, place ockam binary built for aarch64-unknown-linux-gnu in release mode (by default script installs latest released version of Ockam, but expects a manually built binary in the working directory that would be used to replace the released version for easier debugging)
8+
- Use 2 different terminal tabs that logged in into 2 different AWS accounts (e.g. use `export AWS_PROFILE="{PROFILE_NAME}"`)
9+
- Preferably, use the same AWS region to experience the best throughput (e.g., use `export AWS_REGION="{REGION}"`)
10+
- Create 2 subdirectories for 2 AWS accounts and navigate to them in corresponding terminal tabs
11+
- Use the script to create the outlet ec2 machine on the 1st account:
12+
```
13+
aws.sh "{ENROLLMENT TICKET}" inlet
14+
```
15+
- Use the script to create the inlet ec2 machine on the 2nd account. `8` refers to the number of additional ports that the server will advertise as available for MPTCP subflows:
16+
```
17+
aws.sh "{ENROLLMENT TICKET}" outlet 8
18+
```
19+
- Use printed full ssh command to connect to both ec2 machines
20+
- It's recommended to adjust few related variables to achieve the best throughput, e.g.:
21+
- Ockam settings: `export OCKAM_OPENTELEMETRY_EXPORT=false`, `OCKAM_TCP_PORTAL_PAYLOAD_LENGTH=400000` (see `env_info.txt` for the full list)
22+
- Kernel settings (these are already part of the script)
23+
```
24+
sudo sysctl -w net.core.rmem_max=80000000
25+
sudo sysctl -w net.core.wmem_max=80000000
26+
27+
sudo sysctl -w net.ipv4.tcp_rmem="4096 7000000 70000000"
28+
sudo sysctl -w net.ipv4.tcp_wmem="4096 7000000 70000000"
29+
```
30+
- Run the following to create the outlet on the outlet machine:
31+
```
32+
ockam node create --foreground --enable-mptcp \
33+
--enrollment-ticket "$(cat /ticket)" \
34+
--configuration '{
35+
"tcp-listener-address": "0.0.0.0:5555",
36+
"tcp-outlet": {
37+
"to": "localhost:5000"
38+
}
39+
}'
40+
```
41+
- Run the following command to create the inlet on the inlet machine:
42+
```
43+
ockam node create --foreground \
44+
--enrollment-ticket "$(cat /ticket)" \
45+
--configuration '{
46+
"tcp-inlet": {
47+
"from": "0.0.0.0:4000",
48+
"to": "/ip4/{PUBLIC_IP_OF_THE_SECOND_EC2}/mptcp/5555/secure/api/service/outlet",
49+
}
50+
}'
51+
```
52+
- Now the Portal is ready for throughput testing. The scrit will preinstall iperf3 for that (it's also possible to measure the throughput directly without the portal, the preinstalled version of iperf3 has MPTCP support via adding `-m` flag)

‎examples/command/portals/mptcp/aws.sh‎

Lines changed: 280 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,280 @@
1+
#!/usr/bin/env bash
2+
set -ex
3+
4+
run() {
5+
ticket="$1"
6+
type="$2"
7+
number_of_subflows="$3"
8+
machine_type="m8g.48xlarge"
9+
10+
# ----------------------------------------------------------------------------------------------------------------
11+
# CREATE NETWORK
12+
13+
# Create a new VPC and tag it.
14+
vpc_id=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query 'Vpc.VpcId')
15+
aws ec2 create-tags --resources "$vpc_id" --tags "Key=Name,Value=${name}-vpc"
16+
17+
# Create an Internet Gateway and attach it to the VPC.
18+
gw_id=$(aws ec2 create-internet-gateway --query 'InternetGateway.InternetGatewayId')
19+
aws ec2 attach-internet-gateway --vpc-id "$vpc_id" --internet-gateway-id "$gw_id"
20+
21+
# Create a Route Table and a route to the Internet through the Gateway.
22+
rtb_id=$(aws ec2 create-route-table --vpc-id "$vpc_id" --query 'RouteTable.RouteTableId')
23+
aws ec2 create-route --route-table-id "$rtb_id" --destination-cidr-block 0.0.0.0/0 --gateway-id "$gw_id"
24+
25+
# Create a Subnet and associate the Route Table.
26+
az=$(aws ec2 describe-availability-zones --query "AvailabilityZones[-1].ZoneName")
27+
subnet_id=$(aws ec2 create-subnet --vpc-id "$vpc_id" --cidr-block 10.0.0.0/24 \
28+
--availability-zone "$az" --query 'Subnet.SubnetId')
29+
aws ec2 modify-subnet-attribute --subnet-id "$subnet_id" --map-public-ip-on-launch
30+
aws ec2 associate-route-table --subnet-id "$subnet_id" --route-table-id "$rtb_id"
31+
32+
# Create a Security Group.
33+
sg_id=$(aws ec2 create-security-group --group-name "${name}-sg" --vpc-id "$vpc_id" --query 'GroupId' \
34+
--description "Allow ssh ingress and all egress")
35+
# my_ip=$(curl -s http://checkip.amazonaws.com)
36+
# aws ec2 authorize-security-group-ingress --group-id "$sg_id" --cidr "$my_ip/32" --protocol tcp --port 22
37+
aws ec2 authorize-security-group-ingress --group-id "$sg_id" --cidr 0.0.0.0/0 --protocol tcp --port 0-65535
38+
39+
# ----------------------------------------------------------------------------------------------------------------
40+
# CREATE INSTANCE
41+
ami_id=$(aws ec2 describe-images --owners 137112412989 --query "Images | sort_by(@, &CreationDate) | [-1].ImageId" \
42+
--filters "Name=name,Values=al2023-ami-2023.*" "Name=architecture,Values=arm64" \
43+
"Name=virtualization-type,Values=hvm" "Name=root-device-type,Values=ebs")
44+
45+
aws ec2 create-key-pair --key-name "${name}-key" --query 'KeyMaterial' > key.pem
46+
chmod 400 key.pem
47+
48+
cat <<EOF > user_data.sh
49+
#!/bin/bash
50+
set -ex
51+
52+
curl --proto '=https' --tlsv1.2 -sSfL https://install.command.ockam.io | bash -s -- --install-path /opt/ockam
53+
chmod +x /opt/ockam/bin/ockam
54+
cat /opt/ockam/env > /etc/profile.d/ockam.sh
55+
56+
echo "$ticket" > ticket
57+
chmod +r ticket
58+
59+
# sudo yum install iperf3 -y
60+
sudo yum groupinstall -y "Development Tools"
61+
git clone https://github.com/esnet/iperf.git
62+
63+
pushd iperf
64+
./configure; make -j 4; sudo make install
65+
popd
66+
EOF
67+
68+
instance_id=$(aws ec2 run-instances --image-id "$ami_id" --instance-type "$machine_type" \
69+
--subnet-id "$subnet_id" --security-group-ids "$sg_id" \
70+
--user-data file://user_data.sh --key-name "${name}-key" --query 'Instances[0].InstanceId')
71+
aws ec2 create-tags --resources "$instance_id" --tags "Key=Name,Value=${name}-ec2-instance"
72+
aws ec2 wait instance-status-ok --instance-ids "$instance_id"
73+
74+
netwrok_interface_id=$(aws ec2 describe-instances --instance-ids $instance_id \
75+
--query "Reservations[0].Instances[0].NetworkInterfaces[0].NetworkInterfaceId")
76+
77+
# Enable ENA Express
78+
aws ec2 modify-network-interface-attribute --network-interface-id $netwrok_interface_id --ena-srd-specification EnaSrdEnabled=true
79+
80+
public_ip=$(aws ec2 describe-instances --instance-ids "$instance_id" \
81+
--query 'Reservations[0].Instances[0].PublicIpAddress')
82+
83+
scp -o StrictHostKeyChecking=no -i ./key.pem ../ockam "ec2-user@$public_ip:ockam"
84+
ssh -o StrictHostKeyChecking=no -i ./key.pem "ec2-user@$public_ip" \
85+
'bash -s' << EOS
86+
87+
sudo chmod u+x ockam
88+
sudo mv -f ockam /opt/ockam/bin/ockam
89+
EOS
90+
91+
additional_mac_addrs=()
92+
93+
if [[ -z "$number_of_subflows" ]]; then
94+
number_of_subflows=0
95+
fi
96+
97+
if [ "$type" == "inlet" ]; then
98+
for ((i=0; i<$number_of_subflows; i++)); do
99+
eni_id=$(aws ec2 create-network-interface --subnet-id $subnet_id --groups $sg_id --description "Secondary interface $i" --query "NetworkInterface.NetworkInterfaceId")
100+
aws ec2 wait network-interface-available --network-interface-ids $eni_id
101+
aws ec2 create-tags --resources "$eni_id" --tags "Key=Name,Value=${name}-secondary-interface-$i"
102+
103+
mac_address=$(aws ec2 describe-network-interfaces --network-interface-ids $eni_id --query "NetworkInterfaces[-1].MacAddress")
104+
105+
ip_id=$(aws ec2 allocate-address --domain vpc --query "AllocationId")
106+
aws ec2 create-tags --resources "$ip_id" --tags "Key=Name,Value=${name}-additional-ip-$i"
107+
108+
aws ec2 associate-address --network-interface-id $eni_id --allocation-id $ip_id
109+
110+
aws ec2 attach-network-interface --network-interface-id $eni_id --instance-id $instance_id --device-index $((i+1))
111+
112+
additional_mac_addrs+=($mac_address)
113+
done
114+
fi
115+
116+
if [ "$type" == "inlet" ]; then
117+
ssh -o StrictHostKeyChecking=no -i ./key.pem "ec2-user@$public_ip" \
118+
'bash -s' << EOS
119+
120+
endpoints=\$(sudo ip mptcp endpoint show | awk '{print \$3}')
121+
122+
for id in \$endpoints; do
123+
sudo ip mptcp endpoint delete id \$id
124+
done
125+
126+
read -r -a additional_mac_addrs <<< "${additional_mac_addrs[@]}"
127+
for mac in "\${additional_mac_addrs[@]}"; do
128+
dev=\$(ip -o link show | awk -v mac="\$mac" '\$0 ~ mac {print \$2}' | sed 's/://')
129+
ip=\$(ip -o -4 addr show \$dev | awk '{print \$4}' | cut -d/ -f1)
130+
131+
sudo ip mptcp endpoint add \$ip dev \$dev subflow
132+
done
133+
EOS
134+
fi
135+
136+
additional_ports=()
137+
138+
if [ "$type" == "outlet" ]; then
139+
for ((i=0; i<$number_of_subflows; i++)); do
140+
additional_ports+=($((i+6000)))
141+
done
142+
fi
143+
144+
if [ "$type" == "outlet" ]; then
145+
ssh -o StrictHostKeyChecking=no -i ./key.pem "ec2-user@$public_ip" \
146+
'bash -s' << EOS
147+
148+
dev=\$(ip -o link show | awk -F': ' '\$2 != "lo" {print \$2}' | head -n 1)
149+
150+
private_ip=\$(ip -4 addr show \$dev | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
151+
152+
sudo ip addr add $public_ip/32 dev \$dev
153+
154+
read -r -a additional_ports <<< "${additional_ports[@]}"
155+
for port in "\${additional_ports[@]}"; do
156+
sudo ip mptcp endpoint add $public_ip dev \$dev signal port \$port
157+
done
158+
159+
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
160+
161+
sudo yum install -y iptables iptables-services
162+
163+
sudo systemctl enable iptables
164+
165+
sudo tee /etc/sysconfig/iptables <<EOF
166+
*filter
167+
:INPUT ACCEPT [0:0]
168+
:FORWARD ACCEPT [0:0]
169+
:OUTPUT ACCEPT [0:0]
170+
COMMIT
171+
*nat
172+
:PREROUTING ACCEPT [0:0]
173+
:INPUT ACCEPT [0:0]
174+
:OUTPUT ACCEPT [0:0]
175+
:POSTROUTING ACCEPT [0:0]
176+
EOF
177+
178+
for port in "\${additional_ports[@]}"; do
179+
sudo tee -a /etc/sysconfig/iptables <<EOF
180+
-A PREROUTING -d \$private_ip -p tcp -m tcp --dport \$port -j DNAT --to-destination $public_ip:\$port
181+
EOF
182+
done
183+
184+
sudo tee -a /etc/sysconfig/iptables <<EOF
185+
COMMIT
186+
EOF
187+
188+
sudo systemctl start iptables
189+
EOS
190+
fi
191+
192+
193+
ssh -o StrictHostKeyChecking=no -i ./key.pem "ec2-user@$public_ip" \
194+
'bash -s' << EOS
195+
sudo ip mptcp limits set subflows 4
196+
sudo ip mptcp limits set add_addr_accepted 8
197+
198+
sudo sysctl -w net.core.rmem_max=80000000
199+
sudo sysctl -w net.core.wmem_max=80000000
200+
201+
sudo sysctl -w net.ipv4.tcp_rmem="4096 7000000 70000000"
202+
sudo sysctl -w net.ipv4.tcp_wmem="4096 7000000 70000000"
203+
EOS
204+
205+
echo "ssh -o StrictHostKeyChecking=no -i ./key.pem ec2-user@$public_ip"
206+
}
207+
208+
cleanup() {
209+
# ----------------------------------------------------------------------------------------------------------------
210+
# DELETE INSTANCE
211+
212+
instance_ids=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=${name}-ec2-instance" "Name=instance-state-name,Values=pending,running,shutting-down,stopping,stopped" \
213+
--query "Reservations[*].Instances[*].InstanceId")
214+
for i in $instance_ids; do
215+
aws ec2 terminate-instances --instance-ids "$i"
216+
aws ec2 wait instance-terminated --instance-ids "$i"
217+
done
218+
219+
# ----------------------------------------------------------------------------------------------------------------
220+
# DELETE NETWORK INTERFACE
221+
network_ids=$(aws ec2 describe-network-interfaces --filters "Name=tag:Name,Values=${name}-secondary-interface*" \
222+
--query "NetworkInterfaces[*].NetworkInterfaceId")
223+
for i in $network_ids; do
224+
aws ec2 delete-network-interface --network-interface-id "$i"
225+
done
226+
227+
# ----------------------------------------------------------------------------------------------------------------
228+
# DELETE ADDITIONAL IP
229+
additional_ip_ids=$(aws ec2 describe-addresses --filters "Name=tag:Name,Values=${name}-additional-ip*" \
230+
--query "Addresses[*].AllocationId")
231+
for i in $additional_ip_ids; do
232+
aws ec2 release-address --allocation-id "$i"
233+
done
234+
235+
if aws ec2 describe-key-pairs --key-names "${name}-key" &>/dev/null; then
236+
aws ec2 delete-key-pair --key-name "${name}-key"
237+
fi
238+
rm -f key.pem user_data.sh
239+
240+
# ----------------------------------------------------------------------------------------------------------------
241+
# DELETE NETWORK
242+
243+
vpc_ids=$(aws ec2 describe-vpcs --query 'Vpcs[*].VpcId' --filters "Name=tag:Name,Values=${name}-vpc")
244+
245+
for vpc_id in $vpc_ids; do
246+
internet_gateways=$(aws ec2 describe-internet-gateways --query "InternetGateways[*].InternetGatewayId" \
247+
--filters Name=attachment.vpc-id,Values="$vpc_id")
248+
for i in $internet_gateways; do
249+
aws ec2 detach-internet-gateway --internet-gateway-id "$i" --vpc-id "$vpc_id"
250+
aws ec2 delete-internet-gateway --internet-gateway-id "$i"
251+
done
252+
253+
subnet_ids=$(aws ec2 describe-subnets --query "Subnets[*].SubnetId" --filters Name=vpc-id,Values="$vpc_id")
254+
for i in $subnet_ids; do aws ec2 delete-subnet --subnet-id "$i"; done
255+
256+
route_tables=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values="$vpc_id" \
257+
--query 'RouteTables[?length(Associations[?Main!=`true`]) > `0` || length(Associations) == `0`].RouteTableId')
258+
for i in $route_tables; do aws ec2 delete-route-table --route-table-id "$i" || true; done
259+
260+
security_groups=$(aws ec2 describe-security-groups --filters Name=vpc-id,Values="$vpc_id" \
261+
--query "SecurityGroups[?!contains(GroupName, 'default')].[GroupId]")
262+
for i in $security_groups; do aws ec2 delete-security-group --group-id "$i"; done
263+
264+
if aws ec2 describe-vpcs --vpc-ids "$vpc_id" &>/dev/null; then
265+
aws ec2 delete-vpc --vpc-id "$vpc_id"
266+
fi
267+
done
268+
}
269+
270+
export AWS_PAGER="";
271+
export AWS_DEFAULT_OUTPUT="text";
272+
273+
user=""
274+
command -v sha256sum &>/dev/null && user=$(aws sts get-caller-identity | sha256sum | cut -c 1-20)
275+
command -v shasum &>/dev/null && user=$(aws sts get-caller-identity | shasum -a 256 | cut -c 1-20)
276+
export name="ockam-quick-$user"
277+
278+
# Check if the first argument is "cleanup"
279+
# If it is, call the cleanup function. If not, call the run function.
280+
if [ "$1" = "cleanup" ]; then cleanup; else run "$1" "$2" "$3"; fi

0 commit comments

Comments
 (0)