Secure Tunnel WebSocket Protocol V3 Support (#557)

* Secure Tunnel WebSocket Protocol V3 Support

* Updated Secure Tunnel Sample and documentation

* Updated Secure Tunnel Notification Sample and documentation

* Secure Tunnel integration Test

Author: sbSteveK

codebuild/integration-tests.sh

@@ -0,0 +1,19 @@
+set -e
+
+env
+
+pushd $CODEBUILD_SRC_DIR/secure_tunneling/tests
+
+mkdir _build
+cd _build
+cmake -DCMAKE_PREFIX_PATH=/tmp/install ..
+make -j
+
+tunnel_info=$(aws iotsecuretunneling open-tunnel --destination-config services=ssh,ssh2,ssh3 --timeout-config maxLifetimeTimeoutMinutes=10) && echo -e "$tunnel_info" > /tmp/tunnel_info.pem
+
+export SECTUN_SOURCE_TOKEN=$(sed '4!d' /tmp/tunnel_info.pem | cut -d'"' -f4)
+export SECTUN_DESTINATION_TOKEN=$(sed '5!d' /tmp/tunnel_info.pem | cut -d'"' -f4)
+export SECTUN_ENDPOINT="data.tunneling.iot.us-east-1.amazonaws.com"
+
+echo "Running Secure Tunnel Test"
+./secure_tunnel_test

codebuild/linux-integration-tests.yml

@@ -0,0 +1,25 @@
+version: 0.2
+env:
+  shell: bash
+#this buildspec assumes the ubuntu 14 image
+phases:
+  install:
+    commands:
+      - wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
+      - add-apt-repository ppa:ubuntu-toolchain-r/test
+      - apt-add-repository "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-8 main"
+      - apt-get update -y
+      - apt-get install cmake -y -f
+
+  build:
+    commands:
+      - echo Build started on `date`
+      # Building of dependencies happens in setup-linux
+      - $CODEBUILD_SRC_DIR/codebuild/samples/setup-linux.sh
+
+      # Run the integration tests
+      - $CODEBUILD_SRC_DIR/codebuild/integration-tests.sh
+  post_build:
+    commands:
+      - echo Build completed on `date`
+

crt/aws-c-iot

@@ -43,7 +43,7 @@ When a WebSocket upgrade request fails to connect, this callback will return an
 ### OnConnectionShutdown
 When the WebSocket connection shuts down, this callback will be invoked.
 
-### OnSendDataComplete
+### OnSendMessageComplete
 When a message has been completely written to the socket, this callback will be invoked.
 
 ### OnMessageReceived
@@ -55,6 +55,12 @@ When a stream is started by a Source connected to the Destination, the Destinati
 ### OnStreamStopped
 When an open stream is closed, this callback will be invoked and return the stopped stream's information.
 
+### OnConnectionStarted
+When a connection start message is received and a new active connection is established, the Destination will invoke this callback and return the connection information.
+
+### OnConnectionReset
+When a connection has ended either in error or closed intentionally by the secure tunnel peer, the client will invoke this callback and return the connection information.
+
 ### OnSessionReset
 When the Secure Tunnel service requests the Secure Tunnel client fully reset, this callback is invoked.
 
@@ -135,21 +141,24 @@ if (!secureTunnel->Start())
     return -1;
 }
 ```
+### Reconnecting
+A Secure Tunnel Client that has been started will attempt to reconnect upon a failed connection attempt or disconnection until `Stop()` is called. If the secure tunnel closes or the access tokens change, the Secure Tunnel Client will become disconnected and continue to attempt a reconnection until `Stop()` is called. The Secure Tunnel Client implements a Full Jitter Backoff Algorithm along with an exponential back off timer. More information on both can be found here: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
 
 ## Stop
-Invoking `Stop()` on the Secure Tunnel Client breaks the current connection (if any) and moves the client into an idle state.
+Invoking `Stop()` on the Secure Tunnel Client breaks the current connection (if any) and moves the client into an idle state. A Secure Tunnel Client that has been stopped will no longer attempt to reconnect and is ready to be cleaned up.
 ```cpp
 if(!secureTunnel->Stop()){
     fprintf(stdout, "Failed to stop the Secure Tunnel connection session. Exiting..\n");
 }
 ```
-# Multiplexing
-You can use multiple data streams per Secure Tunnel by using the [Multiplexing](https://docs.aws.amazon.com/iot/latest/developerguide/multiplexing.html) feature.
+# Multiplexing and Simultaneous TCP Connections
+You can use multiple data streams per Secure Tunnel by using the [Multiplexing and Simultaneous TCP Connections](https://docs.aws.amazon.com/iot/latest/developerguide/multiplexing.html) features.
+
 
 ## Opening a Secure Tunnel with Multiplexing
 To use Multiplexing, a Secure Tunnel must be created with one to three "services". A Secure Tunnel can be opened through the AWS IoT console [Secure Tunnel Hub](https://console.aws.amazon.com/iot/home#/tunnelhub) or by using the [OpenTunnel API](https://docs.aws.amazon.com/iot/latest/apireference/API_Operations_AWS_IoT_Secure_Tunneling.html). Both of these methods allow you to add services with whichever names suit your needs.
 ## Services Within the Secure Tunnel Client
-On a successfull connection to a Secure Tunnel, the Secure Tunnel Client will invoke the `OnConnectionSuccess` callback. This callback will return `ConnectionSuccessEventData` that will contain any available Service Ids that can be used for multiplexing. Below is an example of how to set the callback using the Secure Tunnel Builder and check whether a Service Id is available.
+On a successfull connection to a Secure Tunnel, the Secure Tunnel Client will invoke the `OnConnectionSuccess` callback. This callback will return `ConnectionSuccessEventData` which contains any available Service Ids that can be used for multiplexing. Below is an example of how to set the callback using the Secure Tunnel Builder and check whether a Service Id is available.
 ```cpp
 // Create Secure Tunnel Builder
 SecureTunnelBuilder builder = SecureTunnelBuilder(...);
@@ -191,6 +200,8 @@ builder.WithOnConnectionSuccess([&](SecureTunnel *secureTunnel, const Connection
 ```
 ## Using Service Ids
 Service Ids can be added to outbound Messages as shown below in the Send Message example. If the Service Id is both available on the current Secure Tunnel and there is an open stream with a Source device on that Service Id, the message will be sent. If the Service Id does not exist on the current Secure Tunnel or there is no currently active stream available on that Service Id, the Message will not be sent and a Warning will be logged. The `OnStreamStarted` callback is invoked when a stream is started and it returns a `StreamStartedEventData` which can be parsed to determine if a stream was started using a Service Id for Multiplexing. Incoming messages can also be parsed to determine if a Service Id has been set as shown above in the [Setting Secure Tunnel Callbacks](#setting-secure-tunnel-callbacks) code example.
+## Using Connection Ids
+Connection Ids can be added to outbound Messages as shown below in the Send Message example. If there is an active stream currently open using the combination of the Service Id and Connection Id, the message will be sent. If a Connection Id is not set on an outbound message, a Connecion Id of 1 is assumed and applied to the Message. When additional streams are activated, the `OnConnectionStarted` callback is invoked and returns a `ConnectionStartedEventData` which can be parsed to determine the Connection Id of the newly activated stream. A Connection Id will also be present in the `StreamStartedEventData` that is returned when the `OnStreamStarted` callback is invoked.
 # Secure Tunnel Operations
 
 ## Send Message
@@ -201,11 +212,16 @@ Crt::String serviceId_string = "ssh";
 Crt::String message_string = "any payload";
 
 ByteCursor serviceId = ByteCursorFromString(serviceId_string);
+uint32_t connectionId = 1
 ByteCursor payload = ByteCursorFromString(message_string);
 
 // Create Message
 std::shared_ptr<Message> message = std::make_shared<Message>();
+// Add a Service Id
 message->withServiceId(serviceId);
+// Add a Connection Id
+message->withConnectionId(connectionId);
+// Add a payload
 message->withPayload(payload);
 
 // Send Message

crt/aws-crt-cpp

@@ -1,72 +1,33 @@
-## Secure Tunnel
+# Secure Tunnel
 
 [**Return to main sample list**](../../README.md)
 
-This sample uses AWS IoT [Secure Tunneling](https://docs.aws.amazon.com/iot/latest/developerguide/secure-tunneling.html) Service to connect a destination and a source with each other through the AWS Secure Tunnel endpoint using access tokens using the [V2WebSocketProtocol](https://github.com/aws-samples/aws-iot-securetunneling-localproxy/blob/main/V2WebSocketProtocolGuide.md). For more information, see the [Secure Tunnel Userguide](../../../documents/Secure_Tunnel_Userguide.md)
-
-Your IoT Core Thing's [Policy](https://docs.aws.amazon.com/iot/latest/developerguide/iot-policies.html) must provide privileges for this sample to connect, subscribe, publish, and receive. Below is a sample policy that can be used on your IoT Core Thing that will allow this sample to run as intended.
-
-<details>
-<summary>(see sample policy)</summary>
-<pre>
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Effect": "Allow",
-      "Action": [
-        "iot:Publish",
-        "iot:Receive"
-      ],
-      "Resource": [
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/things/<thing_name>/tunnels/notify"
-      ]
-    },
-    {
-      "Effect": "Allow",
-      "Action": [
-        "iot:Subscribe"
-      ],
-      "Resource": [
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topicfilter/$aws/things/<thing_name>/tunnels/notify"
-      ]
-    },
-    {
-      "Effect": "Allow",
-      "Action": [
-        "iot:Connect"
-      ],
-      "Resource": [
-        "arn:aws:iot:<b>region</b>:<b>account</b>:client/test-*"
-      ]
-    }
-  ]
-}
-</pre>
-
-Replace with the following with the data from your AWS account:
-* `<region>`: The AWS IoT Core region where you created your AWS IoT Core thing you wish to use with this sample. For example `us-east-1`.
-* `<account>`: Your AWS IoT Core account ID. This is the set of numbers in the top right next to your AWS account name when using the AWS IoT Core website.
-* `<thingname>`: The name of your AWS IoT Core thing you want the device connection to be associated with
-
-Note that in a real application, you may want to avoid the use of wildcards in your ClientID or use them selectively. Please follow best practices when working with AWS on production applications using the SDK. Also, for the purposes of this sample, please make sure your policy allows a client ID of `test-*` to connect or use `--client_id <client ID here>` to send the client ID your policy supports.
+This sample uses AWS IoT [Secure Tunneling](https://docs.aws.amazon.com/iot/latest/developerguide/secure-tunneling.html) Service to connect a destination or a source Secure Tunnel Client to an AWS Secure Tunnel endpoint using access tokens using the [V3WebSocketProtocol](https://github.com/aws-samples/aws-iot-securetunneling-localproxy/blob/main/V3WebSocketProtocolGuide.md). For more information, see the [Secure Tunnel Userguide](../../../documents/Secure_Tunnel_Userguide.md)
 
 ## How to run
 
 Create a new secure tunnel in the AWS IoT console (https://console.aws.amazon.com/iot/) (AWS IoT/Manage/Tunnels/Create tunnel) and retrieve the destination and source access tokens. (https://docs.aws.amazon.com/iot/latest/developerguide/secure-tunneling-tutorial-open-tunnel.html). Once you have these tokens, you are ready to open a secure tunnel.
 
+### Destination Mode
+
 To run the sample with a destination access token in destination mode (default), you can use the following command:
 
 ``` sh
-./secure_tunnel --region <region> --access_token_file <path to destination access token>
+./secure_tunnel --signing_region <signing_region> --access_token_file <path to destination access token>
 ```
 
-However, for this sample to work, it will also need another instance of the sample running in source mode. You can run another instance of the sample in source mode using the same command, but adding the `--localProxyModeSource` flag:
+The sample will create a Secure Tunnel connection and remain connected in `DESTINATION MODE` and will echo any messages it receives through the Secure Tunnel back to the Source Device.
+
+### Source Mode
+
+While the focus of the Secure Tunnel Client for the IoT Device SDK is to connect with Secure Tunnels in `DESTINATION MODE` we also support connecting in `SOURCE MODE`. The token file should be the Source Token in this instance and you must add the `--localProxyModeSource` flag:
 
 ``` sh
-./secure_tunnel --region <region> --access_token_file <path to source access token> --localProxyModeSource
+./secure_tunnel --signing_region <signing_region> --access_token_file <path to source access token> --localProxyModeSource
 ```
 
 Then two samples will then connect to each other through the AWS Secure Tunnel endpoint and establish a stream through which data can be transmitted in either direction.
+The sample will create a Secure Tunnel connection in `SOURCE MODE` and will open a stream using an available `Service Id`. It will then send n messages on the opened stream. It will then create a new simultaneous TCP connection on the stream and send an additional n messages on the new TCP connection. It will then exit.
 
+### Proxy
 Note that a proxy server may be used via the `--proxy_host` and `--proxy_port` argument. If the proxy server requires a user name and password to connect,  you can use `--proxy_user_name` and `--proxy_password` to in the sample to pass the required data to the sample.