Feature/build with gcc9 on ubuntu

docs/container-orchestration/README.md

@@ -0,0 +1,70 @@
+# REDHAWK Container Orchestration
+REDHAWK offers a standardized methodology for running waveforms on container orchestration technologies by means of an extensible plugin class. These capabilities have been integrated into core-framework through modifications to the Application Factory and Sandbox but in a manner that emphaised backwards compatibility; it is still possible to use the standard Domain Manager and GPP combo to run native Executable and SharedLibrary type components on your REDHAWK systems.
+
+Each cluster technology has its own unique ways to launch, configure, and delete containers, so the plugin system allows users to develop their own Sandbox and Application Factory plugins that Domain Manager dynamically loads at run-time to spawn waveforms on a cluster according a global cluster configuration file called `cluster.cfg` that controls the parameters passed to each plugin.
+
+*NOTE* These additions have *not* been reconciled with REDHAWK IDE. Viewing Components with spd.xml files modified in support of using these features will generate visual errors in RHIDE.
+
+# General Installation Dependencies
+
+1. Install the dependencies for REDHAWK with the [REDHAWK Manual](https://redhawksdr.org/2.2.8/manual/appendices/dependencies/)
+2. Yum install the `yaml-cpp-devel` package for the plugins' yaml generation
+3. Docker version 19.03.12 is required for building REDHAWK Component docker images
+4. Download and install [docker-redhawk](https://github.com/Geontech/docker-redhawk.git) tag 2.2.8-0 or [docker-redhawk-ubuntu](https://github.com/Geontech/docker-redhawk-ubuntu.git) branch develop-2.2.1 (Ubuntu is only required for GNURadio components)
+5. The following must be present in your environment (have network connectivity to your system running REDHAWK and proper configuration) in order to use RH components in clusters:
+   * A cluster running the desired orchestration technology
+   * The matching RH cluster plugin installed
+   * Your networking, firewalls and DNS setup to communicate with your cluster's control plane/nodes/API
+   * Your local system with REDHAWK installed must have its `cluster.cfg` file set correctly and the plugin's approprriate environment variables set
+
+For plugin specific instructions, please see that plugin's corresponding documentation.
+
+# Plugin Execution Path
+The Application Factory loops through the list of components in the waveform making note of the component's code type. If the code type is set to "Container", then the spd parsing mechanism checks the component's entrypoint. The entrypoint element has been extended to identify the suffix of a docker image path. Consider the example below pulled from SigGen's spd.xml file.
+```
+<code type="Container">
+      <localfile name="cpp/SigGen"/>
+      <entrypoint>cpp/SigGen::rh.siggen</entrypoint>
+</code>
+```
+In this entrypoint element, everything after the "::" is regarded as the desired image name suffix. The "::" is a delimiter. The prefix of the image is found in the `cluster.cfg` file.
+
+If all components in the waveform are of Container code type, then no GPP is required to run the waveform. Waveforms can be ran in a "hybrid" manner in which components with Executable or SharedLibrary code types will run on the native REDHAWK system while those with Container code type are executed on the cluster dictated by the installed plugin and `cluster.cfg` file. Hybrid waveforms, or any waveform that uses Executable or SharedLibrary code types still required a GPP to run (this is for backwards compatibility).
+
+# Building and Installing core-framework from Source
+```bash
+$ cd /core-framework/redhawk/src
+$ ./build.sh
+$ sudo make install
+```
+## Building and Installing Plugins
+No plugin is installed by default. Three plugins are currently included with REDHAWK:
+1. [Docker](plugin-docker.md)
+2. [DockerSwarm](plugin-dockerswarm.md)
+3. [EksKube](plugin-ekskube.md)
+
+## The cluster.cfg file
+The `$OSSIEHOME/cluster.cfg` file contains sections named according to the cluster orchestration plugin technology (EKS, Docker Swarm, etc). Each plugin-specific section contains variables (key value pairs) used by the specific plugin. You should ensure the values set in this file are correct prior to running a waveform.
+
+#### Installing the Cluster Configuration File
+```bash
+$ cd ./core-framework/redhawk/src/base/cfg
+$ sudo -E ./build.py --cluster <Desired_Plugin_Name> <Arguments to set variables in cluster.cfg>
+```
+
+This will render the template cluster.cfg file into your $OSSIEHOME/cluster.cfg file. The plugin name argument you specify sets the plugin's top section which controls will plugin REDHAWK will use, assuming the plugin is already installed.
+
+Possible values for the plugin name argument are:
+1. [EksKube](plugin-ekskube.md)
+2. [Docker](plugin-docker.md)
+3. [DockerSwarm](plugin-dockerswarm.md)
+
+## Networking
+Networking can get complicated quickly because it will vary depending on what you're trying to run in your cluster, what that cluster's networking setup looks like, and your local system's ability to send traffic to that cluster. Guiding rules of thumb are:
+* If you are running DomMgr and/or GPP with OmniORB on your local system and only running waveforms on a cluster, those launched components need to be able to initiate connections to omniORB. This boils down to there being a "flat" network between where OmniORB runs and where the waveform runs. NAT will break one party's ability to talk to the other.
+* In the scenario where all REDHAWK services (OmniORB, DomMgr, and/or GPP) run inside the cluster alongside the waveform payloads, so long as the containers can network resolve each other (almost always the case barring network security restrictions on the cluster), then there should not be any difficulties with networking.
+
+Please see each plugin's documention for more more network specifics.
+
+## Misc additions
+Each plugin behaves differently. Some require specialized networking, other require special credentials, and some might require environment variables. Please consult the specific plugin's documentation to learn more.

docs/container-orchestration/plugin-class.md

@@ -0,0 +1,49 @@
+# Plugin Class
+The parent plugin class can be used to inherit from and form the skeleton of your own plugin. It can be found at `core-framework/src/base/include/ossie/cluster/ClusterManagerResolver.h` The class' public methods are those that your derivative class can overwrite for you orchestration technology's way of handling containers.
+
+# Public Methods
+## launchComponent(std::string app_id)
+* Launches a component or yaml file of multiple components into the cluster
+* @param app_id the redhawk code passes the plugin the application ID (can be used for namespaces for example)
+* @return An integer representing a pid. A negative pid will throw an error while a pid 0 and greater will succeed
+
+## pollStatusActive(std::string app_id)
+* Polls the component and waits to see that it is active and running (equivalent to REDHAWKs native pid check but for clusters)
+* @param comp_id the key that is used on validNamesMap to find the name of the component that is being checked to be active
+* @return Boolean where true means the component is active and false means the component is not. REDHAWK prints an error to logs if false
+
+## pollStatusTerminated(std::string app_id)
+* Polls the component and waits for terminatation (in clusters this might mean CrashLoopBackoff, Completed, etc...)
+* @param comp_id the key that is used on validNamesMap to find the name of the component that is being checked to be terminated
+* @return Boolean where true means the component is terminated and false means the component is not. REDHAWK prints an error to logs if false
+
+## deleteComponent(std::string comp_id)
+* Deletes a component or multiple components in a yaml file from the namespace
+* @param comp_id the key that is used on validNamesMap to find the name of the component that is being checked to be deleted
+
+## isTerminated(std::string app_id)
+* One-off check for if a component has terminated (in clusters this might mean CrashLoopBackoff, Completed, etc...)
+* @param comp_id the key that is used on validNamesMap to find the name of the component that is being checked to be terminated
+* @return true if terminated and false if not. Throws a ComponentTerminated exception if false on start up
+
+## openComponentConfigFile(redhawk::PropertyMap execParameters, std::string entryPoint, std::string image)
+ Adds a component to the yaml file so that all cluster type components are in the file before launching. This is also the location where non-cluster type deployments would run executables (see DockerResolver.cpp)
+
+`execParameters` The parameters given to the component to be able to execute it. These parameters can instead be baked into the yaml file for the cluster technology to launch itself 
+(i.e /path/to/executable NAMING_CONTEXT_IOR <nameing_context_ior> PROFILE_NAME <profile_name> NAME_BINDING <name_binding> COMPONENT_IDENTIFIER <component_identifier> DEBUG_LEVEL <debug_level>).
+Other params include:
+* NIC
+* RH::GPP::MODIFIED_CPU_RESERVATION_VALUE
+
+`entryPoint` The path to the executable (i.e /var/redhawk/sdr/dom/components/rh/SigGen/cpp/SigGen)
+
+`image` The image name that was attached to the entrypoint in the spd.xml file (i.e in the spd <\entrypoint>/path/to/executable::image<\entrypoint>). 
+This is not the fully qualified path to the image. The registry path will instead be found in /usr/local/redhawk/core/cluster.cfg and combined with this `image` parameter to yield the fully qualified image path.
+
+## closeComponentConfigFile(std::string app_id)
+* Closes the yaml file that was being written to
+* @param app_id the application ID is given so that when the file is saved out it can be unique
+
+
+
+

docs/container-orchestration/plugin-docker.md

@@ -0,0 +1,58 @@
+# Docker
+The Docker plugin is designed to run REDHAWK waveforms on your local system's Docker installation.
+
+# Building and Installing the Plugin
+
+Both the Application Factory and Sandbox plugins for Docker are installed when core-framework is built and installed from source.
+
+Application Factory
+```bash
+$ cd core-framework/redhawk/src/base/plugin/clustermgr/clustertype
+$ ./build.py Docker
+$ ./reconf && ./configure && make && sudo make install
+```
+Sandbox
+```bash
+$ cd core-framework/redhawk/src/base/framework/python/ossie/utils/sandbox
+$ make FILE=Docker
+```
+
+This will compile and install the Application Factory and Sandbox plugins for the user. The plugins are built in a specific location in core-framework (`core-framework/redhawk/src/base/plugin/clustermgr/`and `core-framework/redhawk/src/base/framework/python/ossie/utils/sandbox/`) and are both installed to `/usr/local/redhawk/core/lib`
+
+# Plugin Specifics
+## Dependencies
+1. Docker installed on your system along with REDHAWK
+
+## The cluster.cfg file
+```bash
+cd core-framework/redhawk/src/base/cfg
+sudo -E ./build.py --cluster Docker --docker_dir <where-to-mount-on-container> --mount_dir <where-to-mount-from>
+```
+OR
+```bash
+cd core-framework/redhawk/src/base/cfg
+make Docker DOCKER_DIR="<your-docker-dir>" MOUNT_DIR="<your-mount-location>"
+```
+This will properly set the top section to use the Docker plugin and pass in the assortment of arguments to setup the cluster.cfg file.
+
+## cluster.cfg file variables
+The top section of the file should specify that the Docker plugin is desired like so:
+```
+[CLUSTER]
+name = Docker
+```
+| Variable   | Example Value         | Description                                                |
+|------------|-----------------------|------------------------------------------------------------|
+| docker_dir | /mnt/                 | Path inside of the docker container to mount mount_dir to  |
+| mount_dir  | /home/bob/myshareddir | Path on the docker host to mount into the docker container |
+
+*NOTE*: These variables are only used by the Sandbox Docker plugin and do not work for the Application Factory plugin.
+
+## Credentials
+The Docker plugin needs the following credentials:
+1. Your ~/.docker/config.json updated to allow docker to pull your Component images from your desired Registry if the images are not already present on your local system
+
+## Networking
+Docker uses a bridged network by default by running the containers with the "--network host" option.
+
+This configuration is ideal for enabling your containers to communicate with running REDHAWK services on the same system.

docs/container-orchestration/plugin-dockerswarm.md

@@ -0,0 +1,77 @@
+# DockerSwarm
+The DockerSwarm plugin is designed to run REDHAWK waveforms on Docker Swarm clusters.
+
+# Building and Installing the Plugin
+
+Application Factory
+```bash
+$ cd core-framework/redhawk/src/base/plugin/clustermgr/clustertype
+$ sudo ./build.py DockerSwarm
+$ ./reconf && ./configure && make && sudo make install
+```
+Sandbox
+```bash
+$ cd core-framework/redhawk/src/base/framework/python/ossie/utils/sandbox/clustertype
+$ make FILE=DockerSwarm
+```
+
+This will compile and install the Application Factory and Sandbox plugins for the user. The plugins are built in a specific location in core-framework (`core-framework/redhawk/src/base/plugin/clustermgr/`and `core-framework/redhawk/src/base/framework/python/ossie/utils/sandbox/`) and are both installed to `/usr/local/redhawk/core/lib`
+
+# Plugin Specifics
+## Dependencies
+1. None
+
+## The cluster.cfg file
+```bash
+cd core-framework/redhawk/src/base/cfg
+sudo -E ./build.py --cluster DockerSwarm --registry <your-registry> --ssh_key <your-ssh-key> --server_user <your-server-username> --server_ip <your-server-ip-addr>
+```
+OR
+```bash
+cd core-framework/redhawk/src/base/cfg
+make DockerSwarm REGISTRY="<your-registry>" SSH_KEY="<your-ssh-key>" SERVER_USER="<your-server-username>" SERVER_IP="<your-server-ip-addr>"
+```
+This will properly set the top section to use the DockerSwarm plugin and pass in the assortment of arguments to setup the cluster.cfg file.
+
+## cluster.cfg file variables
+The top section of the file should specify that the DockerSwarm plugin is desired like so:
+```
+[CLUSTER]
+name = DockerSwarm
+```
+| Variable         | Example Value      | Description                                                                                                                                            |
+|------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
+| registry         | geontech           | This value is concatenated with a "/" and then the image suffix found in the entrypoint of the component's spd.xml file. Shared across all components. |
+| tag              | latest             | The image tag used. Shared across all components.                                                                                                      |
+| key              | ~/.ssh/ssh_key.pem | Path to your SSH key. Must be read-accessible by Domain Manager                                                                                        |
+| user             | centos             | User used to log into node via SSH                                                                                                                     |
+| ip               | 10.10.10.10        | IP address or FQDN of the Swarm Master node to connect to via SSH                                                                                      |
+| docker_login_cmd | "docker login"     | The command ran by the plugin to log into the registry hosting the Component container images                                                          |
+
+## Credentials
+The DockerSwarm plugin needs the following credentials:
+1. A linux user on a Docker Swarm Manager node authorized to SSH into it with a corresponding SSH private key
+2. Docker login creds on the Docker Swarm Manager Node
+
+Docker Swarm does not offer a "remote control" capability like K8S does via kubectl, so the Docker Swarm plugin uses SSH to log into a Swarm Manager Node to run/apply the generated docker-compose yaml. For this SSH login to work, a pre-configured user must exist on the targeted Manager node that is allowed to SSH in.
+
+When the plugin logs into the node over SSH, it first runs the command defined in the docker_login_cmd variable in `cluster.cfg`. You can overrite this value to supply your username and password as arguments in the `cluster.cfg` file.
+
+## Networking
+Docker Swarm uses virtual overlay networks by default, which isn't ideal for waveforms on the Swarm and REDHAWK services running on a remote system external to the cluster. It is for this reason that the yaml generated for Docker Swarm creates a host network and attached all Component containers to that network:
+```yaml
+version: '3'
+networks:
+  outside:
+    external:
+      name: host
+services:
+  siggen1:
+    ...
+    networks:
+      - outside
+````
+This makes each Component container share the host node's IP address.
+
+
+This configuration is ideal for running REDHAWK services external to the cluster (on your local REDHAWK system).