OpenShift on OpenStack

About

A collection of documentation, Heat templates, configuration and everything else that’s necessary to deploy OpenShift on OpenStack.

This template uses Heat to create the OpenStack infrastructure components, then calls the OpenShift Ansible installer playbooks to install and configure OpenShift on the VMs.

Architecture

All of the OpenShift VMs will share a private network. This network is connected to the public network by a router.

The deployed OpenShift environment is composed of a replicated set of OpenShift master VMs fronted by a load_balancer. This provides both a single point of access and some HA capabilities. The applications run on one or more OpenShift node VMs. These are connected by a private software defined network (SDN) which can be implemented either with OpenVSwitch or Flannel.

An infrastructure server is used to control the host and service configuration. It can also run local DNS services as needed. The host and service configuration is run using Ansible playbooks executed from the infrastructure host.

Every VM in this configuration is also given a floating IP address on the public network. This provides direct access to the master and node hosts when needed.

All of the OpenShift hosts (master and node) have block storage for Docker images and containers provided by Cinder. OpenShift will run a local Docker registry, also backed by Cinder block storage. Finally all nodes will have access to Cinder volumes which can be created by OpenStack users and mounted into containers by Kubernetes.

Prerequisites

1. OpenStack version Juno or later with the Heat, Neutron, Ceilometer services running:

   • heat-api-cfn service - used for passing heat metadata to nova instances

   • neutron LBaaS service (optional) - used for loadbalancing requests in HA mode, if this service is not available, you can deploy dedicated loadbalancer node, see 'Select Loadbalancer Type' section

   • ceilometer services (optional) - used when autoscaling is enabled

2. CentOS 7.2 cloud image (we leverage cloud-init) loaded in Glance for OpenShift Origin Deployments. RHEL_ 7.2 cloud image if doing Atomic Enterprise or OpenShift Enterprise

3. An SSH keypair loaded to Nova

4. A (Neutron) network with a pool of floating IP addresses available

CentOS and RHEL are the only tested distros for now.

Creating an All-In-One Demo Environment

Following steps can be used to setup all-in-one testing/developer environment:

# OpenStack does not run with NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager

# The Packstack Installer is not supported for production but will work
# for demonstrations
yum -y install openstack-packstack libvirt git

# Add room for images if /varlib is too small
mv /var/lib/libvirt/images /home
ln -s /home/images /var/lib/libvirt/images

# Install openstack demonstrator with no real security
#   This produces the keystonrc_admin file used below
packstack --allinone --provision-all-in-one-ovs-bridge=y \
  --os-heat-install=y --os-heat-cfn-install=y \
  --os-neutron-lbaas-install=y \
  --keystone-admin-passwd=password --keystone-demo-passwd=password

# Retrieve the Heat templates for OpenShift
git clone https://github.com/redhat-openstack/openshift-on-openstack.git

# Retrieve a compatible image for the OpenShift VMs
curl -O http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2

# Set access environment parameters for the new OpenStack service
source keystonerc_admin

# Load the VM image into the store and make it available for creating VMs
glance image-create --name centos72 --is-public True \
  --disk-format qcow2 --container-format bare \
  --file CentOS-7-x86_64-GenericCloud.qcow2
# For newer versions of glance clients, substitute "--is-public True" with "--visibility public"

# Install the current user's SSH key for access to VMs
nova keypair-add --pub-key ~/.ssh/id_rsa.pub default

Deployment

You can pass all environment variables to heat on command line. However, two example environment files have been given.

• env_origin.yaml is an example of the variables to deploy an OpenShift Origin 3 environment.

• env_aop.yaml is an example of the variables to deploy an Atomic Enterprise or OpenShift Enterprise 3 environment. Note deployment type should be openshift-enterprise for OpenShift or atomic-enterprise for Atomic Enterprise. Also, a valid RHN subscription is required for deployment.

Assuming your external network is called public, your SSH key is default and your CentOS 7.2 image is centos72 and your domain name is example.com, this is how you deploy OpenShift Origin:

cat << EOF > openshift_parameters.yaml
parameters:
   # Use OpenShift Origin (vs Openshift Enterprise)
   deployment_type: origin

   # set SSH access to VMs
   ssh_user: centos
   ssh_key_name: default

   # Set the image type and size for the VMs
   server_image: centos72
   flavor: m1.medium

   # Set an existing network for inbound and outbound traffic
   external_network: public
   dns_nameserver: 8.8.4.4,8.8.8.8

   # Define the host name templates for master and nodes
   domain_name: "example.com"
   master_hostname: "origin-master"
   node_hostname: "origin-node"

   # Allocate additional space for Docker images
   master_docker_volume_size_gb: 25
   node_docker_volume_size_gb: 25

   # Specify the (initial) number of nodes to deploy
   node_count: 2

   # Add auxiliary services: OpenStack router and internal Docker registry
   deploy_router: False
   deploy_registry: False

   # If using RHEL image, add RHN credentials for RPM installation on VMs
   rhn_username: ""
   rhn_password: ""
   rhn_pool: '' # OPTIONAL
EOF

# retrieve the Heat template (if you haven't yet)
git clone https://github.com/redhat-openstack/openshift-on-openstack.git

# create a stack named 'my-openshift'
heat stack-create my-openshift -t 180 \
  -e openshift_parameters.yaml \
  -f openshift-on-openstack/openshift.yaml

The node_count parameter specifies how many non-master OpenShift nodes you want to deploy. In the example above, we will deploy one master and two nodes.

The templates will report stack completion back to Heat only when the whole OpenShift setup is finished.

To confirm that everything is indeed ready, look for OpenShift has been installed. in the OpenShift master node data in the stack output:

heat output-show my-openshift master_data

Debugging

Sometimes it’s necessary to find out why a stack was not deployed as expected. Debugging helps you find the root cause of the issue.

Multiple Master Nodes

You can deploy OpenShift with multiple master hosts using the 'native' HA method (see https://docs.openshift.org/latest/install_config/install/advanced_install.html#multiple-masters for details) by increasing number of master nodes. This can be done by setting heat parameter master_count heat parameter:

heat stack-create my-openshift \
   -e openshift_parameters.yaml \
   -P master_count=3 \
   -f openshift-on-openstack/openshift.yaml

Three master nodes will be deployed. Console and API URLs point to the loadbalancer server which distributes requests across all three nodes. You can get the URLs from Heat by running heat output-show my-openshift console_url and heat output-show my-openshift api_url.

Select Loadbalancer Type

When deploying multiple master nodes, both access to the nodes and OpenShift router pods (which run on master nodes) have to be loadbalanced. Openshift-on-openstack provides multiple options for setting up loadbalancing:

• Neutron LBaaS - this loadbalancer is used by default. Neutron loadbalancer serviceis used for loadbalancing console/api requests to master nodes. At the moment Openshift router requests are not loadbalanced and an external loadbalancer has to be used for it. This is default option, but can be set explicitly by including -e openshift-on-openstack/env_loadbalancer_neutron.yaml when creating the stack.

• External loadbalancer - a user is expected to set its own loadbalancer both for master nodes and Openshift routers. This is suggested type for production. To select this type include -e openshift-on-openstack/env_loadbalancer_external.yaml when creating the stack.

• Dedicated loadbalancer node - a dedicated node is created during stack creation and HAProxy loadbalancer is configured on it. This type is useful for demo/testing purposes only because HA is not assured for the single loadbalancer To select this type include -e openshift-on-openstack/env_loadbalancer_dedicated.yaml when creating the stack. node.

• None - if only single master node is deployed, it’s possible to skip loadbalancer creation, then all master node requests and Openshift router requests point to the single master node. To select this type include -e openshift-on-openstack/env_loadbalancer_none.yaml when creating the stack.

Select SDN Type

By default, OpenShift is deployed with OpenShift-SDN. When used with Openstack Neutron with GRE or VXLAN tunnels, packets are encapsulated twice which can have an impact on performances. Those Heat templates allow using Flannel instead of openshift-sdn, with the host-gw backend to avoid the double encapsulation. To do so, you need to include the env_flannel.yaml environment file when you create the stack:

heat stack-create my_openshift \
   -e openshift_parameters.yaml \
   -f openshift-on-openstack/openshift.yaml \
   -e openshift-on-openstack/env_flannel.yaml

To use this feature, the Neutron port_security extension driver needs to be enabled. To do so and when using the ML2 driver, edit the file /etc/neutron/plugins/ml2/ml2_conf.ini and make sure it contains the line:

extension_drivers = port_security

Note that this feature is still in experimental mode.

LDAP authentication

You can use an external LDAP server to authenticate OpenShift users. Update parameters in env_ldap.yaml file and include this environment file when you create the stack.

Example of env_ldap.yaml using an Active Directory server:

parameter_defaults:
   ldap_hostname: <ldap hostname>
   ldap_ip: <ip of ldap server>
   ldap_url: ldap://<ldap hostname>:389/CN=Users,DC=example,DC=openshift,DC=com?sAMAccountName
   ldap_bind_dn: CN=Administrator,CN=Users,DC=example,DC=openshift,DC=com?sAMAccountName
   ldap_bind_password: <admin password>

heat stack-create my-openshift \
  -e openshift_parameters.yaml \
  -e openshift-on-openstack/env_ldap.yaml \
  -f openshift-on-openstack/openshift.yaml

Using Custom Yum Respositories

You can set additional Yum repositories on deployed nodes by passing extra_repository_urls parameter which contains list of Yum repository URLs delimited by comma:

heat stack-create my-openshift \
  -e openshift_parameters.yaml \
  -P extra_repository_urls=http://server/my/own/repo1.repo,http://server/my/own/repo2.repo
  -f openshift-on-openstack/openshift.yaml

Using Custom Docker Respositories

You can set additional Docker repositories on deployed nodes by passing extra_docker_repository_urls parameter which contains list of docker repository URLs delimited by comma, if a repository is insecure you can use #insecure suffix for the repository:

heat stack-create my-openshift \
  -e openshift_parameters.yaml \
  -P extra_docker_repository_urls='user.docker.example.com,custom.user.example.com#insecure'
  -f openshift-on-openstack/openshift.yaml

Accessing the Web UI

You can get the URL for the OpenShift Console (the web UI) from Heat by running heat output-show my-openshift master_console_url.

Currently, the UI and the resolution for the public hostnames that will be associated to services running in OpenShift is dependent on the DNS created internally by these Heat templates.

So to access the UI, you can get the DNS IP address by heat output-show my-openshift dns_ip and put nameserver $DNS_IP as the first entry in your /etc/resolv.conf.

We plan to let you supply your own DNS that has the OpenShift cloud domain and all the nodes pre-configured and also to optionally have the UI server bind to its IP address instead of the hostname.

Retrieving the CA certificate

You can retrieve the CA certificate that was generated during the Openshift installation by running

heat output-show --format=raw my-openshift ca_cert > ca.crt
heat output-show --format=raw my-openshift ca_key > ca.key

Current Status

1. The CA certificate used with OpenShift is currently not configurable.

2. The apps cloud domain is hardcoded for now. We need to make this configurable.

Prebuild images

A customize-disk-image script is provided to preinstall Openshift packages.

./customize-disk-image --disk rhel7.2.qcow2 --sm-credentials user:password

The modified image must be uploaded into Glance and used as the server image for the heat stack with the server_image parameter.

Copyright

Copyright 2016 Red Hat, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.