OpenStack Cluster

In order to have a system to provision virtual machines on demand for various use cases (such as for Jenkins slaves) a small OpenStack installation has been deployed on a XenServer pool of four physical machines.

Architecture and Components

OpenStack defines a terminology for its components, see [1]. It distinguishes between three types of nodes which are used to govern the functioning of the system: Controller Node (controller), Compute Node (compute) and Network Node (network). In the deployment guides these are viewed as separate nodes, in order to support high scalability, but given the small size of our cluster, we decided to go ahead with an all-in-one design that incorporates the functionality of all three nodes into one.

OpenStack provides clear separation between its components, and leaves it up to the user which to install and which not, depending on his needs. The components that we selected are:

• keystone - identity service for users and for other openstack components alike
• nova - compute service, governs VM management
• glance - image service, stores and provisions cloud images to nova
• cinder - volume service, provides persistent storage volumes to VMs
• horizon - web-based ui

We decided to drop the use of neutron (networking service) in favour of the nova-network legacy networking service because of its simplicity.

OpenStack and XenServer

The guide at [2] offers some insight into how OpenStack should be set up together with XenServer. Our XenServer instance consist of four physical machines with a pool master. The XenServer pool master is required because of the iSCSI backend used for VM volume provisioning. To avoid race conditions between XenServer machines, only the pool master is allowed to allocate space, and thus only the pool master can create new VMs (a VM can still be created on any XenServer host, but the request will be routed to the pool master). Having a XenServer pool master forced us to deviate from the deployment guide in some cases pointed out bellow. OpenStack is talking to the hypervisor using the xenapi, which is the same api that the XenCenter management system uses.

Before installing the OpenStack packages the following steps have been taken from [3] on the dom0 :

• Storage: we went ahead with the LVM based iSCSI storage, against the advice. (The only limitation noticed so far is the broken cinder deployment)
• XenAPI Plugins: these python plugins have been copied into /etc/xapi.d/plugins/ on the XenServer pool master (This is crutial, because otherwise nova-compute cannot communicate with XenServer)
• VIF Isolation Rules: not yet installed
• 'resize' functionality: not supported yet
• python 2.6 and packeges: not yet installed, stayed with python 2.4

DomU Setup

Our all-in-one OpenStack node is installed in a paravirtualized VM running in the domU of XenServer. Since there was a 64 bit CentOS 6 VM template with 1 VCPU and 1 GB of memory already configured in XenServer, this became the OS of choice for the all-in-one node. In retrospective, it could have used a more powerful VM (at least with more memory, because the 1GB is nearly all reserved by nova).

Regarding the networking part, we wanted to follow the FlatDHCP[4] design, where a dedicated VM network is created with the openstack node serving as a gateway and DHCP server for it. Since the actual host machines only have two network interfaces (one for management and one for public traffic) we created a dedicated VLAN for the openstack VM network. The new, simplified, version of the network configuration on the host machines looks something like this:

 eth0 (management network)
   |
   --- xenbr0 (virtual bridge)
   |
   --- xapi5 (iscsi network)

 eth1
   |
   --- xenbr1 (virtual bridge)  
   |
   --- xapi17 (openstack VM network) *new*
   |
   --- xapi14 (tw public network) 

Note that the routers behind the host machines have to be configured to forward packets in the newly created VLAN. Next, we create virtual network interfaces (VIF) for the openstack node to every required network. You can create a new VIF with:

xe vif-create vm-uuid=<vm id> network-uuid=<network id> device=0 mac=random 
xe vif-plug <vif id>

If you cannot see the newly created interfaces in the VM, restart the VM. In the end, there should be 4 network interfaces present in the openstack node as follows:

 # OpenStack communicates with XenServer over this network
 eth0 (management network)                                                          -> plugged in xenbr0
 
 # to the internet
 eth1 (public network)                                                              -> plugged in xapi14

 # This interface is configured with a static ip, 
 # the first ip chosen from 172.22.192.0/18 (VM network)
 # Acts as a DHCP server and gateway for the internet for the VM network
 eth2 (VM network) 172.22.192.1 (static ip)                                         -> plugged in xapi17	
 
 # While experimenting with cinder we needed access into our iSCSI backend. 
 # We ended up needing it for the glance backend
 eth3 (iscsi network)                                                               -> plugged in xapi5		

Make sure to configure eth1 to have access to the internet. In our case we dedicated a public ip for it.

In order to support the full functionality of this paravirtualized system first the XenServer Tools have to be installed on it [5]. We still had the xen-tools.iso lying around on the XenServer hosts in the list of VDIs (Virtual Disk Image), so it just needed to be mounted on the newly created VM:

xe vbd-create vm-uuid=<vm id> device=xvdd vdi-uuid=<xen-tools.iso vdi id> bootable=false mode=RO type=Disk 
xe vbd-plug uuid=<new vbd id>

and once logged into the VM

mnt /dev/xvdd /mnt
./mnt/Linux/install.sh

Deploying OpenStack

Before starting the installation make sure to update the OS to its newest version, and set up ntp. Installing the epel packages is also recommended. OpenStack can be installed and set up in a manual[6] or a more automated way. No matter which method you choose, there is a dependency to look out for which might or might not be mentioned in the guide you follow. This dependency is a python library called XenAPI.py, and it's needed to support the XenServer hypervisor. You can get it with:

yum install pip
pip install xenapi

We chose an automated deployment method using PackStack[7]. You can use the packstack command to generate an answer-file (packstack config file) and modify it to your needs before the installation. This takes care of generating most of the required setting and passwords for every component. The important parameters to change are the ones deciding whether to install a service or not, the *_HOST parameters pointing at where to install services and your choice of database (MariaDB for us) and message queue service (qpid for CentOS). Do not lose or post the answers-file, since it contains the passwords for all the openstack services.

Note! You should take special consideration when deploying a multi-node system. The OpenStack components are using the message queue to communicate with each other which is set up insecurely (without ssl) by default. This is ok in an all-in-one setup, because messages are not leaving the machine, but in other cases the message queue security has to be set up.

packstack --gen-answer-file my_answers.txt

#Modification made to the my_answers.txt file
#The default is to install every service, so here we mark unwanted services with 'n'
> CONFIG_NEUTRON_INSTALL=n
> CONFIG_SWIFT_INSTALL=n
> CONFIG_CEILOMETER_INSTALL=n
> CONFIG_NAGIOS_INSTALL=n
> CONFIG_DEBUG_MODE=y
> CONFIG_CONTROLLER_HOST=<public ip>
> CONFIG_COMPUTE_HOSTS=<public ip>
> CONFIG_NETWORK_HOSTS=<public ip>
> CONFIG_STORAGE_HOST=<public ip>
> CONFIG_RH_OPTIONAL=n
> CONFIG_AMQP_BACKEND=qpid
> CONFIG_AMQP_HOST=<public ip>
> CONFIG_MARIADB_HOST=<public ip>
> CONFIG_NOVA_COMPUTE_PRIVIF=eth2
> CONFIG_NOVA_NETWORK_PUBIF=eth1
> CONFIG_NOVA_NETWORK_PRIVIF=eth0
> CONFIG_NOVA_NETWORK_FIXEDRANGE=<ip range> (172.22.192.0/18) 
> CONFIG_NOVA_NETWORK_DEFAULTFLOATINGPOOL=novapool
> CONFIG_NOVA_NETWORK_VLAN_START=
> CONFIG_CINDER_VOLUMES_CREATE=n

packstack --answer-file=my-answers.txt

Grab a cup of coffee. This should set up most of the required software to run OpenStack. The packstack installation also creates two openstack user account: 'admin' and 'demo'. It also creates two files called 'keystonerc_admin' and 'keystonerc_demo' which can be easily sourced to become one user or the other. You can use these useful commands to find out more about your installation.

Note! Packstack installs the libvirt package, which is not really needed if you're setting up OpenStack with XenServer via the xenapi. To avoid confusion we removed it.

openstack-status
openstack-service [list|start|stop]

Configuring OpenStack

Glance

Glance comes configured almost completely, the only setting that you might want to change is the backend used for storing the images. This defaults to a local filesystem store, which tends to quickly fill up the local disk, not a good idea.

We experimented with using cinder as a backend storage for glance, but after numerous tries this was abandoned. We ended up using the filesystem store as a backend after all, but using a bigger logical volume via iSCSI. To manage iSCSI we downloaded the Linux SCSI target framework (tgt)[8]. Make sure to allow connections on the iSCSI port 3260 (if you installed cinder, it is likely that it added the exception for you already in the iptables). The way to set this up to work as a glance backend is as follows[9]:

service tgtd start
tgtadm --lld iscsi --op new --mode target --tid 1 -T prefix.example:server
#you ca verify if the connection works with
tgtadm --lld iscsi --op show --mode target
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sda

#create logical volume to mount
pvcreate /dev/sda
vgcreate vg_glance /dev/sda
lvcreate -V size -n lv_glance vg_glance 

mount /dev/vg_glance/lv_galnce /mnt

Now all that is left to be done is change the following variable in /etc/glance/glance-api.conf

filesystem_store_datadir=/mnt

Nova

The main nova configuration file can be found in /etc/nova/nova.conf.

nova-compute

The important bits of this configuration are the ones regarding XenServer, and are as followed:

compute_driver=xenapi.XenAPIDriver
xenapi_connection_url=https://<XenServer pool master host ip on management network>
xenapi_connection=_username=root
xenapi_connection_password=password

Another important flag is:

sr_matching_filter=default-sr:true

This flag defaults to 'default-sr:true', which is what we need in our case. This flag will make OpenStack provision disk space for its VMs through XenServer, using the SR (Storage Repository) configured as default. In our setup the default SR in XenCenter is an 'lvmoiscsi' type backend.

nova-network

After numerous attempts we finally managed to set up the FlatDHCP network using the VM network created earlier. The difficulty lies in attaching the newly created VMs to the right bridge in dom0. Note, that OpenStack is not aware of the VLAN configuration set up earlier, because it's transparent to it. The setup is as follows:

my_ip=<public ip from eth1>

network_driver=nova.network.linux_net
network_manager=nova.network.manager.FlatDHCPManager
public_interface=eth1
dns_server=<valid dns server>
force_snat_range=0.0.0.0/0
send_arp_for_ha=True
flat_network_bridge=xapi17
flat_injected=false
flat_interface=eth2
fixed_range=172.22.192.0/18
force_dhcp_release=True 

It's important to choose the right flat_network_bridge because this determines to which bridge the new VMs are getting connected to. IP injection is not required because we are using a DHCP server with ips from the provided fixed_range. Traffic from the VM network is going to be snat-ed through the public_interface.

Note! Do not set multi_host to true, unless you really have multiple network hosts.

Note! Do not set share_dhcp_address to true in an all-in-one setup! This flag makes it possible to have multiple gateways with seemingly the same ip address for high availability. As a side-effect it also adds some rules into ebtables that filter out all traffic coming from the VMs. This lead to a scenario where VM instances could ping each other, but not the gateway.

vnc console

If you want OpenStack to provide vnc consoles through the browser for you:

novncproxy_base_url=http://<public_ip>:6080/vnc_auto.html
xvpvncproxy_base_url=http://<public_ip>:6081/console

vncserver_listen=<XenServer_ip>
vncserver_proxyclient_address=<XenServer_ip>

vnc_enabled=True
vnc_keymap=en-us

xvpvncproxy_host=<public_ip>
xvpvncproxy_port=6081
novncproxy_host=<public_ip>
novncproxy_port=6080

This will enable nova to generate an address containing a token with which you can get console access to an instance. Encryption, however is not enabled on the connection by default. Note, that most cloud images do not come with a predefined password for their users, in which case you cannot login using the console, unless you set the password beforehand.

Cinder

Only experimental. Coming Soon!

Horizon

Only experimental. Coming Soon!

Operating OpenStack

Commands issued to OpenStack using the API have to be authenticated with right credentials. By default, the OpenStack installer deploys two user accounts 'demo' and 'admin', with their corresponding rc files 'keystonerc_demo' and 'keystonerc_admin'. If you do not have these files, you can create them according to the following template:

export OS_USERNAME=demo
export OS_TENANT_NAME=demo
export OS_PASSWORD=<demo password>
export OS_AUTH_URL=http://<ip>:5000/v2.0/
export PS1='[\u@\h \W(keystone_demo)]\$ '

Starting VMs

Source the 'keystonerc_demo' file, so that all the following commands get authenticated.

Adding images

In order to start a VM, first you have to make sure you have images configured in glance that can be used to boot. We only experimented with specific cloud images provided by CentOS and Fedora. These images can simply be imported into glance with:

glance image-create --name=<name> --disk-format=raw --container-format=bare --location <image url>

Note that glance will save a local copy of the image, so it's advised to have a backend configured that is large enough. The 'container-format' is a deprecated option still required by this release [10]. The 'disk-format' which worked for us was 'raw'. Many images come in the compressed qcow2 format, but unfortunately we could not boot these because of XenServer incompatibilities. Images can be converted from qcow2 to raw format using the 'qemu-img' tool. You could also, in theory, build your own bootable images but this is not yet tested out by us.

Create Network

Next, you need a predefined network to which the new VM will be attached. We call this the 'tenant-network'. To create a network from the command line you need the admin privileges, so source 'keystonerc_admin'. You can create a new network with:

nova network-create --fixed-range-v4 172.22.192.0/18 --project-id <projectID> tenant-network

This command will use the values of the network configuration defined here as a default, which can be overridden with direct options. OpenStack projects/tenants are a form of user grouping. By default, the demo account should be part of a demo project. You can get the id of this using:

keystone tenant-list

The network created by this command will only have the fixed ips set up. The floating ip setup is still future plan.

Note! Before you create the network you should configure a static IP for the interface that you're going to use for the tenant-network (flat_interface in nova.config). This is the IP that the tenant network will take as the gateway address.

Note! Whenever you're trying to recreate a network with the same ip range, first you have to tear down the old one. Existing networks can be deleted by disassociating them from their projects first, as such

nova-manage project scrub --project <projectID>
nova net-delete <networkID>

To avoid conflicts and confusion between separate networks you should also stop the running 'dnsmasq' processes and remove the existing network bridges created by OpenStack using 'brctl delbr <bridge>'.

Booting VM

Set up a keypair to use for ssh interaction and register it with OpenStack. Now you can switch back to the demo user to start a VM:

nova keypair-add --pub-key <pub-key> demo-key
nova boot --flavor m1.small --image centOS-7-x86_64 --nic net-id=<network id> --security-group default --key-name demo-key centos7--instance