OpenStack Cluster

In order to have a system to provision virtual machines on demand for various use cases (such as for Jenkins slaves) a small OpenStack installation has been deployed on a XenServer pool of four physical machines.

Architecture and Components

OpenStack defines a terminology for its components, see [1]. It distinguishes between three types of nodes which are used to govern the functioning of the system: Controller Node (controller), Compute Node (compute) and Network Node (network). In the deplyment guides these are viewed as separate nodes, in order to support high scalabilty, but given the small size of our cluster, we decided to go ahead with an all-in-one design that incorporates the functionality of all three nodes into one.

OpenStack provides clear separation between its components, and leaves it up to the user which to install and which not, depending on his needs. The components that we selected are:

• keystone - identity service for users and for other openstack components alike
• nova - compute service, governs VM management
• glance - image service, stores and provisions cloud images to nova
• cinder - volume service, provides persistent storage volumes to VMs
• horizon - web-based ui

We decided to drop the use of neutron (networking service) in favor of the nova-network legacy neworking service because of its simplicity.

OpenStack and XenServer

The guide at [2] offers some insight into how OpenStack should be set up together with XenServer. Our XenServer instance consist of four physical machines with a pool master. The XenServer pool master is required because of the iSCSI backend used for VM volume provisioning. To avoid race conditions between XenServer machines, only the pool master is allowed to allocate space, and thus only the pool master can create new VMs (a VM can still be created on any XenServer host, but the request will be routed to the pool master). Having a XenServer pool master froced us to deviate from the deplyment guide in some cases pointed out bellow. OpenStack is talking to the hypervisor using the xenapi, which is the same api that the XenCenter management system uses.

Before installing the OpenStack packages the following steps have been taken from [3] on the dom0 :

• Storage: we went ahead with the LVM based iSCSI storage, against the advice. (The only limitation noticed so far is the broken cinder deployment)
• XenAPI Plugins: these python plugins have been copied into /etc/xapi.d/plugins/ on the XenServer pool master (This is crutial, because otherwise nova-compute cannot communicate with XenServer)
• VIF Isolation Rules: not yet installed
• 'resize' functionality: not supported yet
• python 2.6 and packeges: not yet installed, stayed with python 2.4

DomU Setup

Our all-in-one OpenStack node is installed in a paravirtualized VM running in the domU of XenServer. Since there was a 64 bit CentOS 6 VM template with 1 VCPU and 1 GB of memory already configured in XenServer, this became the OS of choice for the all-in-one node. In retrospective, it could have used a more powerful VM (at least with more memory, because the 1GB is nearly all reserved by nova).

Regarding the networking part, we wanted to follow the FlatDHCP[4] design, where a dedicated VM network is created with the openstack node serving as a gateway and DHCP server for it. Since the actual host machines only have two network interfaces (one for management and one for public traffic) we created a dedicated VLAN for the openstack VM network. The new, simplified, version of the network configuration on the host machines looks something like this:

 eth0 (management network)
   |
   --- xenbr0 (virtual bridge)
   |
   --- xapi5 (iscsi network)

 eth1
   |
   --- xenbr1 (virtual bridge)  
   |
   --- xapi17 (openstack VM network) *new*
   |
   --- xapi14 (tw public network) 

Note that the routers behind the host machines have to be configured to forward packets in the newly created VLAN. Next, we create virtual network interfaces (VIF) for the openstack node to every required network. You can create a new VIF with:

xe vif-create vm-uuid=<vm id> network-uuid=<network id> device=0 mac=random 
xe vif-plug <vif id>

If you cannot see the newly created interfaces in the VM, restart the VM. In the end, there should be 4 network interfaces present in the openstack node as follows:

 # OpenStack communicates with XenServer over this network
 eth0 (management network)                                                          -> plugged in xenbr0
 
 # to the internet
 eth1 (public network)                                                              -> plugged in xapi14

 # This interface is configured with a static ip, 
 # the first ip chosen from 172.22.192.0/18 (VM network)
 # Acts as a DHCP server and gateway for the internet for the VM network
 eth2 (VM network) 172.22.192.1 (static ip)                                         -> plugged in xapi17	
 
 # (optional)
 # While experimenting with cinder we needed access into our iSCSI backend. 
 eth3 (iscsi network)                                                               -> plugged in xapi5		

Make sure to configure eth1 to have access to the internet. In our case we dedicated a public ip for it.

In order to support the full functionality of this paravirtualized system first the XenServer Tools have to be installed on it [5]. We still had the xen-tools.iso lying around on the XenServer hosts in the list of VDIs (Virtual Disk Image), so it just needed to be mounted on the newly created VM:

xe vbd-create vm-uuid=<vm id> device=xvdd vdi-uuid=<xen-tools.iso vdi id> bootable=false mode=RO type=Disk 
xe vbd-plug uuid=<new vbd id>

and once logged into the VM

mnt /dev/xvdd /mnt
./mnt/Linux/install.sh

Deploying OpenStack

Before starting the installation make sure to update the OS to its newest version, and set up ntp. Installing the epel packages is also recommended. OpenStack can be installed and set up in a manual[6] or a more automated way. No matter which method you choose, there is a dependency to look out for which might or might not be mentioned in the guide you follow. This dependency is a python library called XenAPI.py, and it's needed to support the XenServer hypervisor. You can get it with:

yum install pip
pip install xenapi

We chose an automated deployment method using PackStack[7]. You can use the packstack command to generate an answer-file (packstack config file) and modify it to your needs before the installation. This takes care of generating most of the required setting and passwords for every component. The important parameters to change are the ones deciding whether to install a service or not, the *_HOST parameters pointing at where to install services and your choice of database (MariaDB for us) and message queue service (qpid for CentOS). Do not lose or post the answers-file, since it contains the passwords for all the openstack services.

Note! You should take special consideration when deploying a multi-node system. The OpenStack components are using the message queue to communicate with each other which is set up insecurely (without ssl) by default. This is ok in an all-in-one setup, because messages are not leaving the machine, but in other cases the message queue security has to be set up.

packstack --gen-answer-file my_answers.txt

#Modification made to the my_answers.txt file
#The default is to install every service, so here we mark unwanted services with 'n'
> CONFIG_NEUTRON_INSTALL=n
> CONFIG_SWIFT_INSTALL=n
> CONFIG_CEILOMETER_INSTALL=n
> CONFIG_NAGIOS_INSTALL=n
> CONFIG_DEBUG_MODE=y
> CONFIG_CONTROLLER_HOST=<public ip>
> CONFIG_COMPUTE_HOSTS=<public ip>
> CONFIG_NETWORK_HOSTS=<public ip>
> CONFIG_STORAGE_HOST=<public ip>
> CONFIG_RH_OPTIONAL=n
> CONFIG_AMQP_BACKEND=qpid
> CONFIG_AMQP_HOST=<public ip>
> CONFIG_MARIADB_HOST=<public ip>
> CONFIG_NOVA_COMPUTE_PRIVIF=eth2
> CONFIG_NOVA_NETWORK_PUBIF=eth1
> CONFIG_NOVA_NETWORK_PRIVIF=eth0
> CONFIG_NOVA_NETWORK_FIXEDRANGE=<ip range> (172.22.192.0/18) 
> CONFIG_NOVA_NETWORK_DEFAULTFLOATINGPOOL=novapool
> CONFIG_NOVA_NETWORK_VLAN_START=
> CONFIG_CINDER_VOLUMES_CREATE=n

packstack --answer-file=my-answers.txt

Grab a cup of coffee. This should set up most of the required software to run OpenStack. The packstack installation also creates to openstack user account: 'admin' and 'demo'. It also creates two files called 'keystonerc_admin' and 'keystonerc_demo' which can be easily sourced to become one user or the other. You can use these useful commands to find out more about your installation.

Note! Packstack installs the libvirt, which is not really needed if you're setting up OpenStack with XenServer via the xenapi. To avoid confusion we removed it.

openstack-status
openstack-service [list|start|stop]

Configuring OpenStack

Glance

Glance comes configured almost completely, the only setting that you might want to change is the backend used for storing the images. This defaults to a local filesystem store, which tends to quickly fill up the local disk, not a good idea.

We experimented with using cinder as a backend storage for glance, but after numerous tries this was abandoned.

We ended up using the filesystem store as a backend after all, but using a bigger logical volume via iSCSI. To manage iSCSI we downloaded the Linux SCSI target framework (tgt)[8]. Make sure to allow connections on the iSCSI port 3260 (if you installed cinder, it is likely that it added the exception for you already). The way to set this up to work as a glance backend is as follows[9]:

service tgtd start
tgtadm --lld iscsi --op new --mode target --tid 1 -T prefix.example:server
#you ca verify if the connection works with
tgtadm --lld iscsi --op show --mode target
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sda

#create logical volume to mount
pvcreate /dev/sda
vgcreate vg_glance /dev/sda
lvcreate -V size -n lv_glance vg_glance 

mount /dev/vg_glance/lv_galnce /mnt

Now all that is left to be done is change the following variable in /etc/glance/glance-api.conf

filesystem_store_datadir=/mnt

Configuring Nova

The main nova configuration file can be found in /etc/nova/nova.conf.

nova-compute

The important bits of this configuration are the ones regarding XenServer, and are as followed:

compute_driver=xenapi.XenAPIDriver
xenapi_connection_url=https://<XenServer pool master host ip on management network>
xenapi_connection=_username=root
xenapi_connection_password=password

nova-network

After numerous attempts we finally managed to set up the FlatDHCP network using the VM network created earlier. The difficulty lies in attaching the newly created VMs to the right bridge in dom0. Note, that OpenStack is not aware of the VLAN configuration set up earlier. The setup is as follows:

my_ip=<public ip from eth1>

network_driver=nova.network.linux_net
network_manager=nova.network.manager.FlatDHCPManager
public_interface=eth1
dns_server=<valid dns server>
force_snat_range=0.0.0.0/0
send_arp_for_ha=True
flat_network_bridge=xapi17
flat_injected=false
flat_interface=eth2
fixed_range=172.22.192.0/18
force_dhcp_release=True 

It's important to choose the right flat_network_bridge because this determines to which bridge the new VMs are getting connected to. IP injection is not required because we are using a DHCP server with ips from the provided fixed_range. Traffic from the VM network is going to be snat-ed through the public_interface.

Note! Do not set multi_host to true, unless you really have multiple network hosts.

Note! Do not set share_dhcp_address to true in an all-in-one setup! This flag makes it possible to have multiple gateways with seemingly the same ip address for high availability. As a side-effect it also adds some rules into ebtables that filter out all traffic coming from the VMs.

vnc console

If you want OpenStack to provide vnc consoles through the browser for you:

novncproxy_base_url=http://<public_ip>:6080/vnc_auto.html
xvpvncproxy_base_url=http://<public_ip>:6081/console

vncserver_listen=<XenServer_ip>
vncserver_proxyclient_address=<XenServer_ip>

vnc_enabled=True
vnc_keymap=en-us

xvpvncproxy_host=<public_ip>
xvpvncproxy_port=6081
novncproxy_host=<public_ip>
novncproxy_port=6080

This will enable nova to generate an address containing a token with which you can get console access to an instance. Note, that most cloud images do not come with a predefined password for their users, in which case you cannot login using the console, unless you set the password beforehand.