GSP Virtualisation with Xen

The Grid Server Park machines (general services used for Nikhef and BiG Grid) is going to be run with a centrally managed and controlled virtualisation environment. After the testing and evaluation period it is likely (but not yet certain) that the Open Source Xen Cloud Platform (XCP, version 1.5beta) is going to be chosen to run this infrastructure. The aim is to bring all systems under XCP control, managed in a set of clusters:

General information

• http://wiki.xen.org/xenwiki/XCP/XenServer_Feature_Matrix
• http://xen.org/download/xcp/index_1.5.0.html

Hardware

Cluster	qty	system type	VM server hostnames	current master
Piet	16 systems	M610: 12 cores, 24 SMT threads, 96 GiB, 2x600GB 10k SAS, dual 8G FC, dual 1GbE + dual 10GbE	vms-piet-*.inst.ipmi.nikhef.nl
Generic	8 systems	PE2950: 8 cores, 24 GiB, 4x500GB 7k2 SATA, dual 1GbE	vms-gen-*.inst.ipmi.nikhef.nl	vms-gen-05
BL0	5 systems	M610: 8 cores, 32 GiB RAM, 2x300GB 10k SAS, dual 1GbE (+dual 8G FC)	vms-bl0-*.inst.ipmi.nikhef.nl
Security	2 systems	PE2950: 8 cores, 24 GiB, 4x500GB 7k2 SATA, dual 1GbE	vms-sec-*.inst.ipmi.nikhef.nl

Troubleshooting

Dead VM cluster master

Disable HA if it is enabled. We don't use HA (it's not part of XCP), but its harmless to try anyway. Login to any other node in the cluster and type

xe pool-ha-disable

and list all hosts in the pool to find the UUID of a slave host you want to become the new master

xe host-list

and make it happen with

xe pool-designate-new-master host-uuid=UUID

Now since the master is dead (that's why we started in the first place), we make it explicit to the old slave to start working:

xe pool-emergency-transition-to-master

and connect to the slaves again

xe pool-recover-slaves

(from http://blog.carlosgomez.net/2010/01/citrix-xen-server-changing-pool-master.html)

Incinerated VM servers

If the VM server is dead and will never come back, you can 'forget' the host from the CLI (on any server in the pool):

xe host-forget uuid=UUID

but if there are still running VMs assigned to it, it will refuse for forget the server. Remove the VMs first by forcing them off using

xe vm-reset-powerstate uuid=UUID --force

(from http://forums.citrix.com/thread.jspa?threadID=250603)

VMs on a dead server

You can declare a VM 'shut down' via the CLI on a dead VM server:

xe vm-reset-powerstate uuid=UUID --force

(from http://forums.citrix.com/thread.jspa?threadID=250603)