Xen on CentOS 5

Xen Virtualization

This description worked for us, no guarantees that it will for you. Any comments welcome. In the following we describe how to use Xen-Virtualisation for Quattor managed grid middleware (gLite) servers. Quattor is not necessarily needed, any other fabric management tool or simple scripts will do the same. In particular we had an existing Quattor environment and added the virtualisation framework. In the following the steps done by Quattor are marked with a Q.

1. Hardware: PE-1950 with two 500G hard diskes, PE-2950 with four 500G hard disks
2. Software:

• Operating system host (dom0): CentOS-5.x, x.ge.2, x86-64
• Virtualisation Method: Xen, stock CentOS xen kernels
• Operating system guest (domU): CentOS-4.x, i386 (ParaVirtualisedMachine -guest, PVM), CentOS-3.x, i386 (Hardware (assisted) Virtual Machine, HVM
• Fabric Management: Quattor
• Network installation tools: Kickstart, PXE, pypxeboot
• You have IPs for the dom0s and domUs available.

Installation (PXE) Server

We assume you have an installation server with Preboot eXecution Environment (PXE) available.

Images, Kernels

For the dom0 network installation no additional steps are necessary as well as for the HVM domUs, for the PVM domUs you need to have the directories with the images. Example, for network installation of CentOS-4.6 i386 domUs you could do the following:

mkdir -p <base directory>/xen-domU-c4.6/i386

this directory should have the files initrd.img and vmlinuz, they are available from here

DHCP

Add the MAC adresses to /etc/dhcp.conf. For the dom-0 take the one from the network card. For the domU you have to choose a MAC-address. In order to avoid clashes (multiple usage of the same MAC) you can do the following: the first three hex numbers should be 00:16:3e (reserved for xen domUs), the last three can be for example the hex representation of the last three numbers of the IP-address.

Example: 
IP of the DomU is 192.168.1.2
MAC:  00:16:3e:a8:01:02, the result of `printf 00:16:3e:%02x:%02x:%02x"\n" 168 1 2` command

Create PXE Menus

At our site this is created with Quattor, resulting in file like:

 <base directory>/pxelinux.cfg/<hostname.cfg>

default Scientific Linux centos4_i386
   label Scientific Linux centos4_i386
   kernel centos-xen-4/i386/vmlinuz
   append ramdisk=32768 initrd=centos-xen-4/i386/initrd.img ks=http://<host with kickstart files>/<path to ks files>/<hostname>.ks ksdevice=eth0

For all hosts there should be a file like:

<base directory>/pxelinux.cfg/localboot.cfg
default Localdisk
label Localdisk
localboot 0

If you want to install/localboot a host create the links accordingly by hand or with your fabric management tool. At our site this is also done with the Quattor.

Centos-5.2-x86_64 xen-dom0 Installation

The following works for us, there is no guarantee that it will also work for you. All settings might change in future.

• Hardware: Dell PE-1950, HD: 2X500G Soft-RAID-1, PE-2950, HD: 4X500G Hardware-RAID-1-0.
• Network: 2 Interfaces, only one cabled

Installation of Dom-0

Dom0 (Host) needs < 10G for its system, tmp is not extensively used nor is var (besides /var/lib/xen/save see below). To prevent the system to get unresponsive due to full / fs we could give 4GB /tmp and /var each. These values can be smaller if we have hardware which has some disk space restrictions. The saved states of the domUs, created for example during a reboot of the dom0 will be stored in /var/lib/xen/save, this might get filled and is on a separate partition.

Depending on the hardware we have to set up a soft RAID-1 (2 disks) or a hardware RAID-1-0 where we treat the (4-disks) as one disk within the kickstart file.

Filesystem layout Soft-RAID-1

Principle layout:

• RAID-1
• Primary Partitions

/boot 512MB
/swap 4096MB

• LVM

Volumegroup System, Size: rest of available space
I System-Volumes-Dom0 [/, /tmp] fs: ext3
II DomU-lvms added when needed.

To have the possibility to reinstall the dom0 with the fabric management tool (uses pxeboot/kickstart) we postprocess the ks file after first installation. See appendix (ks2reinst.sh).

clearpart --all --initlabel
part raid.01 --size=512   --ondisk=sda --asprimary
part raid.02 --size=8192 --ondisk=sda --asprimary
part raid.03 --size=24576  --ondisk=sda
part raid.04 --size=1 --ondisk=sda --grow
part raid.11 --size=512   --ondisk=sdb --asprimary
part raid.12 --size=8192 --ondisk=sdb --asprimary
part raid.13 --size=24576  --ondisk=sdb
part raid.14 --size=1 --ondisk=sdb --grow
raid /boot     --level=RAID1 --device=md0 --fstype=ext2 raid.01 raid.11
raid /         --level=RAID1 --device=md1 --fstype=ext3 raid.02 raid.12
raid pv.01 --fstype "physical volume (LVM)" --level=RAID1 --device=md2 raid.03 raid.13
raid pv.02 --fstype "physical volume (LVM)" --level=RAID1 --device=md3 raid.04 raid.14
volgroup systemvg       pv.01
volgroup xenvg          pv.02
logvol swap --fstype=swap       --vgname=systemvg       --size=2048     --name=sysswap
logvol /tmp --fstype=ext3       --vgname=systemvg       --size=4096     --name=systmp
logvol /var --fstype=ext3       --vgname=systemvg       --size=4096     --name=sysvar
logvol /var/lib/xen/save --fstype=ext3       --vgname=systemvg       --size=10240     --name=sysxensave

Filesystem layout Harware-RAID-10

Here the disks appear as one and the respective section in the kickstart file gets a lot simpler:

clearpart --all --initlabel
part /boot --size=256 --asprimary --fstype=ext2 --ondisk=sda
part pv.01 --size=73728 --asprimary --fstype=ext3 --ondisk=sda
part pv.02 --size=1 --grow --ondisk=sda
volgroup systemvg pv.01
volgroup xenvg pv.02
logvol / --fstype ext3 --size=8192     --name=sysroot  --vgname=systemvg
logvol /var/lib/xen/save --fstype ext3 --size=16384     --name=sysxensave  --vgname=systemvg
logvol /var --fstype ext3 --size=4096     --name=sysvar  --vgname=systemvg
logvol /tmp --fstype ext3 --size=2048     --name=systmp  --vgname=systemvg
logvol swap --fstype swap --size=4096     --name=sysswap  --vgname=systemvg

For reinstalling the host OS and keep the domUs you could try:

# make sure to have deleted or put a comment on the line saying: clearpart --all --initlabel:
#part /boot --size=256 --asprimary --fstype=ext2 --ondisk=sda
#part pv.01 --size=73728 --asprimary --fstype=ext3 --ondisk=sda
#part pv.02 --size=1 --grow --ondisk=sda
volgroup systemvg pv.01 --useexisting
volgroup xenvg pv.02 --noformat
logvol / --fstype ext3 --size=8192     --name=sysroot  --vgname=systemvg --useexisting
logvol /var/lib/xen/save --fstype ext3 --size=16384     --name=sysxensave  --vgname=systemvg --useexisting
logvol /var --fstype ext3 --size=4096     --name=sysvar  --vgname=systemvg --useexisting
logvol /tmp --fstype ext3 --size=2048     --name=systmp  --vgname=systemvg --useexisting
logvol swap --fstype swap --size=4096     --name=sysswap  --vgname=systemvg --useexisting

Software

We install stock CentOS packages, the following excerpt from the kickstart file should be sufficient.

%packages --resolvedeps --ignoremissing
openssh
openssh-server
wget
perl-libnet
perl-MIME-Base64
perl-URI
perl-Digest-MD5
perl-libwww-perl
perl-XML-Parser
perl-DBI
perl-Crypt-SSLeay
lsof
curl
xen
kernel-xen

== /tftpboot/pxelinux.cfg/netinstall-CentOS-4-i386-xenU==

should read like

# You need the xen-kernels for the DomU, get them from 
default centos4.6_32_xenU
label centos4.6_32_xenU
 kernel vmlinuz-CentOS-4.6_i386_xenU
 append initrd=initrd.img-CentOS-4.6_i386_xenU

Create kickstart file for DomU

Addidtional Software needed to pxeboot domUs

See S. Childs

Xen dom-0 Configuration

Dom0 grub

For performance its best to limit the Memory of the dom0 to 512 MB and to prevent it from ballooning (i.e.dom-0 takes all available memory), and to restrict the dom0 to one cpu. Use some monitoring to make sure that these settings are sufficient.

Therefore modify /boot/grub/menu.lst, the important entries are in bold:

title CentOS (2.*xen)
       root (hd0,0)
       kernel /xen.gz-2.6.18-92.1.6.el5 dom0_mem=512M
       module /vmlinuz-2.6.18-92.1.6.el5xen ro root=/dev/systemvg/sysroot nosmp
       module /initrd-2.6.18-92.1.6.el5xen.img

xend configuration

Set in /etc/xen/xend-config.sxp:

(dom0-min-mem 0)

Installation of a domU using pypxeboot

Create domU config file

You'll have to provide a MAC-address. Calculate last 3 hex numbers from IP (example 192.168.96.43)with: printf ":%02x:%02x:%02x\n" {192,168,43} this must not clash with the MAC-Address of other domUs. create domU config file

<dom0>:/etc/xen/ullr
name ='ullr'
memory ='1024'
bootloader="/usr/bin/pypxeboot"
vif=['mac=00:16:3e:XX:XX:XX']
bootargs=vif[0]
disk = [ "phy:xenvg/<DOMU-NAME>,xvda,w" ]
root="/dev/xvda1 ro"
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Create logical volume for this domU

lvcreate -L XG -n <DOMU-NAME xenvg

Create DomU

on dom0:

xm create <DOMU-NAME>