Xen on CentOS 5

Xen Virtualization

This description worked for us by part it is still work in progress. Any comments welcome. In the following we describe how to use Xen-Virtualisation for Quattor managed grid middleware (gLite) servers. The domUs (guests) are on local disks in logical volumes. Quattor is not necessarily needed, any other fabric management tool or simple scripts will do the same. In particular we had an existing Quattor environment and added the virtualisation framework, so you'll find here no information on howto manage system, glite-service-installation/configuration and host-monitoring, etc. with Quattor, this can be found elsewhere in this wiki. In the following Quattor manages "only" the dhcp.conf, kickstart- and pxe configuration files.

Hardware:

PE-1950 with two 500G hard diskes, PE-2950 with four 500G hard disks.

Software:

• Operating system host (dom0): CentOS-5.x, x.ge.2, x86-64
• Virtualisation Method: Xen, stock CentOS xen kernels
• Operating system guest (domU): CentOS-4.x, i386 (ParaVirtualisedMachine -guest, PVM), CentOS-3.x, i386 (Hardware (assisted) Virtual Machine, HVM)
• LVM
• Fabric Management: Quattor
• Network installation tools: Kickstart, PXE, pypxeboot
• You have IPs for the dom0s and domUs available.

Installation (PXE) Server

We assume you have an installation server with Preboot eXecution Environment (PXE) available.

Images, Kernels

For the dom0 network installation no additional steps are necessary as well as for the HVM domUs, for the PVM domUs you need to have the directories with the images. Example, for network installation of CentOS-4.6 i386 domUs you could do the following:

mkdir -p <base directory>/xen-domU-c4.6/i386

this directory should have the files initrd.img and vmlinuz, they are available from here

DHCP

Add the MAC adresses to /etc/dhcp.conf. For the dom-0 take the one from the network card. For the domU you have to choose a MAC-address. In order to avoid clashes (multiple usage of the same MAC) you can do the following: the first three hex numbers should be 00:16:3e (reserved for xen domUs), the last three can be for example the hex representation of the last three numbers of the IP-address.

Example: 
IP of the DomU is 192.168.1.2
MAC:  00:16:3e:a8:01:02, the result of `printf 00:16:3e:%02x:%02x:%02x"\n" 168 1 2` command

Create PXE Menus

At our site this is created with Quattor, for a CentOS-4 i386 domU the cfg file looks like:

 <base directory>/pxelinux.cfg/<hostname.cfg>

default Scientific Linux centos4_i386
   label Scientific Linux centos4_i386
   kernel xen-domU-c4.6/i386/vmlinuz
   append ramdisk=32768 initrd=xen-domU-c4.6/i386/initrd.img \
   ks=http://<Installation server>/<path to ks files>/<hostname>.ks ksdevice=eth0

For all hosts there should be a file like:

<base directory>/pxelinux.cfg/localboot.cfg
default Localdisk
label Localdisk
localboot 0

If you want to install/localboot a host create the links accordingly by hand or with your fabric management tool. At our site this is also done with Quattor.

Centos-5.2-x86_64 xen-dom0 Installation

The following works for us, there is no guarantee that it will also work for you. All settings might change in future.

• Hardware: Dell PE-1950, HD: 2X500G Soft-RAID-1, PE-2950, HD: 4X500G Hardware-RAID-1-0.
• Network: 2 Interfaces, only one cabled

Installation of Dom-0

Dom0 (Host) needs < 10G for its system, tmp is not extensively used nor is var (besides /var/lib/xen/save see below). To prevent the system to get unresponsive due to full filesystems you could give 4GB /tmp and /var each. These values can be smaller if you have hardware which has some disk space restrictions. The saved states of the domUs, created for example during a reboot of the dom0 will be stored in /var/lib/xen/save, this might get filled and is on a separate partition.

Depending on the hardware we have to set up a soft RAID-1 (2 disks) or a hardware RAID-1-0 where we treat the (4-disks) as one disk within the kickstart file.

Filesystem layout Soft-RAID-1

Principle layout:

• RAID-1
• Primary Partitions

/boot 512MB
/swap 4096MB

• LVM

Volumegroup System, Size: rest of available space
I System-Volumes-Dom0 [/, /tmp] fs: ext3
II DomU-lvms added when needed.

To have the possibility to reinstall the dom0 with the fabric management tool (uses pxeboot/kickstart) we postprocess the ks file after first installation. See appendix (ks2reinst.sh).

clearpart --all --initlabel
part raid.01 --size=512   --ondisk=sda --asprimary
part raid.02 --size=8192 --ondisk=sda --asprimary
part raid.03 --size=24576  --ondisk=sda
part raid.04 --size=1 --ondisk=sda --grow
part raid.11 --size=512   --ondisk=sdb --asprimary
part raid.12 --size=8192 --ondisk=sdb --asprimary
part raid.13 --size=24576  --ondisk=sdb
part raid.14 --size=1 --ondisk=sdb --grow
raid /boot     --level=RAID1 --device=md0 --fstype=ext2 raid.01 raid.11
raid /         --level=RAID1 --device=md1 --fstype=ext3 raid.02 raid.12
raid pv.01 --fstype "physical volume (LVM)" --level=RAID1 --device=md2 raid.03 raid.13
raid pv.02 --fstype "physical volume (LVM)" --level=RAID1 --device=md3 raid.04 raid.14
volgroup systemvg       pv.01
volgroup xenvg          pv.02
logvol swap --fstype=swap       --vgname=systemvg       --size=2048     --name=sysswap
logvol /tmp --fstype=ext3       --vgname=systemvg       --size=4096     --name=systmp
logvol /var --fstype=ext3       --vgname=systemvg       --size=4096     --name=sysvar
logvol /var/lib/xen/save --fstype=ext3       --vgname=systemvg       --size=10240     --name=sysxensave

For reinstallation of the host OS leaving the domUs untouched the following might work: Try at own risk!

# make sure to have deleted or put a comment on the line saying: clearpart --all --initlabel
#part raid.01 --size=512   --ondisk=sda --asprimary
#part raid.02 --size=8192 --ondisk=sda --asprimary
#part raid.03 --size=24576  --ondisk=sda
#part raid.04 --size=1 --ondisk=sda --grow
#part raid.11 --size=512   --ondisk=sdb --asprimary
#part raid.12 --size=8192 --ondisk=sdb --asprimary
#part raid.13 --size=24576  --ondisk=sdb
#part raid.14 --size=1 --ondisk=sdb --grow
raid /boot     --level=RAID1 --device=md0 --fstype=ext2  --useexisting
raid /         --level=RAID1 --device=md1 --fstype=ext3  --useexisting
raid pv.01 --fstype "physical volume (LVM)" --level=RAID1 --device=md2  --useexisting
raid pv.02 --fstype "physical volume (LVM)" --level=RAID1 --device=md3  --useexisting
volgroup systemvg       pv.01 --useexisting
volgroup xenvg          pv.02 --noformat
logvol swap --fstype=swap       --vgname=systemvg       --size=2048     --name=sysswap --useexisting
logvol /tmp --fstype=ext3       --vgname=systemvg       --size=4096     --name=systmp --useexisting
logvol /var --fstype=ext3       --vgname=systemvg       --size=4096     --name=sysvar --useexisting
logvol /var/lib/xen/save --fstype=ext3       --vgname=systemvg       --size=10240     --name=sysxensave --useexisting

Filesystem layout Harware-RAID-10

Here the disks appear as one and the respective section in the kickstart file gets a lot simpler:

clearpart --all --initlabel
part /boot --size=256 --asprimary --fstype=ext2 --ondisk=sda
part pv.01 --size=73728 --asprimary --fstype=ext3 --ondisk=sda
part pv.02 --size=1 --grow --ondisk=sda
volgroup systemvg pv.01
volgroup xenvg pv.02
logvol / --fstype ext3 --size=8192     --name=sysroot  --vgname=systemvg
logvol /var/lib/xen/save --fstype ext3 --size=16384     --name=sysxensave  --vgname=systemvg
logvol /var --fstype ext3 --size=4096     --name=sysvar  --vgname=systemvg
logvol /tmp --fstype ext3 --size=2048     --name=systmp  --vgname=systemvg
logvol swap --fstype swap --size=4096     --name=sysswap  --vgname=systemvg

For reinstalling the host OS and keep the domUs you could try at own risk!:

# make sure to have deleted or put a comment on the line saying: clearpart --all --initlabel
#part /boot --size=256 --asprimary --fstype=ext2 --ondisk=sda
#part pv.01 --size=73728 --asprimary --fstype=ext3 --ondisk=sda
#part pv.02 --size=1 --grow --ondisk=sda
volgroup systemvg pv.01 --useexisting
volgroup xenvg pv.02 --noformat
logvol / --fstype ext3 --size=8192     --name=sysroot  --vgname=systemvg --useexisting
logvol /var/lib/xen/save --fstype ext3 --size=16384     --name=sysxensave  --vgname=systemvg --useexisting
logvol /var --fstype ext3 --size=4096     --name=sysvar  --vgname=systemvg --useexisting
logvol /tmp --fstype ext3 --size=2048     --name=systmp  --vgname=systemvg --useexisting
logvol swap --fstype swap --size=4096     --name=sysswap  --vgname=systemvg --useexisting

Software

We install stock CentOS packages, the following excerpt from the kickstart file should be sufficient.

%packages --resolvedeps --ignoremissing
openssh
openssh-server
wget
perl-libnet
perl-MIME-Base64
perl-URI
perl-Digest-MD5
perl-libwww-perl
perl-XML-Parser
perl-DBI
perl-Crypt-SSLeay
lsof
curl
xen
kernel-xen
lvm2

DomU Kickstart file

Here the whole lvm created on the dom0 will be used as one disk (xvda), the size of the root partition will then depend on the lvm size. For a basic system the following snipped of a ks file could be useful:

clearpart --all --initlabel
part swap  --size=1024 --asprimary --ondisk=xvda
part /tmp  --size=4096 --ondisk=xvda
part /var  --size=4096 --ondisk=xvda
part /     --size=8192 --grow --asprimary --ondisk=xvda
.
.
.
zerombr yes
%packages --resolvedeps --ignoremissing
openssh
openssh-server
wget
perl-libnet
perl-MIME-Base64
perl-URI
perl-Digest-MD5
perl-libwww-perl
perl-XML-Parser
perl-DBI
perl-Crypt-SSLeay
lsof
curl

Xen dom-0 Configuration

Dom0 grub

For performance its best to limit the Memory of the dom0 to 512 MB and to prevent it from ballooning (i.e.dom-0 takes all available memory), and to restrict the dom0 to one cpu. Use some monitoring to make sure that these settings are sufficient.

Therefore modify /boot/grub/menu.lst, the important entries are in bold:

title CentOS (2.*xen)
       root (hd0,0)
       kernel /xen.gz-2.6.18-92.1.6.el5 dom0_mem=512M
       module /vmlinuz-2.6.18-92.1.6.el5xen ro root=/dev/systemvg/sysroot nosmp
       module /initrd-2.6.18-92.1.6.el5xen.img

xend configuration

Set in /etc/xen/xend-config.sxp:

(dom0-min-mem 0)

Installation of a domU using pypxeboot

On the dom0 pypxeboot and the patched version of udhcp has to be available, see here.

On the dom0 hosting the domU, do the following:

1. Create a lvm for the domU, for the size check the domU kickstart file you created above. lvcreate -n <domU name> -L<SIZE>G xenvg
2. Create domU config file:

You'll have to provide the domU MAC-address you created above.dhcp above

<dom0>:/etc/xen/<domU name>
name ='<domU name>'
memory ='1024'
bootloader="/usr/bin/pypxeboot"
vif=['mac=00:16:3e:XX:XX:XX']
bootargs=vif[0]
disk = [ "phy:xenvg/<domU-name>,xvda,w" ]
root="/dev/xvda1 ro"
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Create DomU

on dom0:

xm create <domU name>

should give you something like: