Difference between revisions of "Xen on CentOS 5"
| (9 intermediate revisions by one other user not shown) | |||
| Line 1: | Line 1: | ||
| + | '''This information is no longer actual, please refer to [[GSP_Virtualisation_with_Xen]] for up to date configuration and install instructions.''' | ||
| + | |||
= Xen Virtualization = | = Xen Virtualization = | ||
| − | This description worked for us | + | This description worked for us by part it is still work in progress. Any comments welcome. |
| − | In the following we describe how to use Xen-Virtualisation for Quattor managed grid middleware (gLite) servers. | + | In the following we describe how to use Xen-Virtualisation for Quattor managed grid middleware (gLite) servers. The domUs (guests) are on local disks in logical volumes. |
Quattor is not necessarily needed, any other fabric management tool or simple scripts will do the same. | Quattor is not necessarily needed, any other fabric management tool or simple scripts will do the same. | ||
In particular we had an existing Quattor environment and added the virtualisation framework, so you'll find here no information on howto manage system, glite-service-installation/configuration and host-monitoring, etc. with Quattor, this can be found elsewhere in this wiki. In the following Quattor manages "only" the dhcp.conf, kickstart- and pxe configuration files. | In particular we had an existing Quattor environment and added the virtualisation framework, so you'll find here no information on howto manage system, glite-service-installation/configuration and host-monitoring, etc. with Quattor, this can be found elsewhere in this wiki. In the following Quattor manages "only" the dhcp.conf, kickstart- and pxe configuration files. | ||
==Hardware:== | ==Hardware:== | ||
| − | PE-1950 with two 500G hard diskes, PE-2950 with four 500G hard disks | + | PE-1950 with two 500G hard diskes, PE-2950 with four 500G hard disks. |
| + | |||
== Software:== | == Software:== | ||
* Operating system host (dom0): CentOS-5.x, x.ge.2, x86-64 | * Operating system host (dom0): CentOS-5.x, x.ge.2, x86-64 | ||
* Virtualisation Method: Xen, stock CentOS xen kernels | * Virtualisation Method: Xen, stock CentOS xen kernels | ||
* Operating system guest (domU): CentOS-4.x, i386 (ParaVirtualisedMachine -guest, PVM), CentOS-3.x, i386 (Hardware (assisted) Virtual Machine, HVM) | * Operating system guest (domU): CentOS-4.x, i386 (ParaVirtualisedMachine -guest, PVM), CentOS-3.x, i386 (Hardware (assisted) Virtual Machine, HVM) | ||
| + | * LVM | ||
* Fabric Management: Quattor | * Fabric Management: Quattor | ||
* Network installation tools: Kickstart, PXE, [https://www.cs.tcd.ie/Stephen.Childs/pypxeboot/ pypxeboot] | * Network installation tools: Kickstart, PXE, [https://www.cs.tcd.ie/Stephen.Childs/pypxeboot/ pypxeboot] | ||
| Line 54: | Line 58: | ||
=== Installation of Dom-0 === | === Installation of Dom-0 === | ||
| − | Dom0 (Host) needs < 10G for its system, tmp is not extensively used nor is var (besides /var/lib/xen/save see below). To prevent the system to get unresponsive due to full | + | Dom0 (Host) needs < 10G for its system, tmp is not extensively used nor is var (besides /var/lib/xen/save see below). To prevent the system to get unresponsive due to full filesystems you could give 4GB /tmp and /var each. These values can be smaller if you have hardware which has some disk space restrictions. |
The saved states of the domUs, created for example during a reboot of the dom0 will be stored in /var/lib/xen/save, this might get filled and is on a separate partition. | The saved states of the domUs, created for example during a reboot of the dom0 will be stored in /var/lib/xen/save, this might get filled and is on a separate partition. | ||
| Line 162: | Line 166: | ||
xen | xen | ||
kernel-xen | kernel-xen | ||
| + | lvm2 | ||
== DomU Kickstart file == | == DomU Kickstart file == | ||
| Line 193: | Line 198: | ||
== Xen dom-0 Configuration == | == Xen dom-0 Configuration == | ||
=== Dom0 grub === | === Dom0 grub === | ||
| − | + | We limit the memory of the dom0 to 512 MB and to prevent it from ballooning (i.e.dom-0 takes all available memory), and to restrict the dom0 to one cpu. | |
Use some monitoring to make sure that these settings are sufficient. | Use some monitoring to make sure that these settings are sufficient. | ||
| − | Therefore modify /boot/grub/menu.lst, the important entries are in bold: | + | Therefore modify the dom0 /boot/grub/menu.lst, the important entries are in bold: |
title CentOS (2.*xen) | title CentOS (2.*xen) | ||
| Line 214: | Line 219: | ||
# Create a lvm for the domU, for the size check the domU kickstart file you created above. ''' lvcreate -n <domU name> -L<SIZE>G xenvg ''' | # Create a lvm for the domU, for the size check the domU kickstart file you created above. ''' lvcreate -n <domU name> -L<SIZE>G xenvg ''' | ||
# Create domU config file: | # Create domU config file: | ||
| − | You'll have to provide the domU MAC-address you created above. | + | You'll have to provide the domU MAC-address you created above for the dhcp config on the installation server. |
<dom0>:/etc/xen/<domU name> | <dom0>:/etc/xen/<domU name> | ||
| Line 233: | Line 238: | ||
xm create <domU name> | xm create <domU name> | ||
| − | should give you something like: | + | should give you something like: ... to be continued |
Latest revision as of 11:36, 16 June 2012
This information is no longer actual, please refer to GSP_Virtualisation_with_Xen for up to date configuration and install instructions.
Xen Virtualization
This description worked for us by part it is still work in progress. Any comments welcome. In the following we describe how to use Xen-Virtualisation for Quattor managed grid middleware (gLite) servers. The domUs (guests) are on local disks in logical volumes. Quattor is not necessarily needed, any other fabric management tool or simple scripts will do the same. In particular we had an existing Quattor environment and added the virtualisation framework, so you'll find here no information on howto manage system, glite-service-installation/configuration and host-monitoring, etc. with Quattor, this can be found elsewhere in this wiki. In the following Quattor manages "only" the dhcp.conf, kickstart- and pxe configuration files.
Hardware:
PE-1950 with two 500G hard diskes, PE-2950 with four 500G hard disks.
Software:
- Operating system host (dom0): CentOS-5.x, x.ge.2, x86-64
- Virtualisation Method: Xen, stock CentOS xen kernels
- Operating system guest (domU): CentOS-4.x, i386 (ParaVirtualisedMachine -guest, PVM), CentOS-3.x, i386 (Hardware (assisted) Virtual Machine, HVM)
- LVM
- Fabric Management: Quattor
- Network installation tools: Kickstart, PXE, pypxeboot
- You have IPs for the dom0s and domUs available.
Installation (PXE) Server
We assume you have an installation server with Preboot eXecution Environment (PXE) available.
Images, Kernels
For the dom0 network installation no additional steps are necessary as well as for the HVM domUs, for the PVM domUs you need to have the directories with the images. Example, for network installation of CentOS-4.6 i386 domUs you could do the following:
mkdir -p <base directory>/xen-domU-c4.6/i386
this directory should have the files initrd.img and vmlinuz, they are available from here
DHCP
Add the MAC adresses to /etc/dhcp.conf. For the dom-0 take the one from the network card. For the domU you have to choose a MAC-address. In order to avoid clashes (multiple usage of the same MAC) you can do the following: the first three hex numbers should be 00:16:3e (reserved for xen domUs), the last three can be for example the hex representation of the last three numbers of the IP-address.
Example: IP of the DomU is 192.168.1.2 MAC: 00:16:3e:a8:01:02, the result of `printf 00:16:3e:%02x:%02x:%02x"\n" 168 1 2` command
Create PXE Menus
At our site this is created with Quattor, for a CentOS-4 i386 domU the cfg file looks like:
<base directory>/pxelinux.cfg/<hostname.cfg>
default Scientific Linux centos4_i386 label Scientific Linux centos4_i386 kernel xen-domU-c4.6/i386/vmlinuz append ramdisk=32768 initrd=xen-domU-c4.6/i386/initrd.img \ ks=http://<Installation server>/<path to ks files>/<hostname>.ks ksdevice=eth0
For all hosts there should be a file like:
<base directory>/pxelinux.cfg/localboot.cfg default Localdisk label Localdisk localboot 0
If you want to install/localboot a host create the links accordingly by hand or with your fabric management tool. At our site this is also done with Quattor.
Centos-5.2-x86_64 xen-dom0 Installation
The following works for us, there is no guarantee that it will also work for you. All settings might change in future.
- Hardware: Dell PE-1950, HD: 2X500G Soft-RAID-1, PE-2950, HD: 4X500G Hardware-RAID-1-0.
- Network: 2 Interfaces, only one cabled
Installation of Dom-0
Dom0 (Host) needs < 10G for its system, tmp is not extensively used nor is var (besides /var/lib/xen/save see below). To prevent the system to get unresponsive due to full filesystems you could give 4GB /tmp and /var each. These values can be smaller if you have hardware which has some disk space restrictions. The saved states of the domUs, created for example during a reboot of the dom0 will be stored in /var/lib/xen/save, this might get filled and is on a separate partition.
Depending on the hardware we have to set up a soft RAID-1 (2 disks) or a hardware RAID-1-0 where we treat the (4-disks) as one disk within the kickstart file.
Filesystem layout Soft-RAID-1
Principle layout:
- RAID-1
- Primary Partitions
/boot 512MB /swap 4096MB
- LVM
Volumegroup System, Size: rest of available space I System-Volumes-Dom0 [/, /tmp] fs: ext3 II DomU-lvms added when needed.
To have the possibility to reinstall the dom0 with the fabric management tool (uses pxeboot/kickstart) we postprocess the ks file after first installation. See appendix (ks2reinst.sh).
clearpart --all --initlabel part raid.01 --size=512 --ondisk=sda --asprimary part raid.02 --size=8192 --ondisk=sda --asprimary part raid.03 --size=24576 --ondisk=sda part raid.04 --size=1 --ondisk=sda --grow part raid.11 --size=512 --ondisk=sdb --asprimary part raid.12 --size=8192 --ondisk=sdb --asprimary part raid.13 --size=24576 --ondisk=sdb part raid.14 --size=1 --ondisk=sdb --grow raid /boot --level=RAID1 --device=md0 --fstype=ext2 raid.01 raid.11 raid / --level=RAID1 --device=md1 --fstype=ext3 raid.02 raid.12 raid pv.01 --fstype "physical volume (LVM)" --level=RAID1 --device=md2 raid.03 raid.13 raid pv.02 --fstype "physical volume (LVM)" --level=RAID1 --device=md3 raid.04 raid.14 volgroup systemvg pv.01 volgroup xenvg pv.02 logvol swap --fstype=swap --vgname=systemvg --size=2048 --name=sysswap logvol /tmp --fstype=ext3 --vgname=systemvg --size=4096 --name=systmp logvol /var --fstype=ext3 --vgname=systemvg --size=4096 --name=sysvar logvol /var/lib/xen/save --fstype=ext3 --vgname=systemvg --size=10240 --name=sysxensave
For reinstallation of the host OS leaving the domUs untouched the following might work: Try at own risk!
# make sure to have deleted or put a comment on the line saying: clearpart --all --initlabel #part raid.01 --size=512 --ondisk=sda --asprimary #part raid.02 --size=8192 --ondisk=sda --asprimary #part raid.03 --size=24576 --ondisk=sda #part raid.04 --size=1 --ondisk=sda --grow #part raid.11 --size=512 --ondisk=sdb --asprimary #part raid.12 --size=8192 --ondisk=sdb --asprimary #part raid.13 --size=24576 --ondisk=sdb #part raid.14 --size=1 --ondisk=sdb --grow raid /boot --level=RAID1 --device=md0 --fstype=ext2 --useexisting raid / --level=RAID1 --device=md1 --fstype=ext3 --useexisting raid pv.01 --fstype "physical volume (LVM)" --level=RAID1 --device=md2 --useexisting raid pv.02 --fstype "physical volume (LVM)" --level=RAID1 --device=md3 --useexisting volgroup systemvg pv.01 --useexisting volgroup xenvg pv.02 --noformat logvol swap --fstype=swap --vgname=systemvg --size=2048 --name=sysswap --useexisting logvol /tmp --fstype=ext3 --vgname=systemvg --size=4096 --name=systmp --useexisting logvol /var --fstype=ext3 --vgname=systemvg --size=4096 --name=sysvar --useexisting logvol /var/lib/xen/save --fstype=ext3 --vgname=systemvg --size=10240 --name=sysxensave --useexisting
Filesystem layout Harware-RAID-10
Here the disks appear as one and the respective section in the kickstart file gets a lot simpler:
clearpart --all --initlabel part /boot --size=256 --asprimary --fstype=ext2 --ondisk=sda part pv.01 --size=73728 --asprimary --fstype=ext3 --ondisk=sda part pv.02 --size=1 --grow --ondisk=sda volgroup systemvg pv.01 volgroup xenvg pv.02 logvol / --fstype ext3 --size=8192 --name=sysroot --vgname=systemvg logvol /var/lib/xen/save --fstype ext3 --size=16384 --name=sysxensave --vgname=systemvg logvol /var --fstype ext3 --size=4096 --name=sysvar --vgname=systemvg logvol /tmp --fstype ext3 --size=2048 --name=systmp --vgname=systemvg logvol swap --fstype swap --size=4096 --name=sysswap --vgname=systemvg
For reinstalling the host OS and keep the domUs you could try at own risk!:
# make sure to have deleted or put a comment on the line saying: clearpart --all --initlabel #part /boot --size=256 --asprimary --fstype=ext2 --ondisk=sda #part pv.01 --size=73728 --asprimary --fstype=ext3 --ondisk=sda #part pv.02 --size=1 --grow --ondisk=sda volgroup systemvg pv.01 --useexisting volgroup xenvg pv.02 --noformat logvol / --fstype ext3 --size=8192 --name=sysroot --vgname=systemvg --useexisting logvol /var/lib/xen/save --fstype ext3 --size=16384 --name=sysxensave --vgname=systemvg --useexisting logvol /var --fstype ext3 --size=4096 --name=sysvar --vgname=systemvg --useexisting logvol /tmp --fstype ext3 --size=2048 --name=systmp --vgname=systemvg --useexisting logvol swap --fstype swap --size=4096 --name=sysswap --vgname=systemvg --useexisting
Software
We install stock CentOS packages, the following excerpt from the kickstart file should be sufficient.
%packages --resolvedeps --ignoremissing openssh openssh-server wget perl-libnet perl-MIME-Base64 perl-URI perl-Digest-MD5 perl-libwww-perl perl-XML-Parser perl-DBI perl-Crypt-SSLeay lsof curl xen kernel-xen lvm2
DomU Kickstart file
Here the whole lvm created on the dom0 will be used as one disk (xvda), the size of the root partition will then depend on the lvm size. For a basic system the following snipped of a ks file could be useful:
clearpart --all --initlabel part swap --size=1024 --asprimary --ondisk=xvda part /tmp --size=4096 --ondisk=xvda part /var --size=4096 --ondisk=xvda part / --size=8192 --grow --asprimary --ondisk=xvda . . . zerombr yes %packages --resolvedeps --ignoremissing openssh openssh-server wget perl-libnet perl-MIME-Base64 perl-URI perl-Digest-MD5 perl-libwww-perl perl-XML-Parser perl-DBI perl-Crypt-SSLeay lsof curl
Xen dom-0 Configuration
Dom0 grub
We limit the memory of the dom0 to 512 MB and to prevent it from ballooning (i.e.dom-0 takes all available memory), and to restrict the dom0 to one cpu. Use some monitoring to make sure that these settings are sufficient.
Therefore modify the dom0 /boot/grub/menu.lst, the important entries are in bold:
title CentOS (2.*xen)
root (hd0,0)
kernel /xen.gz-2.6.18-92.1.6.el5 dom0_mem=512M
module /vmlinuz-2.6.18-92.1.6.el5xen ro root=/dev/systemvg/sysroot nosmp
module /initrd-2.6.18-92.1.6.el5xen.img
xend configuration
Set in /etc/xen/xend-config.sxp:
(dom0-min-mem 0)
Installation of a domU using pypxeboot
On the dom0 pypxeboot and the patched version of udhcp has to be available, see here.
On the dom0 hosting the domU, do the following:
- Create a lvm for the domU, for the size check the domU kickstart file you created above. lvcreate -n <domU name> -L<SIZE>G xenvg
- Create domU config file:
You'll have to provide the domU MAC-address you created above for the dhcp config on the installation server.
<dom0>:/etc/xen/<domU name> name ='<domU name>' memory ='1024' bootloader="/usr/bin/pypxeboot" vif=['mac=00:16:3e:XX:XX:XX'] bootargs=vif[0] disk = [ "phy:xenvg/<domU-name>,xvda,w" ] root="/dev/xvda1 ro" on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart'
Create DomU
on dom0:
xm create <domU name>
should give you something like: ... to be continued
