Xen on CentOS 5 - Notes

From PDP/Grid Wiki
Jump to navigationJump to search

Docu on CentOS

Some links:

see Virtualization

check PCI-DMA: Out of SW-IOMMU space ERROR

CentOS 5

These are just notes taken during the installation, it is not meant as a howto or something like that

Hardware Dell 1950

Additional Stuff

mercurial is a distributed version control system. to use it on centos-5 X86_64 do:

compile libvirt

  • wget ftp://ftp.easynet.be/gnu/gettext/gettext-0.17.tar.gz; tar xvfz gettext-0.17.tar.gz; cd gettext-0.17; ./configure; make; make install
  • yum install gnutls gnutls-devel
  • getting latest libvirt see: libvirt cvs
  • cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs login; password anoncvs cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs co libvirt
  • cd to libvirt dir and apply patch patch -p1 < configure.patch
  • in libvirt dir do: ./autogen.sh; make; make install

virt-install needs: 

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

CentOS 5 X86_64, Minimal system

Additional Packages

Minimal System installed via kickstart, packages: 

# Packages groups/list
%packages --resolvedeps --ignoremissing

openssh
openssh-server
wget
perl-libnet
perl-MIME-Base64
perl-URI
perl-Digest-MD5
perl-libwww-perl
perl-XML-Parser
perl-DBI
perl-Crypt-SSLeay
lsof

Post Install, get the following packages

Some of these are not needed, has to be cleaned: 

yum install \
gcc make zlib-devel zlib python-devel curses libncurses-devel ncurses-devel openssl openssl-devel \
`yum search xorg-x11|grep dev|grep x86_64|awk -F . '{print $1}'` \
bridge-utils tetex tetex-latex transfig libtool-ltdl dev86 glibc-devel

xen 3.1.2-rc from mercurial

hg pull ...
cd xen-3.1
wget http://www.broadcom.com/docs/driver_download/NXII/linux-1.5.10c.zip
unzip, use broadcom sources for 
make world
make install
depmod 2.6.18-xen
mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod  --with=megaraid_sas /boot/initrd-2.6.18-xen.img 2.6.18-xen

Note on: Network Problems Broadcom drivers have problems, network is OK when xend started without network bridges, see /etc/xen/xend-config.sxp

Starting bridge with: /etc/xen/scripts/network-bridge start results in a not working network configuration in dom0.

see network-bridge causes network to hang

swiotlb=128 has to be set in menu.list since we have 8GB RAM, without youll get error messages like : 

Sep  7 13:14:22 test-11 kernel: PCI-DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:02:01.0

see also: SW-IOMMU space

  • change/add /boot/grub/menu.list:
 title CentOS (xen-3.1.2-rc2 2.6.18 compiled)
       root (hd0,0)
       kernel /xen-3.1.gz
       module /vmlinuz-2.6-xen ro root=/dev/md1 swiotlb=128
       module /initrd-2.6.18-xen.img

Boot into your new system:

[root@kribbe ~]# xm dmesg 

__  __            _____  _   ____              ____
\ \/ /___ _ __   |___ / / | |___ \    _ __ ___|___ \
 \  // _ \ '_ \    |_ \ | |   __) |__| '__/ __| __) |
 /  \  __/ | | |  ___) || |_ / __/|__| | | (__ / __/
/_/\_\___|_| |_| |____(_)_(_)_____|  |_|  \___|_____|

http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory

Xen version 3.1.2-rc2 (root@nikhef.nl) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Tue Nov 13 15:02:14 CET 2007
Latest ChangeSet: unavailable
.
.
.
(XEN) HVM: VMX enabled
(XEN) VMX: MSR intercept bitmap enabled
(XEN) CPU0: Intel(R) Xeon(R) CPU            5150  @ 2.66GHz stepping 06
.
.
.



LVM 2

Due to performance reasons VMs should be installed in LVMs. For recovering a LVM in a Raid-1 see:Recovery of RAID and LVM2 Volumes

Installed Version: 

Installed Packages
Name   : lvm2
Arch   : i386
Version: 2.02.16
Release: 3.el5
Size   : 5.5 M
Repo   : installed
Summary: Userland logical volume management tools


NOTE: for having the live-migration-posibility we have to look at Cluster-LVM or something like that.

To get started ....

Setting up LVM2 on kribbe

Using /dev/srv resp. /dev/md3 for LVM we have to:

  • comment out /dev/md3 in /etc/fstab
  • umount /dev/srv (only once)
  • Creating the physical volume and XenVG Volumegroup:
[root@kribbe ~]# pvcreate /dev/md3
 Physical volume "/dev/md3" successfully created
[root@kribbe ~]# vgcreate XenVG /dev/md3
 Volume group "XenVG" successfully created

Within this volume-group we can now manage the partitions for the virtual maschines using lvcreate, lvresize, .... commands

Example: lvm for a CentOS5 test VM

Root FS, only one partition 

[root@kribbe tmp]# lvcreate -L10G -nXen_CentOS5_Test XenVG
 Logical volume "Xen_CentOS5_Test" created
root@kribbe tmp]# mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1310720 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
       32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.


SWAP Note: swap in a lvm is not performant 

[root@kribbe tmp]# lvcreate -L1G -nXen_CentOS5_Test_swap XenVG
 Logical volume "Xen_CentOS5_Test_swap" created
[root@kribbe tmp]# mkswap /dev/XenVG/Xen_CentOS5_Test_swap
Setting up swapspace version 1, size = 1073737 kB

Virtualize Physical Nodes

  1. On DOM_0 Create a lvm for the DOM_U
  2. Mount this lvm on DOM_0
  3. On Physical Node change runlevel to s
  4. setup a exclude-list file, for details see eisxen
  5. cd /; tar csp --exclude-from=/rootfs/exclude.lst /* |ssh DOM_0 "cd /<mounted LVM>; tar xsp"
  6. change the fstab of the dom_u
Example exclude.lst, node specific, dont just copy, see also: eisxen
This list must not have an empty line at the end.
less /rootfs/exclude.lst
/boot
/data
/datastore
/datastore2
/proc
/rootfs
/etc/mtab

DOM_U is a centos-3.x

In order to run this as a virtual maschine, the dom_u might need a new version of e2fschk and module-init-tools. Therfore, on DOM_0 mount the DOM_U fs created in an earlier step, get e2fsprogs-1.39.tar.gz module-init-tools-3.2.tar.gz (these versions worked for us) to this directory. The following commands worked for me.

  • chroot to dom_u mountpoint
  • extract, compile, install e2fschk and module-init-tools
  • cp -r DOM_0:/lib/modules/<running version> DOM_U:/lib/modules
  • for e2fs do: tar xvfz e2fsprogs-1.39.tar.gz ;cd e2fsprogs-1.39; ./configure; make; make install
  • for modutils: tar xvfz module-init-tools-3.2.tar.gz; cd module-init-tools-3.2; ./configure --prefix=/ ;make moveold; make; make install

CentOS 5 with Virtualization package

  • Installed Version:
2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
  • Run yum update, Version:
2.6.18-8.1.15.el5xen #1 SMP Mon Oct 22 09:01:12 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
  • xen:
[root@kribbe ~]# xm dmesg
__  __            _____  ___   _____             ____       _ ____
\ \/ /___ _ __   |___ / / _ \ |___ /    _ __ ___| ___|  ___| | ___|
 \  // _ \ '_ \    |_ \| | | |  |_ \ __| '__/ __|___ \ / _ \ |___ \
 /  \  __/ | | |  ___) | |_| | ___) |__| | | (__ ___) |  __/ |___) |
/_/\_\___|_| |_| |____(_)___(_)____/   |_|  \___|____(_)___|_|____/

http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory

Xen version 3.0.3-rc5-8.1.15.el5 (mockbuild@) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Mon Oct 22 08:25:42 EDT 2007
Latest ChangeSet: unavailable
....
(XEN) VMXON is done
(XEN) Dom0 has maximum 4 VCPUs
  • lvm2, bridge-utils
Installed Packages:
Name   : lvm2
Arch   : x86_64
Version: 2.02.16

Name   : bridge-utils
Arch   : x86_64
Version: 1.1


  • yum install vnc
  • use partition /dev/md3 mounted on /project for lvm
comment out ..... /project
umount /project
pvcreate /dev/md3
vgcreate XenVG /dev/md3
lvcreate -L10G -nXen_CentOS5_Test XenVG
mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test
lvcreate -L1G -nXen_CentOS5_Test_swap XenVG
mkswap /dev/XenVG/Xen_CentOS5_Test_swap

Running various centOS Versions as VMs on kribbe

Check if virt-manager tools are working

virt-install can be used for VM installation.

CentOS 5 64-Bit in VM

set up kickstart file for VM, than 

virt-install -p --location=http://www.dutchgrid.nl/mirror/centos/5/os/x86_64 --noautoconsole \
--file=/dev/XenVG/Xen_CentOS5_Test --name=domU001 --ram=1024  \
-x "ks=http://stal.nikhef.nl/ks/xen-test-koebrug.ks ip=194.171.97.11 \
netmask=255.255.255.0 dns=192.16.186.253 gateway=194.171.97.254"

Connect to th econsole of the starting vm:

  • xm list, lists the VMs:
[root@kribbe ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     7025     4 r-----   1056.7
domU001                                    8     1023     1 -b----    490.8
  • xm console <ID> or <Name> of the VM (s.a.) creates a console of the VM

Worked

Note: no vnc option in the virt-install command,

  • --vnc option removed, does not boot.
  • during boot of the VM answer: q: graphics support: no

On VM:

  • koebrug.nikhef.nl
/etc/redhat-release
CentOS release 5 (Final)
[root@koebrug ~]# uname -a
Linux koebrug.nikhef.nl 2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

CentOS 4.5 64-Bit in VM

  • silo.nikhef.nl

[root@silo ~]# uname -a Linux silo.nikhef.nl 2.6.9-55.ELxenU #1 SMP Wed May 2 14:29:26 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux [root@silo ~]# cat /etc/redhat-release CentOS release 4.5 (Final)


64 Bit DOM_U in 64 Bit DOM_0 seems to be no problem. 32Bit installations can not be done in this way, apropriate kernel needed Looking at virsh

Using VIRSH

Information on virsh can be found at: centos-wiki or Red Hat Virtualization

virsh is another virtualization management tool (built around the libvirt management API) which is now capable to deal with xen, future versions might also be able to manage kvm or qemu.

Since we plan to use xen we should stick with xm directly.

Virtual Maschine management with xm

based on: Chapter 17. Managing Virtual Machines Using xm

CentOS 5 i386-System -- Quattor generated

Problems

  1. virt-manager not working centos-bug


  • Post-Quattor steps to prevent quattor from revoking installation/configuration operations.
on kribbe:
chkconfig cdp-listend off
chkconfig ncm-cdispd off

on stal:
[root@stal ~]# aii-shellfe --boot kribbe.nikhef.nl

to give back the control to quattor (fresh install next boot, will also switch on cdp-listen and ncm-cdispd)
[root@stal ~]#  aii-shellfe --install kribbe.nikhef.nl
  • Mount external storage for files that should survive a reboot/installation
mount -tnfs tbn15.nikhef.nl:/export/array2/local/tbadmin /tmpmnt
  • Hostname: kribbe.nikhef.nl
  • Kernel: 2.6.18-8.1.8.el5xen
  • HD-Layout:
[root@kribbe ~]# sfdisk -l /dev/sdb

Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+     32      33-    265041   fd  Linux raid autodetect
/dev/sdb2         33   51025   50993  409601272+  fd  Linux raid autodetect
/dev/sdb3      51026   58580    7555   60685537+  fd  Linux raid autodetect
/dev/sdb4      58581   60800    2220   17832150    5  Extended
/dev/sdb5      58581+  60147    1567-  12586896   fd  Linux raid autodetect
/dev/sdb6      60148+  60669     522-   4192933+  fd  Linux raid autodetect
/dev/sdb7      60670+  60800     131-   1052226   fd  Linux raid autodetect

====================

[root@kribbe ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
     264960 blocks [2/2] [UU]
     
md3 : active raid1 sdb2[1] sda2[0]
     409601152 blocks [2/2] [UU]
     
md5 : active raid1 sdb3[1] sda3[0]
     60685440 blocks [2/2] [UU]
      
md2 : active raid1 sdb6[1] sda6[0]
     4192832 blocks [2/2] [UU]
      
md4 : active raid1 sdb7[1] sda7[0]
     1052160 blocks [2/2] [UU]
      
md1 : active raid1 sdb5[1] sda5[0]
     12586816 blocks [2/2] [UU]
      
unused devices: <none>


Example installation of a CentOS5 VM

Create a VM in lvm partition

  1. Setup lvm as stated above
  2. Mount VM root fs: mount /dev/XenVG/Xen_CentOS5_Test /mnt
  3. extract archive:
  4. [root@kribbe tmp]# bunzip2 centos.5-0.20070424.img.tar.bz2
  5. [root@kribbe tmp]# tar xvf centos.5-0.20070424.img.tar
  6. loopmount image
  7. [root@kribbe tmp]# mkdir /img
  8. [root@kribbe tmp]# mount -o loop centos.5-0.img /img
  9. cp image to logical volume
  10. [root@kribbe img]# cp -a * /mnt/
  11. Change Root PW in VM:
[root@kribbe mnt]# chroot /mnt
bash-3.1# pwconv
bash-3.1# passwd
Changing password for user root.
  1. Set Hostname of VM: /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=koebrug.nikhef.nl
  1. no swap in VM: /etc/fstab
#/dev/sda2               none                    swap    sw       0 0


setup of VM start files

SELinux, Postinstallation steps

  • Cleaning up daemons, pay attention to ntp, do the dom_U has the correct time?
  /sbin/service microcode_ctl stop
  /sbin/chkconfig --del microcode_ctl
  /sbin/service smartd stop
  /sbin/chkconfig --del smartd


If you are using SELinux, it is important to check that the image has the correct security context (xen_image_t), or access to the virtual disk will be denied to the domU system. You can check this with ls: 

# ls -Z /srv/xen/mailserver.img 
-rw-r--r--  root root user_u:object_r:xen_image_t      /srv/xen/mailserver.img

If you are having trouble setting the right file context, please have a look at the Xen tips and tricks page. At any rate, turning off SELinux, as some howtos on this subject advise is a very poor workaround. Reading two manual pages (semanage(8) and restorecon(8)), for an extra layer of security is a good trade!

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Xen_on_CentOS_5_-_Notes&oldid=1345"