Difference between revisions of "Xen on CentOS 5 - Notes"
From PDP/Grid Wiki
Jump to navigationJump to search| (44 intermediate revisions by 4 users not shown) | |||
| Line 2: | Line 2: | ||
Some links: | Some links: | ||
| − | [http://www.centos.org/docs/5/ | + | [http://www.centos.org/docs/5/ CentOS-5 Documentation including docs on Virtualization] |
| − | |||
| + | check [https://bugzilla.redhat.com/show_bug.cgi?id=219216 PCI-DMA: Out of SW-IOMMU space ERROR] | ||
| + | = CentOS 5 = | ||
---- | ---- | ||
'''These are just notes taken during the installation, it is ''not'' meant as a howto or something like that''' | '''These are just notes taken during the installation, it is ''not'' meant as a howto or something like that''' | ||
---- | ---- | ||
| + | Hardware Dell 1950 | ||
| + | == Additional Stuff == | ||
| + | '''mercurial''' is a distributed version control system. | ||
| + | to use it on centos-5 X86_64 do: | ||
| + | * wget http://www.selenic.com/mercurial/release/mercurial-0.9.5.tar.gz | ||
| + | * tar xvfz mercurial-0.9.5.tar.gz | ||
| + | * make install | ||
| + | * Add export PYTHONPATH=/usr/local/lib64/python2.4/site-packages:${PYTHONPATH} to the environment | ||
| − | + | compile '''libvirt''' | |
| − | + | * wget ftp://ftp.easynet.be/gnu/gettext/gettext-0.17.tar.gz; tar xvfz gettext-0.17.tar.gz; cd gettext-0.17; ./configure; make; make install | |
| − | + | * yum install gnutls gnutls-devel | |
| − | + | * getting latest libvirt see: [http://libvirt.org/downloads.html libvirt cvs ] | |
| − | + | * cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs login; '''password anoncvs''' cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs co libvirt | |
| − | + | * cd to libvirt dir and apply [http://www.redhat.com/archives/libvir-list/2007-September/msg00166.html patch] patch -p1 < configure.patch | |
| − | + | * in '''libvirt''' dir do: ./autogen.sh; make; make install | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | * yum install | ||
| − | |||
| − | * | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | * - | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | [http://www. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| + | '''virt-install''' needs: | ||
| + | * virtviewer, this needs | ||
| + | * rpm -ivh http://www.lfarkas.org/linux/packages/centos/5/x86_64/gtk-vnc-0.2.0-4.x86_64.rpm | ||
| + | * autogen.sh needs: | ||
| + | export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig | ||
| − | + | == CentOS 5 X86_64, Minimal system == | |
| − | + | == Additional Packages == | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | == | ||
Minimal System installed via kickstart, packages: | Minimal System installed via kickstart, packages: | ||
# Packages groups/list | # Packages groups/list | ||
| Line 192: | Line 55: | ||
=== Post Install, get the following packages === | === Post Install, get the following packages === | ||
Some of these are not needed, has to be cleaned: | Some of these are not needed, has to be cleaned: | ||
| − | yum install\ | + | yum install \ |
gcc make zlib-devel zlib python-devel curses libncurses-devel ncurses-devel openssl openssl-devel \ | gcc make zlib-devel zlib python-devel curses libncurses-devel ncurses-devel openssl openssl-devel \ | ||
`yum search xorg-x11|grep dev|grep x86_64|awk -F . '{print $1}'` \ | `yum search xorg-x11|grep dev|grep x86_64|awk -F . '{print $1}'` \ | ||
bridge-utils tetex tetex-latex transfig libtool-ltdl dev86 glibc-devel | bridge-utils tetex tetex-latex transfig libtool-ltdl dev86 glibc-devel | ||
| + | |||
=== xen 3.1.2-rc from mercurial === | === xen 3.1.2-rc from mercurial === | ||
| + | '''For 64-bit Dom_0 and 32-bit Dom_U kombination:''', the following is taken from: [http://wiki.xensource.com/xenwiki/InstallationNotes#head-b0c9b7fb7d6a4acc97446f3f7a679299436eb2be xensource install notes ] | ||
| + | |||
| + | Dell 2900 modules to include in mkinitrd | ||
| + | |||
| + | mkinitrd -v -f --with=sd_mod --with=scsi_mod --with=megaraid_mm --with=megaraid_sas --with=megaraid_mbox initrd-2.6.16.29-xen.img 2.6.16.29-xen | ||
| + | |||
| + | dom0 64 bit and domU 32 bit from source code | ||
| + | |||
| + | Build and install xen: | ||
| + | |||
| + | make xen | ||
| + | make install-xen | ||
| + | |||
| + | Build and install the dom0: | ||
| + | |||
| + | make linux-2.6-xen0-prep | ||
| + | cd build-linux-*-xen0-x86_64 | ||
| + | make menuconfig | ||
| + | make | ||
| + | make install modules_install | ||
| + | |||
| + | And finally build and install 32 bit domU: | ||
| + | |||
| + | make XEN_TARGET_ARCH=x86_32 linux-2.6-xenU-prep | ||
| + | cd build-linux-*-xen0-x86_32 | ||
| + | make ARCH=i386 menuconfig | ||
| + | make ARCH=i386 | ||
| + | make ARCH=i386 install modules_install | ||
| + | |||
| + | |||
hg pull ... | hg pull ... | ||
cd xen-3.1 | cd xen-3.1 | ||
| Line 205: | Line 99: | ||
depmod 2.6.18-xen | depmod 2.6.18-xen | ||
mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod --with=megaraid_sas /boot/initrd-2.6.18-xen.img 2.6.18-xen | mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod --with=megaraid_sas /boot/initrd-2.6.18-xen.img 2.6.18-xen | ||
| + | |||
| + | ''' Note on: Network Problems ''' | ||
| + | Broadcom drivers have problems, network is OK when xend started without network bridges, see /etc/xen/xend-config.sxp | ||
| + | |||
| + | Starting bridge with: /etc/xen/scripts/network-bridge start | ||
| + | results in a not working network configuration in dom0. | ||
| + | |||
| + | see [http://lists.xensource.com/archives/html/xen-users/2007-07/msg00768.html network-bridge causes network to hang] | ||
| + | |||
| + | '''swiotlb=128''' has to be set in menu.list since we have 8GB RAM, without youll get error messages like : | ||
| + | Sep 7 13:14:22 test-11 kernel: PCI-DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:02:01.0 | ||
| + | see also: [http://lists.xensource.com/archives/html/xen-devel/2007-09/msg00140.html SW-IOMMU space] | ||
* change/add /boot/grub/menu.list: | * change/add /boot/grub/menu.list: | ||
| Line 210: | Line 116: | ||
root (hd0,0) | root (hd0,0) | ||
kernel /xen-3.1.gz | kernel /xen-3.1.gz | ||
| − | module /vmlinuz-2.6-xen ro root=/dev/md1 | + | module /vmlinuz-2.6-xen ro root=/dev/md1 swiotlb=128 |
module /initrd-2.6.18-xen.img | module /initrd-2.6.18-xen.img | ||
| + | |||
| + | ''' Boot into your new system''': | ||
[root@kribbe ~]# xm dmesg | [root@kribbe ~]# xm dmesg | ||
| Line 236: | Line 144: | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| Line 309: | Line 210: | ||
''' SWAP ''' | ''' SWAP ''' | ||
| + | |||
| + | '''Note:''' swap in a lvm is not performant | ||
[root@kribbe tmp]# lvcreate -L1G -nXen_CentOS5_Test_swap XenVG | [root@kribbe tmp]# lvcreate -L1G -nXen_CentOS5_Test_swap XenVG | ||
Logical volume "Xen_CentOS5_Test_swap" created | Logical volume "Xen_CentOS5_Test_swap" created | ||
| Line 314: | Line 217: | ||
Setting up swapspace version 1, size = 1073737 kB | Setting up swapspace version 1, size = 1073737 kB | ||
| − | === | + | ==== Usage of LVM2 on Centos xvda disks ==== |
| − | + | Taken from [http://lists.xensource.com/archives/html/xen-users/2007-06/msg00656.html xen-users archive]. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| + | If you are using a lvm2 partition as a disk (xvda) in a dom_u and this lvm is itself partitioned you can't use just | ||
| − | ''' | + | '''mount -o loop <logical volume> <mountpoint>''', instead you have to give the first sector of the partition you want to mount. This can be calculated by multiplying the start sector with its size |
| − | |||
| − | |||
| − | + | '''Example''': | |
| − | + | [root@hilde xen]# fdisk -l -u /dev/mapper/xenvg-hooiwagenc4532 | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | ''' | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | # | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | Disk /dev/mapper/xenvg-hooiwagenc4532: 53.6 GB, 53687091200 bytes | |
| + | 255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors | ||
| + | Units = sectors of 1 * 512 = 512 bytes | ||
| − | + | Device Boot Start End Blocks Id System | |
| − | + | /dev/mapper/xenvg-hooiwagenc4532p1 * 63 208844 104391 83 Linux | |
| − | + | /dev/mapper/xenvg-hooiwagenc4532p2 208845 2249099 1020127+ 82 Linux swap / Solaris | |
| − | + | /dev/mapper/xenvg-hooiwagenc4532p3 2249100 104856254 51303577+ 83 Linux | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | [root@hilde xen]# mount -o loop,offset=32256 /dev/mapper/xenvg-hooiwagenc4532 /mnt2 | |
| − | # | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | [root@hilde xen]# ls /mnt2 | |
| − | + | bin dev home lib media opt root selinux sys usr | |
| + | boot etc initrd lost+found mnt proc sbin srv tmp var | ||
| + | == Virtualize Physical Nodes == | ||
| + | # On DOM_0 Create a lvm for the DOM_U | ||
| + | # Mount this lvm on DOM_0 | ||
| + | # On Physical Node change runlevel to s | ||
| + | # setup a exclude-list file, for details see [http://www.eisxen.org/54.html eisxen] | ||
| + | # cd /;tar csp --exclude-from=/rootfs/exclude.lst . |ssh <DOM_0> "cd /<mounted LVM>; tar xsp" | ||
| + | # change the fstab of the dom_u | ||
| + | Example exclude.lst, node specific, dont just copy, see also: [http://www.eisxen.org/54.html eisxen] | ||
| + | This list must not have an empty line at the end. | ||
| + | less /rootfs/exclude.lst | ||
| + | ./boot | ||
| + | ./data | ||
| + | ./datastore | ||
| + | ./datastore2 | ||
| + | ./proc | ||
| + | ./rootfs | ||
| + | ./etc/mtab | ||
| + | ./sys | ||
| + | ./project | ||
| − | + | === DOM_U is a centos-3.x === | |
| − | + | In order to run this as a virtual maschine, the dom_u might need a new version of e2fschk and module-init-tools. | |
| − | + | Therfore, on DOM_0 mount the DOM_U fs created in an earlier step, get e2fsprogs-1.39.tar.gz module-init-tools-3.2.tar.gz (these versions worked for us) to this directory. | |
| − | + | The following commands worked for me. | |
| − | + | * chroot to dom_u mountpoint | |
| − | + | * extract, compile, install e2fschk and module-init-tools | |
| − | + | * cp -r DOM_0:/lib/modules/<running version> DOM_U:/lib/modules | |
| − | + | * for e2fs do: tar xvfz e2fsprogs-1.39.tar.gz ;cd e2fsprogs-1.39; ./configure; make; make install | |
| − | + | * for modutils: tar xvfz module-init-tools-3.2.tar.gz; cd module-init-tools-3.2; ./configure --prefix=/ ;make moveold; make; make install | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | == CentOS 5 with Virtualization package == | |
| − | + | * Installed Version: | |
| − | + | 2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux | |
| − | |||
| − | |||
| − | |||
| − | |||
| + | * Run yum update, Version: | ||
| + | 2.6.18-8.1.15.el5xen #1 SMP Mon Oct 22 09:01:12 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux | ||
| − | + | * xen: | |
| − | + | [root@kribbe ~]# xm dmesg | |
| − | + | __ __ _____ ___ _____ ____ _ ____ | |
| − | + | \ \/ /___ _ __ |___ / / _ \ |___ / _ __ ___| ___| ___| | ___| | |
| − | + | \ // _ \ '_ \ |_ \| | | | |_ \ __| '__/ __|___ \ / _ \ |___ \ | |
| − | + | / \ __/ | | | ___) | |_| | ___) |__| | | (__ ___) | __/ |___) | | |
| − | + | /_/\_\___|_| |_| |____(_)___(_)____/ |_| \___|____(_)___|_|____/ | |
| − | |||
| − | |||
| − | / | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | http://www.cl.cam.ac.uk/netos/xen | |
| − | + | University of Cambridge Computer Laboratory | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | - | + | Xen version 3.0.3-rc5-8.1.15.el5 (mockbuild@) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Mon Oct 22 08:25:42 EDT 2007 |
| − | + | Latest ChangeSet: unavailable | |
| − | + | .... | |
| − | + | (XEN) VMXON is done | |
| − | + | (XEN) Dom0 has maximum 4 VCPUs | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | * lvm2, bridge-utils | |
| − | * | + | Installed Packages: |
| − | + | Name : lvm2 | |
| − | + | Arch : x86_64 | |
| − | + | Version: 2.02.16 | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | Name : bridge-utils | |
| + | Arch : x86_64 | ||
| + | Version: 1.1 | ||
| − | |||
| − | + | * yum install vnc | |
| − | + | * use partition /dev/md3 mounted on /project for lvm | |
| − | + | comment out ..... /project | |
| + | umount /project | ||
| + | pvcreate /dev/md3 | ||
| + | vgcreate XenVG /dev/md3 | ||
| + | lvcreate -L10G -nXen_CentOS5_Test XenVG | ||
| + | mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test | ||
| + | lvcreate -L1G -nXen_CentOS5_Test_swap XenVG | ||
| + | mkswap /dev/XenVG/Xen_CentOS5_Test_swap | ||
| − | ==== | + | == Running various centOS Versions as VMs on kribbe == |
| + | === Virt-manager for installation === | ||
| + | virt-install can be used for VM installation. No pypxeboot and patched udhcp needed. | ||
| − | + | ==== CentOS 5 64-Bit in VM ==== | |
| + | set up kickstart file for VM, than | ||
| + | virt-install -p --location=http://www.dutchgrid.nl/mirror/centos/5/os/x86_64 --noautoconsole \ | ||
| + | --file=/dev/XenVG/Xen_CentOS5_Test --name=domU001 --ram=1024 --mac=00:16:3e:ab:61:0b\ | ||
| + | -x "ks=http://stal.nikhef.nl/ks/xen-test-koebrug.ks ip=194.171.97.11 \ | ||
| + | netmask=255.255.255.0 dns=192.16.186.253 gateway=194.171.97.254" | ||
| − | + | '''MAC-ADDRESS''': can be calculated from last the 3 numbers of you guest IP by (ex:): | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | printf 00:16:3e:%02x:%02x:%02x"\n" 171 97 11 | |
| − | + | ||
| − | + | '''00:16:3e:ab:61:0b''' | |
| − | + | This has to be defined in the dhcp servers dhcp.conf. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | '''Connect to the console of the starting vm:''' | |
| + | * xm list, lists the VMs: | ||
| + | [root@kribbe ~]# xm list | ||
| + | Name ID Mem(MiB) VCPUs State Time(s) | ||
| + | Domain-0 0 7025 4 r----- 1056.7 | ||
| + | domU001 8 1023 1 -b---- 490.8 | ||
| − | + | * xm console <ID> or <Name> of the VM (s.a.) creates a console of the VM | |
| − | |||
| − | + | ''' Worked ''' | |
| − | |||
| − | |||
| − | + | '''Old Note''': no vnc option in the virt-install command, | |
| + | * --vnc option removed, does not boot. | ||
| + | * during boot of the VM answer: q: graphics support: no | ||
| − | + | But this one worked fine with a VNC console (both graphical and text based): | |
| − | + | virt-install --force -r 512 -m 00:16:3e:34:4b:6e -n mestkar -f /dev/vg0/mestkar.sys -l http://spiegel.nikhef.nl/mirror/centos/5/os/x86_64 -x "ks=http://www.nikhef.nl/grid/ndpf/files/local/install/mestkar.nikhef.nl.ks ksdevice=eth0" --vnc --vcpus=1 -b xenbr0 | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | ''' | + | ''' On VM:''' |
| − | * | + | * '''koebrug.nikhef.nl''' |
| − | + | /etc/redhat-release | |
| − | + | CentOS release 5 (Final) | |
| − | + | [root@koebrug ~]# uname -a | |
| − | + | Linux koebrug.nikhef.nl 2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | ==== CentOS 4.5 64-Bit in VM ==== | |
| − | + | * '''silo.nikhef.nl''' | |
| + | [root@silo ~]# uname -a | ||
| + | Linux silo.nikhef.nl 2.6.9-55.ELxenU #1 SMP Wed May 2 14:29:26 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux | ||
| + | [root@silo ~]# cat /etc/redhat-release | ||
| + | CentOS release 4.5 (Final) | ||
| − | |||
| − | + | ''' 64 Bit DOM_U in 64 Bit DOM_0''' seems to be no problem. '''32Bit''' installations '''can not be done''' in this way, apropriate kernel needed''' Looking at '''virsh''' | |
| − | |||
| − | |||
| − | |||
| − | + | === Using VIRSH === | |
| − | + | Information on virsh can be found at: | |
| − | + | [http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU centos-wiki] | |
| − | + | or | |
| − | + | [http://www.linuxtopia.org/online_books/centos5/centos_5_xen_virtualization/index.html Red Hat Virtualization] | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | virsh is another virtualization management tool (built around the libvirt management API) which is now capable to deal with xen, future versions might also be able to manage kvm or qemu. | |
| − | + | Since we plan to use xen we should stick with xm directly. | |
| − | + | === Virtual Maschine management with '''xm''' === | |
| − | + | based on: [http://tx.downloads.xensource.com/downloads/docs/user/#SECTION03120000000000000000 xen-user-docu] | |
| − | |||
| − | = | + | = CentOS 5 i386-System -- Quattor generated= |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | == Problems == | |
| − | * | + | * xenconsole: Could not read tty from store: No such file or directory |
| − | + | root@hilde xen]# xm create koebrug -c | |
| − | + | Using config file "./koebrug". | |
| − | + | Started domain koebrug | |
| − | + | xenconsole: Could not read tty from store: No such file or directory | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | '''Try: ''' /etc/init.d/xend restart | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| + | * virt-manager not working [http://bugs.centos.org/view.php?id=2383 centos-bug] | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | * '''Post-Quattor''' steps to prevent quattor from revoking installation/configuration operations. | |
| − | + | on kribbe: | |
| − | + | chkconfig cdp-listend off | |
| − | + | chkconfig ncm-cdispd off | |
| − | |||
| − | + | on stal: | |
| − | + | [root@stal ~]# aii-shellfe --boot kribbe.nikhef.nl | |
| − | + | to give back the control to quattor (fresh install next boot, will also switch on cdp-listen and ncm-cdispd) | |
| − | + | [root@stal ~]# aii-shellfe --install kribbe.nikhef.nl | |
| − | + | * Mount external storage for files that should survive a reboot/installation | |
| + | mount -tnfs tbn15.nikhef.nl:/export/array2/local/tbadmin /tmpmnt | ||
| + | * '''Hostname''': kribbe.nikhef.nl | ||
| + | * '''Kernel''': 2.6.18-8.1.8.el5xen | ||
| + | * '''HD-Layout''': | ||
| + | [root@kribbe ~]# sfdisk -l /dev/sdb | ||
| − | + | Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track | |
| − | + | Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 | |
| − | |||
| − | + | Device Boot Start End #cyls #blocks Id System | |
| − | + | /dev/sdb1 * 0+ 32 33- 265041 fd Linux raid autodetect | |
| + | /dev/sdb2 33 51025 50993 409601272+ fd Linux raid autodetect | ||
| + | /dev/sdb3 51026 58580 7555 60685537+ fd Linux raid autodetect | ||
| + | /dev/sdb4 58581 60800 2220 17832150 5 Extended | ||
| + | /dev/sdb5 58581+ 60147 1567- 12586896 fd Linux raid autodetect | ||
| + | /dev/sdb6 60148+ 60669 522- 4192933+ fd Linux raid autodetect | ||
| + | /dev/sdb7 60670+ 60800 131- 1052226 fd Linux raid autodetect | ||
| − | + | ==================== | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | ===== | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | [root@kribbe ~]# cat /proc/mdstat | |
| − | + | Personalities : [raid1] | |
| − | + | md0 : active raid1 sdb1[1] sda1[0] | |
| − | + | 264960 blocks [2/2] [UU] | |
| − | + | ||
| − | + | md3 : active raid1 sdb2[1] sda2[0] | |
| − | + | 409601152 blocks [2/2] [UU] | |
| − | + | ||
| − | + | md5 : active raid1 sdb3[1] sda3[0] | |
| − | + | 60685440 blocks [2/2] [UU] | |
| − | + | ||
| − | + | md2 : active raid1 sdb6[1] sda6[0] | |
| − | + | 4192832 blocks [2/2] [UU] | |
| − | + | ||
| + | md4 : active raid1 sdb7[1] sda7[0] | ||
| + | 1052160 blocks [2/2] [UU] | ||
| + | |||
| + | md1 : active raid1 sdb5[1] sda5[0] | ||
| + | 12586816 blocks [2/2] [UU] | ||
| + | |||
| + | unused devices: <none> | ||
| − | |||
| − | |||
| − | |||
| − | + | === Example installation of a CentOS5 VM === | |
| − | + | ''' Create a VM in lvm partition''' | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | # Setup lvm as stated above | |
| − | + | # Mount VM root fs: mount /dev/XenVG/Xen_CentOS5_Test /mnt | |
| − | + | # extract archive: | |
| + | # [root@kribbe tmp]# bunzip2 centos.5-0.20070424.img.tar.bz2 | ||
| + | # [root@kribbe tmp]# tar xvf centos.5-0.20070424.img.tar | ||
| + | # loopmount image | ||
| + | # [root@kribbe tmp]# mkdir /img | ||
| + | # [root@kribbe tmp]# mount -o loop centos.5-0.img /img | ||
| + | # cp image to logical volume | ||
| + | # [root@kribbe img]# cp -a * /mnt/ | ||
| + | # '''Change Root PW''' in VM: | ||
| + | [root@kribbe mnt]# chroot /mnt | ||
| + | bash-3.1# pwconv | ||
| + | bash-3.1# passwd | ||
| + | Changing password for user root. | ||
| + | # Set Hostname of VM: /etc/sysconfig/network | ||
| + | NETWORKING=yes | ||
| + | HOSTNAME=koebrug.nikhef.nl | ||
| − | + | # no swap in VM: /etc/fstab | |
| + | #/dev/sda2 none swap sw 0 0 | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | ''' setup of VM start files''' | |
| − | + | ---- | |
| − | + | ---- | |
| − | |||
| − | === | + | === SELinux, Postinstallation steps === |
| − | + | * Cleaning up daemons, pay attention to ntp, does the dom_U has the correct time? | |
| + | /sbin/service microcode_ctl stop | ||
| + | /sbin/chkconfig --del microcode_ctl | ||
| + | /sbin/service smartd stop | ||
| + | /sbin/chkconfig --del smartd | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | * SELinux considerations: from [http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU SELinux in domU] | |
| + | If you are using SELinux, it is important to check that the image has the correct security context (xen_image_t), or access to the virtual disk will be denied to the domU system. You can check this with ls: | ||
| − | + | # ls -Z /srv/xen/mailserver.img | |
| + | -rw-r--r-- root root user_u:object_r:xen_image_t /srv/xen/mailserver.img | ||
| − | + | If you are having trouble setting the right file context, please have a look at the Xen tips and tricks page. At any rate, turning off SELinux, as some howtos on this subject advise is a very poor workaround. Reading two manual pages (semanage(8) and restorecon(8)), for an extra layer of security is a good trade! | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| + | = Memory and network problems = | ||
| − | + | The CentOS 5 (RHEL5) release of Xen is not really stable; it changed it's behaviour over time from 'not working at all' in CentOS 5.0 and 5.1 to 'working' in '5.2', and with further updates it appears as though rather important changes find their way into the Xen part of this otherwise stable distro. | |
| − | |||
| − | |||
| − | + | Recently, and this is 5.3 on April 22, 2009, I started to see spontaneous reboots of my Dom0. Not much in the syslog, but some suspicious entries about memory squeeze: | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | Apr 22 10:29:09 toom kernel: xen_net: Memory squeeze in netback driver. | |
| + | Apr 22 10:29:14 toom kernel: printk: 8 messages suppressed. | ||
| − | + | This relates to a [https://bugzilla.redhat.com/show_bug.cgi?id=454285 known bug]; a suggested solution is to limit the Dom0 memory to prevent 'ballooning'. This is what I did: | |
| − | |||
| − | |||
| − | + | in /etc/grub.conf: | |
| − | + | kernel /xen.gz-2.6.18-128.1.6.el5 com2=9600,8n1 console=com2,vga dom0_mem=1024M | |
| − | |||
| − | |||
| − | + | and in /etc/xen/xend-config.sxp: | |
| − | + | (dom0-min-mem 1024) | |
| − | + | Making 1024 (megabyte) both the minimum and maximum of the dom0 memory. This could affect performance, but I haven't seen the squeeze message since. | |
| − | |||
Latest revision as of 14:34, 21 October 2009
Docu on CentOS
Some links:
CentOS-5 Documentation including docs on Virtualization
check PCI-DMA: Out of SW-IOMMU space ERROR
CentOS 5
These are just notes taken during the installation, it is not meant as a howto or something like that
Hardware Dell 1950
Additional Stuff
mercurial is a distributed version control system. to use it on centos-5 X86_64 do:
- wget http://www.selenic.com/mercurial/release/mercurial-0.9.5.tar.gz
- tar xvfz mercurial-0.9.5.tar.gz
- make install
- Add export PYTHONPATH=/usr/local/lib64/python2.4/site-packages:${PYTHONPATH} to the environment
compile libvirt
- wget ftp://ftp.easynet.be/gnu/gettext/gettext-0.17.tar.gz; tar xvfz gettext-0.17.tar.gz; cd gettext-0.17; ./configure; make; make install
- yum install gnutls gnutls-devel
- getting latest libvirt see: libvirt cvs
- cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs login; password anoncvs cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs co libvirt
- cd to libvirt dir and apply patch patch -p1 < configure.patch
- in libvirt dir do: ./autogen.sh; make; make install
virt-install needs:
- virtviewer, this needs
- rpm -ivh http://www.lfarkas.org/linux/packages/centos/5/x86_64/gtk-vnc-0.2.0-4.x86_64.rpm
- autogen.sh needs:
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
CentOS 5 X86_64, Minimal system
Additional Packages
Minimal System installed via kickstart, packages:
# Packages groups/list %packages --resolvedeps --ignoremissing openssh openssh-server wget perl-libnet perl-MIME-Base64 perl-URI perl-Digest-MD5 perl-libwww-perl perl-XML-Parser perl-DBI perl-Crypt-SSLeay lsof
Post Install, get the following packages
Some of these are not needed, has to be cleaned:
yum install \
gcc make zlib-devel zlib python-devel curses libncurses-devel ncurses-devel openssl openssl-devel \
`yum search xorg-x11|grep dev|grep x86_64|awk -F . '{print $1}'` \
bridge-utils tetex tetex-latex transfig libtool-ltdl dev86 glibc-devel
xen 3.1.2-rc from mercurial
For 64-bit Dom_0 and 32-bit Dom_U kombination:, the following is taken from: xensource install notes
Dell 2900 modules to include in mkinitrd mkinitrd -v -f --with=sd_mod --with=scsi_mod --with=megaraid_mm --with=megaraid_sas --with=megaraid_mbox initrd-2.6.16.29-xen.img 2.6.16.29-xen dom0 64 bit and domU 32 bit from source code Build and install xen: make xen make install-xen Build and install the dom0: make linux-2.6-xen0-prep cd build-linux-*-xen0-x86_64 make menuconfig make make install modules_install And finally build and install 32 bit domU: make XEN_TARGET_ARCH=x86_32 linux-2.6-xenU-prep cd build-linux-*-xen0-x86_32 make ARCH=i386 menuconfig make ARCH=i386 make ARCH=i386 install modules_install
hg pull ... cd xen-3.1 wget http://www.broadcom.com/docs/driver_download/NXII/linux-1.5.10c.zip unzip, use broadcom sources for make world make install depmod 2.6.18-xen mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod --with=megaraid_sas /boot/initrd-2.6.18-xen.img 2.6.18-xen
Note on: Network Problems Broadcom drivers have problems, network is OK when xend started without network bridges, see /etc/xen/xend-config.sxp
Starting bridge with: /etc/xen/scripts/network-bridge start results in a not working network configuration in dom0.
see network-bridge causes network to hang
swiotlb=128 has to be set in menu.list since we have 8GB RAM, without youll get error messages like :
Sep 7 13:14:22 test-11 kernel: PCI-DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:02:01.0
see also: SW-IOMMU space
- change/add /boot/grub/menu.list:
title CentOS (xen-3.1.2-rc2 2.6.18 compiled)
root (hd0,0)
kernel /xen-3.1.gz
module /vmlinuz-2.6-xen ro root=/dev/md1 swiotlb=128
module /initrd-2.6.18-xen.img
Boot into your new system:
[root@kribbe ~]# xm dmesg
__ __ _____ _ ____ ____ \ \/ /___ _ __ |___ / / | |___ \ _ __ ___|___ \ \ // _ \ '_ \ |_ \ | | __) |__| '__/ __| __) | / \ __/ | | | ___) || |_ / __/|__| | | (__ / __/ /_/\_\___|_| |_| |____(_)_(_)_____| |_| \___|_____|
http://www.cl.cam.ac.uk/netos/xen University of Cambridge Computer Laboratory
Xen version 3.1.2-rc2 (root@nikhef.nl) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Tue Nov 13 15:02:14 CET 2007 Latest ChangeSet: unavailable . . . (XEN) HVM: VMX enabled (XEN) VMX: MSR intercept bitmap enabled (XEN) CPU0: Intel(R) Xeon(R) CPU 5150 @ 2.66GHz stepping 06 . . .
LVM 2
Due to performance reasons VMs should be installed in LVMs. For recovering a LVM in a Raid-1 see:Recovery of RAID and LVM2 Volumes
Installed Version:
Installed Packages Name : lvm2 Arch : i386 Version: 2.02.16 Release: 3.el5 Size : 5.5 M Repo : installed Summary: Userland logical volume management tools
NOTE: for having the live-migration-posibility we have to look at Cluster-LVM or something like that.
To get started ....
Setting up LVM2 on kribbe
Using /dev/srv resp. /dev/md3 for LVM we have to:
- comment out /dev/md3 in /etc/fstab
- umount /dev/srv (only once)
- Creating the physical volume and XenVG Volumegroup:
[root@kribbe ~]# pvcreate /dev/md3 Physical volume "/dev/md3" successfully created [root@kribbe ~]# vgcreate XenVG /dev/md3 Volume group "XenVG" successfully created
Within this volume-group we can now manage the partitions for the virtual maschines using lvcreate, lvresize, .... commands
Example: lvm for a CentOS5 test VM
Root FS, only one partition
[root@kribbe tmp]# lvcreate -L10G -nXen_CentOS5_Test XenVG
Logical volume "Xen_CentOS5_Test" created
root@kribbe tmp]# mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1310720 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
SWAP
Note: swap in a lvm is not performant
[root@kribbe tmp]# lvcreate -L1G -nXen_CentOS5_Test_swap XenVG Logical volume "Xen_CentOS5_Test_swap" created [root@kribbe tmp]# mkswap /dev/XenVG/Xen_CentOS5_Test_swap Setting up swapspace version 1, size = 1073737 kB
Usage of LVM2 on Centos xvda disks
Taken from xen-users archive.
If you are using a lvm2 partition as a disk (xvda) in a dom_u and this lvm is itself partitioned you can't use just
mount -o loop <logical volume> <mountpoint>, instead you have to give the first sector of the partition you want to mount. This can be calculated by multiplying the start sector with its size
Example:
[root@hilde xen]# fdisk -l -u /dev/mapper/xenvg-hooiwagenc4532
Disk /dev/mapper/xenvg-hooiwagenc4532: 53.6 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/mapper/xenvg-hooiwagenc4532p1 * 63 208844 104391 83 Linux
/dev/mapper/xenvg-hooiwagenc4532p2 208845 2249099 1020127+ 82 Linux swap / Solaris
/dev/mapper/xenvg-hooiwagenc4532p3 2249100 104856254 51303577+ 83 Linux
[root@hilde xen]# mount -o loop,offset=32256 /dev/mapper/xenvg-hooiwagenc4532 /mnt2 [root@hilde xen]# ls /mnt2 bin dev home lib media opt root selinux sys usr boot etc initrd lost+found mnt proc sbin srv tmp var
Virtualize Physical Nodes
- On DOM_0 Create a lvm for the DOM_U
- Mount this lvm on DOM_0
- On Physical Node change runlevel to s
- setup a exclude-list file, for details see eisxen
- cd /;tar csp --exclude-from=/rootfs/exclude.lst . |ssh <DOM_0> "cd /<mounted LVM>; tar xsp"
- change the fstab of the dom_u
Example exclude.lst, node specific, dont just copy, see also: eisxen This list must not have an empty line at the end. less /rootfs/exclude.lst ./boot ./data ./datastore ./datastore2 ./proc ./rootfs ./etc/mtab ./sys ./project
DOM_U is a centos-3.x
In order to run this as a virtual maschine, the dom_u might need a new version of e2fschk and module-init-tools. Therfore, on DOM_0 mount the DOM_U fs created in an earlier step, get e2fsprogs-1.39.tar.gz module-init-tools-3.2.tar.gz (these versions worked for us) to this directory. The following commands worked for me.
- chroot to dom_u mountpoint
- extract, compile, install e2fschk and module-init-tools
- cp -r DOM_0:/lib/modules/<running version> DOM_U:/lib/modules
- for e2fs do: tar xvfz e2fsprogs-1.39.tar.gz ;cd e2fsprogs-1.39; ./configure; make; make install
- for modutils: tar xvfz module-init-tools-3.2.tar.gz; cd module-init-tools-3.2; ./configure --prefix=/ ;make moveold; make; make install
CentOS 5 with Virtualization package
- Installed Version:
2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
- Run yum update, Version:
2.6.18-8.1.15.el5xen #1 SMP Mon Oct 22 09:01:12 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
- xen:
[root@kribbe ~]# xm dmesg __ __ _____ ___ _____ ____ _ ____ \ \/ /___ _ __ |___ / / _ \ |___ / _ __ ___| ___| ___| | ___| \ // _ \ '_ \ |_ \| | | | |_ \ __| '__/ __|___ \ / _ \ |___ \ / \ __/ | | | ___) | |_| | ___) |__| | | (__ ___) | __/ |___) | /_/\_\___|_| |_| |____(_)___(_)____/ |_| \___|____(_)___|_|____/ http://www.cl.cam.ac.uk/netos/xen University of Cambridge Computer Laboratory Xen version 3.0.3-rc5-8.1.15.el5 (mockbuild@) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Mon Oct 22 08:25:42 EDT 2007 Latest ChangeSet: unavailable .... (XEN) VMXON is done (XEN) Dom0 has maximum 4 VCPUs
- lvm2, bridge-utils
Installed Packages: Name : lvm2 Arch : x86_64 Version: 2.02.16 Name : bridge-utils Arch : x86_64 Version: 1.1
- yum install vnc
- use partition /dev/md3 mounted on /project for lvm
comment out ..... /project umount /project pvcreate /dev/md3 vgcreate XenVG /dev/md3 lvcreate -L10G -nXen_CentOS5_Test XenVG mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test lvcreate -L1G -nXen_CentOS5_Test_swap XenVG mkswap /dev/XenVG/Xen_CentOS5_Test_swap
Running various centOS Versions as VMs on kribbe
Virt-manager for installation
virt-install can be used for VM installation. No pypxeboot and patched udhcp needed.
CentOS 5 64-Bit in VM
set up kickstart file for VM, than
virt-install -p --location=http://www.dutchgrid.nl/mirror/centos/5/os/x86_64 --noautoconsole \ --file=/dev/XenVG/Xen_CentOS5_Test --name=domU001 --ram=1024 --mac=00:16:3e:ab:61:0b\ -x "ks=http://stal.nikhef.nl/ks/xen-test-koebrug.ks ip=194.171.97.11 \ netmask=255.255.255.0 dns=192.16.186.253 gateway=194.171.97.254"
MAC-ADDRESS: can be calculated from last the 3 numbers of you guest IP by (ex:):
printf 00:16:3e:%02x:%02x:%02x"\n" 171 97 11 00:16:3e:ab:61:0b
This has to be defined in the dhcp servers dhcp.conf.
Connect to the console of the starting vm:
- xm list, lists the VMs:
[root@kribbe ~]# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 7025 4 r----- 1056.7 domU001 8 1023 1 -b---- 490.8
- xm console <ID> or <Name> of the VM (s.a.) creates a console of the VM
Worked
Old Note: no vnc option in the virt-install command,
- --vnc option removed, does not boot.
- during boot of the VM answer: q: graphics support: no
But this one worked fine with a VNC console (both graphical and text based):
virt-install --force -r 512 -m 00:16:3e:34:4b:6e -n mestkar -f /dev/vg0/mestkar.sys -l http://spiegel.nikhef.nl/mirror/centos/5/os/x86_64 -x "ks=http://www.nikhef.nl/grid/ndpf/files/local/install/mestkar.nikhef.nl.ks ksdevice=eth0" --vnc --vcpus=1 -b xenbr0
On VM:
- koebrug.nikhef.nl
/etc/redhat-release CentOS release 5 (Final) [root@koebrug ~]# uname -a Linux koebrug.nikhef.nl 2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
CentOS 4.5 64-Bit in VM
- silo.nikhef.nl
[root@silo ~]# uname -a Linux silo.nikhef.nl 2.6.9-55.ELxenU #1 SMP Wed May 2 14:29:26 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux [root@silo ~]# cat /etc/redhat-release CentOS release 4.5 (Final)
64 Bit DOM_U in 64 Bit DOM_0 seems to be no problem. 32Bit installations can not be done in this way, apropriate kernel needed Looking at virsh
Using VIRSH
Information on virsh can be found at: centos-wiki or Red Hat Virtualization
virsh is another virtualization management tool (built around the libvirt management API) which is now capable to deal with xen, future versions might also be able to manage kvm or qemu.
Since we plan to use xen we should stick with xm directly.
Virtual Maschine management with xm
based on: xen-user-docu
CentOS 5 i386-System -- Quattor generated
Problems
- xenconsole: Could not read tty from store: No such file or directory
root@hilde xen]# xm create koebrug -c Using config file "./koebrug". Started domain koebrug xenconsole: Could not read tty from store: No such file or directory
Try: /etc/init.d/xend restart
- virt-manager not working centos-bug
- Post-Quattor steps to prevent quattor from revoking installation/configuration operations.
on kribbe: chkconfig cdp-listend off chkconfig ncm-cdispd off on stal: [root@stal ~]# aii-shellfe --boot kribbe.nikhef.nl to give back the control to quattor (fresh install next boot, will also switch on cdp-listen and ncm-cdispd) [root@stal ~]# aii-shellfe --install kribbe.nikhef.nl
- Mount external storage for files that should survive a reboot/installation
mount -tnfs tbn15.nikhef.nl:/export/array2/local/tbadmin /tmpmnt
- Hostname: kribbe.nikhef.nl
- Kernel: 2.6.18-8.1.8.el5xen
- HD-Layout:
[root@kribbe ~]# sfdisk -l /dev/sdb
Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdb1 * 0+ 32 33- 265041 fd Linux raid autodetect
/dev/sdb2 33 51025 50993 409601272+ fd Linux raid autodetect
/dev/sdb3 51026 58580 7555 60685537+ fd Linux raid autodetect
/dev/sdb4 58581 60800 2220 17832150 5 Extended
/dev/sdb5 58581+ 60147 1567- 12586896 fd Linux raid autodetect
/dev/sdb6 60148+ 60669 522- 4192933+ fd Linux raid autodetect
/dev/sdb7 60670+ 60800 131- 1052226 fd Linux raid autodetect
====================
[root@kribbe ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
264960 blocks [2/2] [UU]
md3 : active raid1 sdb2[1] sda2[0]
409601152 blocks [2/2] [UU]
md5 : active raid1 sdb3[1] sda3[0]
60685440 blocks [2/2] [UU]
md2 : active raid1 sdb6[1] sda6[0]
4192832 blocks [2/2] [UU]
md4 : active raid1 sdb7[1] sda7[0]
1052160 blocks [2/2] [UU]
md1 : active raid1 sdb5[1] sda5[0]
12586816 blocks [2/2] [UU]
unused devices: <none>
Example installation of a CentOS5 VM
Create a VM in lvm partition
- Setup lvm as stated above
- Mount VM root fs: mount /dev/XenVG/Xen_CentOS5_Test /mnt
- extract archive:
- [root@kribbe tmp]# bunzip2 centos.5-0.20070424.img.tar.bz2
- [root@kribbe tmp]# tar xvf centos.5-0.20070424.img.tar
- loopmount image
- [root@kribbe tmp]# mkdir /img
- [root@kribbe tmp]# mount -o loop centos.5-0.img /img
- cp image to logical volume
- [root@kribbe img]# cp -a * /mnt/
- Change Root PW in VM:
[root@kribbe mnt]# chroot /mnt bash-3.1# pwconv bash-3.1# passwd Changing password for user root.
- Set Hostname of VM: /etc/sysconfig/network
NETWORKING=yes HOSTNAME=koebrug.nikhef.nl
- no swap in VM: /etc/fstab
#/dev/sda2 none swap sw 0 0
setup of VM start files
SELinux, Postinstallation steps
- Cleaning up daemons, pay attention to ntp, does the dom_U has the correct time?
/sbin/service microcode_ctl stop /sbin/chkconfig --del microcode_ctl /sbin/service smartd stop /sbin/chkconfig --del smartd
- SELinux considerations: from SELinux in domU
If you are using SELinux, it is important to check that the image has the correct security context (xen_image_t), or access to the virtual disk will be denied to the domU system. You can check this with ls:
# ls -Z /srv/xen/mailserver.img -rw-r--r-- root root user_u:object_r:xen_image_t /srv/xen/mailserver.img
If you are having trouble setting the right file context, please have a look at the Xen tips and tricks page. At any rate, turning off SELinux, as some howtos on this subject advise is a very poor workaround. Reading two manual pages (semanage(8) and restorecon(8)), for an extra layer of security is a good trade!
Memory and network problems
The CentOS 5 (RHEL5) release of Xen is not really stable; it changed it's behaviour over time from 'not working at all' in CentOS 5.0 and 5.1 to 'working' in '5.2', and with further updates it appears as though rather important changes find their way into the Xen part of this otherwise stable distro.
Recently, and this is 5.3 on April 22, 2009, I started to see spontaneous reboots of my Dom0. Not much in the syslog, but some suspicious entries about memory squeeze:
Apr 22 10:29:09 toom kernel: xen_net: Memory squeeze in netback driver. Apr 22 10:29:14 toom kernel: printk: 8 messages suppressed.
This relates to a known bug; a suggested solution is to limit the Dom0 memory to prevent 'ballooning'. This is what I did:
in /etc/grub.conf:
kernel /xen.gz-2.6.18-128.1.6.el5 com2=9600,8n1 console=com2,vga dom0_mem=1024M
and in /etc/xen/xend-config.sxp:
(dom0-min-mem 1024)
Making 1024 (megabyte) both the minimum and maximum of the dom0 memory. This could affect performance, but I haven't seen the squeeze message since.
