Difference between revisions of "Xen on CentOS 5 - Notes"

From PDP/Grid Wiki
Jump to navigationJump to search
 
(65 intermediate revisions by 4 users not shown)
Line 1: Line 1:
−
= CentOS 5 from pxe/kickstart =
+
= Docu on CentOS =
 +
Some links:
  
 +
[http://www.centos.org/docs/5/ CentOS-5 Documentation including docs on Virtualization]
  
 +
check [https://bugzilla.redhat.com/show_bug.cgi?id=219216 PCI-DMA: Out of SW-IOMMU space ERROR]
 +
 +
= CentOS 5 =
 
----
 
----
 
'''These are just notes taken during the installation, it is ''not'' meant as a howto or something like that'''
 
'''These are just notes taken during the installation, it is ''not'' meant as a howto or something like that'''
 
----
 
----
 +
 +
Hardware Dell 1950
 +
== Additional Stuff ==
 +
'''mercurial''' is a distributed version control system.
 +
to use it on centos-5 X86_64 do:
 +
* wget http://www.selenic.com/mercurial/release/mercurial-0.9.5.tar.gz
 +
* tar xvfz mercurial-0.9.5.tar.gz 
 +
* make install
 +
* Add export PYTHONPATH=/usr/local/lib64/python2.4/site-packages:${PYTHONPATH} to the environment
 +
 +
compile '''libvirt'''
 +
* wget ftp://ftp.easynet.be/gnu/gettext/gettext-0.17.tar.gz; tar xvfz gettext-0.17.tar.gz; cd gettext-0.17; ./configure; make; make install
 +
* yum install gnutls gnutls-devel
 +
* getting latest libvirt see: [http://libvirt.org/downloads.html libvirt cvs ]
 +
* cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs login; '''password anoncvs'''  cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs co libvirt
 +
* cd to libvirt dir and apply [http://www.redhat.com/archives/libvir-list/2007-September/msg00166.html patch] patch -p1 < configure.patch
 +
* in '''libvirt''' dir do: ./autogen.sh; make; make install
 +
 +
'''virt-install''' needs:
 +
* virtviewer, this needs
 +
* rpm -ivh http://www.lfarkas.org/linux/packages/centos/5/x86_64/gtk-vnc-0.2.0-4.x86_64.rpm
 +
* autogen.sh needs:
 +
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
 +
 +
== CentOS 5 X86_64, Minimal system ==
 +
== Additional Packages ==
 +
Minimal System installed via kickstart, packages:
 +
# Packages groups/list
 +
%packages --resolvedeps --ignoremissing
 +
 +
openssh
 +
openssh-server
 +
wget
 +
perl-libnet
 +
perl-MIME-Base64
 +
perl-URI
 +
perl-Digest-MD5
 +
perl-libwww-perl
 +
perl-XML-Parser
 +
perl-DBI
 +
perl-Crypt-SSLeay
 +
lsof
 +
 +
=== Post Install, get the following packages ===
 +
Some of these are not needed, has to be cleaned:
 +
yum install \
 +
gcc make zlib-devel zlib python-devel curses libncurses-devel ncurses-devel openssl openssl-devel \
 +
`yum search xorg-x11|grep dev|grep x86_64|awk -F . '{print $1}'` \
 +
bridge-utils tetex tetex-latex transfig libtool-ltdl dev86 glibc-devel
 +
 +
=== xen 3.1.2-rc from mercurial ===
 +
'''For 64-bit Dom_0 and 32-bit Dom_U kombination:''', the following is taken from: [http://wiki.xensource.com/xenwiki/InstallationNotes#head-b0c9b7fb7d6a4acc97446f3f7a679299436eb2be xensource install notes ]
 +
 +
Dell 2900 modules to include in mkinitrd
 +
 +
mkinitrd -v -f --with=sd_mod --with=scsi_mod --with=megaraid_mm --with=megaraid_sas --with=megaraid_mbox initrd-2.6.16.29-xen.img 2.6.16.29-xen
 +
 +
dom0 64 bit and domU 32 bit from source code
 +
 +
Build and install xen:
 +
 +
make xen
 +
make install-xen
 +
 +
Build and install the dom0:
 +
 +
make linux-2.6-xen0-prep
 +
cd build-linux-*-xen0-x86_64
 +
make menuconfig
 +
make
 +
make install modules_install
 +
 +
And finally build and install 32 bit domU:
 +
 +
make XEN_TARGET_ARCH=x86_32 linux-2.6-xenU-prep
 +
cd build-linux-*-xen0-x86_32
 +
make ARCH=i386 menuconfig
 +
make ARCH=i386
 +
make ARCH=i386 install modules_install
 +
 +
 +
hg pull ...
 +
cd xen-3.1
 +
wget http://www.broadcom.com/docs/driver_download/NXII/linux-1.5.10c.zip
 +
unzip, use broadcom sources for
 +
make world
 +
make install
 +
depmod 2.6.18-xen
 +
mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod  --with=megaraid_sas /boot/initrd-2.6.18-xen.img 2.6.18-xen
 +
 +
''' Note on: Network Problems '''
 +
Broadcom drivers have problems, network is OK when xend started without network bridges, see /etc/xen/xend-config.sxp
 +
 +
Starting bridge with: /etc/xen/scripts/network-bridge start
 +
results in a not working network configuration in dom0.
 +
 +
see [http://lists.xensource.com/archives/html/xen-users/2007-07/msg00768.html network-bridge causes network to hang]
 +
 +
'''swiotlb=128''' has to be set in menu.list since we have 8GB RAM, without youll get error messages like :
 +
Sep  7 13:14:22 test-11 kernel: PCI-DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:02:01.0
 +
see also: [http://lists.xensource.com/archives/html/xen-devel/2007-09/msg00140.html SW-IOMMU space]
 +
 +
* change/add /boot/grub/menu.list:
 +
  title CentOS (xen-3.1.2-rc2 2.6.18 compiled)
 +
        root (hd0,0)
 +
        kernel /xen-3.1.gz
 +
        module /vmlinuz-2.6-xen ro root=/dev/md1 swiotlb=128
 +
        module /initrd-2.6.18-xen.img
 +
 +
''' Boot into your new system''':
 +
 +
[root@kribbe ~]# xm dmesg
 +
__  __            _____  _  ____              ____
 +
\ \/ /___ _ __  |___ / / | |___ \    _ __ ___|___ \
 +
  \  // _ \ '_ \    |_ \ | |  __) |__| '__/ __| __) |
 +
  /  \  __/ | | |  ___) || |_ / __/|__| | | (__ / __/
 +
/_/\_\___|_| |_| |____(_)_(_)_____|  |_|  \___|_____|
 +
 +
http://www.cl.cam.ac.uk/netos/xen
 +
University of Cambridge Computer Laboratory
 +
 +
Xen version 3.1.2-rc2 (root@nikhef.nl) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Tue Nov 13 15:02:14 CET 2007
 +
Latest ChangeSet: unavailable
 +
.
 +
.
 +
.
 +
(XEN) HVM: VMX enabled
 +
(XEN) VMX: MSR intercept bitmap enabled
 +
(XEN) CPU0: Intel(R) Xeon(R) CPU            5150  @ 2.66GHz stepping 06
 +
.
 +
.
 +
.
 +
 +
 +
 +
 +
----
 +
----
 +
 +
=== LVM 2 ===
 +
Due to performance reasons VMs should be installed in LVMs.
 +
For recovering a LVM in a Raid-1 see:[http://www.linuxjournal.com/node/8874/print  Recovery of RAID and LVM2 Volumes]
 +
 +
Installed Version:
 +
Installed Packages
 +
Name  : lvm2
 +
Arch  : i386
 +
Version: 2.02.16
 +
Release: 3.el5
 +
Size  : 5.5 M
 +
Repo  : installed
 +
Summary: Userland logical volume management tools
 +
 +
 +
''' NOTE: '''for having the live-migration-posibility we have to look at Cluster-LVM or something like that.
 +
 +
To get started ....
 +
 +
==== Setting up LVM2 on kribbe ====
 +
Using /dev/srv resp. /dev/md3 for LVM we have to:
 +
* comment out /dev/md3 in /etc/fstab
 +
* umount /dev/srv (only once)
 +
* Creating the physical volume and XenVG Volumegroup:
 +
 +
[root@kribbe ~]# pvcreate /dev/md3
 +
  Physical volume "/dev/md3" successfully created
 +
[root@kribbe ~]# vgcreate XenVG /dev/md3
 +
  Volume group "XenVG" successfully created
 +
 +
Within this volume-group we can now manage the partitions for the virtual maschines using lvcreate, lvresize, .... commands
 +
 +
===== Example: lvm for a CentOS5 test VM =====
 +
''' Root FS, only one partition '''
 +
[root@kribbe tmp]# lvcreate -L10G -nXen_CentOS5_Test XenVG
 +
  Logical volume "Xen_CentOS5_Test" created
 +
root@kribbe tmp]# mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test
 +
mke2fs 1.39 (29-May-2006)
 +
Filesystem label=
 +
OS type: Linux
 +
Block size=4096 (log=2)
 +
Fragment size=4096 (log=2)
 +
1310720 inodes, 2621440 blocks
 +
131072 blocks (5.00%) reserved for the super user
 +
First data block=0
 +
Maximum filesystem blocks=2684354560
 +
80 block groups
 +
32768 blocks per group, 32768 fragments per group
 +
16384 inodes per group
 +
Superblock backups stored on blocks:
 +
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
 +
 +
Writing inode tables: done
 +
Creating journal (32768 blocks): done
 +
Writing superblocks and filesystem accounting information: done
 +
 +
This filesystem will be automatically checked every 28 mounts or
 +
180 days, whichever comes first.  Use tune2fs -c or -i to override.
  
  
 +
''' SWAP '''
 +
 +
'''Note:''' swap in a lvm is not performant
 +
[root@kribbe tmp]# lvcreate -L1G -nXen_CentOS5_Test_swap XenVG
 +
  Logical volume "Xen_CentOS5_Test_swap" created
 +
[root@kribbe tmp]# mkswap /dev/XenVG/Xen_CentOS5_Test_swap
 +
Setting up swapspace version 1, size = 1073737 kB
 +
 +
==== Usage of LVM2 on Centos xvda disks ====
 +
Taken from [http://lists.xensource.com/archives/html/xen-users/2007-06/msg00656.html xen-users archive].
 +
 +
 +
If you are using a lvm2 partition as a disk (xvda) in a dom_u and this lvm is itself partitioned you can't use just
 +
 +
'''mount -o loop <logical volume> <mountpoint>''', instead you have to give the first sector of the partition you want to mount. This can be calculated by multiplying the start sector with its size
 +
 +
'''Example''':
 +
[root@hilde xen]# fdisk -l -u /dev/mapper/xenvg-hooiwagenc4532
 +
 +
Disk /dev/mapper/xenvg-hooiwagenc4532: 53.6 GB, 53687091200 bytes
 +
255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors
 +
Units = sectors of 1 * 512 = 512 bytes
 +
 +
                            Device Boot      Start        End      Blocks  Id  System
 +
/dev/mapper/xenvg-hooiwagenc4532p1  *          63      208844      104391  83  Linux
 +
/dev/mapper/xenvg-hooiwagenc4532p2          208845    2249099    1020127+  82  Linux swap / Solaris
 +
/dev/mapper/xenvg-hooiwagenc4532p3        2249100  104856254    51303577+  83  Linux
 +
 +
 +
[root@hilde xen]# mount -o loop,offset=32256 /dev/mapper/xenvg-hooiwagenc4532 /mnt2
 +
 +
[root@hilde xen]# ls /mnt2
 +
bin  dev  home    lib        media  opt  root  selinux  sys  usr
 +
boot  etc  initrd  lost+found  mnt    proc  sbin  srv      tmp  var
 +
 +
== Virtualize Physical Nodes ==
 +
# On DOM_0 Create a lvm for the DOM_U
 +
# Mount this lvm on DOM_0
 +
# On Physical Node change runlevel to s
 +
# setup a exclude-list file, for details see [http://www.eisxen.org/54.html eisxen]
 +
# cd /;tar csp --exclude-from=/rootfs/exclude.lst . |ssh <DOM_0> "cd /<mounted LVM>; tar xsp"
 +
# change the fstab of the dom_u
 +
 +
Example exclude.lst, node specific, dont just copy, see also: [http://www.eisxen.org/54.html eisxen]
 +
This list must not have an empty line at the end.
 +
less /rootfs/exclude.lst
 +
./boot
 +
./data
 +
./datastore
 +
./datastore2
 +
./proc
 +
./rootfs
 +
./etc/mtab
 +
./sys
 +
./project
 +
 +
=== DOM_U is a centos-3.x ===
 +
In order to run this as a virtual maschine, the dom_u might need a new version of e2fschk and  module-init-tools.
 +
Therfore, on DOM_0 mount the DOM_U fs created in an earlier step, get e2fsprogs-1.39.tar.gz module-init-tools-3.2.tar.gz (these versions worked for us) to this directory.
 +
The following commands worked for me.
 +
* chroot to dom_u mountpoint
 +
* extract, compile, install e2fschk and module-init-tools
 +
* cp -r DOM_0:/lib/modules/<running version> DOM_U:/lib/modules
 +
* for e2fs do: tar xvfz e2fsprogs-1.39.tar.gz ;cd e2fsprogs-1.39; ./configure; make;  make install
 +
* for modutils:  tar xvfz module-init-tools-3.2.tar.gz; cd module-init-tools-3.2; ./configure --prefix=/ ;make moveold; make; make install
 +
 +
== CentOS 5 with Virtualization package ==
 
* Installed Version:
 
* Installed Version:
 
  2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
 
  2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
Line 54: Line 323:
  
 
== Running various centOS Versions as VMs on kribbe ==
 
== Running various centOS Versions as VMs on kribbe ==
−
=== Check if virt-manager tools are working ===
+
=== Virt-manager for installation ===
−
  virt-install can be used for VM installation.
+
  virt-install can be used for VM installation. No pypxeboot and patched udhcp needed.
  
 
==== CentOS 5 64-Bit  in VM ====
 
==== CentOS 5 64-Bit  in VM ====
 
set up kickstart file for VM, than
 
set up kickstart file for VM, than
 
  virt-install -p --location=http://www.dutchgrid.nl/mirror/centos/5/os/x86_64 --noautoconsole \
 
  virt-install -p --location=http://www.dutchgrid.nl/mirror/centos/5/os/x86_64 --noautoconsole \
−
  --file=/dev/XenVG/Xen_CentOS5_Test --name=domU001 --ram=1024  \
+
  --file=/dev/XenVG/Xen_CentOS5_Test --name=domU001 --ram=1024  --mac=00:16:3e:ab:61:0b\
 
  -x "ks=http://stal.nikhef.nl/ks/xen-test-koebrug.ks ip=194.171.97.11 \
 
  -x "ks=http://stal.nikhef.nl/ks/xen-test-koebrug.ks ip=194.171.97.11 \
 
  netmask=255.255.255.0 dns=192.16.186.253 gateway=194.171.97.254"
 
  netmask=255.255.255.0 dns=192.16.186.253 gateway=194.171.97.254"
  
−
'''Connect to th econsole of the starting vm:'''
+
'''MAC-ADDRESS''': can be calculated from last the 3 numbers of you guest IP by (ex:):
 +
 
 +
printf 00:16:3e:%02x:%02x:%02x"\n" 171 97 11
 +
 +
'''00:16:3e:ab:61:0b'''
 +
This has to be defined in the dhcp servers dhcp.conf.
 +
 
 +
 
 +
'''Connect to the console of the starting vm:'''
 
* xm list, lists the VMs:
 
* xm list, lists the VMs:
 
  [root@kribbe ~]# xm list
 
  [root@kribbe ~]# xm list
Line 75: Line 352:
 
''' Worked '''
 
''' Worked '''
  
−
'''Note''': no vnc option in the virt-install command,
+
'''Old Note''': no vnc option in the virt-install command,
 
* --vnc option removed, does not boot.
 
* --vnc option removed, does not boot.
 
* during boot of the VM answer:  q: graphics support: no
 
* during boot of the VM answer:  q: graphics support: no
 +
 +
But this one worked fine with a VNC console (both graphical and text based):
 +
virt-install  --force -r 512 -m 00:16:3e:34:4b:6e -n mestkar -f /dev/vg0/mestkar.sys -l http://spiegel.nikhef.nl/mirror/centos/5/os/x86_64 -x "ks=http://www.nikhef.nl/grid/ndpf/files/local/install/mestkar.nikhef.nl.ks ksdevice=eth0" --vnc --vcpus=1 -b xenbr0
 +
 +
  
 
''' On VM:'''
 
''' On VM:'''
Line 99: Line 381:
 
Information on virsh can be found at:
 
Information on virsh can be found at:
 
[http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU centos-wiki]
 
[http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU centos-wiki]
 +
or
 
[http://www.linuxtopia.org/online_books/centos5/centos_5_xen_virtualization/index.html Red Hat Virtualization]
 
[http://www.linuxtopia.org/online_books/centos5/centos_5_xen_virtualization/index.html Red Hat Virtualization]
  
−
= System -- Quattor generated=
+
virsh is another virtualization management tool (built around the libvirt management API) which is now capable to deal with xen, future versions might also be able to  manage kvm or qemu.
 +
 
 +
Since we plan to use xen we should stick with xm directly.
 +
 
 +
=== Virtual Maschine management with '''xm''' ===
 +
based on: [http://tx.downloads.xensource.com/downloads/docs/user/#SECTION03120000000000000000 xen-user-docu]
 +
 
 +
= CentOS 5 i386-System -- Quattor generated=
  
 
== Problems ==
 
== Problems ==
−
# virt-manager not working [http://bugs.centos.org/view.php?id=2383 centos-bug]
+
 
 +
* xenconsole: Could not read tty from store: No such file or directory
 +
root@hilde xen]# xm create koebrug -c
 +
Using config file "./koebrug".
 +
Started domain koebrug
 +
xenconsole: Could not read tty from store: No such file or directory
 +
 
 +
'''Try: ''' /etc/init.d/xend restart
 +
 
 +
 
 +
* virt-manager not working [http://bugs.centos.org/view.php?id=2383 centos-bug]
  
  
Line 160: Line 460:
 
  unused devices: <none>
 
  unused devices: <none>
  
−
== Added to Quattor System ==
 
−
 
−
=== xen 3.1 ===
 
−
wget http://bits.xensource.com/oss-xen/release/3.1.0/kernel-3.1-rhel5x/RPMS/i386/xen-3.1.0-1.i386.rpm
 
−
wget http://bits.xensource.com/oss-xen/release/3.1.0/kernel-3.1-rhel5x/RPMS/i386/kernel-xen-2.6.18-3.1.0.i386.rpm
 
−
wget http://bits.xensource.com/oss-xen/release/3.1.0/kernel-3.1-rhel5x/RPMS/i386/xen-devel-3.1.0-1.i386.rpm
 
−
wget http://bits.xensource.com/oss-xen/release/3.1.0/kernel-3.1-rhel5x/SRPMS/xen-3.1.0-1.src.rpm
 
−
 
−
'''Notes''':
 
−
# rpm -i --force kernel-xen-2.6.18-3.1.0.i386.rpm (kernel is older than installed one, --force)
 
−
# rpm -Uvh xen-3.1.0-1.i386.rpm
 
−
# sources and dev package not installed
 
−
# change /boot/grub/menu.list:
 
−
title Xen (3.10)
 
−
        root (hd0,0)
 
−
        kernel /xen-3.1.gz
 
−
        module /vmlinuz-2.6.18-xen_3.1.0 ro root=/dev/md1
 
−
        module /initrd-2.6.18-xen_3.1.0.img
 
−
''' reboot resulted in a startet xen dom_0, HVM enabled:'''
 
−
[root@kribbe ~]#xm dmesg:
 
−
__  __            _____  _  ___
 
−
\ \/ /___ _ __  |___ / / | / _ \
 
−
  \  // _ \ '_ \    |_ \ | || | | |
 
−
  /  \  __/ | | |  ___) || || |_| |
 
−
/_/\_\___|_| |_| |____(_)_(_)___/
 
−
 
−
http://www.cl.cam.ac.uk/netos/xen
 
−
University of Cambridge Computer Laboratory
 
−
 
−
Xen version 3.1.0 (avsm@uk.xensource.com) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Thu May 17 05:49:14 EDT 2007
 
−
Latest ChangeSet: unavailable
 
−
 
−
...
 
−
(XEN) System RAM: 8191MB (8387872kB)
 
−
...
 
−
(XEN) HVM: VMX enabled
 
−
(XEN) VMX: MSR intercept bitmap enabled
 
−
...
 
−
(XEN)  Xen  kernel: 32-bit, PAE, lsb
 
−
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0xc0100000 -> 0xc044753c
 
−
...
 
−
(XEN) Dom0 has maximum 4 VCPUs
 
−
 
−
 
−
''' Network '''
 
−
# Network cabeling has to be changed when moving from centOS 5 default kernel to installed xen kernel, '''REALlY?'''
 
−
# eth0 HWaddr 00:15:C5:E9:48:B1,  eth1 HWaddr 00:15:C5:E9:48:B3
 
−
# dhcpd.conf on stal is written by quattor?: .. manualy changed :B1 to :B3
 
−
host kribbe {  # added by aii-dhcp
 
−
              hardware ethernet 00:15:C5:E9:48:B3;
 
−
              fixed-address 194.171.97.4;
 
−
              }
 
−
# Make sure bridgeutlis are installed
 
−
# ifconfig should show eth0/1 as well as the bridges (xenbr0), virtual interface (vif), physical eth (pethX), example:
 
−
[root@kribbe ~]# ifconfig
 
−
eth0      Link encap:Ethernet  HWaddr 00:15:C5:E9:48:B1
 
−
...
 
−
peth0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
 
−
...
 
−
vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
 
−
...
 
−
xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
 
−
 
−
For more information see [http://wiki.xensource.com/xenwiki/XenNetworking Networking in Xen], especially '''Bridging'''
 
−
 
−
==== Using 2 Networks in the Virtual Maschine -not needed at nikhef!====
 
−
The following setup can be used to have 2 network interface available in the VMs, perhaps usefull for a management network.
 
−
If a second Network Interface is needed just uncomment the line
 
−
"$dir/network-bridge" start vifnum=1 netdev=eth1
 
−
in /etc/xen/scripts/network-fzk.xen3
 
−
 
−
Problems will occure when the bridged interface is not cabled.
 
−
 
−
''' Needed XEN Scripts:'''
 
−
 
−
''' /etc/xen/xend-config.sxp '''
 
−
(logfile /var/log/xen/xend.log)
 
−
(loglevel DEBUG)
 
−
(network-script network-fzk.xen3)
 
−
(dom0-min-mem 196)
 
−
(dom0-cpus 0)
 
−
 
−
''' /etc/xen/scripts/network-fzk.xen3'''
 
−
#!/bin/sh
 
−
dir=$(dirname "$0")
 
−
echo "====== $dir ======"
 
−
"$dir/network-bridge" start vifnum=0 netdev=eth0
 
−
#"$dir/network-bridge" start vifnum=1 netdev=eth1
 
−
echo "=== Bridges configured === "
 
−
 
−
'''Note''': The last line is to make sure to have an CR after netdev=eth1,
 
−
otherwise this line will not be executed
 
−
 
−
==== Configure 2 Bridges, but only internal nic has an IP ====
 
−
 
−
'''Debian: /etc/network/interfaces'''
 
−
auto lo
 
−
iface lo inet loopback
 
−
 
−
allow-hotplug eth0 eth1
 
−
 
−
iface eth1 inet manual
 
−
 
−
iface eth0 inet static
 
−
        address 10.97.4.150
 
−
        netmask 255.255.255.0
 
−
        network 10.97.4.0
 
−
        gateway 10.97.4.1
 
−
        broadcast 10.97.4.255
 
−
        # dns-* options are implemented by the resolvconf package, if installed
 
−
        dns-nameservers 10.97.1.191 141.52.8.18
 
−
        dns-search gridka.de fzk.de
 
−
        post-up route add -net 10.0.0.0/9 gw 10.97.4.1
 
−
 
−
=== LVM 2 ===
 
−
Due to performance reasons VMs should be installed in LVMs.
 
−
For recovering a LVM in a Raid-1 see:[http://www.linuxjournal.com/node/8874/print  Recovery of RAID and LVM2 Volumes]
 
−
 
−
Installed Version:
 
−
Installed Packages
 
−
Name  : lvm2
 
−
Arch  : i386
 
−
Version: 2.02.16
 
−
Release: 3.el5
 
−
Size  : 5.5 M
 
−
Repo  : installed
 
−
Summary: Userland logical volume management tools
 
−
 
−
 
−
''' NOTE: '''for having the live-migration-posibility we have to look at Cluster-LVM or something like that.
 
−
 
−
To get started ....
 
−
 
−
==== Setting up LVM2 on kribbe ====
 
−
Using /dev/srv resp. /dev/md3 for LVM we have to:
 
−
* comment out /dev/md3 in /etc/fstab
 
−
* umount /dev/srv (only once)
 
−
* Creating the physical volume and XenVG Volumegroup:
 
−
 
−
[root@kribbe ~]# pvcreate /dev/md3
 
−
  Physical volume "/dev/md3" successfully created
 
−
[root@kribbe ~]# vgcreate XenVG /dev/md3
 
−
  Volume group "XenVG" successfully created
 
−
 
−
Within this volume-group we can now manage the partitions for the virtual maschines using lvcreate, lvresize, .... commands
 
−
 
−
===== Example: lvm for a CentOS5 test VM =====
 
−
''' Root FS, only one partition '''
 
−
[root@kribbe tmp]# lvcreate -L10G -nXen_CentOS5_Test XenVG
 
−
  Logical volume "Xen_CentOS5_Test" created
 
−
root@kribbe tmp]# mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test
 
−
mke2fs 1.39 (29-May-2006)
 
−
Filesystem label=
 
−
OS type: Linux
 
−
Block size=4096 (log=2)
 
−
Fragment size=4096 (log=2)
 
−
1310720 inodes, 2621440 blocks
 
−
131072 blocks (5.00%) reserved for the super user
 
−
First data block=0
 
−
Maximum filesystem blocks=2684354560
 
−
80 block groups
 
−
32768 blocks per group, 32768 fragments per group
 
−
16384 inodes per group
 
−
Superblock backups stored on blocks:
 
−
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
 
−
 
−
Writing inode tables: done
 
−
Creating journal (32768 blocks): done
 
−
Writing superblocks and filesystem accounting information: done
 
−
 
−
This filesystem will be automatically checked every 28 mounts or
 
−
180 days, whichever comes first.  Use tune2fs -c or -i to override.
 
−
 
−
 
−
''' SWAP '''
 
−
[root@kribbe tmp]# lvcreate -L1G -nXen_CentOS5_Test_swap XenVG
 
−
  Logical volume "Xen_CentOS5_Test_swap" created
 
−
[root@kribbe tmp]# mkswap /dev/XenVG/Xen_CentOS5_Test_swap
 
−
Setting up swapspace version 1, size = 1073737 kB
 
  
 
=== Example installation of a CentOS5 VM ===
 
=== Example installation of a CentOS5 VM ===
Line 367: Line 488:
  
 
''' setup of VM start files'''
 
''' setup of VM start files'''
 +
----
 +
----
  
−
= Xen Notes, Old Version, to be cleaned up =
+
=== SELinux, Postinstallation steps ===
−
== XenServer-3.2 Commercial Version ==
+
* Cleaning up daemons, pay attention to ntp, does the dom_U has the correct time?
−
any information can be found on: [http://www.xensource.com/products/xen_enterprise/ xen-doku]
+
  /sbin/service microcode_ctl stop
−
an related links.
+
  /sbin/chkconfig --del microcode_ctl
−
 
+
  /sbin/service smartd stop
−
=== Installation xenserver ===
+
  /sbin/chkconfig --del smartd
−
 
 
−
 
 
−
====Principle setup====
 
−
* VMs are organized in a Virtual Rack (a01-008-210)
 
−
* Network: VLAN 1401 (10.97.210.0) and VLAN 158 (192.108.45.0) are configured
 
−
* in VMs use xenbr1401(eth1401) for eth0 (internal network)
 
−
* in VMs use xenbr1(eth1) for eth1 (external network)
 
−
 
 
−
'''Notes''' To install a new xenserver DOM_0
 
−
Local Disk is '''sdb'''
 
−
SAN is '''sda'''
 
−
If asked '''DONT CREATE A STORAGE REPOSITORY, this will delete ALL VMs in the SAN'''
 
  
−
==== Installation Procedure ====
 
−
# Boot from xenserver CD
 
−
# Set eth0 to ONBOOT=NO, i.e. don't use eth0 during installation
 
−
# Configure eth1 with external IP (192.....)
 
−
# NTP: 10.97.8.200
 
−
# DNS: 10.97.1.192, 10.97.1.191, 141.52.8.18
 
−
# ''' NO XEN-STORAGE, disable FLAGS on sdb as well as on sda (SAN)
 
−
# Use sdb (local disk) for installation of the base system
 
−
# Post Installation:
 
−
## add: /etc/sysconfig/static-routes: any net 10.0.0.0/9 gw 10.97.81
 
−
## edit /etc/resolv.conf add search fzk.de, set 141 as primary nameserver
 
−
## edit /boot/grub/menu.lst, replace hd1,0 with hd0,0
 
−
## Manually setting the default Storage Repository [http://kb.xensource.com/entry!default.jspa?categoryID=18&externalID=329&fromSearchPage=true Manually setting the default Storage Repository (SR)]
 
−
### /etc/smtab: 7c2cbf8e-d22e-46a3-8b3c-7791499e7510 none lvm default auto
 
−
### service srmtab restart
 
−
### service xenagentd start
 
−
## set up vlan:
 
−
### ifup eth0
 
−
### modprobe 8021q
 
−
### vconfig add eth0 1401
 
−
### vconfig add eth0 158
 
−
### Set up the following files accordingly
 
−
'''/etc/sysconfig/network-scripts'''
 
−
 
−
==== ifcfg-eth0 =====
 
−
DEVICE=eth0
 
−
ONBOOT=no
 
−
TYPE=Ethernet
 
−
HWADDR=00:1a:64:12:80:71
 
−
BRIDGE=xenbr0
 
−
LINKDELAY=5
 
−
check_link_down() { return 1 ; }
 
−
==== ifcfg-eth1 ====
 
−
DEVICE=eth1
 
−
BOOTPROTO=none
 
−
ONBOOT=yes
 
−
TYPE=Ethernet
 
−
HWADDR=00:1a:64:12:80:73
 
−
BRIDGE=xenbr1
 
−
LINKDELAY=5
 
−
check_link_down() { return 1 ; }
 
−
==== ifcfg-eth1401 ====
 
−
DEVICE=eth0.1401
 
−
ONBOOT=yes
 
−
TYPE=Ethernet
 
−
HWADDR=00:1a:64:12:80:71
 
−
BRIDGE=xenbr1401
 
−
LINKDELAY=5
 
−
VLAN=yes
 
−
check_link_down() { return 1 ; }
 
−
==== ifcfg-eth158 ====
 
−
DEVICE=eth0.158
 
−
ONBOOT=yes
 
−
TYPE=Ethernet
 
−
HWADDR=00:1a:64:12:80:71
 
−
BRIDGE=xenbr158
 
−
LINKDELAY=5
 
−
VLAN=yes
 
−
check_link_down() { return 1 ; }
 
−
==== ifcfg-xenbr0 ====
 
−
DEVICE=xenbr0
 
−
ONBOOT=no
 
−
TYPE=Bridge
 
−
DELAY=0
 
−
STP=off
 
−
check_link_down() { return 1 ; }
 
−
==== ifcfg-xenbr1 ====
 
−
DEVICE=xenbr1
 
−
ONBOOT=yes
 
−
TYPE=Bridge
 
−
DELAY=0
 
−
STP=off
 
−
NETMASK=255.255.255.0
 
−
IPADDR=192.108.45.86
 
−
GATEWAY=192.108.45.1
 
−
PEERDNS=yes
 
−
check_link_down() { return 1 ; }
 
−
==== ifcfg-xenbr1401 ====
 
−
DEVICE=xenbr1401
 
−
ONBOOT=yes
 
−
TYPE=Bridge
 
−
DELAY=0
 
−
STP=off
 
−
NETMASK=255.255.255.0
 
−
IPADDR=10.97.210.107
 
−
PEERDNS=yes
 
−
check_link_down() { return 1 ; }
 
−
==== ifcfg-xenbr158 ====
 
−
DEVICE=xenbr158
 
−
ONBOOT=yes
 
−
TYPE=Bridge
 
−
DELAY=0
 
−
STP=off
 
−
NETMASK=255.255.255.0
 
−
IPADDR=10.97.8.107
 
−
PEERDNS=yes
 
−
check_link_down() { return 1 ; }
 
  
 +
* SELinux considerations: from [http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU SELinux in domU]
 +
If you are using SELinux, it is important to check that the image has the correct security context (xen_image_t), or access to the virtual disk will be denied to the domU system. You can check this with ls:
  
−
===== Cluster LVM =====
+
# ls -Z /srv/xen/mailserver.img
−
# cd /etc/yum.r*
+
-rw-r--r--  root root user_u:object_r:xen_image_t      /srv/xen/mailserver.img
−
# wget http://ftp.halifax.rwth-aachen.de/centos/4/csgfs/CentOS-csgfs.repo
 
−
# yum install lvm2-cluster
 
  
−
== Debian  / Xen3 ==
+
If you are having trouble setting the right file context, please have a look at the Xen tips and tricks page. At any rate, turning off SELinux, as some howtos on this subject advise is a very poor workaround. Reading two manual pages (semanage(8) and restorecon(8)), for an extra layer of security is a good trade!
−
=== Debian 4.0 / Xen3 - xen-linux-system-2.6.18-4-xen-amd64 ===
 
−
'''Post Installation basic package:'''
 
−
apt-get install ssh xen-linux-system-2.6.18-4-xen-amd64 ntp ntp-server bridge-utils less
 
  
−
''' Needed XEN Scripts:'''
 
  
−
''' /etc/xen/xend-config.sxp '''
+
= Memory and network problems =
−
(logfile /var/log/xen/xend.log)
 
−
(loglevel DEBUG)
 
−
(network-script network-fzk.xen3)
 
−
(dom0-min-mem 196)
 
−
(dom0-cpus 0)
 
  
−
''' /etc/xen/scripts/network-fzk.xen3'''
+
The CentOS 5 (RHEL5) release of Xen is not really stable; it changed it's behaviour over time from 'not working at all' in CentOS 5.0 and 5.1 to 'working' in '5.2', and with further updates it appears as though rather important changes find their way into the Xen part of this otherwise stable distro.
−
#!/bin/sh
 
−
dir=$(dirname "$0")
 
−
echo "====== $dir ======"
 
−
"$dir/network-bridge" start vifnum=0 netdev=eth0
 
−
"$dir/network-bridge" start vifnum=1 netdev=eth1
 
−
echo "=== Bridges configured === "
 
  
−
'''Note''': The last line is to make sure to have an CR after netdev=eth1,
+
Recently, and this is 5.3 on April 22, 2009, I started to see spontaneous reboots of my Dom0. Not much in the syslog, but some suspicious entries about memory squeeze:
−
otherwise this line will not be executed
 
−
 
 
−
 
 
−
==== Configure 2 Bridges, but only internal nic has an IP ====
 
−
 
 
−
'''/etc/network/interfaces'''
 
−
auto lo
 
−
iface lo inet loopback
 
−
 
−
allow-hotplug eth0 eth1
 
−
 
−
iface eth1 inet manual
 
−
 
−
iface eth0 inet static
 
−
        address 10.97.4.150
 
−
        netmask 255.255.255.0
 
−
        network 10.97.4.0
 
−
        gateway 10.97.4.1
 
−
        broadcast 10.97.4.255
 
−
        # dns-* options are implemented by the resolvconf package, if installed
 
−
        dns-nameservers 10.97.1.191 141.52.8.18
 
−
        dns-search gridka.de fzk.de
 
−
        post-up route add -net 10.0.0.0/9 gw 10.97.4.1
 
−
 
 
−
=== ''' OUTDATED''' Debian sarge amd64 / xen-3 nstallation ===
 
−
 
 
−
''' /etc/apt/sources.list'''
 
−
# Debian sources
 
−
 
−
deb http://amd64.debian.net/debian-amd64 sarge          main contrib non-free
 
−
deb http://security.debian.org          sarge/updates  main contrib non-free
 
−
deb http://www.backports.org/debian      sarge-backports main
 
−
 
−
deb-src http://amd64.debian.net/debian-amd64 sarge          main contrib non-free
 
−
deb-src http://www.backports.org/debian      sarge-backports main
 
−
 
−
deb http://ftp.uni-stuttgart.de/debian testing  main
 
−
deb http://ftp.uni-stuttgart.de/debian unstable main
 
−
 
−
deb http://packages.debianbase.de/sarge/amd64/xen3/ ./
 
−
deb-src http://packages.debianbase.de/sources/xen3 ./
 
−
 
 
−
 
 
−
 
 
−
''' /etc/apt/preferences'''
 
−
 
−
Package: *
 
−
Pin: release o=Debian,a=stable,l=Debian-Security
 
−
Pin-Priority: 950
 
−
 
−
Package: *
 
−
Pin: release o=Debian,a=stable
 
−
Pin-Priority: 900
 
−
 
−
Package: *
 
−
Pin: release o=Backports.org archive,a=sarge-backports
 
−
Pin-Priority: 800
 
−
 
−
Package: *
 
−
Pin: release o=Debian,a=testing
 
−
Pin-Priority: 700
 
−
 
−
Package: *
 
−
Pin: release o=Debian,a=unstable
 
−
Pin-Priority: 600
 
−
 
−
Package: *
 
−
Pin: origin ""
 
−
Pin-Priority: 400
 
−
 
 
−
''' /boot/grub/menu.lst'''
 
−
title          Xen 3.0.3-1-amd64 / Debian GNU/Linux, kernel 2.6.18-2-xen-amd64
 
−
root            (hd0,0)
 
−
kernel          /boot/xen-3.0.3-1-amd64.gz
 
−
module          /boot/vmlinuz-2.6.18-2-xen-amd64 root=/dev/sda1 ro console=tty0 console=tty0 max_loop=64
 
−
module          /boot/initrd.img-2.6.18-2-xen-amd64
 
−
savedefault
 
−
 
 
−
 
 
−
==== Install xen ====
 
−
 
 
−
apt-cache search xen-linux-system-2.6
 
−
 
 
−
apt-get install xen-linux-system-2.6.18-2-xen-amd64
 
−
apt-get install bridge-utils
 
−
 
 
−
===== Make sure to have brctl installed =====
 
−
apt-get install bridge-utils
 
−
/etc/init.d/xend resart
 
−
 
 
−
ifconfig should show the bridges xenbr0/1
 
−
 
 
−
==== Scripts for dual homed network ====
 
−
 
−
/etc/xen/scripts/network-fzk.xen3
 
−
#!/bin/sh
 
−
dir=$(dirname "$0")
 
−
echo "====== $dir ======"
 
−
"$dir/network-bridge" start vifnum=0 netdev=eth0
 
−
"$dir/network-bridge" start vifnum=1 netdev=eth1
 
−
 
 
−
'''patch /etc/xen/xend-config.sxp '''
 
−
--- /etc/xen/xend-config.sxp.orig      :58:47.000000000 +0100
 
−
+++ /etc/xen/xend-config.sxp  :11:42.000000000 +0100
 
−
@@ -11,8 +11,8 @@
 
−
# Commented out entries show the default for that entry, unless otherwise
 
−
# specified.
 
−
 
−
-#(logfile /var/log/xen/xend.log)
 
−
-#(loglevel DEBUG)
 
−
+(logfile /var/log/xen/xend.log)
 
−
+(loglevel DEBUG)
 
−
 
 
−
#(xend-http-server no)
 
−
#(xend-unix-server no)
 
−
@@ -84,8 +84,8 @@
 
−
# two fake interfaces per guest domain.  To do things like this, write
 
−
# yourself a wrapper script, and call network-bridge from it, as appropriate.
 
−
#
 
−
-(network-script network-dummy)
 
−
-
 
−
+#(network-script network-dummy)
 
−
+(network-script network-fzk.xen3)
 
−
# The script used to control virtual interfaces.  This can be overridden on a
 
−
# per-vif basis when creating a domain or a configuring a new vif.  The
 
−
# vif-bridge script is designed for use with the network-bridge script, or
 
−
@@ -99,7 +99,7 @@
 
−
# If you are using only one bridge, the vif-bridge script will discover that,
 
−
# so there is no need to specify it explicitly.
 
−
#
 
−
-(vif-script vif-bridge)
 
−
+#(vif-script vif-bridge)
 
−
 
−
 
−
## Use the following if network traffic is routed, as an alternative to the
 
−
 
 
−
=== Creating an SL4-Image ===
 
−
* Install SL4 somewhere, use minimal setup
 
−
* create exclude.list (/proc, /sys, /rootfs, /..
 
−
* tar cvfpz /
 
−
 
 
−
=== On xen3 dom_0 ===
 
−
a01-004-151:~# lvcreate -L1G -nXen_SL4_raw XenVG
 
−
  Logical volume "Xen_SL4_raw" created
 
−
a01-004-151:~# lvcreate -L1G -nXen_SL4_Dirs XenVG
 
−
  Logical volume "Xen_SL4_Dirs" created
 
−
a01-004-151:~# lvcreate -L1G -nXen_SL4_swap XenVG
 
−
  Logical volume "Xen_SL4_swap" created
 
−
a01-004-151:~# mkswap /dev/XenVG/Xen_SL4_swap
 
−
Setting up swapspace version 1, size = kB
 
−
no label, UUID=377313c6-498b-4525-b0d3-f9d9fb2c46bd
 
−
 
 
−
 
 
−
a01-004-151:~# mkfs.ext3 /dev/XenVG/Xen_SL4_raw 
 
−
 
−
a01-004-151:~# mkfs.ext3 /dev/XenVG/Xen_SL4_Dirs
 
−
 
 
−
=== Create Image ===
 
−
 
 
−
==== SL3 image ====
 
−
 
 
−
# module_init_tools
 
−
#
 
−
 
 
−
==== SL4 image ====
 
−
 
 
−
out of the box ?
 
−
 
 
−
== Debian/Xen2 - outdated ==
 
−
=== OS-Installation ===
 
−
==== Ganglia on debian ====
 
−
To install and configure ganglia on the dom0 do:
 
−
* apt-get install ganglia-monitor
 
−
* scp ce-fzk:/opt/glite/yaim/rpms/gmond.conf /etc/
 
−
* /etc/init.d/ganglia-monitor restart
 
−
==== New xen-Kernels ====
 
−
To compile new dom-U kernels, cd to xen/linux-*-xenU and do
 
−
make menuconfig ....choose your options
 
−
make ARCH=xen dep bzImage modules modules_install
 
−
==== Debian -- XEN ====
 
−
*Boot from Debian 3.1r1 CD
 
−
*Configure first network interface with the external IP (nameserver e.g. 141.52.8.18)
 
−
*Partitioning: Whole disc in 2 partitions (50 GB / ; rest /srv (for using lvm see below)
 
−
*Installation Type: Manual selection
 
−
*additional packages: run script install_packages.sh
 
−
 
 
−
As the debian default installation is quite small addidional packages are needed, the following script should get them:
 
−
#!/bin/bash
 
−
apt-get install -y make
 
−
apt-get install -y gcc
 
−
apt-get install -y subversion
 
−
apt-get install -y bridge-utils
 
−
apt-get install -y curl
 
−
apt-get install -y libcurl3-dev
 
−
apt-get install -y zlib1g-dev
 
−
apt-get install -y zlib1g
 
−
apt-get install -y python-dev
 
−
apt-get install -y python-twisted
 
−
apt-get install -y bzip2
 
−
apt-get install -y linuxdoc-tools-latex
 
−
apt-get install -y gs-common
 
−
apt-get install -y transfig
 
−
apt-get install -y host
 
−
apt-get install -y iptables-dev
 
−
apt-get install -y ntpdate
 
−
apt-get install -y module-init-tools
 
−
apt-get install -y iproute
 
−
apt-get install -y lvm2
 
−
apt-get install -y dmsetup
 
−
mkdir /opt/xen
 
−
cd /opt/xen
 
−
wget http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0.7-src.tgz
 
−
tar xvfz xen-2.0.7-src.tgz
 
−
cd xen-2.0
 
−
exit
 
−
 
 
−
Untill I find some time or somebody else puts the following in a sed line do: edit Makefile, change the line: Kernels ?= ... so that the 2.4 Kernels are also build, i.e.
 
−
KERNELS ?= linux-2.6-xen0 linux-2.6-xenU linux-2.4-xen0 linux-2.4-xenU
 
−
 
 
−
Then build the xen-kernels:
 
−
 
 
−
make world
 
−
make install
 
−
 
 
−
===== Setup /boot/grub/menu.lst =====
 
−
Configure this file to boot the xen kernel and to set the max_loop Device parameter.
 
−
Change below the line
 
−
 
 
−
'''## ## End Default Options ##'''
 
−
 
 
−
Example:
 
−
#### XEN ####
 
−
title Xen 2.0 / XenLinux 2.6
 
−
root  (hd0,0)
 
−
kernel /boot/xen-2.0.gz dom0_mem=64000 console=vga
 
−
module /boot/vmlinuz-2.6-xen0 root=/dev/sda1 ro console=tty0 max_loop=64
 
−
boot
 
−
 
 
−
===== Configure Network =====
 
−
The network is centraly configured in '''/etc/network/interfaces '''.
 
−
# This file describes the network interfaces available on your system
 
−
# and how to activate them. For more information, see interfaces(5).
 
−
 
−
# The loopback network interface
 
−
auto lo
 
−
iface lo inet loopback
 
−
 
−
# The primary network interface
 
−
auto eth0
 
−
iface eth0 inet static
 
−
        address 192.108.45.66
 
−
        netmask 255.255.255.0
 
−
        network 192.108.45.0
 
−
        broadcast 192.108.45.255
 
−
        gateway 192.108.45.1
 
−
      # dns-* options are implemented by the resolvconf package, if installed
 
−
        dns-nameservers 10.97.1.191 141.52.8.18
 
−
 
 
−
====== Configure secound nic ======
 
−
 
 
−
''' edit /etc/network/interfaces '''
 
−
* add eth1 in the line '''auto eth0'''
 
−
* add a secound interface block like:
 
−
iface eth1 inet static
 
−
        address 10.97.4.130
 
−
        netmask 255.255.255.0
 
−
        network 10.97.4.0
 
−
        broadcast 10.97.4.255
 
−
        #gateway 10.97.4.1
 
−
        # dns-* options are implemented by the resolvconf package, if installed
 
−
        dns-nameservers 10.97.1.191 141.52.8.18
 
−
        dns-search gridka.de fzk.de
 
−
====== Add a route ======
 
−
 
 
−
append to the interfaces file a line like:
 
−
post-up route add -net 10.0.0.0/9 gw 10.97.4.1
 
−
 
 
−
to add a route
 
−
 
 
−
===== Make sure to have enough loop devices =====
 
−
for minor in `seq 8 1 63` ; do mknod /dev/loop$minor b 7 $minor ; done
 
−
===== LVM  Logical Volume Manager ==== =
 
−
Make sure that lvm2 and dmsetup are installed.
 
−
 
 
−
Create physical volume:
 
−
*pvcreate /dev/sdb1
 
−
here we use the sec har disc with one partition, any other setup is possible.
 
−
All partitions can be used for a physical volume, from these a Volume Group is created
 
−
*vgcreate XenVG /dev/sdb1 /dev/<and all other pv created before>
 
−
In this VolumeGroup create the logical volumes to be used in the images, for example
 
−
*lvcreate -L20G -nXen_g_CE_home XenVG
 
−
To get a list of available logical volumes:
 
−
*lvscan -v
 
−
 
 
−
'''Note''' Take care to set the permissions of the lv apropriatly in the virtual servers, e.g. chmod 1777 /tmp for imported tmp directories in the virtual servers.
 
−
 
 
−
====== Images in lvm ======
 
−
 
 
−
Create a suitable lvm volume with ext3 fs:
 
−
lvcreate -L4GB -n Xen_SE_DC_Sys XenVG
 
−
mkfs.ext3 /dev/XenVG/Xen_SE_DC_Sys
 
−
 
 
−
===== XEN POSTINSTALLATION STEPS -- CONFIGURATION of automatic domain start=====
 
−
====== The files which are needed to start the images (xm create <image-start-file>) should be available through nfs.======
 
−
This procedure is slightly different from default configuration, as we want to have multiple nics configured in the virtual servers.
 
−
Procedure on client:
 
−
* ''' NFS''':
 
−
add to /etc/fstab
 
−
a01-004-155:/nfs/etc_xen /etc/xen_nfs nfs ro 0 0
 
−
 
 
−
'''command:''' mount -a
 
−
 
 
−
* Create xen-autostart structure
 
−
* mv /etc/xen /etc/xen.local
 
−
* mkdir -p /etc/xen/auto
 
−
* mkdir /etc/xen_nfs
 
−
* ln -s /etc/xen_nfs/scripts /etc/xen/scripts
 
−
* ln -s /etc/xen_nfs/auto /etc/xen/auto/<IMAGES TO START ON HOST>
 
−
* ln -s /etc/xen_nfs/xend-config.sxp /etc/xen/xend-config.sxp
 
−
* ln -s /etc/init.d/xendomains /etc/rc2.d/S91xendomains
 
−
* ln -s /etc/init.d/xend /etc/rc2.d/S90xend
 
−
====== Timeservers ======
 
−
Virtual servers are using the time of the xen host, make sure that ntpdate is running and is configured coeectly, i.e. use 10.97.4.200 in /etc/default/ntpdate.
 
−
''' Make sure that the time is upated in a cron job:'''
 
−
crontab -e
 
−
58 */6 * * * /usr/sbin/ntpdate 10.97.4.200 > /tmp/zeit.log
 
−
/etc/init.d/cron restart
 
−
 
 
−
====== Activating Second Network Interface ======
 
−
To activate the second network interface the following procedure was applied:
 
−
http://wiki.xensource.com/xenwiki/XenNetworking
 
−
 
 
−
''if you want to use multiple bridges, you must create them yourself, either manually, or via your own startup script,
 
−
or via a custom script to replace network-bridge. For example:
 
−
The second network interface is started  in dom_0 with the script /root/xen-net-init:''
 
−
 
 
−
 
 
−
#!/bin/bash
 
−
# A.G.
 
−
if [ $# != 3 ] ; then
 
−
        echo -e "Usage: $0 <DOMAIN-NUMBER> <Internal IP> <External IP>\n"
 
−
        exit 1
 
−
fi
 
−
XEN_S=/etc/xen/scripts
 
−
D=$1
 
−
# Test if the domain number given really exists
 
−
xm list | tail -n +2 | awk '{print $2}' | grep -q $D
 
−
if [ $? == 1 ] ; then
 
−
        echo -e "Domain $D does not exist! Please provide a DOMAIN-NUMBER\n"
 
−
        exit 2
 
−
fi
 
−
$XEN_S/network start bridge=xen-br0 netdev=eth0 antispoof=no
 
−
$XEN_S/network start bridge=xen-br1 netdev=eth1 antispoof=no
 
−
$XEN_S/vif-bridge up domain=${D} vif=vif${D}.0 bridge=xen-br0 mac="FE:FF:FF:FF:FF:FF" ip="$2"
 
−
$XEN_S/vif-bridge up domain=${D} vif=vif${D}.1 bridge=xen-br1 mac="FE:FF:FF:FF:FF:FF" ip="$3"
 
−
 
 
−
====== Xen Dom_U Start Configuration Files for the Virtual Maschines ======
 
−
Create in /etc/xen/auto/ the configuration files for each virtual maschine to run on the host. The auto directory will be evaluated at dom_0 boot time.
 
−
Example /etc/xen/auto/pps-bdii-fzk:
 
−
 
 
−
#  -*- mode: python; -*-
 
−
 
−
name = "pps-bdii-fzk"
 
−
vmid = 1
 
−
 
−
kernel = "/boot/vmlinuz-2.4.30-xenU"
 
−
memory = 256
 
−
cpu = 1
 
−
 
−
# Number of network interfaces. Default is 1.
 
−
nics=2
 
−
vif=[ 'mac=00:16:3e:70:%d:01,bridge=br0' % vmid,'mac=00:16:3e:70:%d:02,bridge=br1' % vmid]
 
−
 
−
disk = [ 'file:/data/xen-%s,hda1,w' % name,
 
−
          'file:/data/swap-%d,hda2,w' % vmid]
 
−
 
−
root = "/dev/hda1 ro"
 
−
 
 
−
'''Note:''' To avoid collisions of mac addresses it is better use constructions like:
 
−
vif=[ 'mac=00:16:3e:70:Hex1:Hex2,bridge=br0','mac=00:16:3e:70:Hex3:Hex4,bridge=br1']
 
−
Where Hex 1-4 are the last  2 tuples of the internal and external IP
 
−
 
 
−
'''Note:''' If you want use lvm partitions from dom_0 you can use:
 
−
disk = [ 'file:/data/xen-%s,hda1,w' % name,
 
−
          'file:/data/swap-%d,hda2,w' % vmid,
 
−
          'phy:/dev/XenVG/XenCE_home,hdb1,w',
 
−
          'phy:/dev/XenVG/XenCE_tmp,hdb2,w']
 
−
 
 
−
hdb1/2 is then to included in the fstab file in the virt. maschine, for example:
 
−
/dev/hdb1              /home                  ext3    defaults        1 1
 
−
/dev/hdb2              /tmp                    ext3    defaults        1 1
 
−
 
 
−
'''Note:''' Before /dev/hdb1/2 can be used in the virtual maschine a filesystem has to be created there. Therefore the following can be done:
 
−
*comment out /dev/hdb lines in /ets/fstab in the image
 
−
*boot the virtual maschine
 
−
*xm console to this <ID>
 
−
*mkfs.ext3 /dev/hdb1
 
−
*mkfs.ext3 /dev/hdb2
 
−
*uncomment the lines in /etc/fstab
 
−
 
 
−
===== Images for the virtual maschines =====
 
−
A basic ~2GB image can be optained with:
 
−
dd if=/dev/zero of=/<Image-Name> bs=512 count=4000000
 
−
mkfs.ext3 <Image-Name>
 
−
mount -o loop <Image-file> /mnt
 
−
from an existing linux system which is to be virtualized do
 
−
cd /;tar <all dirs except nfs mounted files>, proc|ssh <where created image is mounted> "cd /mnt; tar xsp"
 
−
 
 
−
wget http://savannah.fzk.de/websites/fzk/gridkaschool/2005/xen-images/sl-3.0.5.img.ready-to-yaim--reloaded.gz
 
−
wget http://savannah.fzk.de/websites/fzk/gridkaschool/2005/xen-images/swap-1.gz
 
−
Unzip the image and the swap file, and move them to /data/<Virt-hostname-Name>.
 
−
Virt-Node-Name can be e.g. pps-bdii-fzk. For each Virtual host one image and swap file has to be created with the name used in the /etc/xen/auto/<start-config-file>.
 
−
Before this image can be used several changes in network configuration files have to be done, Therefore loop mount the image and adapt the files
 
−
 
 
−
====== Changing the network configurationof the images ======
 
−
To manipulate files within the image it has to be mounted:
 
−
 
 
−
mount -o loop /<PATH_TO>/<IMAGE_NAME> /tmp
 
−
 
 
−
adapt the following files:
 
−
*/etc/ssh/ssh* to use Port 24
 
−
* /etc/apt/apt.conf Default a proxy is used if not needed change apt.conf to:
 
−
// User customizable configuration
 
−
 
−
RPM
 
−
{
 
−
  // Uncomment to disable GPG-signature checking for packages
 
−
  // GPG-Check "false";
 
−
};
 
−
 
−
// Options for the downloading routines
 
−
Acquire
 
−
{
 
−
  // Retries "1";
 
−
  // http::Proxy "http://user:password@proxy-server.domain.tld:port/";
 
−
  //http::Proxy "http://proxy.fzk.de:8000/";
 
−
  //ftp::Proxy "http://proxy.fzk.de:8000/";
 
−
};
 
−
 
 
−
*/etc/fstab remove the nfs line gks..., otherwise the booting will take till the nfs mount timeouts, i.e. ages, I'll create a new image soon.
 
−
*/etc/sysconfig/network-scripts/route-eth0/1: set the default route of the internal networ
 
−
*/etc/sysconfig/network-scripts/ifcfg-eth0/1}
 
−
 
 
−
Configure the network interfaces according to which bridge it is bound.
 
−
In the examples below it is assumed that the virtual hosts are running in the admin rack 4
 
−
*ifcfg-eth0:
 
−
DEVICE=eth0
 
−
BOOTPROTO=static
 
−
ONBOOT=yes
 
−
IPADDR=192.108.45.XYZ
 
−
NETMASK=255.255.255.0
 
−
GATEWAY=192.108.45.1
 
−
TYPE=Ethernet
 
−
*ifcfg-eth1:
 
−
DEVICE=eth1
 
−
BOOTPROTO=static
 
−
ONBOOT=yes
 
−
IPADDR=10.97.4.ABC
 
−
NETMASK=255.255.255.0
 
−
TYPE=Ethernet
 
−
 
 
−
===== Resizing of a image =====
 
−
To increase an Image one can use.
 
−
dd if=/dev/zero bs=512  count=20000000 >> <IMAGE>
 
−
 
 
−
This will add 10 GB to the image
 
−
 
 
−
Next the filesystem has to be checked:
 
−
e2fsck -f xen-pps-ce-fzk_torque_14G
 
−
and resized
 
−
resize2fs xen-pps-ce-fzk_torque_14G
 
−
 
 
−
===== [[Converting a physical server in a virtual]] =====
 
−
*create image file
 
−
*on server: cd /; tar csp <all without home, tmp, nfs-dirs> | ssh virt. server:/virt-server-image-file "cat - tar xsp"
 
−
* set values in the network config files (eth0/1, fstab, ..) in the virt. server accordingly
 
−
 
 
−
==== Starting the Virtual Maschines ====
 
−
For the first time the following steps have to performed manualy, they can be included in the boot procedure.
 
−
 
 
−
The following commands have to be executed:
 
−
/etc/init.d/xend start
 
−
xm create /etc/xen/auto/<Virt. Maschine Name> to be started
 
−
/root/xen-net-init <ID> <IP 1> <IP 2>
 
−
 
 
−
The <ID> can be evaluated with '''xm list'''. IP 1 and IP 2 have to set according to where eth0 and eth1 are bound to (internal 10.X.X.X, or external 192.X.X.X network)
 
−
 
 
−
Now the virtual maschine should be ssh-accessable from your desktop, if not you can login to the dom_0 host and start a virtual-host-console there with: '''xm create <ID>'''. Now correct the network configuration of the virtual host.
 
−
 
 
−
===== Configuration of the virtual hosts =====
 
−
====== Network configuration ======
 
−
====== Updating the base OS ======
 
−
To update the base OS (Scientific Linux Cern 3.0.X) to the actual version do:
 
−
*check proxy settings (see above)
 
−
*check /etc/hosts for not needed entries
 
−
*check the repository settings in /etc/apt/sources.list
 
−
*apt-get update
 
−
*apt-get upgrade
 
−
 
 
−
 
 
−
 
 
−
==== Problems ====
 
−
===== On nfs Client: =====
 
−
[root@bdii-fzk root]# mount -a
 
−
 
 
−
gives:
 
−
mount: RPC: Program not registered
 
−
 
 
−
''' Is the nfsserver running correctly''', is the nfsd module loaded (modprobe nfsd), possible problem: wrong modules, have to be the same as the kernel was compiled for. Can hapen when you just cp the kernels.
 
−
 
 
−
Is the nfs mount line in /etc/fstab correct?
 
−
 
 
−
===== xm create doesnt work =====
 
−
check if module-init-tools is installed
 
−
 
 
−
===== network bridges cant be created =====
 
−
Network package iproute not installed, causes that the start of the xend creates  a not usable routing table, i.e. route cannot be setup through bridge, just do apt-get install iproute.
 
−
 
 
−
===== Virtual server with lvm in fstab doesnt start =====
 
−
at first boot/mount of the lvm devices a filesystem has to be created on the mounted devices,  
 
−
starting a console gives:
 
−
xm console <ID>
 
−
Checking all file systems.
 
−
[/sbin/fsck.ext3 (1) -- /home] fsck.ext3 -a /dev/hdb1
 
−
[/sbin/fsck.ext3 (1) -- /tmp] fsck.ext3 -a /dev/hdb2
 
−
fsck.ext3: Bad magic number in super-block while trying to open /dev/hdb2
 
−
[FAILED]
 
−
 
 
−
*** An error occurred during the file system check.
 
−
*** Dropping you to a shell; the system will reboot
 
−
*** when you leave the shell.
 
−
Give root password for maintenance
 
−
 
 
−
 
−
This can happen if you have in the xm create start file you have something like:
 
−
 
 
−
'phy:/dev/XenVG/Xen_l_CE_home,hdb1,w',
 
−
 
 
−
in the fstab of the virtual maschine you have something like
 
−
/dev/hdb1 /home
 
−
 
 
−
then give root passwort and create the filesystems with e.g.
 
−
mkfs.ext3 /dev/hdb1
 
−
 
 
−
===== MySQL =====
 
−
 
 
−
Aus meinen Notizen, ohne weitere Formatierung.
 
−
 
 
−
Passwort fuer einen Benutzer einer Datenbank aendern:
 
−
1) als root einloggen:
 
−
  mysql -h localhost -u root -p
 
−
 
−
  2) setzen des Passworts fuer den Benutzer ->accounting<- auf das Passwort
 
−
->accountingspw<-:
 
−
 
 
−
  mysql> set password for accounting@localhost=Password('accountingspw');
 
−
 
 
−
 
 
−
  USE mysql
 
−
GRANT SELECT, INSERT, UPDATE, DELETE ON accounting.* TO 'accounting'
 
−
IDENTIFIED BY 'accounting';
 
−
DELETE FROM user WHERE user LIKE 'accounting';
 
−
DELETE FROM db WHERE user LIKE 'accounting';
 
−
INSERT INTO user (host,user,password,create_priv,drop_priv) VALUES
 
−
('localhost','accounting',password('accounting'),'Y','Y');
 
−
INSERT INTO user (host,user,password,create_priv,drop_priv) VALUES
 
−
('127.0.0.1','accounting',password('accounting'),'Y','Y');
 
−
INSERT INTO user (host,user,password,create_priv,drop_priv) VALUES
 
−
('localhost.localdomain','accounting',password('accounting'),'Y','Y');
 
−
INSERT INTO user (host,user,password,create_priv,drop_priv) VALUES
 
−
('a01-004-123.gridka.de','accounting',password('accounting'),'Y','Y');
 
−
INSERT INTO user (host,user,password,create_priv, drop_priv) VALUES
 
−
('lcg-gridka-ce','accounting',password('accounting'),'Y','Y');
 
−
INSERT INTO db
 
−
(host,db,user,select_priv,insert_priv,update_priv,delete_priv,create_priv,
 
−
alter_priv) VALUES ('%','accounting','accounting','Y','Y','Y','Y','Y','Y');
 
−
FLUSH PRIVILEGES;
 
−
DROP DATABASE IF EXISTS accounting;
 
−
CREATE DATABASE accounting;
 
−
USE accounting;
 
−
 
−
========== Um von meinem Rechner darauf zu arbeiten ===========
 
−
[root@lcg-gridka-ce yaim]# mysql -h localhost -u root -p
 
−
Enter password:
 
−
Welcome to the MySQL monitor.  Commands end with ; or \g.
 
−
Your MySQL connection id is 4 to server version: 4.0.25-standard
 
  
−
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
+
Apr 22 10:29:09 toom kernel: xen_net: Memory squeeze in netback driver.
 +
Apr 22 10:29:14 toom kernel: printk: 8 messages suppressed.
  
−
mysql> USE mysql
+
This relates to a [https://bugzilla.redhat.com/show_bug.cgi?id=454285 known bug]; a suggested solution is to limit the Dom0 memory to prevent 'ballooning'. This is what I did:
−
Reading table information for completion of table and column names
 
−
You can turn off this feature to get a quicker startup with -A
 
  
−
Database changed
+
in /etc/grub.conf:
−
mysql> INSERT INTO user (host,user,password,create_priv, drop_priv) VALUES
+
kernel /xen.gz-2.6.18-128.1.6.el5 com2=9600,8n1 console=com2,vga dom0_mem=1024M
−
    -> ('iwrgabriel.ka.fzk.de','accounting',password('accounting'),'Y','Y');
 
−
Query OK, 1 row affected (0.10 sec)
 
  
−
mysql> FLUSH PRIVILEGES;
+
and in /etc/xen/xend-config.sxp:
−
Query OK, 0 rows affected (0.02 sec)
+
(dom0-min-mem 1024)
  
−
mysql> quit
+
Making 1024 (megabyte) both the minimum and maximum of the dom0 memory. This could affect performance, but I haven't seen the squeeze message since.
−
Bye
 

Latest revision as of 14:34, 21 October 2009

Docu on CentOS

Some links:

CentOS-5 Documentation including docs on Virtualization

check PCI-DMA: Out of SW-IOMMU space ERROR

CentOS 5

These are just notes taken during the installation, it is not meant as a howto or something like that

Hardware Dell 1950

Additional Stuff

mercurial is a distributed version control system. to use it on centos-5 X86_64 do:

compile libvirt

  • wget ftp://ftp.easynet.be/gnu/gettext/gettext-0.17.tar.gz; tar xvfz gettext-0.17.tar.gz; cd gettext-0.17; ./configure; make; make install
  • yum install gnutls gnutls-devel
  • getting latest libvirt see: libvirt cvs
  • cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs login; password anoncvs cvs -d :pserver:anoncvs@libvirt.org:2401/data/cvs co libvirt
  • cd to libvirt dir and apply patch patch -p1 < configure.patch
  • in libvirt dir do: ./autogen.sh; make; make install

virt-install needs: 

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

CentOS 5 X86_64, Minimal system

Additional Packages

Minimal System installed via kickstart, packages: 

# Packages groups/list
%packages --resolvedeps --ignoremissing

openssh
openssh-server
wget
perl-libnet
perl-MIME-Base64
perl-URI
perl-Digest-MD5
perl-libwww-perl
perl-XML-Parser
perl-DBI
perl-Crypt-SSLeay
lsof

Post Install, get the following packages

Some of these are not needed, has to be cleaned: 

yum install \
gcc make zlib-devel zlib python-devel curses libncurses-devel ncurses-devel openssl openssl-devel \
`yum search xorg-x11|grep dev|grep x86_64|awk -F . '{print $1}'` \
bridge-utils tetex tetex-latex transfig libtool-ltdl dev86 glibc-devel

xen 3.1.2-rc from mercurial

For 64-bit Dom_0 and 32-bit Dom_U kombination:, the following is taken from: xensource install notes 

Dell 2900 modules to include in mkinitrd

mkinitrd -v -f --with=sd_mod --with=scsi_mod --with=megaraid_mm --with=megaraid_sas --with=megaraid_mbox initrd-2.6.16.29-xen.img 2.6.16.29-xen

dom0 64 bit and domU 32 bit from source code

Build and install xen:

make xen
make install-xen 

Build and install the dom0:

make linux-2.6-xen0-prep
cd build-linux-*-xen0-x86_64
make menuconfig
make
make install modules_install

And finally build and install 32 bit domU: 

make XEN_TARGET_ARCH=x86_32 linux-2.6-xenU-prep
cd build-linux-*-xen0-x86_32
make ARCH=i386 menuconfig
make ARCH=i386
make ARCH=i386 install modules_install



hg pull ...
cd xen-3.1
wget http://www.broadcom.com/docs/driver_download/NXII/linux-1.5.10c.zip
unzip, use broadcom sources for 
make world
make install
depmod 2.6.18-xen
mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod  --with=megaraid_sas /boot/initrd-2.6.18-xen.img 2.6.18-xen

Note on: Network Problems Broadcom drivers have problems, network is OK when xend started without network bridges, see /etc/xen/xend-config.sxp

Starting bridge with: /etc/xen/scripts/network-bridge start results in a not working network configuration in dom0.

see network-bridge causes network to hang

swiotlb=128 has to be set in menu.list since we have 8GB RAM, without youll get error messages like : 

Sep  7 13:14:22 test-11 kernel: PCI-DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:02:01.0

see also: SW-IOMMU space

  • change/add /boot/grub/menu.list:
 title CentOS (xen-3.1.2-rc2 2.6.18 compiled)
       root (hd0,0)
       kernel /xen-3.1.gz
       module /vmlinuz-2.6-xen ro root=/dev/md1 swiotlb=128
       module /initrd-2.6.18-xen.img

Boot into your new system:

[root@kribbe ~]# xm dmesg 

__  __            _____  _   ____              ____
\ \/ /___ _ __   |___ / / | |___ \    _ __ ___|___ \
 \  // _ \ '_ \    |_ \ | |   __) |__| '__/ __| __) |
 /  \  __/ | | |  ___) || |_ / __/|__| | | (__ / __/
/_/\_\___|_| |_| |____(_)_(_)_____|  |_|  \___|_____|

http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory

Xen version 3.1.2-rc2 (root@nikhef.nl) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Tue Nov 13 15:02:14 CET 2007
Latest ChangeSet: unavailable
.
.
.
(XEN) HVM: VMX enabled
(XEN) VMX: MSR intercept bitmap enabled
(XEN) CPU0: Intel(R) Xeon(R) CPU            5150  @ 2.66GHz stepping 06
.
.
.



LVM 2

Due to performance reasons VMs should be installed in LVMs. For recovering a LVM in a Raid-1 see:Recovery of RAID and LVM2 Volumes

Installed Version: 

Installed Packages
Name   : lvm2
Arch   : i386
Version: 2.02.16
Release: 3.el5
Size   : 5.5 M
Repo   : installed
Summary: Userland logical volume management tools


NOTE: for having the live-migration-posibility we have to look at Cluster-LVM or something like that.

To get started ....

Setting up LVM2 on kribbe

Using /dev/srv resp. /dev/md3 for LVM we have to:

  • comment out /dev/md3 in /etc/fstab
  • umount /dev/srv (only once)
  • Creating the physical volume and XenVG Volumegroup:
[root@kribbe ~]# pvcreate /dev/md3
 Physical volume "/dev/md3" successfully created
[root@kribbe ~]# vgcreate XenVG /dev/md3
 Volume group "XenVG" successfully created

Within this volume-group we can now manage the partitions for the virtual maschines using lvcreate, lvresize, .... commands

Example: lvm for a CentOS5 test VM

Root FS, only one partition 

[root@kribbe tmp]# lvcreate -L10G -nXen_CentOS5_Test XenVG
 Logical volume "Xen_CentOS5_Test" created
root@kribbe tmp]# mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1310720 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
       32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.


SWAP

Note: swap in a lvm is not performant 

[root@kribbe tmp]# lvcreate -L1G -nXen_CentOS5_Test_swap XenVG
 Logical volume "Xen_CentOS5_Test_swap" created
[root@kribbe tmp]# mkswap /dev/XenVG/Xen_CentOS5_Test_swap
Setting up swapspace version 1, size = 1073737 kB

Usage of LVM2 on Centos xvda disks

Taken from xen-users archive.


If you are using a lvm2 partition as a disk (xvda) in a dom_u and this lvm is itself partitioned you can't use just

mount -o loop <logical volume> <mountpoint>, instead you have to give the first sector of the partition you want to mount. This can be calculated by multiplying the start sector with its size

Example: 

[root@hilde xen]# fdisk -l -u /dev/mapper/xenvg-hooiwagenc4532

Disk /dev/mapper/xenvg-hooiwagenc4532: 53.6 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes

                           Device Boot      Start         End      Blocks   Id  System
/dev/mapper/xenvg-hooiwagenc4532p1   *          63      208844      104391   83  Linux
/dev/mapper/xenvg-hooiwagenc4532p2          208845     2249099     1020127+  82  Linux swap / Solaris
/dev/mapper/xenvg-hooiwagenc4532p3         2249100   104856254    51303577+  83  Linux



[root@hilde xen]# mount -o loop,offset=32256 /dev/mapper/xenvg-hooiwagenc4532 /mnt2

[root@hilde xen]# ls /mnt2
bin   dev  home    lib         media  opt   root  selinux  sys  usr
boot  etc  initrd  lost+found  mnt    proc  sbin  srv      tmp  var

Virtualize Physical Nodes

  1. On DOM_0 Create a lvm for the DOM_U
  2. Mount this lvm on DOM_0
  3. On Physical Node change runlevel to s
  4. setup a exclude-list file, for details see eisxen
  5. cd /;tar csp --exclude-from=/rootfs/exclude.lst . |ssh <DOM_0> "cd /<mounted LVM>; tar xsp"
  6. change the fstab of the dom_u
Example exclude.lst, node specific, dont just copy, see also: eisxen
This list must not have an empty line at the end.
less /rootfs/exclude.lst
./boot
./data
./datastore
./datastore2
./proc
./rootfs
./etc/mtab
./sys
./project

DOM_U is a centos-3.x

In order to run this as a virtual maschine, the dom_u might need a new version of e2fschk and module-init-tools. Therfore, on DOM_0 mount the DOM_U fs created in an earlier step, get e2fsprogs-1.39.tar.gz module-init-tools-3.2.tar.gz (these versions worked for us) to this directory. The following commands worked for me.

  • chroot to dom_u mountpoint
  • extract, compile, install e2fschk and module-init-tools
  • cp -r DOM_0:/lib/modules/<running version> DOM_U:/lib/modules
  • for e2fs do: tar xvfz e2fsprogs-1.39.tar.gz ;cd e2fsprogs-1.39; ./configure; make; make install
  • for modutils: tar xvfz module-init-tools-3.2.tar.gz; cd module-init-tools-3.2; ./configure --prefix=/ ;make moveold; make; make install

CentOS 5 with Virtualization package

  • Installed Version:
2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
  • Run yum update, Version:
2.6.18-8.1.15.el5xen #1 SMP Mon Oct 22 09:01:12 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
  • xen:
[root@kribbe ~]# xm dmesg
__  __            _____  ___   _____             ____       _ ____
\ \/ /___ _ __   |___ / / _ \ |___ /    _ __ ___| ___|  ___| | ___|
 \  // _ \ '_ \    |_ \| | | |  |_ \ __| '__/ __|___ \ / _ \ |___ \
 /  \  __/ | | |  ___) | |_| | ___) |__| | | (__ ___) |  __/ |___) |
/_/\_\___|_| |_| |____(_)___(_)____/   |_|  \___|____(_)___|_|____/

http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory

Xen version 3.0.3-rc5-8.1.15.el5 (mockbuild@) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Mon Oct 22 08:25:42 EDT 2007
Latest ChangeSet: unavailable
....
(XEN) VMXON is done
(XEN) Dom0 has maximum 4 VCPUs
  • lvm2, bridge-utils
Installed Packages:
Name   : lvm2
Arch   : x86_64
Version: 2.02.16

Name   : bridge-utils
Arch   : x86_64
Version: 1.1


  • yum install vnc
  • use partition /dev/md3 mounted on /project for lvm
comment out ..... /project
umount /project
pvcreate /dev/md3
vgcreate XenVG /dev/md3
lvcreate -L10G -nXen_CentOS5_Test XenVG
mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test
lvcreate -L1G -nXen_CentOS5_Test_swap XenVG
mkswap /dev/XenVG/Xen_CentOS5_Test_swap

Running various centOS Versions as VMs on kribbe

Virt-manager for installation

virt-install can be used for VM installation. No pypxeboot and patched udhcp needed.

CentOS 5 64-Bit in VM

set up kickstart file for VM, than 

virt-install -p --location=http://www.dutchgrid.nl/mirror/centos/5/os/x86_64 --noautoconsole \
--file=/dev/XenVG/Xen_CentOS5_Test --name=domU001 --ram=1024  --mac=00:16:3e:ab:61:0b\
-x "ks=http://stal.nikhef.nl/ks/xen-test-koebrug.ks ip=194.171.97.11 \
netmask=255.255.255.0 dns=192.16.186.253 gateway=194.171.97.254"

MAC-ADDRESS: can be calculated from last the 3 numbers of you guest IP by (ex:): 

printf 00:16:3e:%02x:%02x:%02x"\n" 171 97 11

00:16:3e:ab:61:0b

This has to be defined in the dhcp servers dhcp.conf.


Connect to the console of the starting vm:

  • xm list, lists the VMs:
[root@kribbe ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     7025     4 r-----   1056.7
domU001                                    8     1023     1 -b----    490.8
  • xm console <ID> or <Name> of the VM (s.a.) creates a console of the VM

Worked

Old Note: no vnc option in the virt-install command,

  • --vnc option removed, does not boot.
  • during boot of the VM answer: q: graphics support: no

But this one worked fine with a VNC console (both graphical and text based): 

virt-install  --force -r 512 -m 00:16:3e:34:4b:6e -n mestkar -f /dev/vg0/mestkar.sys -l http://spiegel.nikhef.nl/mirror/centos/5/os/x86_64 -x "ks=http://www.nikhef.nl/grid/ndpf/files/local/install/mestkar.nikhef.nl.ks ksdevice=eth0" --vnc --vcpus=1 -b xenbr0


On VM:

  • koebrug.nikhef.nl
/etc/redhat-release
CentOS release 5 (Final)
[root@koebrug ~]# uname -a
Linux koebrug.nikhef.nl 2.6.18-8.el5xen #1 SMP Thu Mar 15 19:56:43 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

CentOS 4.5 64-Bit in VM

  • silo.nikhef.nl

[root@silo ~]# uname -a Linux silo.nikhef.nl 2.6.9-55.ELxenU #1 SMP Wed May 2 14:29:26 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux [root@silo ~]# cat /etc/redhat-release CentOS release 4.5 (Final)


64 Bit DOM_U in 64 Bit DOM_0 seems to be no problem. 32Bit installations can not be done in this way, apropriate kernel needed Looking at virsh

Using VIRSH

Information on virsh can be found at: centos-wiki or Red Hat Virtualization

virsh is another virtualization management tool (built around the libvirt management API) which is now capable to deal with xen, future versions might also be able to manage kvm or qemu.

Since we plan to use xen we should stick with xm directly.

Virtual Maschine management with xm

based on: xen-user-docu

CentOS 5 i386-System -- Quattor generated

Problems

  • xenconsole: Could not read tty from store: No such file or directory
root@hilde xen]# xm create koebrug -c
Using config file "./koebrug".
Started domain koebrug
xenconsole: Could not read tty from store: No such file or directory

Try: /etc/init.d/xend restart



  • Post-Quattor steps to prevent quattor from revoking installation/configuration operations.
on kribbe:
chkconfig cdp-listend off
chkconfig ncm-cdispd off

on stal:
[root@stal ~]# aii-shellfe --boot kribbe.nikhef.nl

to give back the control to quattor (fresh install next boot, will also switch on cdp-listen and ncm-cdispd)
[root@stal ~]#  aii-shellfe --install kribbe.nikhef.nl
  • Mount external storage for files that should survive a reboot/installation
mount -tnfs tbn15.nikhef.nl:/export/array2/local/tbadmin /tmpmnt
  • Hostname: kribbe.nikhef.nl
  • Kernel: 2.6.18-8.1.8.el5xen
  • HD-Layout:
[root@kribbe ~]# sfdisk -l /dev/sdb

Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+     32      33-    265041   fd  Linux raid autodetect
/dev/sdb2         33   51025   50993  409601272+  fd  Linux raid autodetect
/dev/sdb3      51026   58580    7555   60685537+  fd  Linux raid autodetect
/dev/sdb4      58581   60800    2220   17832150    5  Extended
/dev/sdb5      58581+  60147    1567-  12586896   fd  Linux raid autodetect
/dev/sdb6      60148+  60669     522-   4192933+  fd  Linux raid autodetect
/dev/sdb7      60670+  60800     131-   1052226   fd  Linux raid autodetect

====================

[root@kribbe ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
     264960 blocks [2/2] [UU]
     
md3 : active raid1 sdb2[1] sda2[0]
     409601152 blocks [2/2] [UU]
     
md5 : active raid1 sdb3[1] sda3[0]
     60685440 blocks [2/2] [UU]
      
md2 : active raid1 sdb6[1] sda6[0]
     4192832 blocks [2/2] [UU]
      
md4 : active raid1 sdb7[1] sda7[0]
     1052160 blocks [2/2] [UU]
      
md1 : active raid1 sdb5[1] sda5[0]
     12586816 blocks [2/2] [UU]
      
unused devices: <none>


Example installation of a CentOS5 VM

Create a VM in lvm partition

  1. Setup lvm as stated above
  2. Mount VM root fs: mount /dev/XenVG/Xen_CentOS5_Test /mnt
  3. extract archive:
  4. [root@kribbe tmp]# bunzip2 centos.5-0.20070424.img.tar.bz2
  5. [root@kribbe tmp]# tar xvf centos.5-0.20070424.img.tar
  6. loopmount image
  7. [root@kribbe tmp]# mkdir /img
  8. [root@kribbe tmp]# mount -o loop centos.5-0.img /img
  9. cp image to logical volume
  10. [root@kribbe img]# cp -a * /mnt/
  11. Change Root PW in VM:
[root@kribbe mnt]# chroot /mnt
bash-3.1# pwconv
bash-3.1# passwd
Changing password for user root.
  1. Set Hostname of VM: /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=koebrug.nikhef.nl
  1. no swap in VM: /etc/fstab
#/dev/sda2               none                    swap    sw       0 0


setup of VM start files

SELinux, Postinstallation steps

  • Cleaning up daemons, pay attention to ntp, does the dom_U has the correct time?
  /sbin/service microcode_ctl stop
  /sbin/chkconfig --del microcode_ctl
  /sbin/service smartd stop
  /sbin/chkconfig --del smartd


If you are using SELinux, it is important to check that the image has the correct security context (xen_image_t), or access to the virtual disk will be denied to the domU system. You can check this with ls: 

# ls -Z /srv/xen/mailserver.img 
-rw-r--r--  root root user_u:object_r:xen_image_t      /srv/xen/mailserver.img

If you are having trouble setting the right file context, please have a look at the Xen tips and tricks page. At any rate, turning off SELinux, as some howtos on this subject advise is a very poor workaround. Reading two manual pages (semanage(8) and restorecon(8)), for an extra layer of security is a good trade!


Memory and network problems

The CentOS 5 (RHEL5) release of Xen is not really stable; it changed it's behaviour over time from 'not working at all' in CentOS 5.0 and 5.1 to 'working' in '5.2', and with further updates it appears as though rather important changes find their way into the Xen part of this otherwise stable distro.

Recently, and this is 5.3 on April 22, 2009, I started to see spontaneous reboots of my Dom0. Not much in the syslog, but some suspicious entries about memory squeeze: 

Apr 22 10:29:09 toom kernel: xen_net: Memory squeeze in netback driver.
Apr 22 10:29:14 toom kernel: printk: 8 messages suppressed.

This relates to a known bug; a suggested solution is to limit the Dom0 memory to prevent 'ballooning'. This is what I did:

in /etc/grub.conf: 

kernel /xen.gz-2.6.18-128.1.6.el5 com2=9600,8n1 console=com2,vga dom0_mem=1024M

and in /etc/xen/xend-config.sxp: 

(dom0-min-mem 1024)

Making 1024 (megabyte) both the minimum and maximum of the dom0 memory. This could affect performance, but I haven't seen the squeeze message since.

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Xen_on_CentOS_5_-_Notes&oldid=7076"