Xen on CentOS 5 - Notes

From PDP/Grid Wiki
Jump to navigationJump to search

System -- Quattor generated

Problems

  1. virt-manager not working centos-bug


  • Post-Quattor steps to prevent quattor from revoking installation/configuration operations.
on kribbe:
chkconfig cdp-listend off
chkconfig ncm-cdispd off

on stal:
[root@stal ~]# aii-shellfe --boot kribbe.nikhef.nl

to give back the control to quattor (fresh install next boot, will also switch on cdp-listen and ncm-cdispd)
[root@stal ~]#  aii-shellfe --install kribbe.nikhef.nl
  • Mount external storage for files that should survive a reboot/installation
mount -tnfs tbn15.nikhef.nl:/export/array2/local/tbadmin /tmpmnt
  • Hostname: kribbe.nikhef.nl
  • Kernel: 2.6.18-8.1.8.el5xen
  • HD-Layout:
[root@kribbe ~]# sfdisk -l /dev/sdb

Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+     32      33-    265041   fd  Linux raid autodetect
/dev/sdb2         33   51025   50993  409601272+  fd  Linux raid autodetect
/dev/sdb3      51026   58580    7555   60685537+  fd  Linux raid autodetect
/dev/sdb4      58581   60800    2220   17832150    5  Extended
/dev/sdb5      58581+  60147    1567-  12586896   fd  Linux raid autodetect
/dev/sdb6      60148+  60669     522-   4192933+  fd  Linux raid autodetect
/dev/sdb7      60670+  60800     131-   1052226   fd  Linux raid autodetect

====================

[root@kribbe ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
     264960 blocks [2/2] [UU]
     
md3 : active raid1 sdb2[1] sda2[0]
     409601152 blocks [2/2] [UU]
     
md5 : active raid1 sdb3[1] sda3[0]
     60685440 blocks [2/2] [UU]
      
md2 : active raid1 sdb6[1] sda6[0]
     4192832 blocks [2/2] [UU]
      
md4 : active raid1 sdb7[1] sda7[0]
     1052160 blocks [2/2] [UU]
      
md1 : active raid1 sdb5[1] sda5[0]
     12586816 blocks [2/2] [UU]
      
unused devices: <none>

Added to Quattor System

xen 3.1

wget http://bits.xensource.com/oss-xen/release/3.1.0/kernel-3.1-rhel5x/RPMS/i386/xen-3.1.0-1.i386.rpm
wget http://bits.xensource.com/oss-xen/release/3.1.0/kernel-3.1-rhel5x/RPMS/i386/kernel-xen-2.6.18-3.1.0.i386.rpm
wget http://bits.xensource.com/oss-xen/release/3.1.0/kernel-3.1-rhel5x/RPMS/i386/xen-devel-3.1.0-1.i386.rpm
wget http://bits.xensource.com/oss-xen/release/3.1.0/kernel-3.1-rhel5x/SRPMS/xen-3.1.0-1.src.rpm

Notes:

  1. rpm -i --force kernel-xen-2.6.18-3.1.0.i386.rpm (kernel is older than installed one, --force)
  2. rpm -Uvh xen-3.1.0-1.i386.rpm
  3. sources and dev package not installed
  4. change /boot/grub/menu.list:
title Xen (3.10)
       root (hd0,0)
       kernel /xen-3.1.gz
       module /vmlinuz-2.6.18-xen_3.1.0 ro root=/dev/md1
       module /initrd-2.6.18-xen_3.1.0.img

reboot resulted in a startet xen dom_0, HVM enabled: 

[root@kribbe ~]#xm dmesg:
__  __            _____  _   ___
\ \/ /___ _ __   |___ / / | / _ \
 \  // _ \ '_ \    |_ \ | || | | |
 /  \  __/ | | |  ___) || || |_| |
/_/\_\___|_| |_| |____(_)_(_)___/

http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory

Xen version 3.1.0 (avsm@uk.xensource.com) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) Thu May 17 05:49:14 EDT 2007
Latest ChangeSet: unavailable

...
(XEN) System RAM: 8191MB (8387872kB)
...
(XEN) HVM: VMX enabled
(XEN) VMX: MSR intercept bitmap enabled
...
(XEN)  Xen  kernel: 32-bit, PAE, lsb
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0xc0100000 -> 0xc044753c
...
(XEN) Dom0 has maximum 4 VCPUs


Network

  1. Network cabeling has to be changed when moving from centOS 5 default kernel to installed xen kernel, REALlY?
  2. eth0 HWaddr 00:15:C5:E9:48:B1, eth1 HWaddr 00:15:C5:E9:48:B3
  3. dhcpd.conf on stal is written by quattor?: .. manualy changed :B1 to :B3
host kribbe {  # added by aii-dhcp
             hardware ethernet 00:15:C5:E9:48:B3;
             fixed-address 194.171.97.4;
             }
  1. Make sure bridgeutlis are installed
  2. ifconfig should show eth0/1 as well as the bridges (xenbr0), virtual interface (vif), physical eth (pethX), example:
[root@kribbe ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:15:C5:E9:48:B1
...
peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
...
vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
...
xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF

For more information see Networking in Xen, especially Bridging

Using 2 Networks in the Virtual Maschine -not needed at nikhef!

The following setup can be used to have 2 network interface available in the VMs, perhaps usefull for a management network. If a second Network Interface is needed just uncomment the line 

"$dir/network-bridge" start vifnum=1 netdev=eth1

in /etc/xen/scripts/network-fzk.xen3

Problems will occure when the bridged interface is not cabled.

Needed XEN Scripts:

/etc/xen/xend-config.sxp 

(logfile /var/log/xen/xend.log)
(loglevel DEBUG)
(network-script network-fzk.xen3)
(dom0-min-mem 196)
(dom0-cpus 0)

/etc/xen/scripts/network-fzk.xen3 

#!/bin/sh
dir=$(dirname "$0")
echo "====== $dir ======"
"$dir/network-bridge" start vifnum=0 netdev=eth0
#"$dir/network-bridge" start vifnum=1 netdev=eth1
echo "=== Bridges configured === "

Note: The last line is to make sure to have an CR after netdev=eth1, otherwise this line will not be executed

Configure 2 Bridges, but only internal nic has an IP

Debian: /etc/network/interfaces 

auto lo
iface lo inet loopback

allow-hotplug eth0 eth1

iface eth1 inet manual

iface eth0 inet static
       address 10.97.4.150
       netmask 255.255.255.0
       network 10.97.4.0
       gateway 10.97.4.1
       broadcast 10.97.4.255
       # dns-* options are implemented by the resolvconf package, if installed
       dns-nameservers 10.97.1.191 141.52.8.18
       dns-search gridka.de fzk.de
       post-up route add -net 10.0.0.0/9 gw 10.97.4.1

LVM 2

Due to performance reasons VMs should be installed in LVMs. For recovering a LVM in a Raid-1 see:Recovery of RAID and LVM2 Volumes

Installed Version: 

Installed Packages
Name   : lvm2
Arch   : i386
Version: 2.02.16
Release: 3.el5
Size   : 5.5 M
Repo   : installed
Summary: Userland logical volume management tools


NOTE: for having the live-migration-posibility we have to look at Cluster-LVM or something like that.

To get started ....

Setting up LVM2 on kribbe

Using /dev/srv resp. /dev/md3 for LVM we have to:

  • comment out /dev/md3 in /etc/fstab
  • umount /dev/srv (only once)
  • Creating the physical volume and XenVG Volumegroup:
[root@kribbe ~]# pvcreate /dev/md3
 Physical volume "/dev/md3" successfully created
[root@kribbe ~]# vgcreate XenVG /dev/md3
 Volume group "XenVG" successfully created

Within this volume-group we can now manage the partitions for the virtual maschines using lvcreate, lvresize, .... commands

Example: lvm for a CentOS5 test VM

Root FS, only one partition 

[root@kribbe tmp]# lvcreate -L10G -nXen_CentOS5_Test XenVG
 Logical volume "Xen_CentOS5_Test" created
root@kribbe tmp]# mkfs.ext3 /dev/XenVG/Xen_CentOS5_Test
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1310720 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
       32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.


SWAP 

[root@kribbe tmp]# lvcreate -L1G -nXen_CentOS5_Test_swap XenVG
 Logical volume "Xen_CentOS5_Test_swap" created
[root@kribbe tmp]# mkswap /dev/XenVG/Xen_CentOS5_Test_swap
Setting up swapspace version 1, size = 1073737 kB

Example installation of a CentOS5 VM

Create a VM in lvm partition

  1. Setup lvm as stated above
  2. Mount VM root fs: mount /dev/XenVG/Xen_CentOS5_Test /mnt
  3. extract archive:
  4. [root@kribbe tmp]# bunzip2 centos.5-0.20070424.img.tar.bz2
  5. [root@kribbe tmp]# tar xvf centos.5-0.20070424.img.tar
  6. loopmount image
  7. [root@kribbe tmp]# mkdir /img
  8. [root@kribbe tmp]# mount -o loop centos.5-0.img /img
  9. cp image to logical volume
  10. [root@kribbe img]# cp -a * /mnt/
  11. Change Root PW in VM:
[root@kribbe mnt]# chroot /mnt
bash-3.1# pwconv
bash-3.1# passwd
Changing password for user root.
  1. Set Hostname of VM: /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=koebrug.nikhef.nl
  1. no swap in VM: /etc/fstab
#/dev/sda2               none                    swap    sw       0 0


setup of VM start files

Xen Notes, Old Version, to be cleaned up

XenServer-3.2 Commercial Version

any information can be found on: xen-doku an related links.

Installation xenserver

Principle setup

  • VMs are organized in a Virtual Rack (a01-008-210)
  • Network: VLAN 1401 (10.97.210.0) and VLAN 158 (192.108.45.0) are configured
  • in VMs use xenbr1401(eth1401) for eth0 (internal network)
  • in VMs use xenbr1(eth1) for eth1 (external network)

Notes To install a new xenserver DOM_0 

Local Disk is sdb
SAN is sda
If asked DONT CREATE A STORAGE REPOSITORY, this will delete ALL VMs in the SAN

Installation Procedure

  1. Boot from xenserver CD
  2. Set eth0 to ONBOOT=NO, i.e. don't use eth0 during installation
  3. Configure eth1 with external IP (192.....)
  4. NTP: 10.97.8.200
  5. DNS: 10.97.1.192, 10.97.1.191, 141.52.8.18
  6. NO XEN-STORAGE, disable FLAGS on sdb as well as on sda (SAN)
  7. Use sdb (local disk) for installation of the base system
  8. Post Installation:
    1. add: /etc/sysconfig/static-routes: any net 10.0.0.0/9 gw 10.97.81
    2. edit /etc/resolv.conf add search fzk.de, set 141 as primary nameserver
    3. edit /boot/grub/menu.lst, replace hd1,0 with hd0,0
    4. Manually setting the default Storage Repository Manually setting the default Storage Repository (SR)
      1. /etc/smtab: 7c2cbf8e-d22e-46a3-8b3c-7791499e7510 none lvm default auto
      2. service srmtab restart
      3. service xenagentd start
    5. set up vlan:
      1. ifup eth0
      2. modprobe 8021q
      3. vconfig add eth0 1401
      4. vconfig add eth0 158
      5. Set up the following files accordingly
/etc/sysconfig/network-scripts

==== ifcfg-eth0 =====
DEVICE=eth0
ONBOOT=no
TYPE=Ethernet
HWADDR=00:1a:64:12:80:71
BRIDGE=xenbr0
LINKDELAY=5
check_link_down() { return 1 ; }
==== ifcfg-eth1 ==== 
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
HWADDR=00:1a:64:12:80:73
BRIDGE=xenbr1
LINKDELAY=5
check_link_down() { return 1 ; }
==== ifcfg-eth1401 ==== 
DEVICE=eth0.1401
ONBOOT=yes
TYPE=Ethernet
HWADDR=00:1a:64:12:80:71
BRIDGE=xenbr1401
LINKDELAY=5
VLAN=yes
check_link_down() { return 1 ; }
==== ifcfg-eth158 ==== 
DEVICE=eth0.158
ONBOOT=yes
TYPE=Ethernet
HWADDR=00:1a:64:12:80:71
BRIDGE=xenbr158
LINKDELAY=5
VLAN=yes
check_link_down() { return 1 ; }
==== ifcfg-xenbr0 ==== 
DEVICE=xenbr0
ONBOOT=no
TYPE=Bridge
DELAY=0
STP=off
check_link_down() { return 1 ; }
==== ifcfg-xenbr1 ==== 
DEVICE=xenbr1
ONBOOT=yes
TYPE=Bridge
DELAY=0
STP=off
NETMASK=255.255.255.0
IPADDR=192.108.45.86
GATEWAY=192.108.45.1
PEERDNS=yes
check_link_down() { return 1 ; }
==== ifcfg-xenbr1401 ==== 
DEVICE=xenbr1401
ONBOOT=yes
TYPE=Bridge
DELAY=0
STP=off
NETMASK=255.255.255.0
IPADDR=10.97.210.107
PEERDNS=yes
check_link_down() { return 1 ; }
==== ifcfg-xenbr158 ==== 
DEVICE=xenbr158
ONBOOT=yes
TYPE=Bridge
DELAY=0
STP=off
NETMASK=255.255.255.0
IPADDR=10.97.8.107
PEERDNS=yes
check_link_down() { return 1 ; }


Cluster LVM
  1. cd /etc/yum.r*
  2. wget http://ftp.halifax.rwth-aachen.de/centos/4/csgfs/CentOS-csgfs.repo
  3. yum install lvm2-cluster

Debian / Xen3

Debian 4.0 / Xen3 - xen-linux-system-2.6.18-4-xen-amd64

Post Installation basic package: 

apt-get install ssh xen-linux-system-2.6.18-4-xen-amd64 ntp ntp-server bridge-utils less

Needed XEN Scripts:

/etc/xen/xend-config.sxp 

(logfile /var/log/xen/xend.log)
(loglevel DEBUG)
(network-script network-fzk.xen3)
(dom0-min-mem 196)
(dom0-cpus 0)

/etc/xen/scripts/network-fzk.xen3 

#!/bin/sh
dir=$(dirname "$0")
echo "====== $dir ======"
"$dir/network-bridge" start vifnum=0 netdev=eth0
"$dir/network-bridge" start vifnum=1 netdev=eth1
echo "=== Bridges configured === "

Note: The last line is to make sure to have an CR after netdev=eth1, otherwise this line will not be executed


Configure 2 Bridges, but only internal nic has an IP

/etc/network/interfaces 

auto lo
iface lo inet loopback

allow-hotplug eth0 eth1

iface eth1 inet manual

iface eth0 inet static
       address 10.97.4.150
       netmask 255.255.255.0
       network 10.97.4.0
       gateway 10.97.4.1
       broadcast 10.97.4.255
       # dns-* options are implemented by the resolvconf package, if installed
       dns-nameservers 10.97.1.191 141.52.8.18
       dns-search gridka.de fzk.de
       post-up route add -net 10.0.0.0/9 gw 10.97.4.1

OUTDATED Debian sarge amd64 / xen-3 nstallation

/etc/apt/sources.list 

# Debian sources

deb http://amd64.debian.net/debian-amd64 sarge           main contrib non-free
deb http://security.debian.org           sarge/updates   main contrib non-free
deb http://www.backports.org/debian      sarge-backports main

deb-src http://amd64.debian.net/debian-amd64 sarge           main contrib non-free
deb-src http://www.backports.org/debian      sarge-backports main

deb http://ftp.uni-stuttgart.de/debian testing  main
deb http://ftp.uni-stuttgart.de/debian unstable main

deb http://packages.debianbase.de/sarge/amd64/xen3/ ./
deb-src http://packages.debianbase.de/sources/xen3 ./


/etc/apt/preferences 

Package: *
Pin: release o=Debian,a=stable,l=Debian-Security
Pin-Priority: 950 

Package: *
Pin: release o=Debian,a=stable
Pin-Priority: 900

Package: *
Pin: release o=Backports.org archive,a=sarge-backports
Pin-Priority: 800

Package: *
Pin: release o=Debian,a=testing
Pin-Priority: 700

Package: *
Pin: release o=Debian,a=unstable
Pin-Priority: 600

Package: *
Pin: origin ""
Pin-Priority: 400

/boot/grub/menu.lst 

title           Xen 3.0.3-1-amd64 / Debian GNU/Linux, kernel 2.6.18-2-xen-amd64
root            (hd0,0)
kernel          /boot/xen-3.0.3-1-amd64.gz
module          /boot/vmlinuz-2.6.18-2-xen-amd64 root=/dev/sda1 ro console=tty0 console=tty0 max_loop=64
module          /boot/initrd.img-2.6.18-2-xen-amd64
savedefault


Install xen

apt-cache search xen-linux-system-2.6

apt-get install xen-linux-system-2.6.18-2-xen-amd64
apt-get install bridge-utils
Make sure to have brctl installed
apt-get install bridge-utils
/etc/init.d/xend resart

ifconfig should show the bridges xenbr0/1

Scripts for dual homed network

/etc/xen/scripts/network-fzk.xen3 

#!/bin/sh
dir=$(dirname "$0")
echo "====== $dir ======"
"$dir/network-bridge" start vifnum=0 netdev=eth0
"$dir/network-bridge" start vifnum=1 netdev=eth1

patch /etc/xen/xend-config.sxp 

--- /etc/xen/xend-config.sxp.orig      :58:47.000000000 +0100
+++ /etc/xen/xend-config.sxp   :11:42.000000000 +0100
@@ -11,8 +11,8 @@
# Commented out entries show the default for that entry, unless otherwise
# specified.

-#(logfile /var/log/xen/xend.log)
-#(loglevel DEBUG)
+(logfile /var/log/xen/xend.log)
+(loglevel DEBUG)
 
#(xend-http-server no)
#(xend-unix-server no)
@@ -84,8 +84,8 @@
# two fake interfaces per guest domain.  To do things like this, write
# yourself a wrapper script, and call network-bridge from it, as appropriate.
#
-(network-script network-dummy)
-
+#(network-script network-dummy)
+(network-script network-fzk.xen3)
# The script used to control virtual interfaces.  This can be overridden on a
# per-vif basis when creating a domain or a configuring a new vif.  The
# vif-bridge script is designed for use with the network-bridge script, or
@@ -99,7 +99,7 @@
# If you are using only one bridge, the vif-bridge script will discover that,
# so there is no need to specify it explicitly.
#
-(vif-script vif-bridge)
+#(vif-script vif-bridge)


## Use the following if network traffic is routed, as an alternative to the

Creating an SL4-Image

  • Install SL4 somewhere, use minimal setup
  • create exclude.list (/proc, /sys, /rootfs, /..
  • tar cvfpz /

On xen3 dom_0

a01-004-151:~# lvcreate -L1G -nXen_SL4_raw XenVG
 Logical volume "Xen_SL4_raw" created
a01-004-151:~# lvcreate -L1G -nXen_SL4_Dirs XenVG
 Logical volume "Xen_SL4_Dirs" created
a01-004-151:~# lvcreate -L1G -nXen_SL4_swap XenVG
 Logical volume "Xen_SL4_swap" created
a01-004-151:~# mkswap /dev/XenVG/Xen_SL4_swap 
Setting up swapspace version 1, size = kB
no label, UUID=377313c6-498b-4525-b0d3-f9d9fb2c46bd



a01-004-151:~# mkfs.ext3 /dev/XenVG/Xen_SL4_raw  

a01-004-151:~# mkfs.ext3 /dev/XenVG/Xen_SL4_Dirs

Create Image

SL3 image

  1. module_init_tools

SL4 image

out of the box ?

Debian/Xen2 - outdated

OS-Installation

Ganglia on debian

To install and configure ganglia on the dom0 do: 

* apt-get install ganglia-monitor
* scp ce-fzk:/opt/glite/yaim/rpms/gmond.conf /etc/
* /etc/init.d/ganglia-monitor restart

New xen-Kernels

To compile new dom-U kernels, cd to xen/linux-*-xenU and do 

make menuconfig ....choose your options
make ARCH=xen dep bzImage modules modules_install

Debian -- XEN

*Boot from Debian 3.1r1 CD
*Configure first network interface with the external IP (nameserver e.g. 141.52.8.18)
*Partitioning: Whole disc in 2 partitions (50 GB / ; rest /srv (for using lvm see below)
*Installation Type: Manual selection
*additional packages: run script install_packages.sh

As the debian default installation is quite small addidional packages are needed, the following script should get them: 

#!/bin/bash
apt-get install -y make
apt-get install -y gcc
apt-get install -y subversion
apt-get install -y bridge-utils
apt-get install -y curl
apt-get install -y libcurl3-dev
apt-get install -y zlib1g-dev
apt-get install -y zlib1g
apt-get install -y python-dev
apt-get install -y python-twisted
apt-get install -y bzip2
apt-get install -y linuxdoc-tools-latex
apt-get install -y gs-common
apt-get install -y transfig
apt-get install -y host
apt-get install -y iptables-dev
apt-get install -y ntpdate
apt-get install -y module-init-tools
apt-get install -y iproute
apt-get install -y lvm2
apt-get install -y dmsetup
mkdir /opt/xen
cd /opt/xen
wget http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0.7-src.tgz
tar xvfz xen-2.0.7-src.tgz
cd xen-2.0
exit

Untill I find some time or somebody else puts the following in a sed line do: edit Makefile, change the line: Kernels ?= ... so that the 2.4 Kernels are also build, i.e. 

KERNELS ?= linux-2.6-xen0 linux-2.6-xenU linux-2.4-xen0 linux-2.4-xenU

Then build the xen-kernels: 

make world
make install
Setup /boot/grub/menu.lst

Configure this file to boot the xen kernel and to set the max_loop Device parameter. Change below the line 

## ## End Default Options ##

Example: 

#### XEN ####
title Xen 2.0 / XenLinux 2.6
root   (hd0,0)
kernel /boot/xen-2.0.gz dom0_mem=64000 console=vga
module /boot/vmlinuz-2.6-xen0 root=/dev/sda1 ro console=tty0 max_loop=64
boot
Configure Network

The network is centraly configured in /etc/network/interfaces . 

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
       address 192.108.45.66
       netmask 255.255.255.0
       network 192.108.45.0
       broadcast 192.108.45.255
       gateway 192.108.45.1
      # dns-* options are implemented by the resolvconf package, if installed
       dns-nameservers 10.97.1.191 141.52.8.18
Configure secound nic

edit /etc/network/interfaces

  • add eth1 in the line auto eth0
  • add a secound interface block like:
iface eth1 inet static
       address 10.97.4.130
       netmask 255.255.255.0
       network 10.97.4.0
       broadcast 10.97.4.255
       #gateway 10.97.4.1
       # dns-* options are implemented by the resolvconf package, if installed
       dns-nameservers 10.97.1.191 141.52.8.18
       dns-search gridka.de fzk.de
Add a route

append to the interfaces file a line like: 

post-up route add -net 10.0.0.0/9 gw 10.97.4.1

to add a route

Make sure to have enough loop devices
for minor in `seq 8 1 63` ; do mknod /dev/loop$minor b 7 $minor ; done

==== LVM Logical Volume Manager ====

Make sure that lvm2 and dmsetup are installed. 

Create physical volume: 
*pvcreate /dev/sdb1
here we use the sec har disc with one partition, any other setup is possible.
All partitions can be used for a physical volume, from these a Volume Group is created
*vgcreate XenVG /dev/sdb1 /dev/<and all other pv created before>
In this VolumeGroup create the logical volumes to be used in the images, for example
*lvcreate -L20G -nXen_g_CE_home XenVG
To get a list of available logical volumes:
*lvscan -v

Note Take care to set the permissions of the lv apropriatly in the virtual servers, e.g. chmod 1777 /tmp for imported tmp directories in the virtual servers.

Images in lvm

Create a suitable lvm volume with ext3 fs: 

lvcreate -L4GB -n Xen_SE_DC_Sys XenVG
mkfs.ext3 /dev/XenVG/Xen_SE_DC_Sys
XEN POSTINSTALLATION STEPS -- CONFIGURATION of automatic domain start
The files which are needed to start the images (xm create <image-start-file>) should be available through nfs.

This procedure is slightly different from default configuration, as we want to have multiple nics configured in the virtual servers. Procedure on client:

  • NFS:

add to /etc/fstab 

a01-004-155:/nfs/etc_xen /etc/xen_nfs nfs ro 0 0

command: mount -a

  • Create xen-autostart structure
* mv /etc/xen /etc/xen.local
* mkdir -p /etc/xen/auto
* mkdir /etc/xen_nfs
* ln -s /etc/xen_nfs/scripts /etc/xen/scripts
* ln -s /etc/xen_nfs/auto /etc/xen/auto/<IMAGES TO START ON HOST>
* ln -s /etc/xen_nfs/xend-config.sxp /etc/xen/xend-config.sxp
* ln -s /etc/init.d/xendomains /etc/rc2.d/S91xendomains
* ln -s /etc/init.d/xend /etc/rc2.d/S90xend
Timeservers

Virtual servers are using the time of the xen host, make sure that ntpdate is running and is configured coeectly, i.e. use 10.97.4.200 in /etc/default/ntpdate. Make sure that the time is upated in a cron job: 

crontab -e
58 */6 * * * /usr/sbin/ntpdate 10.97.4.200 > /tmp/zeit.log
/etc/init.d/cron restart
Activating Second Network Interface

To activate the second network interface the following procedure was applied: 

http://wiki.xensource.com/xenwiki/XenNetworking

if you want to use multiple bridges, you must create them yourself, either manually, or via your own startup script, or via a custom script to replace network-bridge. For example: The second network interface is started in dom_0 with the script /root/xen-net-init: 


#!/bin/bash
# A.G.
if [ $# != 3 ] ; then
       echo -e "Usage: $0 <DOMAIN-NUMBER> <Internal IP> <External IP>\n"
       exit 1
fi
XEN_S=/etc/xen/scripts
D=$1
# Test if the domain number given really exists
xm list | tail -n +2 | awk '{print $2}' | grep -q $D
if [ $? == 1 ] ; then
       echo -e "Domain $D does not exist! Please provide a DOMAIN-NUMBER\n"
       exit 2 
fi
$XEN_S/network start bridge=xen-br0 netdev=eth0 antispoof=no
$XEN_S/network start bridge=xen-br1 netdev=eth1 antispoof=no
$XEN_S/vif-bridge up domain=${D} vif=vif${D}.0 bridge=xen-br0 mac="FE:FF:FF:FF:FF:FF" ip="$2"
$XEN_S/vif-bridge up domain=${D} vif=vif${D}.1 bridge=xen-br1 mac="FE:FF:FF:FF:FF:FF" ip="$3"
Xen Dom_U Start Configuration Files for the Virtual Maschines

Create in /etc/xen/auto/ the configuration files for each virtual maschine to run on the host. The auto directory will be evaluated at dom_0 boot time. Example /etc/xen/auto/pps-bdii-fzk: 

#  -*- mode: python; -*-

name = "pps-bdii-fzk"
vmid = 1

kernel = "/boot/vmlinuz-2.4.30-xenU"
memory = 256
cpu = 1

# Number of network interfaces. Default is 1.
nics=2
vif=[ 'mac=00:16:3e:70:%d:01,bridge=br0' % vmid,'mac=00:16:3e:70:%d:02,bridge=br1' % vmid]

disk = [ 'file:/data/xen-%s,hda1,w' % name,
         'file:/data/swap-%d,hda2,w' % vmid]

root = "/dev/hda1 ro"

Note: To avoid collisions of mac addresses it is better use constructions like: 

vif=[ 'mac=00:16:3e:70:Hex1:Hex2,bridge=br0','mac=00:16:3e:70:Hex3:Hex4,bridge=br1']

Where Hex 1-4 are the last 2 tuples of the internal and external IP

Note: If you want use lvm partitions from dom_0 you can use: 

disk = [ 'file:/data/xen-%s,hda1,w' % name,
         'file:/data/swap-%d,hda2,w' % vmid,
         'phy:/dev/XenVG/XenCE_home,hdb1,w',
         'phy:/dev/XenVG/XenCE_tmp,hdb2,w']

hdb1/2 is then to included in the fstab file in the virt. maschine, for example: 

/dev/hdb1               /home                   ext3    defaults        1 1
/dev/hdb2               /tmp                    ext3    defaults        1 1

Note: Before /dev/hdb1/2 can be used in the virtual maschine a filesystem has to be created there. Therefore the following can be done:

  • comment out /dev/hdb lines in /ets/fstab in the image
  • boot the virtual maschine
  • xm console to this <ID>
  • mkfs.ext3 /dev/hdb1
  • mkfs.ext3 /dev/hdb2
  • uncomment the lines in /etc/fstab
Images for the virtual maschines

A basic ~2GB image can be optained with: 

dd if=/dev/zero of=/<Image-Name> bs=512 count=4000000
mkfs.ext3 <Image-Name>
mount -o loop <Image-file> /mnt

from an existing linux system which is to be virtualized do 

cd /;tar <all dirs except nfs mounted files>, proc|ssh <where created image is mounted> "cd /mnt; tar xsp"

wget http://savannah.fzk.de/websites/fzk/gridkaschool/2005/xen-images/sl-3.0.5.img.ready-to-yaim--reloaded.gz
wget http://savannah.fzk.de/websites/fzk/gridkaschool/2005/xen-images/swap-1.gz

Unzip the image and the swap file, and move them to /data/<Virt-hostname-Name>. Virt-Node-Name can be e.g. pps-bdii-fzk. For each Virtual host one image and swap file has to be created with the name used in the /etc/xen/auto/<start-config-file>. Before this image can be used several changes in network configuration files have to be done, Therefore loop mount the image and adapt the files

Changing the network configurationof the images

To manipulate files within the image it has to be mounted: 

mount -o loop /<PATH_TO>/<IMAGE_NAME> /tmp

adapt the following files:

  • /etc/ssh/ssh* to use Port 24
  • /etc/apt/apt.conf Default a proxy is used if not needed change apt.conf to:
// User customizable configuration

RPM
{
 // Uncomment to disable GPG-signature checking for packages
 // GPG-Check "false";
};

// Options for the downloading routines
Acquire
{
  // Retries "1";
  // http::Proxy "http://user:password@proxy-server.domain.tld:port/";
  //http::Proxy "http://proxy.fzk.de:8000/";
  //ftp::Proxy "http://proxy.fzk.de:8000/";
};
  • /etc/fstab remove the nfs line gks..., otherwise the booting will take till the nfs mount timeouts, i.e. ages, I'll create a new image soon.
  • /etc/sysconfig/network-scripts/route-eth0/1: set the default route of the internal networ
  • /etc/sysconfig/network-scripts/ifcfg-eth0/1}

Configure the network interfaces according to which bridge it is bound. In the examples below it is assumed that the virtual hosts are running in the admin rack 4

  • ifcfg-eth0:
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.108.45.XYZ
NETMASK=255.255.255.0
GATEWAY=192.108.45.1
TYPE=Ethernet
  • ifcfg-eth1:
DEVICE=eth1
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.97.4.ABC
NETMASK=255.255.255.0
TYPE=Ethernet
Resizing of a image

To increase an Image one can use. 

dd if=/dev/zero bs=512  count=20000000 >> <IMAGE>

This will add 10 GB to the image

Next the filesystem has to be checked: 

e2fsck -f xen-pps-ce-fzk_torque_14G

and resized 

resize2fs xen-pps-ce-fzk_torque_14G
Converting a physical server in a virtual
  • create image file
  • on server: cd /; tar csp <all without home, tmp, nfs-dirs> | ssh virt. server:/virt-server-image-file "cat - tar xsp"
  • set values in the network config files (eth0/1, fstab, ..) in the virt. server accordingly

Starting the Virtual Maschines

For the first time the following steps have to performed manualy, they can be included in the boot procedure.

The following commands have to be executed: 

/etc/init.d/xend start
xm create /etc/xen/auto/<Virt. Maschine Name> to be started
/root/xen-net-init <ID> <IP 1> <IP 2>

The <ID> can be evaluated with xm list. IP 1 and IP 2 have to set according to where eth0 and eth1 are bound to (internal 10.X.X.X, or external 192.X.X.X network)

Now the virtual maschine should be ssh-accessable from your desktop, if not you can login to the dom_0 host and start a virtual-host-console there with: xm create <ID>. Now correct the network configuration of the virtual host.

Configuration of the virtual hosts
Network configuration
Updating the base OS

To update the base OS (Scientific Linux Cern 3.0.X) to the actual version do:

  • check proxy settings (see above)
  • check /etc/hosts for not needed entries
  • check the repository settings in /etc/apt/sources.list
  • apt-get update
  • apt-get upgrade


Problems

On nfs Client:

[root@bdii-fzk root]# mount -a

gives: 

mount: RPC: Program not registered

Is the nfsserver running correctly, is the nfsd module loaded (modprobe nfsd), possible problem: wrong modules, have to be the same as the kernel was compiled for. Can hapen when you just cp the kernels.

Is the nfs mount line in /etc/fstab correct?

xm create doesnt work

check if module-init-tools is installed

network bridges cant be created

Network package iproute not installed, causes that the start of the xend creates a not usable routing table, i.e. route cannot be setup through bridge, just do apt-get install iproute.

Virtual server with lvm in fstab doesnt start

at first boot/mount of the lvm devices a filesystem has to be created on the mounted devices, starting a console gives: 

xm console <ID>

Checking all file systems. [/sbin/fsck.ext3 (1) -- /home] fsck.ext3 -a /dev/hdb1 [/sbin/fsck.ext3 (1) -- /tmp] fsck.ext3 -a /dev/hdb2 fsck.ext3: Bad magic number in super-block while trying to open /dev/hdb2 [FAILED]

      • An error occurred during the file system check.
      • Dropping you to a shell; the system will reboot
      • when you leave the shell.

Give root password for maintenance


This can happen if you have in the xm create start file you have something like: 

'phy:/dev/XenVG/Xen_l_CE_home,hdb1,w',

in the fstab of the virtual maschine you have something like
/dev/hdb1 /home

then give root passwort and create the filesystems with e.g. 

mkfs.ext3 /dev/hdb1
MySQL

Aus meinen Notizen, ohne weitere Formatierung. 

Passwort fuer einen Benutzer einer Datenbank aendern:
1) als root einloggen:
 mysql -h localhost -u root -p

 2) setzen des Passworts fuer den Benutzer ->accounting<- auf das Passwort 
->accountingspw<-:
 
 mysql> set password for accounting@localhost=Password('accountingspw');
 

 USE mysql
GRANT SELECT, INSERT, UPDATE, DELETE ON accounting.* TO 'accounting' 
IDENTIFIED BY 'accounting';
DELETE FROM user WHERE user LIKE 'accounting';
DELETE FROM db WHERE user LIKE 'accounting';
INSERT INTO user (host,user,password,create_priv,drop_priv) VALUES 
('localhost','accounting',password('accounting'),'Y','Y');
INSERT INTO user (host,user,password,create_priv,drop_priv) VALUES 
('127.0.0.1','accounting',password('accounting'),'Y','Y');
INSERT INTO user (host,user,password,create_priv,drop_priv) VALUES 
('localhost.localdomain','accounting',password('accounting'),'Y','Y');
INSERT INTO user (host,user,password,create_priv,drop_priv) VALUES 
('a01-004-123.gridka.de','accounting',password('accounting'),'Y','Y');
INSERT INTO user (host,user,password,create_priv, drop_priv) VALUES 
('lcg-gridka-ce','accounting',password('accounting'),'Y','Y');
INSERT INTO db 
(host,db,user,select_priv,insert_priv,update_priv,delete_priv,create_priv, 
alter_priv) VALUES ('%','accounting','accounting','Y','Y','Y','Y','Y','Y');
FLUSH PRIVILEGES;
DROP DATABASE IF EXISTS accounting;
CREATE DATABASE accounting;
USE accounting;
==== Um von meinem Rechner darauf zu arbeiten =====
[root@lcg-gridka-ce yaim]# mysql -h localhost -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4 to server version: 4.0.25-standard

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> USE mysql Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A

Database changed mysql> INSERT INTO user (host,user,password,create_priv, drop_priv) VALUES 

   -> ('iwrgabriel.ka.fzk.de','accounting',password('accounting'),'Y','Y');

Query OK, 1 row affected (0.10 sec)

mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.02 sec)

mysql> quit Bye

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Xen_on_CentOS_5_-_Notes&oldid=1214"