Difference between revisions of "Agile testbed"

From PDP/Grid Wiki
Jump to navigationJump to search
(move hardware inventory down, to the hardware index)
Line 11: Line 11:
  
  
−
== Hardware inventory ==
 
−
 
−
Changes here should probably also go to [[NDPF System Functions]].
 
−
 
−
{| class="wikitable" border="0" cellpadding="8"
 
−
|-style="background-color: #ccc;"
 
−
! name
 
−
! ipmi name*
 
−
! type
 
−
! chipset
 
−
! #cores
 
−
! mem
 
−
! OS
 
−
! disk
 
−
! [http://www.dell.com/support/ service tag]
 
−
! location
 
−
! remarks
 
−
|-style="background-color: #cfc;"
 
−
| bleek
 
−
| bleek
 
−
| PE1950
 
−
| Intel 5150  @ 2.66GHz
 
−
| 2×2
 
−
| align="right"|8GB
 
−
| CentOS 5
 
−
| software raid1 2×500GB disks
 
−
| CQ9NK2J
 
−
| C10
 
−
| High Availability, dual power supply; precious data; [[#backup|backed up]].
 
−
|-style="background: #cfc;"
 
−
| storage
 
−
| put
 
−
| PE2950
 
−
| Intel E5150 @ 2.66GHz
 
−
| 2×2
 
−
| align="right"|8GB
 
−
| FreeNAS 8.3
 
−
| 6× 500 GB SATA, raidz (ZFS)
 
−
| HMXP93J
 
−
| C03
 
−
| former garitxako
 
−
|-style="background-color: #ffc;"
 
−
| blade13
 
−
| bl0-13
 
−
| PEM610
 
−
| E5504  @ 2.00GHz
 
−
| 2×4
 
−
| align="right"|16GB
 
−
| Debian 6, KVM
 
−
| 70 GB + 1 TB Fibre Channel (shared)
 
−
| 5NZWF4J
 
−
| C08 blade13
 
−
|
 
−
|-style="background-color: #ffc;"
 
−
| blade14
 
−
| bl0-14
 
−
| PEM610
 
−
| E5504  @ 2.00GHz
 
−
| 2×4
 
−
| align="right"|16GB
 
−
| Debian 6, KVM
 
−
| 70 GB + 1 TB Fibre Channel (shared)
 
−
| 4NZWF4J
 
−
| C08 blade13
 
−
|
 
−
|-style="background-color: #ffc;"
 
−
| arrone
 
−
| arrone
 
−
| PE1950
 
−
| Intel E5320  @ 1.86GHz
 
−
| 2×4
 
−
| align="right"|8GB
 
−
| Debian 6, KVM
 
−
| 70 GB + 400 GB iSCSI (shared)
 
−
| 982MY2J
 
−
| C10
 
−
| storage shared with aulnes
 
−
|-style="background-color: #ffc;"
 
−
| aulnes
 
−
| aulnes
 
−
| PE1950
 
−
| Intel E5320  @ 1.86GHz
 
−
| 2×4
 
−
| align="right"|8GB
 
−
| Debian 6, KVM
 
−
| 70 GB + 400 GB iSCSI (shared)
 
−
| B82MY2J
 
−
| C10
 
−
| storage shared with arrone
 
−
|-style="background-color: #ffc;"
 
−
| toom
 
−
| toom
 
−
| PE1950
 
−
| Intel E5440  @ 2.83GHz
 
−
| 2×4
 
−
| align="right"|16GB
 
−
| Debian 6, KVM
 
−
| Hardware raid1 2×715GB disks
 
−
| DC8QG3J
 
−
| C10
 
−
| current Xen 3 hypervisor with mktestbed scripts
 
−
|-style="background-color: #ffc;"
 
−
| span
 
−
| span
 
−
| PE2950
 
−
| Intel  E5440  @ 2.83GHz
 
−
| 2×4
 
−
| align="right"|24GB
 
−
| Debian 6, KVM
 
−
| Hardware raid10 on 4×470GB disks (950GB net)
 
−
| FP1BL3J
 
−
| C10
 
−
| plus [[#Squid|squid proxy]]
 
−
|-style="background-color: #fcc;"
 
−
| melkbus
 
−
| bl0-02
 
−
| PEM600
 
−
| Intel E5450 @3.00GHz
 
−
| 2×4
 
−
| align="right"|32GB
 
−
| VMWare ESXi
 
−
| 2× 320GB SAS disks
 
−
| 76T974J
 
−
| C08, blade 1, slot 2
 
−
| Managed by Oscar
 
−
|-style="color: #444;"
 
−
| kudde
 
−
| kudde
 
−
| PE1950
 
−
| Intel E5440  @ 2.83GHz
 
−
| 2×4
 
−
| align="right"|16GB
 
−
| CentOS 5, Xen
 
−
| Hardware raid1 2×715GB disks
 
−
| CC8QG3J
 
−
| C10
 
−
| Contains hardware encryption tokens for robot certificates; managed by Jan Just
 
−
|-
 
−
| ent
 
−
| —
 
−
| Mac Mini
 
−
| Intel Core Duo  @1.66GHz
 
−
| 2
 
−
| align="right"|2GB
 
−
| OS X 10.6
 
−
| SATA 80GB
 
−
| —
 
−
| C24
 
−
| OS X box (no virtualisation)
 
−
|}
 
−
 
−
 
−
 
−
* *ipmi name is used for IPMI access; use <code><name>.ipmi.nikhef.nl</code>.
 
−
* System details such as serial numbers can be retrieved from the command line with <code>dmidecode -t 1</code>.
 
−
* The service-tags can be retrieved through IPMI, but unless you want to send raw commands with ipmitool first you need freeipmi-tools. This contains ipmi-oem that can be called thus:
 
−
ipmi-oem -h host.ipmi.nikhef.nl -u username -p password dell get-system-info service-tag
 
−
 
−
Most machines all run [http://www.debian.org/releases/stable/ Debian squeeze] with [http://www.linux-kvm.org/page/Main_Page KVM] for virtualization, managed by [http://libvirt.org/ libvirt].
 
−
 
−
See [[NDPF_Node_Functions#P4CTB|the official list]] of machines for the most current view.
 
−
 
−
=== Installing Debian and libvirt on new hardware ===
 
−
 
−
Just a few notes. After setting up the basics by adding the hardware address to /etc/dnsmasq.d/pxeboot (see the examples there) and setting up links in /srv/tftpboot/pxelinux.cfg/ to debian6-autoinstall, pxeboot the machine and wait for the installation to complete. Then, set up bridge configurations like so in /etc/network/interfaces:
 
−
auto br0
 
−
iface br0 inet dhcp
 
−
        bridge_ports eth0
 
−
 
−
auto br2
 
−
iface br2 inet manual
 
−
        bridge_ports eth0.2
 
−
 
−
auto br8
 
−
iface br8 inet manual
 
−
        bridge_ports eth0.8
 
−
 
−
Install the vlan package:
 
−
apt-get install vlan
 
−
 
−
In /etc/sysctl.conf:
 
−
net.ipv6.conf.all.forwarding=1
 
−
net.ipv6.conf.all.autoconf = 0
 
−
net.ipv6.conf.all.accept_ra = 0
 
−
 
−
=== Migrating Xen VMs to KVM ===
 
−
 
−
These are a few temporary notes made during the conversion of VMs on toom to blade14.
 
−
 
−
==== preparation ====
 
−
 
−
# Log in to the VM;
 
−
# if this is a paravirtualized  machine, install the 'kernel' package in addition to the 'kernel-xen' package or the machine won't run at all.
 
−
## Edit /etc/grub to remove the xvc0 console setting.
 
−
## Edit /etc/inittab to remove the lone serial console and re-enable the normal ttys.
 
−
## Restore /etc/sysconfig/network-scripts/ifcfg-eth0.bkp to start the network ''(actually this must be done '''after''' the migration, strangely)''.
 
−
# Then shut down the machine.
 
−
# After shutdown, copy the VM disk image to put:/mnt/put/stampede or (if LVM was used) dd the volume to put.
 
−
 
−
 
−
Non-Xen-VMs are easier to transfer ;-)
 
−
 
−
==== create a new machine ====
 
−
 
−
Connect to blade14.testbed with virt-manager and start the new machine wizard.
 
−
* Use an existing image (the one you just made)
 
−
* set a fixed MAC address (found in /root/xen/''machine-definition.vm''
 
−
* set the disk emulation to IDE
 
−
* On first boot, intercept the grub menu to choose the non-Xen kernel
 
−
* After booting up:
 
−
** fix the network (/etc/sysconfig/network-scripts/ifcfg-eth0)
 
−
** remove the xen kernels
 
−
rpm -q kernel-xen | xargs rpm -e
 
−
 
−
Command-line example:
 
−
virt-install --import --disk vol=put.testbed/cert-debian6-64.testbed.img \
 
−
--name cert-debian6-64.testbed --network bridge=br0,mac=00:16:3E:C6:00:16 \
 
−
--os-type linux --os-variant debiansqueeze --ram 1024
 
−
 
−
Or a former RHEL machine:
 
−
virt-install --import --disk vol=put.testbed/genome3.testbed.img,bus=ide \
 
−
--name genome3.testbed --network bridge=br0,mac=00:16:3E:C6:02:0E \
 
−
--os-type linux --os-variant rhel5.4 --ram 800
 
−
 
−
Mind that if the host goes out-of-memory you get a weird error message like:
 
−
libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/12
 
−
 
−
This just means qemu could not create the domain!
 
  
 
== Accessing the machines ==
 
== Accessing the machines ==
Line 469: Line 241:
  
 
At least this seems to work.
 
At least this seems to work.
 +
 +
== Hardware index ==
 +
 +
Changes here should probably also go to [[NDPF System Functions]].
 +
 +
{| class="wikitable" border="0" cellpadding="8"
 +
|-style="background-color: #ccc;"
 +
! name
 +
! ipmi name*
 +
! type
 +
! chipset
 +
! #cores
 +
! mem
 +
! OS
 +
! disk
 +
! [http://www.dell.com/support/ service tag]
 +
! location
 +
! remarks
 +
|-style="background-color: #cfc;"
 +
| bleek
 +
| bleek
 +
| PE1950
 +
| Intel 5150  @ 2.66GHz
 +
| 2&times;2
 +
| align="right"|8GB
 +
| CentOS 5
 +
| software raid1 2&times;500GB disks
 +
| CQ9NK2J
 +
| C10
 +
| High Availability, dual power supply; precious data; [[#backup|backed up]].
 +
|-style="background: #cfc;"
 +
| storage
 +
| put
 +
| PE2950
 +
| Intel E5150 @ 2.66GHz
 +
| 2&times;2
 +
| align="right"|8GB
 +
| FreeNAS 8.3
 +
| 6&times; 500 GB SATA, raidz (ZFS)
 +
| HMXP93J
 +
| C03
 +
| former garitxako
 +
|-style="background-color: #ffc;"
 +
| blade13
 +
| bl0-13
 +
| PEM610
 +
| E5504  @ 2.00GHz
 +
| 2&times;4
 +
| align="right"|16GB
 +
| Debian 6, KVM
 +
| 70 GB + 1 TB Fibre Channel (shared)
 +
| 5NZWF4J
 +
| C08 blade13
 +
|
 +
|-style="background-color: #ffc;"
 +
| blade14
 +
| bl0-14
 +
| PEM610
 +
| E5504  @ 2.00GHz
 +
| 2&times;4
 +
| align="right"|16GB
 +
| Debian 6, KVM
 +
| 70 GB + 1 TB Fibre Channel (shared)
 +
| 4NZWF4J
 +
| C08 blade13
 +
|
 +
|-style="background-color: #ffc;"
 +
| arrone
 +
| arrone
 +
| PE1950
 +
| Intel E5320  @ 1.86GHz
 +
| 2&times;4
 +
| align="right"|8GB
 +
| Debian 6, KVM
 +
| 70 GB + 400 GB iSCSI (shared)
 +
| 982MY2J
 +
| C10
 +
| storage shared with aulnes
 +
|-style="background-color: #ffc;"
 +
| aulnes
 +
| aulnes
 +
| PE1950
 +
| Intel E5320  @ 1.86GHz
 +
| 2&times;4
 +
| align="right"|8GB
 +
| Debian 6, KVM
 +
| 70 GB + 400 GB iSCSI (shared)
 +
| B82MY2J
 +
| C10
 +
| storage shared with arrone
 +
|-style="background-color: #ffc;"
 +
| toom
 +
| toom
 +
| PE1950
 +
| Intel E5440  @ 2.83GHz
 +
| 2&times;4
 +
| align="right"|16GB
 +
| Debian 6, KVM
 +
| Hardware raid1 2&times;715GB disks
 +
| DC8QG3J
 +
| C10
 +
| current Xen 3 hypervisor with mktestbed scripts
 +
|-style="background-color: #ffc;"
 +
| span
 +
| span
 +
| PE2950
 +
| Intel  E5440  @ 2.83GHz
 +
| 2&times;4
 +
| align="right"|24GB
 +
| Debian 6, KVM
 +
| Hardware raid10 on 4&times;470GB disks (950GB net)
 +
| FP1BL3J
 +
| C10
 +
| plus [[#Squid|squid proxy]]
 +
|-style="background-color: #fcc;"
 +
| melkbus
 +
| bl0-02
 +
| PEM600
 +
| Intel E5450 @3.00GHz
 +
| 2&times;4
 +
| align="right"|32GB
 +
| VMWare ESXi
 +
| 2&times; 320GB SAS disks
 +
| 76T974J
 +
| C08, blade 1, slot 2
 +
| Managed by Oscar
 +
|-style="color: #444;"
 +
| kudde
 +
| kudde
 +
| PE1950
 +
| Intel E5440  @ 2.83GHz
 +
| 2&times;4
 +
| align="right"|16GB
 +
| CentOS 5, Xen
 +
| Hardware raid1 2&times;715GB disks
 +
| CC8QG3J
 +
| C10
 +
| Contains hardware encryption tokens for robot certificates; managed by Jan Just
 +
|-
 +
| ent
 +
| &mdash;
 +
| Mac Mini
 +
| Intel Core Duo  @1.66GHz
 +
| 2
 +
| align="right"|2GB
 +
| OS X 10.6
 +
| SATA 80GB
 +
| &mdash;
 +
| C24
 +
| OS X box (no virtualisation)
 +
|}
 +
 +
 +
 +
* *ipmi name is used for IPMI access; use <code><name>.ipmi.nikhef.nl</code>.
 +
* System details such as serial numbers can be retrieved from the command line with <code>dmidecode -t 1</code>.
 +
* The service-tags can be retrieved through IPMI, but unless you want to send raw commands with ipmitool first you need freeipmi-tools. This contains ipmi-oem that can be called thus:
 +
ipmi-oem -h host.ipmi.nikhef.nl -u username -p password dell get-system-info service-tag
 +
 +
Most machines all run [http://www.debian.org/releases/stable/ Debian squeeze] with [http://www.linux-kvm.org/page/Main_Page KVM] for virtualization, managed by [http://libvirt.org/ libvirt].
 +
 +
See [[NDPF_Node_Functions#P4CTB|the official list]] of machines for the most current view.
 +
 +
=== Installing Debian and libvirt on new hardware ===
 +
 +
Just a few notes. After setting up the basics by adding the hardware address to /etc/dnsmasq.d/pxeboot (see the examples there) and setting up links in /srv/tftpboot/pxelinux.cfg/ to debian6-autoinstall, pxeboot the machine and wait for the installation to complete. Then, set up bridge configurations like so in /etc/network/interfaces:
 +
auto br0
 +
iface br0 inet dhcp
 +
        bridge_ports eth0
 +
 +
auto br2
 +
iface br2 inet manual
 +
        bridge_ports eth0.2
 +
 +
auto br8
 +
iface br8 inet manual
 +
        bridge_ports eth0.8
 +
 +
Install the vlan package:
 +
apt-get install vlan
 +
 +
In /etc/sysctl.conf:
 +
net.ipv6.conf.all.forwarding=1
 +
net.ipv6.conf.all.autoconf = 0
 +
net.ipv6.conf.all.accept_ra = 0
 +
 +
=== Migrating Xen VMs to KVM ===
 +
 +
These are a few temporary notes made during the conversion of VMs on toom to blade14.
 +
 +
==== preparation ====
 +
 +
# Log in to the VM;
 +
# if this is a paravirtualized  machine, install the 'kernel' package in addition to the 'kernel-xen' package or the machine won't run at all.
 +
## Edit /etc/grub to remove the xvc0 console setting.
 +
## Edit /etc/inittab to remove the lone serial console and re-enable the normal ttys.
 +
## Restore /etc/sysconfig/network-scripts/ifcfg-eth0.bkp to start the network ''(actually this must be done '''after''' the migration, strangely)''.
 +
# Then shut down the machine.
 +
# After shutdown, copy the VM disk image to put:/mnt/put/stampede or (if LVM was used) dd the volume to put.
 +
 +
 +
Non-Xen-VMs are easier to transfer ;-)
 +
 +
==== create a new machine ====
 +
 +
Connect to blade14.testbed with virt-manager and start the new machine wizard.
 +
* Use an existing image (the one you just made)
 +
* set a fixed MAC address (found in /root/xen/''machine-definition.vm''
 +
* set the disk emulation to IDE
 +
* On first boot, intercept the grub menu to choose the non-Xen kernel
 +
* After booting up:
 +
** fix the network (/etc/sysconfig/network-scripts/ifcfg-eth0)
 +
** remove the xen kernels
 +
rpm -q kernel-xen | xargs rpm -e
 +
 +
Command-line example:
 +
virt-install --import --disk vol=put.testbed/cert-debian6-64.testbed.img \
 +
--name cert-debian6-64.testbed --network bridge=br0,mac=00:16:3E:C6:00:16 \
 +
--os-type linux --os-variant debiansqueeze --ram 1024
 +
 +
Or a former RHEL machine:
 +
virt-install --import --disk vol=put.testbed/genome3.testbed.img,bus=ide \
 +
--name genome3.testbed --network bridge=br0,mac=00:16:3E:C6:02:0E \
 +
--os-type linux --os-variant rhel5.4 --ram 800
 +
 +
Mind that if the host goes out-of-memory you get a weird error message like:
 +
libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/12
 +
 +
This just means qemu could not create the domain!

Revision as of 15:46, 22 February 2013

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Agile_testbed&oldid=8851"