HooiMaanden

How to manage and configure the Hooi Maanden

First some specs about the cluster:

• 12 x Dell R710
  • 2 x Intel L5520 quadcore
  • 72GB DDR3 ram
  • 2 x 154GB 10K RPM SAS disks in hardware raid 1
  • Intel 10Ge 82598EB network card
  • Mellanox Infiniband MT25208 card
  • iDRAC6 enterprice management card
  • Onboard 1Ge network cards are disabled in the BIOS
• 6 x DDN SA9900
  • 2 Controllers per cabinet
  • Each controller has 20Gbit/s Infiniband connection to one of the Dell machines
  • 5 disk drawers per cabinet, each drawer can have 60 disks
  • Western Digital 5400RPM 2TB disks

The hostnames of the disk controllers:

• hooimaand-mngt-C37-1.ipmi.nikhef.nl
• hooimaand-mngt-C37-2.ipmi.nikhef.nl
• hooimaand-mngt-C38-1.ipmi.nikhef.nl
• hooimaand-mngt-C38-2.ipmi.nikhef.nl
• hooimaand-mngt-C39-1.ipmi.nikhef.nl
• hooimaand-mngt-C39-2.ipmi.nikhef.nl

Basic commands

When you're logged into a disk controller you'll find the following commands usefull. host, lun, disk, faults,

How to setup the Hooi Maanden

First off install the machines (for example with a base EL5 Quattor install) When you have a working system, install these packages:

yum install srptools opensm xfsprogs

This will install the packages you'll need for the Infiniband stack. After that were going to activate the Infiniband stack, so that the DDN Controller will see the connection and to hook them up to the disks.

/etc/init.d/openibd start
sleep 3
/etc/init.d/opensmd start
sleep 20
/usr/sbin/ibsrpdm -d /dev/infiniband/umad1 -c | sed -e "s/\(service_id\)/max_sect=4096,max_cmd_per_lun=4,\1/" > /sys/class/infiniband_srp/srp-mthca0-2/add_target

Now were going to login into the DDN disk controllers. There two options to use: serial cable or a telnet session. Our disk controllers have all ready a network connection, but where going to use the serial connection.

Configuring the DDN setup

Log into one of the DDN controllers, (the one where the infiniband connection to the machine is connected to). I expect that you allready did the commands of HooiMaanden#How_to_setup_the_Hooi_Maanden