Hooikar/hooiwagen

This article describes the setup of the DPM disk servers hooikar and hooiwagen. These disk servers export file systems which are mounted via iSCSI from the Sun X4500 hosts (hooi-ei-*).

Motivation

The Sun X4500 hosts were installed with operating system CentOS-4 x86-64 and the gLite 3.1 version of DPM. gLite decided to drop support for the gLite 3.1 version of DPM in the course of 2011. Existing installations had to be upgraded to the gLite 3.2 version of DPM under EL5.

Unfortunately, the Sun X4500 series are only certified for use with RHEL4. The combination with RHEL5 is explicitly not supported by the vendor.

Although aware of this support issue, we installed CentOS-5 x86-64 on a test machine and started testing. The results were disappointing. When used under modest load (using rsync to copy from another host, a single-threaded operation), the test machine would halt with a kernel panic in ~6 hours. Leaving the test machine without any load would lead to a kernel panic in a week or so.

These findings and the lack of support by the vendor made us search for another way to use the ~280 TB of disk space using the gLite 3.2 version of DPM.

iSCSI-based setup

The new setup uses a CentOS-4 x86-64 installation of the Sun X4500 hosts as iscsi-target hosts. Each of these hosts exports 20 TB disk space as a single block device via iSCSI.

Two new hosts come into play: hooikar and hooiwagen. They are Dell R610 hosts with dual Myricom 10 Gbps network cards. These hosts are installed with CentOS-5 x86-64 and the gLite 3.2 version of the DPM disk server middleware. They are configured as iSCSI clients and connected via a private network to the the Sun X4500 hosts.

Each of the 2 disk servers mounts block devices from 7 X4500 hosts and exports them to DPM. (That brings an interesting issue for DPM's file selection mechanism; see below).

Configuration of the iscsi-target

Download the rpm source package from http://www.cryptoforge.net/iscsi/RPMS, maintained by Bastiaan Bakker. As of this writing use version 0.4.12 as 0.4.13 will fail with the updated kernel

Use the version "-6" and rebuild on an RHEL4 system with the kernel module build source RPM installed.

The result for kernel 2.6.9-89.0.16.EL is at:

 http://www.nikhef.nl/grid/ndpf/files/iSCSI/

alongside the source.

Install the kernel module RPMs, run "depmod -a" and "modprobe iscsi_trgt"

On the Sun X4500 hosts, create 4 physical volumes and combine them in a single volume group "data" (the script mkdataordered handles that). Then, create a logical volume in the volume group. This volume is the block device that is exported.

Set that block device in the configure file /etc/ietd.conf file, containing for example:

 Target iqn.2010-11.nl.nikhef:opnstorage.easteregg.host-hooi-ei-13.lun1
       IncomingUser username twelvecharss
       OutgoingUser
       Lun 0 Path=/dev/data/ei13,Type=fileio
       Alias iDISK0

In this case the password is sent in the clear ovr the network -- so must be on a private SAN network!

Finally, (re)start the service:

  /etc/init.d/iscsi-target restart