DTConsole

Background

This page documents how to do management tasks -- like looking at the console, powering machines on and off, etc. -- from your desktop, instead of going into the machine room.

IPMI

IPMI is a "standard" interface that can be used to do remote management tasks. Most of the physical machines in our farm segment our connected to a special IPMI network (which is different from the standard "internet" network). Via this special management network, you can do things like turn the machine off and on, or look at its console in some cases, or read the temperature of the box. If you know the physical machine you want to access, you can find the corresponding IPMI address on this page: NDPF_Node_Functions, in the table called "IPMI dedicated management network". Here is an example: you want to access wn-val-003.farm.nikhef.nl. You can see from that table the following line:

0.20-0.121	 wn-val-(001-102)		 valentine	 LCG2ELPROD

read this as follows: the first field gives the last two parts of the IP address (the first part is 172.20), so wn-val-003 will have IP address 172.20.0.22. The second field gives the IPMI hostname, so this node has hostname "wn-val-003.ipmi.nikhef.nl".

bosui:~> host wn-val-003.ipmi.nikhef.nl
wn-val-003.ipmi.nikhef.nl has address 172.20.0.22

This information can be used as hostname/ip input for the various IPMI client tools.

Important Note: the IPMI network is not accessible from everywhere. From your desktop, you should have the OpenVPN tunnel running, otherwise you won't be able to connect. The IPMI network is not accessible from all nodes in the farm; it does work from e.g. the install server (stal).

Remote Consoles and Switches

Unfortunately there are very many ways to remotely connect to a machine console, and as far as I can tell, NONE of them work for all machines. One must be prepared to deal with the soup : several different tools, several different "administrator usernames", in some cases going through a console switch, in some cases connecting directly to the machine, in some cases even both options are possible. Also, there are a lot of "ifs" : some machines would appear from the wiki to be IPMI/console ready but are in reality not.

One way that works on all machines that are a) in the IPMI network and b) have IPMI2.0 or higher support and c) are properly configured (in the kernel and the BIOS), is to use something called Serial-over-LAN. The easiest way to connect to a console using this is via ipmitool:

 ipmitool -I lanplus  -U root -H hilde.ipmi.nikhef.nl sol activate

will connect you to the console interface on hilde.

Another way is to use something called KVM. This is a bit nicer to look at, but is vendor specific. You can't in general use the same "KVM viewer" to look at a Dell as a Supermicro, the protocol used to send the KVM information across the IPMI network is not the same in the two cases.

IMPIView (mentioned on the Serial Consoles page) will let you look at the KVM console of Supermicro machines. To look at the KVM console of Dell machines, you need to use the Dell Remote Console Switch software. This software only works under Linux and Windows. Note if you do it from your desktop, you need the OpenVPN tunnel open. Mac users can resort to a virtual machine, or they can log in to a machine on the right network (stal works) and launch the software there, the standard X forwarding in ssh will put the window on your desktop.

Command-line operations via ipmitool

There are a lot of commands you can run via ipmitool. One example is to turn the power of a machine off or on. Here is how to turn one on.

 ipmitool -P $P -U root -H wn-lui1-001.ipmi.nikhef.nl chassis power on

will turn on the power for wn-lui1-001. $P is the password and not listed on this page.

Usernames and passwords

There is no standard username for the IPMI interface on our farm.

• for valentine machines, the username is ADMIN
• for luilak machines, the username is root
• for the Dell Remote Console Switches, the username is apparently Admin although as of this writing I have not been able to verify this.

Good luck and happy system management!