Nagios Monitoring Setup

This article describes the current setup of the Nagios monitoring setup. Note that this setup is being modified, so the information described here may soon be outdated.

Hosts:

• spade: Nagios master server, collects information from active Nagios servers via NSCA and presents it via the web interface and actively schedules checks for grid monitoring (either via UI horige, or fetching results from SAM, GOC, GGUS, gStat).
• riek: Nagios server for production grid servers (including storage servers) and generic servers.
• eg: Nagios server for the worker nodes. The check results are published via NSCA to spade.
• tbn06: Nagios server for the ITB. The check results are published via NSCA to spade.
• horige: gLite 3.2 UI dedicated to grid monitoring. Nagios server schedules the grid checks via NRPE. To run the grid checks, a valid grid proxy must exist on host horige. This UI does not run a Nagios server.

Nagios configuration

At present, all of the Nagios configuration is handled by the quattor setup. The grid-specific configuration is also defined in the quattor setup, but handled on the Nagios master server spade and the NRPE UI horige via Yaim.

At Nagios master spade, Yaim runs the script ncg.pl to regenerate the hosts, services and commands for grid checks. When new (SAM) tests appear, it may be needed to regenerate by hand:

/usr/sbin/ncg.pl

The configuration files for ncg.pl are present in /etc/ncg and they are generated by Yaim. The output is now written to /etc/nagios/wlcg.d (Nagios server configuration) and /etc/nagios/nrpe (NRPE configuration for the UI). The generated configuration files require some editing before they can be used:

• wlcg.d/commands-edited.cfg: this is a copy of the generated file wlcg.d/commands.cfg; the 2 commands for pnp4nagios have to be commented out since they are already defined by the Quattor setup (and cannot be removed there).

After making the changes, run

nagios -v /etc/nagios/nagios.cfg

to verify that there are no errors. Then, the Nagios service can be restarted.

It is not needed to make any changes on the UI horige. Changes on spade are automatically propagated to horige.

Note: the quattor setup defines checks for the hosts spade.nikhef.nl and horige.nikhef.nl (with domain name) for load, disk usage etc. The Yaim-generated setup uses the short host names (spade and horige) to prevent doubly defined host names. These names should not be mixed to prevent Nagios that will refuse to start!

Nagios grid proxy

A valid dteam proxy has to exist at the UI horige in the location:

/etc/nagios/globus/userproxy-dteam.pem

If no valid proxy exists (anymore), create a regular dteam proxy on any UI:

voms-proxy-init -voms dteam

and copy it to the above location:

scp /tmp/x509up_u500 root@horige:/etc/nagios/globus/userproxy-dteam.pem

Login as root@horige and change the ownership and group to nagios:nagios:

chown nagios:nagios /etc/nagios/globus/userproxy.pem-dteam

Then register a dteam proxy in the MyProxy server wierde. Easiest is to execute the wrapper script

~ronalds/bin/nagios-refresh 

This script can also be used to renew an existing valid proxy.

Note that the if a proxy was registered before with the MyProxy server by another user, that user will have to delete his/her proxy before it can be registered with different credentials!

Verification of the validity of the currently registered proxy:

• Point your browser to the location of the Nagios server at spade
• Show host wierde
• Check service hr.srce.MyProxy-ProxyLifetime-dteam
• The check result will show the date and time of expiration and an estimate of the time left as measured on the moment of execution the check.
• It is possible to force the check of service hr.srce.MyProxy-ProxyLifetime-dteam to get an update (e.g. after renewing the registration of the proxy) by selecting "Re-schedule the next check of this service" in the panel on the right.

When the proxy will expire in less than 24 hours, the check result for hr.srce.MyProxy-ProxyLifetime-dteam will be a warning.

Note : to get your proxy OUT of the server:

GT_PROXY_MODE=old myproxy-destroy -s wierde -l nagios -k NagiosRetrieve-horige-dteam

[ with the proxy of the person whose proxy is in wierde ... ]

Emergency Procedure

Only apply the emergeny procedure if the proxy registered at the MyProxy server has expired and the owner of that proxy is not on-line!

Login as root at MyProxy server wierde and move the existing credentials out of the way:

cd /var/myproxy/
ls -l nagios-NagiosRetrieve-horige-dteam.*
-rw-------  1 root root 4256 Apr 11 10:02 nagios-NagiosRetrieve-horige-dteam.creds
-rw-------  1 root root  203 Apr 11 10:02 nagios-NagiosRetrieve-horige-dteam.data
mkdir OLD
mv nagios-NagiosRetrieve-horige-dteam.* OLD/

It is then possible to register a proxy owned by a different user at the MyProxy server wierde.

External documentation:

Yaim Based Installation of Nagios & NCG