Nagios Monitoring Setup

This article describes the current setup of the Nagios monitoring setup. Note that this setup is being modified, so the information described here may soon be outdated.

Hosts:

• spade: Nagios master server, collects information from active Nagios servers via NSCA and presents it via the web interface and actively schedules checks for grid monitoring (either via UI horige, or fetching results from SAM, GOC, GGUS, gStat).
• riek: Nagios server for production grid servers (including storage servers) and generic servers.
• eg: Nagios server for the worker nodes. The check results are published via NSCA to spade.
• tbn06: Nagios server for the ITB. The check results are published via NSCA to spade.
• horige: gLite 3.2 UI dedicated to grid monitoring. Nagios server schedules the grid checks via NRPE. To run the grid checks, a valid grid proxy must exist on host horige. This UI does not run a Nagios server.

Nagios configuration

At present, all of the Nagios configuration is handled by the quattor setup, except for the grid-specific checks. These checks are generated by the script ncg.pl at host spade via the following command:

ncg.pl

The configuration files for ncg.pl are present in /etc/ncg and they are generated by Yaim. The output is now written to /etc/nagios/wlcg.d (Nagios server configuration) and /etc/nagios/nrpe (NRPE configuration for the UI). The generated configuration files require some editing before they can be used:

• wlcg.d/commands-edited.cfg: this is a copy of the generated file wlcg.d/commands.cfg; the 2 commands for pnp4nagios have to be commented out since they are already defined by the Quattor setup (and cannot be removed there).

After making the changes, run

nagios -v /etc/nagios/nagios.cfg

to verify that there are no errors. Then, the Nagios service can be restarted.

Nagios grid proxy

~ronalds/bin/nagios-refresh

horige inloggen als wortel

service httpd start

dan kijken op https://tbn12.nikhef.nl/nagios don't forget to have an openVPN tunnel first

show host wierde

myproxylifetime check

reschedule the next check of this service

should then show a valid proxy ... then you can stop the httpd daemon on tbn12.

make a dteam voms proxy

scp this proxy to

scp /tmp/x509up_u500 root@tbn12:/etc/nagios/globus/userproxy.pem

warning comes 24 hours beforehand : proxy will expire

run this:

~ronalds/bin/nagios-refresh

if the proxy completely expires then this does not work ... need to redo the scp.

Note : to get your proxy OUT of the server:

GT_PROXY_MODE=old myproxy-destroy -s wierde -l nagios -k NagiosRefresh

[ with the proxy of the person whose proxy is in wierde ... ]