NDPF GS environment

From PDP/Grid Wiki
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

The Grid Services environment contains nodes and virtual machines that run special or dedicated services for grid and grid-related work: web servers, the EUGridPMA Repository, the CA and RA systems, et cetera. These service nodes are ‘one-off’ systems, not under quattor control, installed separately, and updating themselves using yum or apt. They do not even all run the same OS version or flavour.

They mostly live on a separate network (194.171.96.64/28), and at the Remote Housing Location.

Machine overview

Machine (real or virtual) overview
machine responsible Level Tasks Comments
rooier sveng low web server for EGEE Security SSCs
beerput davidg medium rsync backup service with ADSM client and backup
gierput davidg low no useful purpose left spare for beerput
sikkel davidg high NDPF subversion service
zeis davidg critical www.eugridpma.org web site (with dynamic content) a hot spare is available on dodo, re-point the DNS (hosted at https://access.enom.com/) in case it really does not come back
weikuip davidg critical dist.eugridpma.info web (IGTF CA distribution) a hot spare is available on lama, re-point the DNS (hosted at https://access.enom.com/) in case it really does not come back
keerder davidg critical physical host system serves: zeis, weikuip, rooier, sikkel
hek davidg high DutchGrid CA 'internal' system ra.dutchgrid.nl, used by the CA admins
kaasvat davidg critical ca.dutchgrid.nl (DutchGrid CRL distribution) a hot spare is available on vink, re-point the DNS for ca.dutchgrid.nl, ask PaulKS
rakel davidg high physical host system Blade #1 (top left, in c15). Hosts: mestkar
mestkar davidg high web server for dutchgrid (and some NDPF stats)
rijf davidg medium NDPF mirror service stalkaars-02, in 2nd valentine rack
salado davidg high network management host in cabinet of deel. Makes the cricket graphs. Warning: disk is NOT raided!

Web sites

EUGridPMA and IGTF

For the EUGridPMA and IGTF web sites, also Anders Waananen (NBI, DK) has the access rights and methods to get into it. He could potentially also do the system swap in DNS with ENOM, but had never tried that one yet.

These web sites *really* have a high profile, so please take care of them for me. Mails sent to the EUGridPMA Operations email address get forwarded to the grid sysadmin list as well.

DutchGrid CA

The Dutchgrid CA has, besides its off-line signing system, 2 (two) on-line systems: the 'RA' box that serves the internal web management console that Djuhaeri, Andre and Dennis can use; and the 'public' box that serves the web site for user requests, as well as the CRL download location. This latter function (CRL downloads) is *really* critical and gets noticed by each and every site in the grid. Please keep it running, and look for complaints sent to ca@dutchgrid.nl. Dennis, Djuhaeri and Andre get these mails.

Neither of the two boxes has a redundant power supply, but they do have redundant RAID-1 disks (on a 3ware controller)

DutchGrid web site, BiG Grid and the VL-e PoC

These web sites have (just!) been migrated to mestkar.nikhef.nl, a VM(Xen,PV) hosted on the first blade top-left in the new chassis, on a host called rakel. This machine also does the CVS service for now. New here: the uids are taken from the NDPF LDAP, and no longer follow the ikonet assignments.

The only service that was NOT yet migrated away from beerput is the ADSM backup. Even more: mestkar is now backed-up TO beerput on a daily basis.

CVS

The CVS service, using ssh access only, is now provided from mestkar (was: beerput)

SVN

The SVN service runs on sikkel, a VM(Xen,PV) on keerder.

ADSM and backup

The rsync backup service runs on beerput. In /export/data/backups/FQDN/ you find the mos recent backup. The time stamp of the top-level directory is the time the backup last ran.

This area is again backed-up through ADSM to SARA on a daily basis, with 100 days history. In case of trouble with ADSM, contact Ton.

Cricket and network monitoring and control

The 'salado' host (a.k.a. schoffel) is directly connected over a private 192.168.254.0/24 lan to the management blades in deel and nikopn, and uses several of its other interfacs to connect to guestnet, the public-farmnet(sec) and to theipmi network.

This hosts also runs the cricket grapher (the site http://www.dutchgrid.nl/ndpf/cricket/ is just a proxy forward), and runs it from there. If the machine is completely hosed, the cricket config (and graphs up to and including June 5th, and INCLUDING the new hef-router collectors) is copied in a tar-ball to /global/ices/grid/nikhef/network/. Unpack in /project/cricket and restart the cron job (from a host with the same network addresses).

The cron job is

*/5 * * * *     /project/cricket/deploy/cricket/collect.sh > /dev/null 2>&1

Also add to /etc/hosts the correct guestnet-side address of salado:

192.16.192.80   hef-router.nikhef.nl hef-router

and of course, enable cgi and the web server on the new salado

lrwxrwxrwx  1 cricket cricket 43 Aug 18  2008 grapher.cgi -> /project/cricket/deploy/cricket/grapher.cgi
lrwxrwxrwx  1 cricket cricket 38 Aug 18  2008 images -> /project/cricket/deploy/cricket/images
lrwxrwxrwx  1 cricket cricket 35 Aug 18  2008 lib -> /project/cricket/deploy/cricket/lib
lrwxrwxrwx  1 cricket cricket 46 Aug 18  2008 mini-graph.cgi -> /project/cricket/deploy/cricket/mini-graph.cgi

The firewall of this box is really strict, make sure to make any new box as paranoid as this one.

The Real Hosts

Most of the grid services run off 2 (two) physical hosts: keerder, a PE1950-III with a software-raid-1 serup from the HA-GRID series systems; the other is rakel, a M600e blade with hardware raid-1 over SATA in position 1 of the enclosure. Physical hosts left are: beerput, gierput, hek, kaasvat, rijf/stalkaars-02.

Decommissioned services

The following services have been decommissioned:

  • VO LDAP services at grid-vo.nikhef.nl
  • SecureGrid.org web site

Also, all running services that used to run on beerput.nikhef.nl, except for the ADSM backup have been migrated to "mestkar.nikhef.nl".

Older documentation that still has validity

For the non-migrated services (mainly the DutchGrid CA and the rsync-based backup service, the attached document (PDF) is still valid! File:Grid-Service-Systems-Guide-20070518.pdf