Difference between revisions of "NDPF GS environment"
m |
|||
| (9 intermediate revisions by the same user not shown) | |||
| Line 9: | Line 9: | ||
|- style="background:green; color:white" | |- style="background:green; color:white" | ||
| ''machine''||''responsible''||''Level''||''Tasks''||''Comments'' | | ''machine''||''responsible''||''Level''||''Tasks''||''Comments'' | ||
| − | |- | + | |- style="background: gray" |
| − | | rooier || sveng || low || web server for EGEE Security SSCs | + | | rooier || sveng || low || web server for EGEE Security SSCs || |
| − | |- | + | |- style="background: lightgray" |
| beerput || davidg || medium || rsync backup service || with ADSM client and backup | | beerput || davidg || medium || rsync backup service || with ADSM client and backup | ||
| − | |- | + | |- style="background: gray" |
| gierput || davidg || low || no useful purpose left || spare for beerput | | gierput || davidg || low || no useful purpose left || spare for beerput | ||
| − | |- | + | |- style="background: yellow" |
| − | | sikkel || davidg || high || NDPF subversion service | + | | sikkel || davidg || high || NDPF subversion service || |
| − | |- | + | |- style="background: red" |
| − | | zeis || davidg || critical || www.eugridpma.org web site (with dynamic content) || a hot spare is available on dodo, re-point the DNS (hosted at [ | + | | zeis || davidg || critical || www.eugridpma.org web site (with dynamic content) || a hot spare is available on dodo, re-point the DNS (hosted at [https://access.enom.com/ https://access.enom.com/]) in case it really does not come back |
| − | |- | + | |- style="background: red" |
| − | | weikuip || davidg || critical || dist.eugridpma.info web (IGTF CA distribution) || a hot spare is available on lama, re-point the DNS (hosted at [ | + | | weikuip || davidg || critical || dist.eugridpma.info web (IGTF CA distribution) || a hot spare is available on lama, re-point the DNS (hosted at [https://access.enom.com/ https://access.enom.com/]) in case it really does not come back |
| − | |- | + | |- style="background: red" |
| keerder || davidg || critical || physical host system || serves: zeis, weikuip, rooier, sikkel | | keerder || davidg || critical || physical host system || serves: zeis, weikuip, rooier, sikkel | ||
| − | |- | + | |- style="background: yellow" |
| hek || davidg || high || DutchGrid CA 'internal' system || ra.dutchgrid.nl, used by the CA admins | | hek || davidg || high || DutchGrid CA 'internal' system || ra.dutchgrid.nl, used by the CA admins | ||
| − | |- | + | |- style="background: red" |
| kaasvat || davidg || critical || ca.dutchgrid.nl (DutchGrid CRL distribution) || a hot spare is available on vink, re-point the DNS for ca.dutchgrid.nl, ask PaulKS | | kaasvat || davidg || critical || ca.dutchgrid.nl (DutchGrid CRL distribution) || a hot spare is available on vink, re-point the DNS for ca.dutchgrid.nl, ask PaulKS | ||
| − | |- | + | |- style="background: yellow" |
| rakel || davidg || high || physical host system || Blade #1 (top left, in c15). Hosts: mestkar | | rakel || davidg || high || physical host system || Blade #1 (top left, in c15). Hosts: mestkar | ||
| − | |- | + | |- style="background: yellow" |
| − | | mestkar || davidg || high || web server for dutchgrid (and some NDPF stats) | + | | mestkar || davidg || high || web server for dutchgrid (and some NDPF stats) || |
| − | |- | + | |- style="background: lightgray" |
| rijf || davidg || medium || NDPF mirror service || stalkaars-02, in 2nd valentine rack | | rijf || davidg || medium || NDPF mirror service || stalkaars-02, in 2nd valentine rack | ||
| − | |- | + | |- style="background: yellow" |
| salado || davidg || high || network management host || in cabinet of deel. Makes the cricket graphs. Warning: disk is NOT raided! | | salado || davidg || high || network management host || in cabinet of deel. Makes the cricket graphs. Warning: disk is NOT raided! | ||
|} | |} | ||
| Line 41: | Line 41: | ||
== EUGridPMA and IGTF == | == EUGridPMA and IGTF == | ||
| − | For the EUGridPMA and IGTF | + | For the EUGridPMA and IGTF web sites, also Anders Waananen (NBI, DK) has the access rights and methods to get into it. He could potentially also do the system swap in DNS with ENOM, but had never tried that one yet. |
| + | |||
| + | These web sites *really* have a high profile, so please take care of them for me. Mails sent to the EUGridPMA Operations email address get forwarded to the grid sysadmin list as well. | ||
| + | |||
| + | == DutchGrid CA == | ||
| + | |||
| + | The Dutchgrid CA has, besides its off-line signing system, 2 (two) on-line systems: the 'RA' box that serves the internal web management console that Djuhaeri, Andre and Dennis can use; and the 'public' box that serves the web site for user requests, as well as the CRL download location. This latter function (CRL downloads) is *really* critical and gets noticed by each and every site in the grid. Please keep it running, and look for complaints sent to ca@dutchgrid.nl. Dennis, Djuhaeri and Andre get these mails. | ||
| + | |||
| + | Neither of the two boxes has a redundant power supply, but they do have redundant RAID-1 disks (on a 3ware controller) | ||
| + | |||
| + | == DutchGrid web site, BiG Grid and the VL-e PoC == | ||
| + | |||
| + | These web sites have (just!) been migrated to mestkar.nikhef.nl, a VM(Xen,PV) hosted on the first blade top-left in the new chassis, on a host called rakel. This machine also does the CVS service for now. | ||
| + | New here: the uids are taken from the NDPF LDAP, and no longer follow the ikonet assignments. | ||
| + | |||
| + | The only service that was NOT yet migrated away from beerput is the ADSM backup. Even more: mestkar is now backed-up TO beerput on a daily basis. | ||
| + | |||
| + | = CVS = | ||
| + | |||
| + | The CVS service, using ssh access only, is now provided from mestkar (was: beerput) | ||
| + | |||
| + | = SVN = | ||
| + | |||
| + | The SVN service runs on sikkel, a VM(Xen,PV) on keerder. | ||
| + | |||
| + | = ADSM and backup = | ||
| + | |||
| + | The rsync backup service runs on beerput. In /export/data/backups/''FQDN''/ you find the mos recent backup. The time stamp of the top-level directory is the time the backup last ran. | ||
| + | |||
| + | This area is again backed-up through ADSM to SARA on a daily basis, with 100 days history. In case of trouble with ADSM, contact Ton. | ||
| + | |||
| + | = Cricket and network monitoring and control = | ||
| + | |||
| + | The 'salado' host (a.k.a. schoffel) is directly connected over a private 192.168.254.0/24 lan to the management blades in deel and nikopn, and uses several of its other interfacs to connect to guestnet, the public-farmnet(sec) and to theipmi network. | ||
| + | |||
| + | This hosts also runs the cricket grapher (the site http://www.dutchgrid.nl/ndpf/cricket/ is just a proxy forward), and runs it from there. | ||
| + | If the machine is completely hosed, the cricket config (and graphs up to and including June 5th, and INCLUDING the new hef-router collectors) | ||
| + | is copied in a tar-ball to <tt>/global/ices/grid/nikhef/network/</tt>. Unpack in /project/cricket and restart the cron job (from a host with the same network addresses). | ||
| + | |||
| + | The cron job is | ||
| + | */5 * * * * /project/cricket/deploy/cricket/collect.sh > /dev/null 2>&1 | ||
| + | |||
| + | Also add to /etc/hosts the correct guestnet-side address of salado: | ||
| + | 192.16.192.80 hef-router.nikhef.nl hef-router | ||
| + | |||
| + | and of course, enable cgi and the web server on the new salado | ||
| + | lrwxrwxrwx 1 cricket cricket 43 Aug 18 2008 grapher.cgi -> /project/cricket/deploy/cricket/grapher.cgi | ||
| + | lrwxrwxrwx 1 cricket cricket 38 Aug 18 2008 images -> /project/cricket/deploy/cricket/images | ||
| + | lrwxrwxrwx 1 cricket cricket 35 Aug 18 2008 lib -> /project/cricket/deploy/cricket/lib | ||
| + | lrwxrwxrwx 1 cricket cricket 46 Aug 18 2008 mini-graph.cgi -> /project/cricket/deploy/cricket/mini-graph.cgi | ||
| + | |||
| + | The firewall of this box is really strict, make sure to make any new box as paranoid as this one. | ||
| + | |||
| + | = The Real Hosts = | ||
| + | |||
| + | Most of the grid services run off 2 (two) physical hosts: keerder, a PE1950-III with a software-raid-1 serup from the HA-GRID series systems; the other is rakel, a M600e blade with hardware raid-1 over SATA in position 1 of the enclosure. | ||
| + | Physical hosts left are: beerput, gierput, hek, kaasvat, rijf/stalkaars-02. | ||
= Decommissioned services = | = Decommissioned services = | ||
| Line 52: | Line 108: | ||
Also, all running services that used to run on <tt>beerput.nikhef.nl</tt>, '''except for the ADSM backup''' have been migrated to "mestkar.nikhef.nl". | Also, all running services that used to run on <tt>beerput.nikhef.nl</tt>, '''except for the ADSM backup''' have been migrated to "mestkar.nikhef.nl". | ||
| − | = Older documentation = | + | = Older documentation that still has validity = |
For the non-migrated services (mainly the DutchGrid CA and the rsync-based backup service, the attached document (PDF) is still valid! | For the non-migrated services (mainly the DutchGrid CA and the rsync-based backup service, the attached document (PDF) is still valid! | ||
[[Image:Grid-Service-Systems-Guide-20070518.pdf||Grid Service Guide]] | [[Image:Grid-Service-Systems-Guide-20070518.pdf||Grid Service Guide]] | ||
Latest revision as of 14:31, 5 June 2009
The Grid Services environment contains nodes and virtual machines that run special or dedicated services for grid and grid-related work: web servers, the EUGridPMA Repository, the CA and RA systems, et cetera. These service nodes are ‘one-off’ systems, not under quattor control, installed separately, and updating themselves using yum or apt. They do not even all run the same OS version or flavour.
They mostly live on a separate network (194.171.96.64/28), and at the Remote Housing Location.
Machine overview
| Machine (real or virtual) overview | ||||
|---|---|---|---|---|
| machine | responsible | Level | Tasks | Comments |
| rooier | sveng | low | web server for EGEE Security SSCs | |
| beerput | davidg | medium | rsync backup service | with ADSM client and backup |
| gierput | davidg | low | no useful purpose left | spare for beerput |
| sikkel | davidg | high | NDPF subversion service | |
| zeis | davidg | critical | www.eugridpma.org web site (with dynamic content) | a hot spare is available on dodo, re-point the DNS (hosted at https://access.enom.com/) in case it really does not come back |
| weikuip | davidg | critical | dist.eugridpma.info web (IGTF CA distribution) | a hot spare is available on lama, re-point the DNS (hosted at https://access.enom.com/) in case it really does not come back |
| keerder | davidg | critical | physical host system | serves: zeis, weikuip, rooier, sikkel |
| hek | davidg | high | DutchGrid CA 'internal' system | ra.dutchgrid.nl, used by the CA admins |
| kaasvat | davidg | critical | ca.dutchgrid.nl (DutchGrid CRL distribution) | a hot spare is available on vink, re-point the DNS for ca.dutchgrid.nl, ask PaulKS |
| rakel | davidg | high | physical host system | Blade #1 (top left, in c15). Hosts: mestkar |
| mestkar | davidg | high | web server for dutchgrid (and some NDPF stats) | |
| rijf | davidg | medium | NDPF mirror service | stalkaars-02, in 2nd valentine rack |
| salado | davidg | high | network management host | in cabinet of deel. Makes the cricket graphs. Warning: disk is NOT raided! |
Web sites
EUGridPMA and IGTF
For the EUGridPMA and IGTF web sites, also Anders Waananen (NBI, DK) has the access rights and methods to get into it. He could potentially also do the system swap in DNS with ENOM, but had never tried that one yet.
These web sites *really* have a high profile, so please take care of them for me. Mails sent to the EUGridPMA Operations email address get forwarded to the grid sysadmin list as well.
DutchGrid CA
The Dutchgrid CA has, besides its off-line signing system, 2 (two) on-line systems: the 'RA' box that serves the internal web management console that Djuhaeri, Andre and Dennis can use; and the 'public' box that serves the web site for user requests, as well as the CRL download location. This latter function (CRL downloads) is *really* critical and gets noticed by each and every site in the grid. Please keep it running, and look for complaints sent to ca@dutchgrid.nl. Dennis, Djuhaeri and Andre get these mails.
Neither of the two boxes has a redundant power supply, but they do have redundant RAID-1 disks (on a 3ware controller)
DutchGrid web site, BiG Grid and the VL-e PoC
These web sites have (just!) been migrated to mestkar.nikhef.nl, a VM(Xen,PV) hosted on the first blade top-left in the new chassis, on a host called rakel. This machine also does the CVS service for now. New here: the uids are taken from the NDPF LDAP, and no longer follow the ikonet assignments.
The only service that was NOT yet migrated away from beerput is the ADSM backup. Even more: mestkar is now backed-up TO beerput on a daily basis.
CVS
The CVS service, using ssh access only, is now provided from mestkar (was: beerput)
SVN
The SVN service runs on sikkel, a VM(Xen,PV) on keerder.
ADSM and backup
The rsync backup service runs on beerput. In /export/data/backups/FQDN/ you find the mos recent backup. The time stamp of the top-level directory is the time the backup last ran.
This area is again backed-up through ADSM to SARA on a daily basis, with 100 days history. In case of trouble with ADSM, contact Ton.
Cricket and network monitoring and control
The 'salado' host (a.k.a. schoffel) is directly connected over a private 192.168.254.0/24 lan to the management blades in deel and nikopn, and uses several of its other interfacs to connect to guestnet, the public-farmnet(sec) and to theipmi network.
This hosts also runs the cricket grapher (the site http://www.dutchgrid.nl/ndpf/cricket/ is just a proxy forward), and runs it from there. If the machine is completely hosed, the cricket config (and graphs up to and including June 5th, and INCLUDING the new hef-router collectors) is copied in a tar-ball to /global/ices/grid/nikhef/network/. Unpack in /project/cricket and restart the cron job (from a host with the same network addresses).
The cron job is
*/5 * * * * /project/cricket/deploy/cricket/collect.sh > /dev/null 2>&1
Also add to /etc/hosts the correct guestnet-side address of salado:
192.16.192.80 hef-router.nikhef.nl hef-router
and of course, enable cgi and the web server on the new salado
lrwxrwxrwx 1 cricket cricket 43 Aug 18 2008 grapher.cgi -> /project/cricket/deploy/cricket/grapher.cgi lrwxrwxrwx 1 cricket cricket 38 Aug 18 2008 images -> /project/cricket/deploy/cricket/images lrwxrwxrwx 1 cricket cricket 35 Aug 18 2008 lib -> /project/cricket/deploy/cricket/lib lrwxrwxrwx 1 cricket cricket 46 Aug 18 2008 mini-graph.cgi -> /project/cricket/deploy/cricket/mini-graph.cgi
The firewall of this box is really strict, make sure to make any new box as paranoid as this one.
The Real Hosts
Most of the grid services run off 2 (two) physical hosts: keerder, a PE1950-III with a software-raid-1 serup from the HA-GRID series systems; the other is rakel, a M600e blade with hardware raid-1 over SATA in position 1 of the enclosure. Physical hosts left are: beerput, gierput, hek, kaasvat, rijf/stalkaars-02.
Decommissioned services
The following services have been decommissioned:
- VO LDAP services at grid-vo.nikhef.nl
- SecureGrid.org web site
Also, all running services that used to run on beerput.nikhef.nl, except for the ADSM backup have been migrated to "mestkar.nikhef.nl".
Older documentation that still has validity
For the non-migrated services (mainly the DutchGrid CA and the rsync-based backup service, the attached document (PDF) is still valid! File:Grid-Service-Systems-Guide-20070518.pdf
