Difference between revisions of "ADC Operation NL"

From Atlas Wiki
Jump to navigation Jump to search
 
(53 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Introduction ==
 
== Introduction ==
−
This page is summarizing the current setup/configuration of the NL cloud of the ATLAS Distributed Computing (ADC).  
+
This page is to summarize the current setup/configuration in the NL cloud for supporting the ATLAS Distributed Computing (ADC).
  
−
Apart from that, it also logs the used-to-happen issues during the day-to-day operation works.
+
== Sites ==
 +
<ul>
 +
<li/>combined Tier1 center:
 +
{| class="wikitable sortable" border="1" cellpadding="5" cellspacing="0"
 +
|-
 +
! width="10%" | Country
 +
! width="10%" | Institute
 +
! width="30%" | GOC site name
 +
|-
 +
| Netherlands || NIKHEF    || NIKHEF-ELPROD
 +
|-
 +
| Netherlands || SARA      || SARA-MATRIX
 +
|}
  
−
== Sites ==
+
<li/>Tier2 centers:
−
* Tier1: combining SARA (SARA-MATRIX) and NIKHEF (NIKHEF-ELPROD) resources/services
+
{| class="wikitable sortable" border="1" cellpadding="5" cellspacing="0"
−
* Tier2:
 
−
{| class="wikitable" border="1" cellpadding="5" cellspacing="0"
 
 
|-
 
|-
−
! Country || Institute || GOC site name
+
! width="10%" | Country
 +
! width="10%" | Institute
 +
! width="30%" | GOC site name
 
|-
 
|-
 
| Russia  || ITEP      || ITEP
 
| Russia  || ITEP      || ITEP
Line 26: Line 38:
 
|-
 
|-
 
| Turkey  || ULAKBIM  || TR-10-ULAKBIM
 
| Turkey  || ULAKBIM  || TR-10-ULAKBIM
 +
|-
 +
| Israel  || WEIZMANN  || WEIZMANN-LCG2
 +
|-
 +
| Israel  || TECHNION  || TECHNION-HEP
 
|}
 
|}
 +
</ul>
  
−
== DDM Services ==
+
== Data Management Services ==
 
* Storage Resource Manager v2.2: deployed on each site as common interface to storage elements
 
* Storage Resource Manager v2.2: deployed on each site as common interface to storage elements
 
* gLite File Transfer Service (FTS): deployed at SARA serving the T1-T1 and T1-T2 data transfers to the sites of the NL cloud
 
* gLite File Transfer Service (FTS): deployed at SARA serving the T1-T1 and T1-T2 data transfers to the sites of the NL cloud
 
* LCG File Catalog Service (LFC): deployed at SARA for cataloging grid files stored on the sites of the NL cloud
 
* LCG File Catalog Service (LFC): deployed at SARA for cataloging grid files stored on the sites of the NL cloud
 +
 +
== ATLAS defined SRMv2 space tokens ==
 +
The following table is taken from [http://indico.cern.ch/materialDisplay.py?contribId=87&sessionId=15&materialId=slides&confId=22137 Stephane's talk on ATLAS Computing & Software Week, November 2008] showing the SRMv2 space tokens needed for ADC operations. More up-to-date information can be found at [https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsGroup the DDM operations twiki].
 +
 +
[[Image:ATLAS space tokens.001.png|ATLAS space tokens]]
 +
 +
== ATLAS SRMv2 space tokens deployed in NL cloud ==
 +
{| class="wikitable sortable" border="1" cellpadding="5" cellspacing="0"
 +
|-
 +
!  width="10%" | Institute
 +
!  width="10%" | ATLASDATATAPE
 +
!  width="10%" | ATLASDATADISK
 +
!  width="10%" | ATLASMCTAPE
 +
!  width="10%" | ATLASMCDISK
 +
!  width="10%" | ATLASPRODDISK
 +
!  width="10%" style="background:grey" | ATLASROFVAL
 +
!  width="10%" | ATLASGROUPDISK
 +
!  width="10%" | ATLASUSERDISK
 +
!              | ATLASLOCALGROUPDISK
 +
|-
 +
| style="background:grey"  | SARA
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
|-
 +
| style="background:grey"  | NIKHEF
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
|-
 +
| ITEP
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
|-
 +
| SINP
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
|-
 +
| IHEP
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
|-
 +
| JINR
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
|-
 +
| PNPI
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
|-
 +
| RRC-KI
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
|-
 +
| CSTCDIE
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
|-
 +
| ULAKBIM
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
| style="background:green" | &nbsp;
 +
| style="background:white" | &nbsp;
 +
|}
  
 
== Data locations among SARA and NIKHEF ==
 
== Data locations among SARA and NIKHEF ==
 +
[[Image:ATLAS datalocations NL Tier1.001.001.001.png|ATLAS data locations at NL Tier1]]
  
−
== SRMv2 space tokens ==
+
== Monitoring pages ==
 +
* [http://atlddm02.cern.ch/dq2/accounting/cloud_view/NLSITES/30/ NL cloud storage usage summary]
 +
* [http://dashb-atlas-data-tier0.cern.ch/dashboard/request.py/site?statsInterval=24&name=SARA Data transfers (T0-T1, T1-T1) in last 24 hrs]
 +
* [http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=24&name=SARA Data transfers (T1-T2) in last 24 hrs]
 +
* [http://panda.cern.ch:25880/server/pandamon/query?dash=task&cloud=NL&show=active active PanDA tasks in NL cloud]
 +
* specific storage usage of essential space tokens at NL T1 (require CERN NICE account)
 +
** [https://sls.cern.ch/sls/service.php?id=NIKHEF-ELPROD_ATLASDATADISK NIKHEF-ELPROD_DATADISK]
 +
** [https://sls.cern.ch/sls/service.php?id=NIKHEF-ELPROD_ROFVAL NIKHEF-ELPROD_ROFVAL]
 +
** [https://sls.cern.ch/sls/service.php?id=SARA-MATRIX_DATADISK SARA-MATRIX_DATADISK]
 +
** [https://sls.cern.ch/sls/service.php?id=SARA-MATRIX_MCDISK SARA-MATRIX_MCDISK]
  
−
== Useful links ==
+
== More information ==
 +
* [https://twiki.cern.ch/twiki/bin/view/Atlas/AtlasGridDowntime ATLAS site downtime]
 +
* [https://prod-grid-logger.cern.ch/elog/ATLAS+Computer+Operations+Logbook/?Cloud=NL ADC eLog entries concerning NL cloud]
 +
* [https://twiki.cern.ch/twiki/bin/view/Atlas/ADCoS#Controlling_Panda_Queues Instructions for moving T2 sites online for MC production]
 +
* [https://twiki.cern.ch/twiki/bin/view/Atlas/StorageSetUp ATLAS storage setup guidline]
 +
* DDM operation wiki
 +
* DDM browser
 +
 
 +
== Daily operation logs ==
 +
{| class="wikitable" border="1" cellpadding="5" cellspacing="0"
 +
|-
 +
! Date        || Actions || Remarks
 +
|-
 +
|  9 Oct. 2008 || requests cosmic and 1beam reprocessed ESD to NIKHEF-ELPROD_DATADISK || needed by NIKHEF physics group
 +
|-
 +
| 10 Oct. 2008 || NL Tier2s become on-line again for MC production                    || FTS performance issue at SARA fixed
 +
|}
  
 
== Trouble shooting logs ==
 
== Trouble shooting logs ==
 +
=== Sept. 2008 - huge transfer backlog from T2s to SARA ===
 +
Problem fixed by running the FTS admin tool to slim down the FTS job history db table.
 +
 +
=== Sept. 2008 - SRM request timeout reading data from SARA ===
 +
A cron job fixing orphan file issue of dCache loads PNFS server so it was stopped.
 +
Also observe a broken network switch involving 4 new dCache node. The problematic network switch has been replaced.
 +
A broken dCache node is also replaced.

Latest revision as of 11:58, 7 July 2009

Introduction

This page is to summarize the current setup/configuration in the NL cloud for supporting the ATLAS Distributed Computing (ADC).

Sites

  • combined Tier1 center:
    Country Institute GOC site name
    Netherlands NIKHEF NIKHEF-ELPROD
    Netherlands SARA SARA-MATRIX
  • Tier2 centers:
    Country Institute GOC site name
    Russia ITEP ITEP
    Russia IHEP RU-PROTVINO-IHEP
    Russia JINR JINR-LCG2
    Russia PNPI RU-PNPI
    Russia SINP SINP
    Russia RRC-KI RRC-KI
    Ireland CST CSTCDIE
    Turkey ULAKBIM TR-10-ULAKBIM
    Israel WEIZMANN WEIZMANN-LCG2
    Israel TECHNION TECHNION-HEP

Data Management Services

  • Storage Resource Manager v2.2: deployed on each site as common interface to storage elements
  • gLite File Transfer Service (FTS): deployed at SARA serving the T1-T1 and T1-T2 data transfers to the sites of the NL cloud
  • LCG File Catalog Service (LFC): deployed at SARA for cataloging grid files stored on the sites of the NL cloud

ATLAS defined SRMv2 space tokens

The following table is taken from Stephane's talk on ATLAS Computing & Software Week, November 2008 showing the SRMv2 space tokens needed for ADC operations. More up-to-date information can be found at the DDM operations twiki.

ATLAS space tokens

ATLAS SRMv2 space tokens deployed in NL cloud

Institute ATLASDATATAPE ATLASDATADISK ATLASMCTAPE ATLASMCDISK ATLASPRODDISK ATLASROFVAL ATLASGROUPDISK ATLASUSERDISK ATLASLOCALGROUPDISK
SARA                  
NIKHEF                  
ITEP                  
SINP                  
IHEP                  
JINR                  
PNPI                  
RRC-KI                  
CSTCDIE                  
ULAKBIM                  

Data locations among SARA and NIKHEF

ATLAS data locations at NL Tier1

Monitoring pages

More information

Daily operation logs

Date Actions Remarks
9 Oct. 2008 requests cosmic and 1beam reprocessed ESD to NIKHEF-ELPROD_DATADISK needed by NIKHEF physics group
10 Oct. 2008 NL Tier2s become on-line again for MC production FTS performance issue at SARA fixed

Trouble shooting logs

Sept. 2008 - huge transfer backlog from T2s to SARA

Problem fixed by running the FTS admin tool to slim down the FTS job history db table.

Sept. 2008 - SRM request timeout reading data from SARA

A cron job fixing orphan file issue of dCache loads PNFS server so it was stopped. Also observe a broken network switch involving 4 new dCache node. The problematic network switch has been replaced. A broken dCache node is also replaced.

Retrieved from "https://wiki.nikhef.nl/atlas/index.php?title=ADC_Operation_NL&oldid=4815"