Difference between revisions of "DQ2Client quick usage"

From Atlas Wiki
Jump to navigation Jump to search
 
(39 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Introduction ==
 
== Introduction ==
−
This page aims to help ATLAS users to quickly browse, search, retrieve ATLAS datasets for analysis. The page is organized by several use cases.
+
This page aims to give a quick usage of DQ2Client commands for browsing, searching and retrieving the ATLAS datasets on the Grid.
  
 
== Setup environment ==
 
== Setup environment ==
 +
<ul>
 +
<li/>Setup DQ2 and LCG UI environments using CVMFS:
 +
 +
<pre>
 +
% source /project/atlas/nikhef/cvmfs/setup.sh
 +
% setupATLAS
 +
% localSetupDQ2Client
 +
</pre>
 +
 +
<!--
 +
If running on <tt>ribble</tt>, you need to do one of the follows to ensure you can obtain a proper VOMS server certificate:
 +
 +
<ol>
 +
<li/>preparing a file <tt>$HOME/.glite/vomses</tt> containing at least the following line:
 +
<pre>
 +
"atlas" "voms.cern.ch" "15001" "/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch" "atlas"
 +
</pre>
 +
 +
<li/>pointing voms related commands to look for VOMS server certificates from CERN AFS area.
 +
<pre>
 +
% export X509_VOMS_DIR=/afs/cern.ch/project/gd/LCG-share/3.1.14-0/external/etc/grid-security/vomsdir (for BASH)
 +
</pre>
 +
<pre>
 +
% setenv X509_VOMS_DIR /afs/cern.ch/project/gd/LCG-share/3.1.14-0/external/etc/grid-security/vomsdir (for CSH)
 +
</pre>
 +
</ol> -->
 +
<li/>Initiate the atlas voms-proxy
 
<pre>
 
<pre>
−
% source /project/atlas/nikhef/dq2/dq2_setup.(c)sh.NIKHEF
 
 
% voms-proxy-init -voms atlas
 
% voms-proxy-init -voms atlas
 
</pre>
 
</pre>
 +
 +
<li/>Check if you have a proper atlas voms-proxy
 +
<pre>
 +
% voms-proxy-info -all
 +
subject  : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE/CN=proxy
 +
issuer    : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
 +
identity  : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
 +
type      : proxy
 +
strength  : 512 bits
 +
path      : /tmp/x509up_u7838
 +
timeleft  : 11:59:56
 +
=== VO atlas extension information ===
 +
VO        : atlas
 +
subject  : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
 +
issuer    : /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch
 +
attribute : /atlas/Role=NULL/Capability=NULL
 +
attribute : /atlas/lcg1/Role=NULL/Capability=NULL
 +
attribute : /atlas/nl/Role=NULL/Capability=NULL
 +
timeleft  : 11:59:56
 +
</pre>
 +
 +
</ul>
 +
 +
== Search datasets ==
 +
You can search for ATLAS dataset by [https://atlastagcollector.in2p3.fr:8443/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?linkId=246 Atlas Metadata Interface (AMI)]
  
 
== List datasets ==
 
== List datasets ==
Line 24: Line 75:
 
</pre>
 
</pre>
  
−
The ones with <b>"<tt>/</tt>"</b> at the end indicates a dataset container.
+
The one with <b>"<tt>/</tt>"</b> at the end indicates a dataset container.
  
−
== Check the replica locations of a particular dataset ==
+
== List dataset locations ==
 
The following command lists the replica locations of dataset <tt>fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10</tt>
 
The following command lists the replica locations of dataset <tt>fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10</tt>
 
<pre>
 
<pre>
Line 39: Line 90:
 
   NIKHEF-ELPROD_DATADISK
 
   NIKHEF-ELPROD_DATADISK
 
   ...
 
   ...
 +
</pre>
 +
 +
== List of all possible locations ==
 +
The following command lists all possible locations with files
 +
<pre>
 +
dq2-destinations
 +
</pre>
 +
For example, you will see several Nikhef disks in this list.
 +
 +
== List dataset files ==
 +
The following command lists the files in a particular dataset
 +
<pre>
 +
% dq2-ls -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 +
 +
fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 +
[X] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1 9C32CDB7-7638-DD11-8B46-0030487C782A ad:6906bd22 3044690419
 +
[X] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0007.1 8AEBE6D3-7638-DD11-A9B9-0030487CD836 ad:72aacdf0 3013049170
 +
[X] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0002.1 96E34CB5-7638-DD11-B584-0019DB29C602 ad:be45ed57 2990039386
 +
[ ] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1 5C4857B2-7638-DD11-87BB-0030487CD762 ad:19ebf510 2991127754
 +
[X] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1 5E0F0BCC-7638-DD11-B69A-0030487CD938 ad:e96929bb 2008226831
 +
[ ] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1 526C74BC-7638-DD11-9959-0030487CD16C ad:72859987 2994945608
 +
...
 +
</pre>
 +
 +
{| class="wikitable" border="1" cellpadding="5" cellspacing="0"
 +
|+ Meaning of the returned table
 +
|-
 +
! column # || example || meaning
 +
|-
 +
| 1 || <tt>[X]</tt> or <tt>[ ]</tt> || existence of the file in local site. Cross means it exists.
 +
|-
 +
| 2 || <tt>fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1</tt> || logical file name
 +
|-
 +
| 3 || <tt>9C32CDB7-7638-DD11-8B46-0030487C782A</tt> || Global Unique Identifier (GUID)
 +
|-
 +
| 4 || <tt>ad:6906bd22</tt> || checksum
 +
|-
 +
| 5 || <tt>3044690419</tt> || file size in byte
 +
|}
 +
 +
== List dataset files in PFNs ==
 +
The following command lists the physical SURLs of the files of a particular dataset.
 +
<pre>
 +
% dq2-ls -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 +
 +
fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 +
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1
 +
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1
 +
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1
 +
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1
 +
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1
 +
...
 +
</pre>
 +
 +
It returns a list of physical files located at your local site defined by <tt>$DQ2_LOCAL_SITE_ID</tt> environment variable.
 +
 +
You can however list the physical files located at another site by using the <tt>-L</tt> option:
 +
 +
<pre>
 +
% dq2-ls -L CERN-PROD_DATADISK -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 +
 +
fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 +
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1
 +
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1
 +
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1
 +
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1
 +
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1
 +
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1
 +
...
 +
</pre>
 +
 +
== Get dataset ==
 +
The following command downloads a dataset from your local site to your current working directory.
 +
 +
<pre>
 +
% dq2-get fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 +
</pre>
 +
 +
The following command downloads a dataset from a specific remote site to your current working directory.
 +
 +
<pre>
 +
% dq2-get -L CERN-PROD_DATADISK fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 
</pre>
 
</pre>

Latest revision as of 08:25, 28 September 2012

Introduction

This page aims to give a quick usage of DQ2Client commands for browsing, searching and retrieving the ATLAS datasets on the Grid.

Setup environment

  • Setup DQ2 and LCG UI environments using CVMFS: 
    % source /project/atlas/nikhef/cvmfs/setup.sh
% setupATLAS
% localSetupDQ2Client
  • Initiate the atlas voms-proxy 
    % voms-proxy-init -voms atlas
  • Check if you have a proper atlas voms-proxy 
    % voms-proxy-info -all
subject   : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE/CN=proxy
issuer    : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
identity  : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
type      : proxy
strength  : 512 bits
path      : /tmp/x509up_u7838
timeleft  : 11:59:56
=== VO atlas extension information ===
VO        : atlas
subject   : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
issuer    : /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch
attribute : /atlas/Role=NULL/Capability=NULL
attribute : /atlas/lcg1/Role=NULL/Capability=NULL
attribute : /atlas/nl/Role=NULL/Capability=NULL
timeleft  : 11:59:56

Search datasets

You can search for ATLAS dataset by Atlas Metadata Interface (AMI)

List datasets

The following command lists the merged FDRII AOD datasets of the Muon stream. 

% dq2-ls fdr08_run2*Muon*merge*AOD*
...
fdr08_run2.0052280.physics_Muon.merge.AOD.o3_r508_t45
fdr08_run2.0052301.physics_Muon.merge.AOD.o3_r508_t45/
fdr08_run2.0052283.physics_Muon.merge.AOD.o3_r508_t45
fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
fdr08_run2.0052290.physics_Muon.merge.AOD.o3_f8_m10
fdr08_run2.0052293.physics_Muon.merge.AOD.o3_r508_t45_tid024345
fdr08_run2.0052283.physics_Muon.merge.AOD.o3_r508_t45/
fdr08_run2.0052300.physics_Muon.merge.AOD.o3_f26_m15
...

The one with "/" at the end indicates a dataset container.

List dataset locations

The following command lists the replica locations of dataset fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 

% dq2-ls -r fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 INCOMPLETE:
   RAL-LCG2_DATADISK
 COMPLETE:
   PRAGUELCG2_DATADISK
   GOEGRID_DATADISK
   TRIUMF-LCG2_DATADISK
   ...
   NIKHEF-ELPROD_DATADISK
   ...

List of all possible locations

The following command lists all possible locations with files 

dq2-destinations

For example, you will see several Nikhef disks in this list.

List dataset files

The following command lists the files in a particular dataset 

% dq2-ls -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
[X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1	9C32CDB7-7638-DD11-8B46-0030487C782A	ad:6906bd22	3044690419
[X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0007.1	8AEBE6D3-7638-DD11-A9B9-0030487CD836	ad:72aacdf0	3013049170
[X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0002.1	96E34CB5-7638-DD11-B584-0019DB29C602	ad:be45ed57	2990039386
[ ]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1	5C4857B2-7638-DD11-87BB-0030487CD762	ad:19ebf510	2991127754
[X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1	5E0F0BCC-7638-DD11-B69A-0030487CD938	ad:e96929bb	2008226831
[ ]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1	526C74BC-7638-DD11-9959-0030487CD16C	ad:72859987	2994945608
...
Meaning of the returned table
column # example meaning
1 [X] or [ ] existence of the file in local site. Cross means it exists.
2 fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1 logical file name
3 9C32CDB7-7638-DD11-8B46-0030487C782A Global Unique Identifier (GUID)
4 ad:6906bd22 checksum
5 3044690419 file size in byte

List dataset files in PFNs

The following command lists the physical SURLs of the files of a particular dataset. 

% dq2-ls -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1
...

It returns a list of physical files located at your local site defined by $DQ2_LOCAL_SITE_ID environment variable.

You can however list the physical files located at another site by using the -L option: 

% dq2-ls -L CERN-PROD_DATADISK -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1
...

Get dataset

The following command downloads a dataset from your local site to your current working directory. 

% dq2-get fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

The following command downloads a dataset from a specific remote site to your current working directory. 

% dq2-get -L CERN-PROD_DATADISK fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
Retrieved from "https://wiki.nikhef.nl/atlas/index.php?title=DQ2Client_quick_usage&oldid=4813"