Difference between revisions of "DQ2Client quick usage"
Jump to navigation
Jump to search
| (7 intermediate revisions by the same user not shown) | |||
| Line 4: | Line 4: | ||
== Setup environment == | == Setup environment == | ||
<ul> | <ul> | ||
| − | <li/>Setup DQ2 and LCG UI environments: | + | <li/>Setup DQ2 and LCG UI environments using CVMFS: |
<pre> | <pre> | ||
| − | % source /project/atlas/nikhef/ | + | % source /project/atlas/nikhef/cvmfs/setup.sh |
| + | % setupATLAS | ||
| + | % localSetupDQ2Client | ||
</pre> | </pre> | ||
| + | <!-- | ||
If running on <tt>ribble</tt>, you need to do one of the follows to ensure you can obtain a proper VOMS server certificate: | If running on <tt>ribble</tt>, you need to do one of the follows to ensure you can obtain a proper VOMS server certificate: | ||
| Line 25: | Line 28: | ||
% setenv X509_VOMS_DIR /afs/cern.ch/project/gd/LCG-share/3.1.14-0/external/etc/grid-security/vomsdir (for CSH) | % setenv X509_VOMS_DIR /afs/cern.ch/project/gd/LCG-share/3.1.14-0/external/etc/grid-security/vomsdir (for CSH) | ||
</pre> | </pre> | ||
| − | </ol> | + | </ol> --> |
| − | |||
<li/>Initiate the atlas voms-proxy | <li/>Initiate the atlas voms-proxy | ||
<pre> | <pre> | ||
| Line 53: | Line 55: | ||
</ul> | </ul> | ||
| + | |||
| + | == Search datasets == | ||
| + | You can search for ATLAS dataset by [https://atlastagcollector.in2p3.fr:8443/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?linkId=246 Atlas Metadata Interface (AMI)] | ||
== List datasets == | == List datasets == | ||
| Line 71: | Line 76: | ||
The one with <b>"<tt>/</tt>"</b> at the end indicates a dataset container. | The one with <b>"<tt>/</tt>"</b> at the end indicates a dataset container. | ||
| − | |||
| − | |||
| − | |||
== List dataset locations == | == List dataset locations == | ||
| Line 100: | Line 102: | ||
The following command lists the files in a particular dataset | The following command lists the files in a particular dataset | ||
<pre> | <pre> | ||
| − | % dq2-ls - | + | % dq2-ls -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 |
fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 | fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 | ||
| Line 169: | Line 171: | ||
<pre> | <pre> | ||
| − | % dq2-get - | + | % dq2-get -L CERN-PROD_DATADISK fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 |
</pre> | </pre> | ||
Latest revision as of 08:25, 28 September 2012
Introduction
This page aims to give a quick usage of DQ2Client commands for browsing, searching and retrieving the ATLAS datasets on the Grid.
Setup environment
-
Setup DQ2 and LCG UI environments using CVMFS:
% source /project/atlas/nikhef/cvmfs/setup.sh % setupATLAS % localSetupDQ2ClientInitiate the atlas voms-proxy
% voms-proxy-init -voms atlasCheck if you have a proper atlas voms-proxy
% voms-proxy-info -all subject : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE/CN=proxy issuer : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE identity : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE type : proxy strength : 512 bits path : /tmp/x509up_u7838 timeleft : 11:59:56 === VO atlas extension information === VO : atlas subject : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE issuer : /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch attribute : /atlas/Role=NULL/Capability=NULL attribute : /atlas/lcg1/Role=NULL/Capability=NULL attribute : /atlas/nl/Role=NULL/Capability=NULL timeleft : 11:59:56
Search datasets
You can search for ATLAS dataset by Atlas Metadata Interface (AMI)
List datasets
The following command lists the merged FDRII AOD datasets of the Muon stream.
% dq2-ls fdr08_run2*Muon*merge*AOD* ... fdr08_run2.0052280.physics_Muon.merge.AOD.o3_r508_t45 fdr08_run2.0052301.physics_Muon.merge.AOD.o3_r508_t45/ fdr08_run2.0052283.physics_Muon.merge.AOD.o3_r508_t45 fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 fdr08_run2.0052290.physics_Muon.merge.AOD.o3_f8_m10 fdr08_run2.0052293.physics_Muon.merge.AOD.o3_r508_t45_tid024345 fdr08_run2.0052283.physics_Muon.merge.AOD.o3_r508_t45/ fdr08_run2.0052300.physics_Muon.merge.AOD.o3_f26_m15 ...
The one with "/" at the end indicates a dataset container.
List dataset locations
The following command lists the replica locations of dataset fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
% dq2-ls -r fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 INCOMPLETE: RAL-LCG2_DATADISK COMPLETE: PRAGUELCG2_DATADISK GOEGRID_DATADISK TRIUMF-LCG2_DATADISK ... NIKHEF-ELPROD_DATADISK ...
List of all possible locations
The following command lists all possible locations with files
dq2-destinations
For example, you will see several Nikhef disks in this list.
List dataset files
The following command lists the files in a particular dataset
% dq2-ls -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 [X] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1 9C32CDB7-7638-DD11-8B46-0030487C782A ad:6906bd22 3044690419 [X] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0007.1 8AEBE6D3-7638-DD11-A9B9-0030487CD836 ad:72aacdf0 3013049170 [X] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0002.1 96E34CB5-7638-DD11-B584-0019DB29C602 ad:be45ed57 2990039386 [ ] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1 5C4857B2-7638-DD11-87BB-0030487CD762 ad:19ebf510 2991127754 [X] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1 5E0F0BCC-7638-DD11-B69A-0030487CD938 ad:e96929bb 2008226831 [ ] fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1 526C74BC-7638-DD11-9959-0030487CD16C ad:72859987 2994945608 ...
| column # | example | meaning |
|---|---|---|
| 1 | [X] or [ ] | existence of the file in local site. Cross means it exists. |
| 2 | fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1 | logical file name |
| 3 | 9C32CDB7-7638-DD11-8B46-0030487C782A | Global Unique Identifier (GUID) |
| 4 | ad:6906bd22 | checksum |
| 5 | 3044690419 | file size in byte |
List dataset files in PFNs
The following command lists the physical SURLs of the files of a particular dataset.
% dq2-ls -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1 srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1 srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1 srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1 srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1 ...
It returns a list of physical files located at your local site defined by $DQ2_LOCAL_SITE_ID environment variable.
You can however list the physical files located at another site by using the -L option:
% dq2-ls -L CERN-PROD_DATADISK -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1 srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1 srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1 srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1 srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1 srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1 ...
Get dataset
The following command downloads a dataset from your local site to your current working directory.
% dq2-get fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
The following command downloads a dataset from a specific remote site to your current working directory.
% dq2-get -L CERN-PROD_DATADISK fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10