DQ2Client quick usage

From Atlas Wiki
Jump to navigation Jump to search

Introduction

This page aims to help ATLAS users to quickly browse, search, retrieve ATLAS datasets for analysis. The page is organized by several use cases.

Setup environment

% source /project/atlas/nikhef/dq2/dq2_setup.(c)sh.NIKHEF
% voms-proxy-init -voms atlas

List datasets

The following command lists the merged FDRII AOD datasets of the Muon stream. 

% dq2-ls fdr08_run2*Muon*merge*AOD*
...
fdr08_run2.0052280.physics_Muon.merge.AOD.o3_r508_t45
fdr08_run2.0052301.physics_Muon.merge.AOD.o3_r508_t45/
fdr08_run2.0052283.physics_Muon.merge.AOD.o3_r508_t45
fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
fdr08_run2.0052290.physics_Muon.merge.AOD.o3_f8_m10
fdr08_run2.0052293.physics_Muon.merge.AOD.o3_r508_t45_tid024345
fdr08_run2.0052283.physics_Muon.merge.AOD.o3_r508_t45/
fdr08_run2.0052300.physics_Muon.merge.AOD.o3_f26_m15
...

The one with "/" at the end indicates a dataset container.

List dataset locations

The following command lists the replica locations of dataset fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10 

% dq2-ls -r fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
 INCOMPLETE:
   RAL-LCG2_DATADISK
 COMPLETE:
   PRAGUELCG2_DATADISK
   GOEGRID_DATADISK
   TRIUMF-LCG2_DATADISK
   ...
   NIKHEF-ELPROD_DATADISK
   ...

List dataset files

The following command lists the files in a particular dataset 

% dq2-ls -l fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
[X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1	9C32CDB7-7638-DD11-8B46-0030487C782A	ad:6906bd22	3044690419
[X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0007.1	8AEBE6D3-7638-DD11-A9B9-0030487CD836	ad:72aacdf0	3013049170
[X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0002.1	96E34CB5-7638-DD11-B584-0019DB29C602	ad:be45ed57	2990039386
[ ]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1	5C4857B2-7638-DD11-87BB-0030487CD762	ad:19ebf510	2991127754
[X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1	5E0F0BCC-7638-DD11-B69A-0030487CD938	ad:e96929bb	2008226831
[ ]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1	526C74BC-7638-DD11-9959-0030487CD16C	ad:72859987	2994945608
...
Meaning of the returned table
column # example meaning
1 [X] or [ ] existence of the file in local site. Cross means it exists.
2 fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1 logical file name
3 9C32CDB7-7638-DD11-8B46-0030487C782A Global Unique Identifier (GUID)
4 ad:6906bd22 checksum
5 3044690419 file size in byte

List dataset files in PFNs

The following command lists the physical SURLs of the files of a particular dataset. 

% dq2-ls -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1
srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1
...

It returns a list of physical files located at your local site defined by $DQ2_LOCAL_SITE_ID environment variable.

You can however list the physical files located at another site by using the -L option: 

% dq2-ls -L CERN-PROD_DATADISK -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1
srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1
...

Get dataset

The following command downloads a dataset from your local site to your current working directory. 

% dq2-get fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

The following command downloads a dataset from a specific remote site to your current working directory. 

% dq2-get -s CERN-PROD_DATADISK fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
Retrieved from "https://wiki.nikhef.nl/atlas/index.php?title=DQ2Client_quick_usage&oldid=2418"