DQ2Client quick usage

From Atlas Wiki
Jump to navigation Jump to search

Introduction

This page aims to give a quick usage of DQ2Client commands for browsing, searching and retrieving the ATLAS datasets on the Grid.

Setup environment

  1. If running on ribble, you need to prepare a file $HOME/.glite/vomses containing at least the following line:
    "atlas" "voms.cern.ch" "15001" "/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch" "atlas"
    
  2. Initiate the atlas voms-proxy
    % source /project/atlas/nikhef/dq2/dq2_setup.(c)sh.NIKHEF
    % voms-proxy-init -voms atlas
    
  3. Check the atlas voms-proxy
    % voms-proxy-info -all
    subject   : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE/CN=proxy
    issuer    : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
    identity  : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
    type      : proxy
    strength  : 512 bits
    path      : /tmp/x509up_u7838
    timeleft  : 11:59:56
    === VO atlas extension information ===
    VO        : atlas
    subject   : /O=dutchgrid/O=users/O=nikhef/CN=Hurng-Chun LEE
    issuer    : /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch
    attribute : /atlas/Role=NULL/Capability=NULL
    attribute : /atlas/lcg1/Role=NULL/Capability=NULL
    attribute : /atlas/nl/Role=NULL/Capability=NULL
    timeleft  : 11:59:56
    

    List datasets

    The following command lists the merged FDRII AOD datasets of the Muon stream.

    % dq2-ls fdr08_run2*Muon*merge*AOD*
    ...
    fdr08_run2.0052280.physics_Muon.merge.AOD.o3_r508_t45
    fdr08_run2.0052301.physics_Muon.merge.AOD.o3_r508_t45/
    fdr08_run2.0052283.physics_Muon.merge.AOD.o3_r508_t45
    fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
    fdr08_run2.0052290.physics_Muon.merge.AOD.o3_f8_m10
    fdr08_run2.0052293.physics_Muon.merge.AOD.o3_r508_t45_tid024345
    fdr08_run2.0052283.physics_Muon.merge.AOD.o3_r508_t45/
    fdr08_run2.0052300.physics_Muon.merge.AOD.o3_f26_m15
    ...
    

    The one with "/" at the end indicates a dataset container.

    List dataset locations

    The following command lists the replica locations of dataset fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10

    % dq2-ls -r fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
     INCOMPLETE:
       RAL-LCG2_DATADISK
     COMPLETE:
       PRAGUELCG2_DATADISK
       GOEGRID_DATADISK
       TRIUMF-LCG2_DATADISK
       ...
       NIKHEF-ELPROD_DATADISK
       ...
    

    List of all possible locations

    The following command lists all possible locations with files

    dq2-destinations
    

    For example, you will see several Nikhef disks in this list.

    List dataset files

    The following command lists the files in a particular dataset

    % dq2-ls -l fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
    
    fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
    [X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1	9C32CDB7-7638-DD11-8B46-0030487C782A	ad:6906bd22	3044690419
    [X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0007.1	8AEBE6D3-7638-DD11-A9B9-0030487CD836	ad:72aacdf0	3013049170
    [X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0002.1	96E34CB5-7638-DD11-B584-0019DB29C602	ad:be45ed57	2990039386
    [ ]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1	5C4857B2-7638-DD11-87BB-0030487CD762	ad:19ebf510	2991127754
    [X]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1	5E0F0BCC-7638-DD11-B69A-0030487CD938	ad:e96929bb	2008226831
    [ ]	fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1	526C74BC-7638-DD11-9959-0030487CD16C	ad:72859987	2994945608
    ...
    
    Meaning of the returned table
    column # example meaning
    1 [X] or [ ] existence of the file in local site. Cross means it exists.
    2 fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0004.1 logical file name
    3 9C32CDB7-7638-DD11-8B46-0030487C782A Global Unique Identifier (GUID)
    4 ad:6906bd22 checksum
    5 3044690419 file size in byte

    List dataset files in PFNs

    The following command lists the physical SURLs of the files of a particular dataset.

    % dq2-ls -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
    
    fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
    srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1
    srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1
    srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1
    srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1
    srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1
    ...
    

    It returns a list of physical files located at your local site defined by $DQ2_LOCAL_SITE_ID environment variable.

    You can however list the physical files located at another site by using the -L option:

    % dq2-ls -L CERN-PROD_DATADISK -p -f fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
    
    fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
    srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0006.1
    srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0008.1
    srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0001.1
    srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0005.1
    srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0009.1
    srm://srm-atlas.cern.ch/castor/cern.ch/grid/atlas/tzero/atlasdatadisk/fdr08_run2/AOD/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10/fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10._0003.1
    ...
    

    Get dataset

    The following command downloads a dataset from your local site to your current working directory.

    % dq2-get fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10
    

    The following command downloads a dataset from a specific remote site to your current working directory.

    % dq2-get -s CERN-PROD_DATADISK fdr08_run2.0052293.physics_Muon.merge.AOD.o3_f8_m10