Difference between revisions of "Checksumming support in SRM implementations"

From PDP/Grid Wiki
Jump to navigationJump to search
Line 27: Line 27:
 
** can be computed using the <tt>lcg-get-checksum</tt> command
 
** can be computed using the <tt>lcg-get-checksum</tt> command
 
** are stored in the DPM namespace but can never be retrieved ( !! )
 
** are stored in the DPM namespace but can never be retrieved ( !! )
−
** are not displayed when using the <tt>srmls -l</tt> command
+
** are '''not''' displayed when using the <tt>srmls -l</tt> command
 
** are recalculated '''every time'''
 
** are recalculated '''every time'''
  
Line 35: Line 35:
 
* checksums
 
* checksums
 
** can be computed using the <tt>lcg-get-checksum</tt> command; however, the command fails for large files
 
** can be computed using the <tt>lcg-get-checksum</tt> command; however, the command fails for large files
−
** are not displayed when using the <tt>srmls -l</tt> command
+
** are '''not''' displayed when using the <tt>srmls -l</tt> command
 
** are recalculated '''every time'''
 
** are recalculated '''every time'''
  
Line 46: Line 46:
 
** are displayed when using the <tt>srmls -l</tt> command
 
** are displayed when using the <tt>srmls -l</tt> command
 
** are stored in the CASTOR name space
 
** are stored in the CASTOR name space
 +
* gridftp checksums are '''not''' supported
 +
 +
lcg-cr output:
 +
$ lcg-cr -d $SRM/myfile2 -l lfn:/grid/pvier/janjust/my-castor-file2 file://$PWD/myfile --checksum
 +
[LCG-UTIL][lcg_get_checksum_surls][]
 +
srm://srm-dteam.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/test/dteam/disk0tape0/janjust/myfile2:
 +
Communication error on send
 +
guid:64344a07-f3f3-41e5-9079-c86bf61a874e
 +
lcg_cr: Communication error on send
 +
 +
Afterwards the file is stored in SRM but not in the LFC.
 +
srmls output:
 +
$ srmls -l $SRM/myfile
 +
  8508 /castor/ads.rl.ac.uk/test/dteam/disk0tape0/janjust/myfile
 +
  space token(s) :none found
 +
  type: null
 +
  retentionpolicyinfo : null
 +
  locality:ONLINE_AND_NEARLINE
 +
  - Checksum value:  0xcd5d9820
 +
  - Checksum type:  adler32
 +
    UserPermission: uid=dteam001 PermissionsRWX
 +
    GroupPermission: gid=dteam PermissionsRWX
 +
  WorldPermission: RW
 +
  created at:2009/08/20 17:11:06

Revision as of 21:48, 21 August 2009

Some applications have requested the availability of checksumming support on files stored on the storage elements in the Netherlands. The first VO to request this was Atlas, but other VOs are also very interested. Unfortunately, checksumming support seems to be non-homogenously implemented across the different storage backend systems.

This Wiki page is the result of a short investigation into the checksumming differences between dCache, DPM, StoRM and CASTOR.

To store a file with a checksum in SRM I've used the commands 

lcg-cr -d $SRM -l lfn:/grid/pvier/janjust/myfile file://$PWD/myfile --checksum
lcg-cr -d $SRM -l lfn:/grid/pvier/janjust/myfile file://$PWD/myfile --checksum --checksum-type adler32
lcg-cr -d $SRM -l lfn:/grid/pvier/janjust/myfile file://$PWD/myfile --checksum --checksum-type md5

where $SRM points to the appropriate storage system.

Note: so-called gridftp checksums are checksums that can be calculated on-the-fly using a special GridFTP command

dCache

  • srmping: v2.2 dCache production-1.9.3-3
  • seems to support only adler32 checksums (which is what WLCG seems to want) ; according to Ron md5 is also support but this requires a server reconfiguration
  • checksums
    • are computed when the file is transferred to dCache or during a move between pool nodes.
    • are stored in the dCache namespace
    • can be retrieved using the srmls -l command
  • gridftp checksums are not supported

DPM

  • srmping: v2.2 DPM 1.7.0-5
  • supports adler32, md5 and crc32 checksums
  • supports only gridftp checksums; adler32 and crc32 are supported using a special DPM-DSI GridFTP plugin
  • checksums
    • can be computed using the lcg-get-checksum command
    • are stored in the DPM namespace but can never be retrieved ( !! )
    • are not displayed when using the srmls -l command
    • are recalculated every time

StoRM

  • srmping: v2.2 StoRM <FE:1.4.0-01.sl4><BE:1.4.0-00>
  • supports only md5 gridftp checksums
  • checksums
    • can be computed using the lcg-get-checksum command; however, the command fails for large files
    • are not displayed when using the srmls -l command
    • are recalculated every time

CASTOR

  • srmping: v2.2 CASTOR v2_7_15 2.1.7
  • supports only adler32 checksums
  • checksums
    • are computed in the background
    • cannot be specified with the --checksum option
    • are displayed when using the srmls -l command
    • are stored in the CASTOR name space
  • gridftp checksums are not supported

lcg-cr output: 

$ lcg-cr -d $SRM/myfile2 -l lfn:/grid/pvier/janjust/my-castor-file2 file://$PWD/myfile --checksum
[LCG-UTIL][lcg_get_checksum_surls][]
srm://srm-dteam.gridpp.rl.ac.uk/castor/ads.rl.ac.uk/test/dteam/disk0tape0/janjust/myfile2:
Communication error on send 
guid:64344a07-f3f3-41e5-9079-c86bf61a874e
lcg_cr: Communication error on send

Afterwards the file is stored in SRM but not in the LFC. srmls output: 

$ srmls -l $SRM/myfile
 8508 /castor/ads.rl.ac.uk/test/dteam/disk0tape0/janjust/myfile
 space token(s) :none found
 type: null
 retentionpolicyinfo : null
  locality:ONLINE_AND_NEARLINE
  - Checksum value:  0xcd5d9820
  - Checksum type:  adler32 
   UserPermission: uid=dteam001 PermissionsRWX
   GroupPermission: gid=dteam PermissionsRWX
  WorldPermission: RW
 created at:2009/08/20 17:11:06
Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Checksumming_support_in_SRM_implementations&oldid=3205"