Checksumming support in SRM implementations
From PDP/Grid Wiki
Revision as of 15:14, 21 August 2009 by Janjust@nikhef.nl (talk | contribs)
Some applications have requested the availability of checksumming support on files stored on the storage elements in the Netherlands. The first VO to request this was Atlas, but other VOs are also very interested. Unfortunately, checksumming support seems to be non-homogenously implemented across the different storage backend systems.
This Wiki page is the result of a short investigation into the checksumming differences between dCache, DPM, StoRM and CASTOR.
To store a file with a checksum in SRM I've used the commands
lcg-cr -d $SRM -l lfn:/grid/pvier/janjust/myfile file://$PWD/myfile --checksum lcg-cr -d $SRM -l lfn:/grid/pvier/janjust/myfile file://$PWD/myfile --checksum --checksum-type adler32 lcg-cr -d $SRM -l lfn:/grid/pvier/janjust/myfile file://$PWD/myfile --checksum --checksum-type md5
where $SRM points to the appropriate storage system.
Note: so-called gridftp checksums are checksums that can be calculated on-the-fly using a special GridFTP command
dCache
- srmping: v2.2 dCache production-1.9.3-3
- seems to support only adler32 checksums (which is what WLCG seems to want) ; according to Ron md5 is also support but this requires a server reconfiguration
- checksums
- are computed when the file is transferred to dCache or during a move between pool nodes.
- are stored in the dCache namespace
- can be retrieved using the srmls -l command
- gridftp checksums are not supported
DPM
- srmping: v2.2 DPM 1.7.0-5
- supports adler32, md5 and crc32 checksums
- supports only gridftp checksums; adler32 and crc32 are supported using a special DPM-DSI GridFTP plugin
- checksums
- can be computed using the lcg-get-checksum command
- are stored in the DPM namespace but can never be retrieved ( !! )
- are not displayed when using the srmls -l command
- are recalculated every time
StoRM
- srmping: v2.2 StoRM <FE:1.4.0-01.sl4><BE:1.4.0-00>
- supports only md5 gridftp checksums
- checksums
- can be computed using the lcg-get-checksum command; however, the command fails for large files
- are not displayed when using the srmls -l command
- are recalculated every time