Dans Data Upload
Revision as of 10:50, 10 May 2012 by Janjust@nikhef.nl (talk | contribs) (→How to upload a DANS archive to the grid)
How to upload a DANS archive to the grid
- create a new directory with the name of the archive. As an example we use the 'Crome' archive. We refer to the name of the archive using the environment variable '$ARCHIVE' :
$ export ARCHIVE=Crome $ mkdir -p ~/dans/$ARCHIVE/
- In this directory create another directory with the same name; this directory will contain the list of files and directories that need to be uploaded
$ cd ~/dans/$ARCHIVE $ mkdir $ARCHIVE
- copy over the scripts from the repository
$ cp -a ~/dans/scripts/* .
- generate a sorted list of files. Note: All further actions are done based on this list!
$ find -L $ARCHIVE -type f | sort > ${ARCHIVE}-files.txt
- check the list of files , remove any entries such as '.Trash' folders if desired.
- generate a list of tarball.lst files. Each tarball.lst file contains a subset of entries from the ${ARCHIVE}-files.txt file that , when tarred up into a single .tar file, is roughly 8 GB in size. The output files are named '$ARCHIVE-<N>.tar.lst' where <N> is a 4 digit counter starting at 1:
$ ./gen-tar-list ${ARCHIVE}-files.txt Crome-0001.tar.lst Crome-0002.tar.lst ... Crome-0072.tar.lst
- Now the final big step: run the 'upload-tar' script, which will
- generate the tarballs
- generate md5 checksums for all files in each tarball
- upload each tarball to the grid.
Note: For this step a valid grid proxy is required!
$ ./upload-tar Checksumming tarball contents