Dans Data Upload

How to upload a DANS archive to the grid

Note: as a convention, commands that need to be typed in are preceded by a UNIX prompt-sign '$' . Output if shown without a preceding prompt sign.

• create a new directory with the name of the archive. As an example we use the 'Crome' archive. We refer to the name of the archive using the environment variable '${ARCHIVE}' :

 $ export ARCHIVE=Crome
 $ mkdir -p ~/dans/${ARCHIVE}

• In this directory create another directory with the same name; this directory will contain the list of files and directories that need to be uploaded

 $ cd ~/dans/${ARCHIVE}
 $ mkdir ${ARCHIVE}

• copy over the scripts from the repository

 $ cp -a ~/dans/scripts/* .

• generate a sorted list of files. Note: All further actions are done based on this list!

 $ find -L ${ARCHIVE} -type f | sort > ${ARCHIVE}-files.txt

• check the list of files , remove any entries such as '.Trash' folders if desired.
• generate a list of tarball.lst files. Each tarball.lst file contains a subset of entries from the ${ARCHIVE}-files.txt file that , when tarred up into a single .tar file, is roughly 8 GB in size. The output files are named '${ARCHIVE}-<N>.tar.lst' where <N> is a 4 digit counter starting at 1:

 $ ./gen-tar-list ${ARCHIVE}-files.txt
 Crome-0001.tar.lst
 Crome-0002.tar.lst
 ...
 Crome-0072.tar.lst

• Now the final big step: run the 'upload-tar' script, which will
  • generate the tarballs
  • generate md5 checksums for all files in each tarball
  • upload each tarball to the grid.

This script will take a long time to run, depending on how many tarballs there are.

Note: For this step a valid grid proxy is required!

 $ ./upload-tar
 Checksumming tarball contents
 Generating ${ARCHIVE}-0001.tar
 Uploading ${ARCHIVE}-0001.tar
 guid:86f11fb4-9b57-4fae-a787-8019663e248c
 Moving ${ARCHIVE}-0001.tar.lst and ${ARCHIVE}-0001.tar.md5sum to directory "done"
 Checksumming tarball contents
 Generating ${ARCHIVE}-0002.tar
 Uploading ${ARCHIVE}-0002.tar
 ...
 Generating ${ARCHIVE}-0072.tar
 Uploading ${ARCHIVE}-0072.tar
 guid:e6781de9-9709-4bf6-a019-b5fa4a1fb3c8
 Moving ${ARCHIVE}-0072.tar.lst and ${ARCHIVE}-0072.tar.md5sum to directory "done"

For each tarball that is successfully processed the '${ARCHIVE}-<N>.tar.lst' file is moved to a separate directory 'done' . This way the 'upload-tar' script can be stopped and restarted at will, as it will continue processing '${ARCHIVE}-<N>.tar.lst' files until all have been moved to the 'done' directory.

• Check the contents of the 'done' directory , especially the contents of the '${ARCHIVE}-<N>.tar.md5sum' files:

 $