Difference between revisions of "Dans Data Upload"
(10 intermediate revisions by the same user not shown) | |||
Line 16: | Line 16: | ||
$ find -L ${ARCHIVE} -type f | sort > ${ARCHIVE}-files.txt | $ find -L ${ARCHIVE} -type f | sort > ${ARCHIVE}-files.txt | ||
* check the list of files , remove any entries such as '.Trash' folders if desired. | * check the list of files , remove any entries such as '.Trash' folders if desired. | ||
− | * generate a list of tarball.lst files. Each tarball.lst file contains a subset of entries from the ${ARCHIVE}-files.txt file that , when tarred up into a single .tar file, is | + | * Use the <tt>gen-tar-list</tt> script to generate a list of tarball.lst files. Each tarball.lst file contains a subset of entries from the ${ARCHIVE}-files.txt file that , when tarred up into a single .tar file, is at least 8 GB in size. The output files are named '${ARCHIVE}-<N>.tar.lst' where <N> is a 4 digit counter starting at 1: |
$ ./gen-tar-list ${ARCHIVE}-files.txt | $ ./gen-tar-list ${ARCHIVE}-files.txt | ||
Crome-0001.tar.lst | Crome-0001.tar.lst | ||
Line 23: | Line 23: | ||
Crome-0073.tar.lst | Crome-0073.tar.lst | ||
− | * Now the final big step: run the 'upload-tar' script, which will | + | * Now the final big step: run the '<tt>upload-tar</tt>' script, which will |
** generate the tarballs | ** generate the tarballs | ||
** generate md5 checksums for all files in each tarball | ** generate md5 checksums for all files in each tarball | ||
** upload each tarball to the grid. | ** upload each tarball to the grid. | ||
− | This script will take a long time to run, depending on how many tarballs there are. | + | This script will take a long time to run, depending on how many tarballs there are. It is often useful to run the next command inside a <tt>screen</tt> session. |
'''Note''': For this step a valid grid proxy is required! | '''Note''': For this step a valid grid proxy is required! | ||
Line 45: | Line 45: | ||
Moving ${ARCHIVE}-0073.tar.lst and ${ARCHIVE}-0073.tar.md5sum to directory "done" | Moving ${ARCHIVE}-0073.tar.lst and ${ARCHIVE}-0073.tar.md5sum to directory "done" | ||
− | For each tarball that is successfully processed the '${ARCHIVE}-<N>.tar.lst' file is moved to a separate directory 'done' . This way the 'upload-tar' script can be stopped and restarted at will, as it will continue processing '${ARCHIVE}-<N>.tar.lst' files until all have been moved to the 'done' directory. | + | For each tarball that is successfully processed the '${ARCHIVE}-<N>.tar.lst' file is moved to a separate directory 'done' . This way the '<tt>upload-tar</tt>' script can be stopped and restarted at will, as it will continue processing '${ARCHIVE}-<N>.tar.lst' files until all have been moved to the 'done' directory. |
+ | |||
+ | For more details on the usage of the '<tt>upload-tar</tt>' script , see [[DANS_Job_Scripts|DANS Job Scripts]]. | ||
* Check the contents of the 'done' directory , especially the contents of the '${ARCHIVE}-<N>.tar.md5sum' files: | * Check the contents of the 'done' directory , especially the contents of the '${ARCHIVE}-<N>.tar.md5sum' files: | ||
Line 63: | Line 65: | ||
# Crome-0001.tar END | # Crome-0001.tar END | ||
− | By combining all '${ARCHIVE}-0001.tar.md5sum' files a full list of all md5sums can be generated and compared to the output of an 'md5deep' command: | + | By combining all '${ARCHIVE}-0001.tar.md5sum' files a full list of all md5sums can be generated and compared to the output of an '<tt>md5deep</tt>' command: |
$ for i in done/${ARCHIVE}-*.tar.md5sum ; do grep -v '#' $i ; done | sort > total.md5sum | $ for i in done/${ARCHIVE}-*.tar.md5sum ; do grep -v '#' $i ; done | sort > total.md5sum | ||
+ | |||
+ | * After verifying that all '${ARCHIVE}-0001.tar.md5sum' files are correct, move them (or symlink them) to another directory '<tt>checksums</tt>'. The latter directory is used by the '<tt>compare-checksums</tt>' script from [[DANS_Data_Management#DANS_Workflow_phase_3:_Verifying_data|Phase 3: Verifying data]]. | ||
+ | $ mkdir checksums | ||
+ | $ cp done/*.tar.md5sum checksums | ||
+ | |||
+ | * Once the upload has completed successfully you can proceed to [[DANS_Data_Management#DANS_Workflow_phase_2:_Compressing_data|Phase2: Compressing Data]]. |
Latest revision as of 11:26, 20 November 2012
How to upload a DANS archive to the grid
Note: as a convention, commands that need to be typed in are preceded by a UNIX prompt-sign '$' . Output if shown without a preceding prompt sign.
- create a new directory with the name of the archive. As an example we use the 'Crome' archive. We refer to the name of the archive using the environment variable '${ARCHIVE}' :
$ export ARCHIVE=Crome $ mkdir -p ~/dans/${ARCHIVE}
- copy over the scripts from the repository
$ cd ~/dans/${ARCHIVE} $ cp -a ~/dans/scripts/* .
- In this directory create another directory with the same name; this directory will contain the list of files and directories that need to be uploaded
$ mkdir ${ARCHIVE}
- in this second ${ARCHIVE} directory, create a list of all files and directories that need to be uploaded. If an entire directory needs to be uploaded then create a symlink to it. For the 'Crome' archive the following command was used to create the symlinks:
$ for i in <fullpath>/Crome/crome*/ ; do echo ln -fs $i ; done
- generate a sorted list of files. Note: All further actions are done based on this list!
$ find -L ${ARCHIVE} -type f | sort > ${ARCHIVE}-files.txt
- check the list of files , remove any entries such as '.Trash' folders if desired.
- Use the gen-tar-list script to generate a list of tarball.lst files. Each tarball.lst file contains a subset of entries from the ${ARCHIVE}-files.txt file that , when tarred up into a single .tar file, is at least 8 GB in size. The output files are named '${ARCHIVE}-<N>.tar.lst' where <N> is a 4 digit counter starting at 1:
$ ./gen-tar-list ${ARCHIVE}-files.txt Crome-0001.tar.lst Crome-0002.tar.lst ... Crome-0073.tar.lst
- Now the final big step: run the 'upload-tar' script, which will
- generate the tarballs
- generate md5 checksums for all files in each tarball
- upload each tarball to the grid.
This script will take a long time to run, depending on how many tarballs there are. It is often useful to run the next command inside a screen session.
Note: For this step a valid grid proxy is required!
$ ./upload-tar Checksumming tarball contents Generating ${ARCHIVE}-0001.tar Uploading ${ARCHIVE}-0001.tar guid:86f11fb4-9b57-4fae-a787-8019663e248c Moving ${ARCHIVE}-0001.tar.lst and ${ARCHIVE}-0001.tar.md5sum to directory "done" Checksumming tarball contents Generating ${ARCHIVE}-0002.tar Uploading ${ARCHIVE}-0002.tar ... Generating ${ARCHIVE}-0073.tar Uploading ${ARCHIVE}-0073.tar guid:e6781de9-9709-4bf6-a019-b5fa4a1fb3c8 Moving ${ARCHIVE}-0073.tar.lst and ${ARCHIVE}-0073.tar.md5sum to directory "done"
For each tarball that is successfully processed the '${ARCHIVE}-<N>.tar.lst' file is moved to a separate directory 'done' . This way the 'upload-tar' script can be stopped and restarted at will, as it will continue processing '${ARCHIVE}-<N>.tar.lst' files until all have been moved to the 'done' directory.
For more details on the usage of the 'upload-tar' script , see DANS Job Scripts.
- Check the contents of the 'done' directory , especially the contents of the '${ARCHIVE}-<N>.tar.md5sum' files:
$ cat done/${ARCHIVE}-0001.tar.md5sum # Crome-0001.tar START 939131fac0d40184b5681e18f7b9856c Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_01/407_0001_01.MP4 03a1c56e97923f083bd554981a555a0f Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_01/407_0001_01.SMI 462e8ebbfe70127e46a9b447a14706f4 Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_01/407_0001_01I01.PPN 581c66b2da8852111d449300926c1d52 Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_01/407_0001_01M01.XML 9d5c4d62b4d410e4e1f44372cefe2132 Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_01/407_0001_01R01.BIM b0ca419552a5bb9004ff1849c7edbb3e Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_02/407_0002_01.MP4 68c36464994cebe84df5ff2b38320a32 Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_02/407_0002_01.SMI 6d72650cd3cfd3481a97af6d1aacfef7 Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_02/407_0002_01I01.PPN 1c1be7c0bd0d11302536663079c0fd69 Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_02/407_0002_01M01.XML 0924e3973a329a0824a400d8ac78d0b8 Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_02/407_0002_01R01.BIM d1d08d0818ffe4c5ae3ebb0d0ea349a0 Crome/crome_0045_zeljko_obradovic/crome_0045_zeljko_obradovic_03/407_0002_02.MP4 # Crome-0001.tar END
By combining all '${ARCHIVE}-0001.tar.md5sum' files a full list of all md5sums can be generated and compared to the output of an 'md5deep' command:
$ for i in done/${ARCHIVE}-*.tar.md5sum ; do grep -v '#' $i ; done | sort > total.md5sum
- After verifying that all '${ARCHIVE}-0001.tar.md5sum' files are correct, move them (or symlink them) to another directory 'checksums'. The latter directory is used by the 'compare-checksums' script from Phase 3: Verifying data.
$ mkdir checksums $ cp done/*.tar.md5sum checksums
- Once the upload has completed successfully you can proceed to Phase2: Compressing Data.