Difference between revisions of "DANS Data Management"
Jump to navigation
Jump to search
| Line 3: | Line 3: | ||
One of the main goals of [http://www.dans.knaw.nl DANS] is to to longer-term management of a variety of data. The goal of the DANS Data Management project is to allow DANS to reliably back up their data onto BiG Grid storage resources. | One of the main goals of [http://www.dans.knaw.nl DANS] is to to longer-term management of a variety of data. The goal of the DANS Data Management project is to allow DANS to reliably back up their data onto BiG Grid storage resources. | ||
| − | 'Reliably' means that the consistency of the data can be verified by DANS engineers at any given time. For this, a set of tools and procedures | + | 'Reliably' means that the consistency of the data can be verified by DANS engineers at any given time. For this, a set of tools and procedures have been developed to allow DANS engineers to |
* upload data from DANS to the grid (in 'tarball' format) | * upload data from DANS to the grid (in 'tarball' format) | ||
* compress the data stored on the grid | * compress the data stored on the grid | ||
* verify the MD5 checksums of all data stored inside the tarballs on the grid | * verify the MD5 checksums of all data stored inside the tarballs on the grid | ||
| + | |||
| + | ===Uploading data=== | ||
| + | |||
| + | Data can be uploaded from 'twister11.dans.knaw.nl' using the following procedure: | ||
| + | * create a new directory with the name of the archive. As an example we use the 'Crome' archive: | ||
| + | mkdir -p ~/dans/Crome/ | ||
| + | * In this directory create another directory with the same name; this directory will contain the list of files and directories that need to be uploaded | ||
| + | cd ~/dans/Crome | ||
| + | mkdir Crome | ||
| + | * copy over the scripts from repository | ||
| + | cp -a ~/dans/scripts/* . | ||
| + | |||
| + | |||
| + | ===Compressing data=== | ||
| + | |||
| + | |||
| + | ===Verifying data=== | ||
Revision as of 09:42, 10 May 2012
DANS Data Management
One of the main goals of DANS is to to longer-term management of a variety of data. The goal of the DANS Data Management project is to allow DANS to reliably back up their data onto BiG Grid storage resources.
'Reliably' means that the consistency of the data can be verified by DANS engineers at any given time. For this, a set of tools and procedures have been developed to allow DANS engineers to
- upload data from DANS to the grid (in 'tarball' format)
- compress the data stored on the grid
- verify the MD5 checksums of all data stored inside the tarballs on the grid
Uploading data
Data can be uploaded from 'twister11.dans.knaw.nl' using the following procedure:
- create a new directory with the name of the archive. As an example we use the 'Crome' archive:
mkdir -p ~/dans/Crome/
- In this directory create another directory with the same name; this directory will contain the list of files and directories that need to be uploaded
cd ~/dans/Crome mkdir Crome
- copy over the scripts from repository
cp -a ~/dans/scripts/* .