Difference between revisions of "DANS Data Management"
| Line 11: | Line 11: | ||
===DANS Workflow phase1: Uploading data=== | ===DANS Workflow phase1: Uploading data=== | ||
| − | [[File: | + | [[File:DANS_Workflow1.png]] |
Data can be uploaded from 'DANS archief' using the [[Dans Data Upload]] procedure. | Data can be uploaded from 'DANS archief' using the [[Dans Data Upload]] procedure. | ||
Revision as of 13:01, 10 October 2012
DANS Data Management
One of the main goals of DANS is to to longer-term management of a variety of data. The goal of the DANS Data Management project is to allow DANS to reliably back up their data onto BiG Grid storage resources.
'Reliably' means that the consistency of the data can be verified by DANS engineers at any given time. For this, a set of tools and procedures have been developed to allow DANS engineers to
- upload data from DANS to the grid (in 'tarball' format)
- compress the data stored on the grid
- verify the MD5 checksums of all data stored inside the tarballs on the grid
DANS Workflow phase1: Uploading data
Data can be uploaded from 'DANS archief' using the Dans Data Upload procedure.
Compressing data
After successfully uploading the data , the data stored on the grid can be compressed using the Dans Data Compress procedure. This is primarily done to save disk space on the grid storage infrastructure. It will also help in improving data download speeds under certain circumstances.
Verifying data
Periodically the integrity of the data stored on "the grid" needs to be verified. For this, an extensive 'md5sum' verification procedure is available. Read more about it in the Dans Data Verify procedure.