Difference between revisions of "Dans Data Compress"
Jump to navigation
Jump to search
| Line 16: | Line 16: | ||
After the jobs have been submitted to the grid you can track the status of these jobs using the '<tt>job-status</tt>' script: | After the jobs have been submitted to the grid you can track the status of these jobs using the '<tt>job-status</tt>' script: | ||
$ ./job-status | $ ./job-status | ||
| − | + | 00149 https://wms1.grid.sara.nl:9000/5To794h9GaRL-mPH9E7TpQ Status=Running | |
| − | + | 00150 https://wms1.grid.sara.nl:9000/W9YklyQ6MFeKsvzeHBVZXg Status=Running | |
| − | + | 00151 https://wms1.grid.sara.nl:9000/bmcj7Ja548EAHT4NmrNnPg Status=Running | |
| − | + | 00152 https://wms1.grid.sara.nl:9000/O0BB1AQuQQ8llmYEmAdIZQ Status=Running | |
| − | + | 00153 https://wms1.grid.sara.nl:9000/UdxeMCitwmPKKhoVXcB_ug Status=Running | |
| − | + | 00154 https://wms1.grid.sara.nl:9000/LlxU9gTVTMvxi2sSjsqr9g Status=Running | |
| − | + | ||
===Notes=== | ===Notes=== | ||
* the order in which the job are executed on the grid is not necessarily the same as the order in which they are submitted. | * the order in which the job are executed on the grid is not necessarily the same as the order in which they are submitted. | ||
Revision as of 14:23, 13 November 2012
After the tarballs have been uploaded to the grid, the next step is to compress the tarballs to so-called .tar.gz files to save space. This compression step is done on the grid itself, hence we need to submit a set of jobs to the grid. The 'compress-tar' script does this automatically.
$ ./compress-tar Found 84 tar balls in /grid/dans/soundbites Splitting into 6 jobs, start=1, end=84 Delegating proxy Submitting DANS job 149: https://wms1.grid.sara.nl:9000/5To794h9GaRL-mPH9E7TpQ Submitting DANS job 150: https://wms1.grid.sara.nl:9000/W9YklyQ6MFeKsvzeHBVZXg Submitting DANS job 151: https://wms1.grid.sara.nl:9000/bmcj7Ja548EAHT4NmrNnPg Submitting DANS job 152: https://wms1.grid.sara.nl:9000/O0BB1AQuQQ8llmYEmAdIZQ Submitting DANS job 153: https://wms1.grid.sara.nl:9000/UdxeMCitwmPKKhoVXcB_ug Submitting DANS job 154: https://wms1.grid.sara.nl:9000/LlxU9gTVTMvxi2sSjsqr9g
The 'soundbites' archive consists of 84 tarballs which need to be compressed. Each gridjob will compress 15 tarballs, hence a total of 6 jobs were submitted. After the jobs have been submitted to the grid you can track the status of these jobs using the 'job-status' script:
$ ./job-status 00149 https://wms1.grid.sara.nl:9000/5To794h9GaRL-mPH9E7TpQ Status=Running 00150 https://wms1.grid.sara.nl:9000/W9YklyQ6MFeKsvzeHBVZXg Status=Running 00151 https://wms1.grid.sara.nl:9000/bmcj7Ja548EAHT4NmrNnPg Status=Running 00152 https://wms1.grid.sara.nl:9000/O0BB1AQuQQ8llmYEmAdIZQ Status=Running 00153 https://wms1.grid.sara.nl:9000/UdxeMCitwmPKKhoVXcB_ug Status=Running 00154 https://wms1.grid.sara.nl:9000/LlxU9gTVTMvxi2sSjsqr9g Status=Running
Notes
- the order in which the job are executed on the grid is not necessarily the same as the order in which they are submitted.
- the grid job ids, starting with https://, look like URLs and that's exactly what they are. The user who submits the job can view the status of that job using a webbrowser, provided that the user's grid certificate is installed in that browser.
See the section DANS Job Scripts for more details on both the 'compress-tar' and the 'job-status' scripts.
Important notes
- The tarballs need to be compressed only once. It is done on the grid because it is much faster to do it that way, instead of on the DANS dataserver.
- After the .tar.gz files have been created and have been verified (see "Phase 3" for more details) the original tarballs need to be deleted from the grid storage. This is not done automatically. An effective commandline to delete all files named '.tar' from a single directory on the LFC is
lfc-ls /grid/dans/$ARCHIVE | grep ".tar$" > tarball-list
lcg-del -a `cat tarball-list`
CHECK!
The 'lcg-del' command can take quite some time to complete.