Difference between revisions of "DCache"

From PDP/Grid Wiki
Jump to navigationJump to search
 
(24 intermediate revisions by one other user not shown)
Line 1: Line 1:
 +
'''This page is being moved to a new system, please don't make edits here'''
 +
 
= Introduction =
 
= Introduction =
  
[http://www.dcache.org dCache] is filesystem designed for storing large amounts of data. The data is stored amongst many server nodes, accessed through a single virtual filesystem by a variety of methods. The dCache filesystem used on stoomboot can be accessed via on nfs mount point from any of the stoomboot interactive or worker nodes. All file stored in dCache can be found under the mount point <code>/dcache</code>.
+
[http://www.dcache.org dCache] is filesystem designed for storing large amounts of data. The data is stored amongst many server nodes, accessed through a single virtual filesystem by a variety of methods. The dCache filesystem is available on stoomboot and can be found under the directory <code>/dcache</code>.
 +
 
 +
There are no backups of made of the dCache file system.
 +
 
 +
Files written to dCache are fixed and not changeable, see [[#Immutable Files]]. Other than this limitation dCache can be used pretty much as a normal file system.
 +
 
 +
Issues and problems with the file system should be sent to <code>stbc-admin[AT]nikhef[DOT]nl</code>.
 +
 
 +
= Getting Started =
 +
 
 +
To get a storage directory created for yourself please send a request to <code>stbc-admin[AT]nikhef[DOT]nl</code>. Please include
 +
which group you are working with and where in the dcache directory hierarchy to create your directory as the directory organisation within
 +
each group subdirectory is up to each group. And finally don't forget a name for the new directory.
  
= Directory Allocation =
+
= Access =
  
To get a storage directory created for yourself please send a request to <code>stbc-admin@nikhef.nl</code>. Don't forget to include which group you are working with as that's used to determine were in the storage hierarchy your directory is created.
+
dCache can be accessed from any of the stoomboot nodes, both the interactive and compute nodes. The file system is mounted via an nfs mount point <code>/dcache</code>. This mount point is the same across all the stoomboot nodes. Running jobs can also read and write to dCache as jobs run under the identity of the user who submitted the job and so have the same access rights as the same user in an interactive shell. '''There is no access to the /dcache mount from outside stoomboot'''.
  
= Usage =
+
= Immutable Files =
  
dCache can be used pretty much as a normal file system. Access to files is controlled by the usual unix file permissions and the main difference is that you can not change the contents of a file stored in dCache once the file is written. So for example you can't overwrite a file:
+
'''Once a file is written into dCache the contents of the file can not be changed.'''
 +
 
 +
This means a file '''CAN NOT''' be overwritten:
  
 
<code>
 
<code>
Line 18: Line 34:
 
</code>
 
</code>
  
or append to a file:
+
and '''CAN NOT''' be appended to:
  
 
<code>
 
<code>
Line 27: Line 43:
 
</code>
 
</code>
  
and when deleting a file that file is always treated as being write protected (even if the writable bit is set):
+
and when deleting a file is always treated as being write protected (even if the writable bit is set):
  
 
<code>
 
<code>
Line 37: Line 53:
 
</code>
 
</code>
  
These limitations mean that dCache is ''not'' suited for such things as compiling code on, storing large files being actively edited (e.g. word documents) or storing files under a version control system.
+
These limitations mean that dCache is '''NOT''' suited for such things as compiling code on, storing large files being actively edited (e.g. word documents) or storing files under a version control system.
 +
 
 +
= No Backups =
 +
 
 +
There are '''NO BACKUPS''' of files stored on the dCache file system.
  
 
= Storage Space Organisation =
 
= Storage Space Organisation =
  
The total storage space on the dCache storage nodes is about 550 TB. This space is divided up amongst the various groups using the system and is organised into a directory hierarchy:
+
The total storage space on the dCache storage nodes is about 550 TB. Each group has a directory under the main dcache directory:
 +
 
 
<code>
 
<code>
 
  <nowiki>
 
  <nowiki>
[andrewp@stbc-i4 ~]$ ll /dcache/
+
[stbc-i4 ~]$ ll /dcache/
 
total 2
 
total 2
 
drwxr-xr-x. 4 root alice    512 Apr  1 11:46 alice
 
drwxr-xr-x. 4 root alice    512 Apr  1 11:46 alice
 
drwxr-sr-x. 5 root atlas    512 May  4 09:48 atlas
 
drwxr-sr-x. 5 root atlas    512 May  4 09:48 atlas
drwxrwsr-x. 3 root datagrid 512 Nov 13  2015 datagrid
+
drwxrwsr-x. 3 root datagrid 512 Nov 13  2015 datagrid</nowiki>
drwxr-xr-x. 3 root root    512 Jan 21 16:10 test</nowiki>
 
 
</code>
 
</code>
  
Line 59: Line 79:
 
total 1
 
total 1
 
drwxr-sr-x. 1094 andrewp datagrid 512 May  4 09:59 andrewp</nowiki>
 
drwxr-sr-x. 1094 andrewp datagrid 512 May  4 09:59 andrewp</nowiki>
 +
</code>
 +
 +
Storage space is assigned to each group under there group directory so all people within a group share the same group storage space allocation.
 +
 +
''TO-DO: df doesn't give useful results for dcache so a web page with current data usage will be setup''
 +
 +
= Remote access =
 +
 +
Remote access to the dCache storage using grid certificate authorisation is also available. To register your grid certificate for remote access please send your grid certificate's DN to <code>stbc-admin[AT]nikhef[DOT]nl</code> with a request for remote access to dcache. To get the DN of a certificate from the certificate file, run the following command:
 +
 +
<code>
 +
openssl x509 -in __cert_file__ -noout -subject
 +
</code>
 +
 +
Replacing <code>__cert_file__</code> with the file name of the certificate.
 +
 +
== xrootd ==
 +
 +
Once your certificate DN is registered as described above, the xrootd door can be accessed at <code>dcache.nikhef.nl</code>.
 +
The directory paths are the same as for nfs access. So for example the directory <code>/dcache/test</code> on stoomboot can
 +
be seen via xrootd at <code>root://dcache.nikhef.nl/dcache/test</code>. To list the directory use the command <code>xrd</code>
 +
for example:
 +
 +
<code>
 +
[bosui ~]$ xrdfs dcache.nikhef.nl ls -l /dcache/test
 +
</code>
 +
 +
OR
 +
 +
<code>
 +
[bosui ~]$ xrd dcache.nikhef.nl dirlist /dcache/test
 +
</code>
 +
 +
And to copy a file:
 +
 +
<code>
 +
[bosui ~]$ xrdcp root://dcache.nikhef.nl/dcache/test/test1 test1
 +
</code>
 +
 +
== grid-ftp ==
 +
 +
Similarly to the xrootd door, once your certificate DN is registered as described above, the xrootd door can be accessed at <code>dcache.nikhef.nl</code>. To list a directory use the command <code>globus-url-copy -list</code>, for example:
 +
 +
<code>
 +
[bosui ~]$ globus-url-copy -list gsiftp://dcache.nikhef.nl/dcache/test/
 +
</code>
 +
 +
And to copy a file:
 +
 +
<code>
 +
[bosui ~]$ globus-url-copy gsiftp://dcache.nikhef.nl/dcache/test/test1 test1
 
</code>
 
</code>

Latest revision as of 10:03, 14 February 2023

This page is being moved to a new system, please don't make edits here

Introduction

dCache is filesystem designed for storing large amounts of data. The data is stored amongst many server nodes, accessed through a single virtual filesystem by a variety of methods. The dCache filesystem is available on stoomboot and can be found under the directory /dcache.

There are no backups of made of the dCache file system.

Files written to dCache are fixed and not changeable, see #Immutable Files. Other than this limitation dCache can be used pretty much as a normal file system.

Issues and problems with the file system should be sent to stbc-admin[AT]nikhef[DOT]nl.

Getting Started

To get a storage directory created for yourself please send a request to stbc-admin[AT]nikhef[DOT]nl. Please include which group you are working with and where in the dcache directory hierarchy to create your directory as the directory organisation within each group subdirectory is up to each group. And finally don't forget a name for the new directory.

Access

dCache can be accessed from any of the stoomboot nodes, both the interactive and compute nodes. The file system is mounted via an nfs mount point /dcache. This mount point is the same across all the stoomboot nodes. Running jobs can also read and write to dCache as jobs run under the identity of the user who submitted the job and so have the same access rights as the same user in an interactive shell. There is no access to the /dcache mount from outside stoomboot.

Immutable Files

Once a file is written into dCache the contents of the file can not be changed.

This means a file CAN NOT be overwritten:

[stbc-i1 /dcache/test]$ echo 123 > test.file
[stbc-i1 /dcache/test]$ echo 1234 > test.file
-bash: test.file: Permission denied

and CAN NOT be appended to:

[stbc-i2 /dcache/test]$ echo 123 > test.file
[stbc-i2 /dcache/test]$ echo 4 >> test.file
-bash: test.file: Permission denied

and when deleting a file is always treated as being write protected (even if the writable bit is set):

[stbc-i2 /dcache/test]$ ll test.file
-rw-r--r--. 1 andrewp datagrid 4 May  4 09:57 test.file
[stbc-i2 /dcache/test]$ rm test.file
rm: remove write-protected regular file `test.file'? y

These limitations mean that dCache is NOT suited for such things as compiling code on, storing large files being actively edited (e.g. word documents) or storing files under a version control system.

No Backups

There are NO BACKUPS of files stored on the dCache file system.

Storage Space Organisation

The total storage space on the dCache storage nodes is about 550 TB. Each group has a directory under the main dcache directory:

[stbc-i4 ~]$ ll /dcache/
total 2
drwxr-xr-x. 4 root alice    512 Apr  1 11:46 alice
drwxr-sr-x. 5 root atlas    512 May  4 09:48 atlas
drwxrwsr-x. 3 root datagrid 512 Nov 13  2015 datagrid

User storage directories are under the appropriate group directory:

[stbc-i4 ~]$ ll /dcache/datagrid/
total 1
drwxr-sr-x. 1094 andrewp datagrid 512 May  4 09:59 andrewp

Storage space is assigned to each group under there group directory so all people within a group share the same group storage space allocation.

TO-DO: df doesn't give useful results for dcache so a web page with current data usage will be setup

Remote access

Remote access to the dCache storage using grid certificate authorisation is also available. To register your grid certificate for remote access please send your grid certificate's DN to stbc-admin[AT]nikhef[DOT]nl with a request for remote access to dcache. To get the DN of a certificate from the certificate file, run the following command:

openssl x509 -in __cert_file__ -noout -subject

Replacing __cert_file__ with the file name of the certificate.

xrootd

Once your certificate DN is registered as described above, the xrootd door can be accessed at dcache.nikhef.nl. The directory paths are the same as for nfs access. So for example the directory /dcache/test on stoomboot can be seen via xrootd at root://dcache.nikhef.nl/dcache/test. To list the directory use the command xrd for example:

[bosui ~]$ xrdfs dcache.nikhef.nl ls -l /dcache/test

OR

[bosui ~]$ xrd dcache.nikhef.nl dirlist /dcache/test

And to copy a file:

[bosui ~]$ xrdcp root://dcache.nikhef.nl/dcache/test/test1 test1

grid-ftp

Similarly to the xrootd door, once your certificate DN is registered as described above, the xrootd door can be accessed at dcache.nikhef.nl. To list a directory use the command globus-url-copy -list, for example:

[bosui ~]$ globus-url-copy -list gsiftp://dcache.nikhef.nl/dcache/test/

And to copy a file:

[bosui ~]$ globus-url-copy gsiftp://dcache.nikhef.nl/dcache/test/test1 test1