VO-specific software and modules

Introduction

VOs sometimes have a need to deploy their own software at a particular site. For this the Software Group Manager VOMS role is available. Not all VOs have this role and not all VOs with this role are supported at each site. If you are a Software Group Manager for a particular VO then the VOMS role will have been assigned to you and you can generate a SGM-specific proxy.

For these Software Group Managers this HOWTO will explain how you can deploy your own software and add your own module to the available modules for the modules command.

VOMS Role

The HEP VOs have an SGM role, usually of the form

 /Role=lcgadmin

However, in this HOWTO the VO vlemed was chosen as the example. This VO has a role

/Role=sgm

available, which gives users who possess that role the right to install software in the VO specific software area. To generate an SGM-proxy use

$ voms-proxy-init --voms vlemed:/vlemed/Role=sgm

You can view your current VOMS roles using

$ voms-proxy-info -all | sed 's/^/ /'
subject   : /O=dutchgrid/O=users/O=nikhef/CN=Jan Just Keijser/CN=proxy
issuer    : /O=dutchgrid/O=users/O=nikhef/CN=Jan Just Keijser
identity  : /O=dutchgrid/O=users/O=nikhef/CN=Jan Just Keijser
type      : proxy
strength  : 1024 bits
path      : /tmp/x509up_u7651
timeleft  : 11:59:33
=== VO vlemed extension information ===
VO        : vlemed
subject   : /O=dutchgrid/O=users/O=nikhef/CN=Jan Just Keijser
issuer    : /O=dutchgrid/O=hosts/OU=sara.nl/CN=voms.grid.sara.nl
attribute : /vlemed/Role=sgm/Capability=NULL
attribute : /vlemed/Role=NULL/Capability=NULL
timeleft  : 11:59:33
uri       : voms.grid.sara.nl:30003

VO-specific software area

The VO-specific software area is denoted using the environment variable

VO_<VO>_SW_DIR

e.g. for vlemed it is VO_VLEMED_SW_DIR.

At Nikhef the $VO_VLEMED_SW_DIR has the following permissions:

$ cd $VO_VLEMED_SW_DIR
$ ls -ald .
drwxrwsr-t 4 root vlemedsm 4096 Dec 11 16:17 .

This means the directory is writable only for members of the Unix group vlemedsm. When submitting a job to Nikhef using the SGM-proxy a special pool-account is chosen:

$ id vlemsm00
uid=70960(vlemsm00) gid=2058(vlemedsm) groups=2024(vlemed),2058(vlemedsm)

So the SGM-proxy causes a mapping to a pool-account with access rights to install software in $VO_VLEMED_SW_DIR.

Just to verify: a "regular" vlemed poolaccount does not have these permissions:

$ id vlemed00
uid=53200(vlemed00) gid=2024(vlemed) groups=2024(vlemed)

Note Checking this before proceeding is a very good practice, as sites misconfigurations at this level occur quite frequently.

Building the software

Before deploying new software we need to build and package it first on a local system:

• Notes on building
  • The target platform is a RHEL5 64bit compatible system. The easiest approach is to build the software on such a system (e.g. ui.grid.sara.nl)
  • It is not possible to hardcode paths into the software: the VO_<VO>_SW_DIR points to different directories on different clusters. Try to ensure that your software is relocatable using environment variables. Most software allows for this.
• Notes on packaging
  • In this HOWTO all software is packaged as a single tarball
  • The installation job then only has to download and extract the tarball in the right location.

For this HOWTO guide we will package a non-existing version 5.0 of the fsl package. The following directory structure has been set up for this package:

<local-dir>/fsl-5.0
<local-dir>/fsl-5.0/bin
<local-dir>/fsl-5.0/bin/fsl

where fsl-5.0/bin/fsl is a dummy script.

Packaging this software is very easy

$ cd <local-dir>
$ tar czvf ~/my-fsl-5.0.tar.gz fsl-5.0

We then upload this tarball to a public webserver (or gridftp server):

$ scp ~/my-fsl-5.0.tar.gz ~/public_html

Deploying the software

To deploy the software we use this JDL file:

Executable = "deploy.sh";
Stdoutput = "stdout";
StdError = "stderr";
InputSandbox = {"deploy.sh"};
OutputSandbox = {"stdout","stderr"};

with this deploy.sh script:

#!/bin/bash
# Set a sane umask, just in case
umask 0022

cd $VO_VLEMED_SW_DIR
wget http://www.nikhef.nl/~janjust/my-fsl-5.0.tar.gz || exit 1
tar xzf my-fsl-5.0.tar.gz || exit 2
# make sure the permissions are right
chmod -R u+rw,g+rw,o+r-w fsl-5.0
# List the directory afterwards for inspection
ls -l

Run this job with an SGM-proxy like any other job at the cluster where you want to install it, e.g.

$ glite-wms-job-submit -d janjust.sgm -r gazon.nikhef.nl:2119/jobmanager-pbs-short deploy.jdl

Wait for completion and check the stdout and stderr files for any errors.

Adding a module

All VO-specific modules need to be installed in

VO_<VO>_SW_DIR/modules

Only modulefiles installed in this directory will be automagically picked up by the worker node login scripts.

Here is the listing of a sample modulefile

#%Module1.0#####################################################################
##
## fsl 5.0 modulefile
##

proc ModulesHelp { } {
        global fslversion

        puts stderr "\tSet up the environment for FSL"
        puts stderr "\n\tVersion $fslversion\n"
}

module-whatis   "sets FSL environment"

# for Tcl script use only
set     fslversion      5.0

set fsldir      "$env(VO_VLEMED_SW_DIR)/fsl-5.0"

prepend-path    PATH    "$fsldir/bin"
setenv          FSLDIR  "$fsldir"

which adds support for a (non-existing) version 5.0 of the package fsl.

Note especially the

$env(VO_VLEMED_SW_DIR)

in this modulefile: all software needs be installed (and thus, relocatable!) relative to this directory. The $env(...) command is the modules (TCL) method to import an environment variable.

It is by far the easiest to develop and test new modulefiles on a local system. After making sure that the modulefile works you can then include it in the software tarball or create a new tarball, e.g. mypackage-X.Y-module.tar.gz.

For this HOWTO we will do the latter:

$ cd <local-dir>
$ ls 
fsl-5.0
$ mkdir -p modules/fsl
$ cp .../my-new-module-file modules/fsl/5.0

Before packaging it we test it first

$ export MODULEPATH=$PWD/modules
$ module avail
$ module load fsl/5.0

When finished we package it:

$ tar czf ~/my-fsl-5.0-module.tar.gz modules
$ cp ~/my-fsl-5.0-module.tar.gz ~/public_html

And we deploy it in exactly the same manner as the actual software (see above).

Using the module

In order to use our shiny new module we launch a normal job. Here is a listing of a sample job which checks the MODULEPATH parameter and a few other modules-related things:

#!/bin/bash -l

id
ls -l $VO_VLEMED_SW_DIR
echo "MODULEPATH=$MODULEPATH"
echo "## Listing available modules:"
module avail 2>&1
echo "## Loading fsl"
module load fsl 2>&1
echo "## Which modules are now loaded:"
module list 2>&1

module unload fsl 2>&1
echo "## Loading MY fsl module"
module load fsl/5.0 2>&1
echo "## Which modules are now loaded:"
module list 2>&1

echo "## which fsl:"
which fsl

The output of this job when run at the Nikhef cluster is:

uid=53207(vlemed07) gid=2024(vlemed) groups=2024(vlemed)
total 16
drwxr-sr-x 3 vlemsm05 vlemedsm 4096 Dec 11 16:17 fsl-5.0
drwxr-sr-x 4 vlemsm05 vlemedsm 4096 Dec 11 16:22 modules
MODULEPATH=/opt/vl-e/modules/Modules/versions:/opt/vl-e/modules/Modules/$MODULE_VERSION/modulefiles:
           /etc/opt/vl-e/modulefiles::/data/esia/vlemed/modules
## Listing available modules:
---------------------- /opt/vl-e/modules/Modules/versions ----------------------
3.2.6

----------------- /opt/vl-e/modules/Modules/3.2.6/modulefiles ------------------
dot         module-cvs  module-info modules     null        use.own

-------------------------- /etc/opt/vl-e/modulefiles ---------------------------
fsl/4.0              javagat/1.7.1        pl/5.6.64
fsl/4.0.4            lam/7.1              r/2.6
fsl/4.1              lam/7.1.4            r/2.6.2
fsl/4.1.4            mcr/7.11             r/2.9
gat/1.8              mesa3d/6.4           r/2.9.2
gat/1.8.2            mesa3d/6.4.2         rmpi/0.5
graphviz/2.18        mpitb/2.1            srb/3.4
gt/4.0               mpitb/2.1.73         srb/3.4.2
gt/4.0.8             mricro/1.39          vlet/1.0
ibis/1.4             mricro/1.39.3        vlet/1.0.2
itk/3.14             octave/2.1           vtk/4.4
itk/3.14.0           octave/2.1.73        vtk/4.4.2
itk/3.4              openmpi/1.3.2        vtk/5.4
itk/3.4.0            openrdf-sesame/2.0   vtk/5.4.0
java/1.6             openrdf-sesame/2.0.1 weka/3.4
javagat/1.7          pl/5.6               weka/3.4.12

-------------------------- /data/esia/vlemed/modules ---------------------------
fsl/5.0       mypackage/5.0
## Loading fsl
## Which modules are now loaded:
Currently Loaded Modulefiles:
  1) fsl/4.1.4
## Loading MY fsl module
## Which modules are now loaded:
Currently Loaded Modulefiles:
  1) fsl/5.0
## which fsl:
/data/esia/vlemed/fsl-5.0/bin/fsl

Notes

• when adding a new version (e.g. 5.0) of an existing system-wide package (e.g. fsl) this version does not become the default, as can be seen in the script output. To use the new version you have to explicitly specify the version number.
• Completely new packages are picked up automatically:

$ module load mypackage
$ module list
Currently Loaded Modulefiles:
  1) fsl/4.1.4       2) mypackage/5.0