How to setup the qsub tunnel for use with Ganga and the LHCb software

General information

The qsub tunnel is a new service supplied by the PDP group that allows Nikhef users to submit directly to the "medium" queue of the Nikhef Tier1. For general information and instructions, see this twiki page: qsub tunnel wiki pagina.

First steps

• Make sure that you are included in the list of people who are allowed to submit jobs using the qsub tunnel by asking the PDP group if you are.
• Make sure that you have a grid proxy if you want to use the Oracle conditions database or if you want to access files on the Nikhef or Sara storage elements.
• Make sure that you are using Ganga version 5.5.16 (v505r16)

Preparing your environment on the submission machine

• To make your grid proxy available on the worker node, set the following environment variable before you create your proxy. It is easiest to put this in your login scripts.
• The pbs server must also be known (Set this back to allier.nikhef.nl if you want to submit to stoomboot):
• For bash-like shells:

export X509_USER_PROXY=/data/tunnel/user/yourusername/.x509tmp
export PBS_SERVER=stro.nikhef.nl

• For csh-like shells:

setenv X509_USER_PROXY /data/tunnel/user/yourusername/.x509tmp
setenv PBS_SERVER stro.nikhef.nl

Preparing the environment for the worker nodes

The LHCb software available on the Tier1 worker nodes is not the same installation as is available in the /project/bfys area. The worker nodes have all versions available which are distributed to the Grid by the LHCb production and software management team. Login scripts need to be prepared:

• Create a file called .bash_profile in /data/tunnel/user/yourusername.
• Put the following lines into it:

if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

• Create a file named .bashrc in /data/tunnel/user/yourusername.
• Put the following lines into it:

export X509_USER_PROXY=/data/tunnel/user/yourusername/.x509tmp
. $VO_LHCB_SW_DIR/group_login.sh

Ganga configuration

Edit your ~/.gangarc file and look for the [PBS] section, put in the following bits:

postexecute =
 import os
 env = os.environ
 tmpdir = None
 if "TMPDIR" in env: tmpdir = env[ "TMPDIR" ].rstrip( "/" )
 else: tmpdir = "."
 jobnumid = env["PBS_JOBID"]
 dir = "%s/%s" % ( tmpdir, jobnumid )
 os.chdir( tmpdir )
 os.system("rm -rf %s" % dir )

preexecute =
 import os
 env = os.environ
 jobnumid = env["PBS_JOBID"]
 tmpdir = None
 if "TMPDIR" in env: tmpdir = env[ "TMPDIR" ].rstrip( "/" )
 else: tmpdir = "."
 dir = "%s/%s" % ( tmpdir, jobnumid )
 os.system("mkdir %s" % dir )
 os.chdir( "%s" % dir )
 os.environ["PATH"]+=":."

submit_res_pattern = ^(?P<id>\d*)\.\w+\.nikhef\.nl

When using Ganga

• When submitting jobs, use the PBS backend with the following settings:

yourjob.backend = PBS( queue = 'medium' )