Ganga with AMAAthena

Introduction

This page will describe how to run AMAAthena jobs with Ganga on different computing platforms (Stoomboot, lxbatch, Grid).

Preparation

• Make sure you can run AMAAthena standalone on local desktop. Here are instructions about doing it at NIKHEF: Using Athena at NIKHEF
• Make sure you manage to submit HelloWorld jobs to different computing platforms. Here are instructions: Ganga: basic usage

Starting Ganga

Before starting Ganga, set CMT environment properly. Here is the example commands presuming that you have the setup scripts for CMT in $HOME/cmthome directory.

The first 2 lines in the following example is only needed if you want to use non-default DBRelease available in the release.

% export ATLAS_DB_AREA=/data/atlas/offline/db
% export DBRELEASE_OVERRIDE=7.8.1
% source $HOME/cmthome/setup.sh -tag=15.6.1,32
% source $TestArea/PhysicsAnalysis/AnalysisCommon/AMA/AMAAthena/cmt/setup.sh

Then start Ganga in $TestArea/PhysicsAnalysis/AnalysisCommon/AMA/AMAAthena/run directory.

% cd $TestArea/PhysicsAnalysis/AnalysisCommon/AMA/AMAAthena/run
% source /project/atlas/nikhef/ganga/etc/setup.[c]sh
% ganga --config-path=/project/atlas/nikhef/ganga/config/Atlas.ini.nikhef

Templates for quick start

There are ready-to-go Ganga scripts made for this tutorial. Following the instructions below to copy them into your AMAAthena run directory:

% cd $TestArea/PhysicsAnalysis/AnalysisCommon/AMA/AMAAthena/run
% cp /project/atlas/nikhef/ganga/tutorial_2010/job_options/* .
% cp /project/atlas/nikhef/ganga/tutorial_2010/ama_config/* .
% cp /project/atlas/nikhef/ganga/tutorial_2010/ganga_scripts/* .

• data_D3PD_simple2.conf: simple AMA configuration file for converting AOD into D3PD and dumping NTuples
• data_D3PD_filler_v4.conf: part of the AMA configuration file included by data_D3PD_simple2.conf
• ama_d3pd_maker.local.gpi: Ganga script for creating a ready-to-submit job to run on local desktop
• ama_d3pd_maker.pbs.gpi: Ganga script for creating a ready-to-submit job to run on Stoomboot
• ama_d3pd_maker.lcg.gpi: Ganga script for creating a ready-to-submit job to run on the grid through gLite WMS
• ama_d3pd_maker.panda.gpi: Ganga script for creating a ready-to-submit job to run on the grid through Panda

Apart from the files mentioned above, there are also few files prepared so that one can submit the jobs right away. They are listed below. In general, those files are prepared by user as mentioned in the Application pre-configuration below.

• AMAAthena_jobOptions_new.py: top-level AMAAthena job option file without AutoConfig/RecExCommon
• AMAAthena_jobOptions_AUTO.py: top-level AMAAthena job option file with AutoConfig/RecExCommon
• data_D3PD_simple2.py: user-level AMAAthena job option file converted from the AMA configuration file
• rundef.py: run definition job option file of AMAAthena

With all those files ready in the run directory, you can just load one of the Ganga script and submit the analysis jobs right away.

For example:

In [n]: execfile('ama_d3pd_maker.lcg.gpi')
In [n]: j.submit()

will create and submit a LCG job to generate D3PD files using AMAAthena.

The rest of the wiki will give you detail explanations on what has been done within those scripts.

Ganga jobs by yourself

Ganga job creation

The first step is to create a new (empty) job in Ganga, do

In [n]: j = Job()

and you can set job's name as

In [n]: j.name = 'my_ama_job'

Application configuration

Pre-configuration

AMAAthena is an Athena "Algorithm", so you can just use the Athena application object in Ganga to configure your AMAAthena job. However, there are steps to be done before setting the Athena application object in Ganga:

1. copy the top-level job option file of AMAAthena to your working directory:
   • with AutoConfig/RecExCommon

     % get_files -jo AMAAthena_jobOptions_AUTO.py

   • without AutoConfig/RecExCommon

     % get_files -jo AMAAthena_jobOptions_new.py

2. convert user-level AMA configuration file into a Athena job option file. For example, if you have a configuration file called data_D3PD_simple2.conf, do:

   % AMAConfigfileConverter data_D3PD_simple2.conf data_D3PD_simple2.py

3. create a AMA runtime definition job option called rundef.py and edit it as the following example:

   SampleName = 'data09_900GeV_00140541_MuonswBeam'
   ConfigFile = 'data_D3PD_simple2.py'
   FlagList = ''
   EvtMax = -1
   AMAAthenaFlags = ['DATA', 'TRIG']
   

   The variables in rundef.py is explained in the following:

   • SampleName: the user defined sample name. This name will be used in composing the AMA summary output files.
   • ConfigFile: the job option file name converted from the user-level configuration file (the output of step 2)
   • FlagList: legacy AMA flags
   • EvtMax: the maximum number of event to be processed in the job
   • AMAAthenaFlags: the additional AMA job option files to be included by the top-level AMA job option file. This is ignored if using AutoConfig/RecExCommon.

Configuration

Once you have the above steps done, you can proceed in Ganga to set up the Athena application:

In [n]: j.application = Athena()
In [n]: j.application.max_events = -1
In [n]: j.application.option_file += [ File('rundef.py'), File('AMAAthena_jobOptions_new.py'), File('data_D3PD_simple2.py') ]
In [n]: j.application.prepare()

The j.application.prepare() method automatically detects the input/output files by virtually run through the job option files given above. As the outputs are controlled internally by AMA, it's suggested to always add the following two lines to avoid possible confusion (e.g. with Panda). So if you run AMAAthena, always do the following lines after j.application.prepare().

In [n]: j.application.atlas_run_config['output']['outHist'] = False
In [n]: j.application.atlas_run_config['output']['alloutputs'] = []

Optional configurations

Override default DBRelease

By default, the job will pick up the DBRelease shipped together with the Athena release that you are using to run the job. In some cases, you may want to override it, for example, when you encounter the following error:

T_AthenaPoolCnv     ERROR poolToObject: caught error: 
FID "74981861-8AD2-DE11-95BD-001CC466D3D3" is not existing in the catalog 
( POOL : "PersistencySvc::UserDatabase::connectForRead" from "PersistencySvc" )

For local jobs, you can simply set the following lines in order to force the job to load a proper DBRelease version from certain area, presuming that the DBRelease area is on a shared file system:

In [n]: j.application.atlas_environment = ['ATLAS_DB_AREA=/data/atlas/offline/db', 'DBRELEASE_OVERRIDE=7.8.1']

For grid jobs, you cannot do that as you don't know the path on the remote machine in advance. To achieve it, one needs to do:

In [n]: j.application.atlas_dbrelease   = 'ddo.000001.Atlas.Ideal.DBRelease.v070801:DBRelease-7.8.1.tar.gz'
In [n]: j.application.atlas_environment =['DBRELEASE_OVERRIDE=7.8.1']

where the j.application.atlas_dbrelease points the job to download the DBRelease tarball "DBRelease-7.8.1.tar.gz" in the ATLAS dataset "ddo.000001.Atlas.Ideal.DBRelease.v070801"; while j.application.atlas_environment enforces the Athena job to use its version of DBRelease instead of the default one.

InputDataset configuration

It is encouraged to enable FileStager with your analysis job as it has been proved to be more efficient in majority of cases. To do so, there are two InputDataset object to use in Ganga depending on where you submit your jobs.

StagerDataset for local jobs

Presuming you have a dataset "data09_900GeV.00140541.physics_MuonswBeam.merge.AOD.f170_m268" located at "NIKHEF-ELPROD_DATADISK", you can set the inputdata attribute of the Ganga job object as the following:

In [n]: config.DQ2.DQ2_LOCAL_SITE_ID = 'NIKHEF-ELPROD_DATADISK'
In [n]: j.inputdata = StagerDataset()
In [n]: j.inputdata.type = 'DQ2'
In [n]: j.inputdata.dataset += [ 'data09_900GeV.00140541.physics_MuonswBeam.merge.AOD.f170_m268' ]

Remarks

• Always use StagerDataset with LSF and PBS backends for local jobs.
• StagerDataset is aimed for copying files from local storage. You need to find the local location of the dataset in terms of DDM site name and set it properly in Ganga by config.DQ2.DQ2_LOCAL_SITE_ID

You can also use StagerDataset to access the dataset files already existing on local disk. The following example assumes that you have dataset files already sitting in the directory /data/atlas3/users/hclee/data/data09_900GeV.00140541.physics_MuonswBeam.merge.AOD.f170_m268

In [n]: j.inputdata = StagerDataset()
In [n]: j.inputdata.type = 'LOCAL'
In [n]: j.inputdata.dataset = ['/data/atlas3/users/hclee/data/data09_900GeV.00140541.physics_MuonswBeam.merge.AOD.f170_m268']

DQ2Dataset for grid jobs

Presuming you want to run on a dataset data09_900GeV.00140541.physics_MuonswBeam.merge.AOD.f170_m268 on the grid, you can set the InputDataset object as the following in Ganga:

In [n]: j.inputdata = DQ2Dataset()
In [n]: j.inputdata.dataset += [ 'data09_900GeV.00140541.physics_MuonswBeam.merge.AOD.f170_m268' ]
In [n]: j.inputdata.type = 'FILE_STAGER'

Remarks

• Always use DQ2Dataset with Panda and LCG backends.

Splitter configuration

The examples below ask each subjob to process on 2 files in maximum.

StagerJobSplitter for StagerDataset

In [n]: j.splitter = StagerJobSplitter()
In [n]: j.splitter.numfiles = 2

DQ2JobSplitter for DQ2Dataset

In [n]: j.splitter = DQ2JobSplitter()
In [n]: j.splitter.numfiles = 2

Backend (platform) configuration

You should be able to switch to different computing platform (Backend in Ganga terminology) by simply change the backend attribute of a job object. The available backends are:

• Local: for running jobs locally right on your desktop
• PBS: for running jobs on a PBS-based computer cluster (e.g. the Stoomboot)
• LSF: for running jobs on a LSF-based computer cluster (e.g. lxbatch@CERN)
• LCG: for running jobs on the grid (EGEE sites), jobs are brokered by gLite Workload Management System (WMS)
• Panda: for running jobs on the grid (EGEE, OSG, NorduGrid sites), jobs are brokered by Panda

For example, to switch to submit jobs to the grid through Panda:

In [n]: j.backend = Panda()

Local

Ask the job to be executed locally right on the desktop. This is the default backend of a newly created Ganga job.

In [n]: j.backend = Local()

PBS

ask job to be submitted to the "qlong" of the Stoomboot.

In [n]: j.backend = PBS()
In [n]: j.queue = 'qlong'

LSF

Ask job to be submitted to the "1nh" (1 hour) queue on the lxbatch@CERN. You need to run it from lxplus@CERN.

In [n]: j.backend = LSF()
In [n]: j.queue = '1nh'

LCG

Ask job to be submitted to a EGEE site wherever the dataset given above is available and with the queue supporting 12 hours long jobs.

In [n]: j.backend = LCG()
In [n]: j.backend.requirements.cloud = 'ALL'
In [n]: j.backend.requirements.walltime = 720

Panda

Ask the job to be submitted to Panda and then being brokered to whatever site being able to process this job in the "US" cloud.

In [n]: j.backend = Panda()
In [n]: j.backend.libds = ''
In [n]: j.backend.requirements.cloud = 'US'

Submit the job

This is as simple as you can imagine:

In [n]: j.submit()

Next step

For more details about different Athena use cases, you can refer to the following twiki to get more information:

• Full GangaAtlas tutorial: the up-to-date official tutorial wiki for GangaAtlas
• GangaAtlas FAQ: the Q&A page with questions collected by the global user analysis support team

Or you are encouraged to subscribe to atlas-dist-analysis-help forum where you can send GangaAtlas related issues to ask for supports from experts and/or global users. Take it as the HelpDesk of the global user analysis support.