StructNtuple making

Introduction

User analysis of NIKHEF Top group requires data format from AOD/D2PD to so-called StructNtuple. The intermittent by-product is so-called AMAEvent. The size ratio of AOD:AMAEvent:StructNtuple is roughly 100:10:1.

The conversion from AOD/D2PD to AMAEvent requires Athena framework and as the AOD/D2PD are widely distributed on the Grid; while the step from AMAEvent to StructNtuple needs only ROOT environment (in principle). For the collision data, we decided to start D2PD instead of AOD as the non-interesting events for Top analysis have been skimmed, slimmed and ...; while for MC samples, one needs to start from AOD.

Here after, the technical steps to go through for the AMAEvent/StructNtuple generations are described together with instructions to take actions on those steps.

The job submission tools is based on Ganga using the WMS and/or Panda for job brokering to the Grid.

Workflow

Task definition

{
'input'  : 'group.phys-top.D2PD_rel15.6.11.3.1_data10_7TeV.00162620.physics_Egamma.merge.AOD.f287_m588.20100914_00_D2AODM_TOPEL',
'amaconfig' : './config/Structs_data10_7TeV_Common.conf', 
'amaconfig_flags' : ['LepFlavor=11.', 'NTupleType=0'],
'ama_userarea' : 'amaathena_trunk_20100901.tar.gz',

'backend' : 'PANDA',
'cloud' : 'NL',
'site' : '', 

'metadata' : {  'atlas_grlinfo'  : {'url' : 'https://atlas-top-grls.web.cern.ch/atlas-top-grls/xmlGRL/DetStatus-v03-pass1-analysis-2010F/top_allchannels_7TeV_DetStatus_v03_pass1_analysis_2010F.xml',
                                                      'name' : 'top_allchannels_7TeV_DetStatus_v03_pass1_analysis_2010F'},
                                      'atlas_project'  : 'TopPhys',
                                      'run'                  : '162620', 
                                      'algorithm'       : 'none', 
                                      'signal'              : 'data10_7TeV.period.physics_Sample5',
                                      'tag'                   : 'f287_m588.20100914_00_D2AODM_TOPEL', 
                                      'ama_flags'       : ['AUTODATA', 'IGNORETAGDIFF', 'AUTOTOPINPUTS', 'JES', 'GRL'], 
                                      'sample'            : 'group.phys-top.D2PD_rel15.6.11.3.1_data10_7TeV.00162620.physics_Egamma.merge.AOD.f287_m588.20100914_00_D2AODM_TOPEL', 
                                      'version'            : 1, 
                                      'nickname'        : 'test_struct_maker', 
                                      'atlas_release'   : '15.6.12.4.1'}
}

The tree view of the above task definition:

Actions

Freezing AMAAthena release

All the code change in AMAUser/AMACore/AMAAthena should be consolidated with all possible dependencies clearly defined in SVN.

Freezing AMA configurations

Following the frozen of AMAAthena release, the proper configuration file for AMAEvent and StructNtuple generations should be also made available and frozen.

Installing Athena at NIKHEF

When a new Athena release is needed and not installed at NIKHEF's /data/atlas, install it first.

Making UserArea tarball

This step contains 2 works:

1. making a self-contained UserArea tarball to be shipped along with the job to the remote grid worker nodes.
2. converting the AMA configuration files into standalone Athena job options, those job options are only for AMAEvent Making.

Making validation samples

At this step, one small squad job will be sent to the Grid to make StructNtuple. It should take only few hours to arrive on /data/atlas3. When the test sample becomes available, everybody is invited to check the quality of the data to make sure:

1. AMA codes are OK.
2. AMA configurations are OK

If something not correctly set or implemented, one should go back to the corresponding steps above to repeat the loop.

Validation meeting

If needed, a dedicate meeting to coordinate the validation process can be organized.

Massive production

Submitting bunch of jobs to the Grid for StructNtuple generation. It's not clear yet if we go straight to StructNtuple or keep the intermittent AMAEvent. Therefore it can be a 2 steps works:

1. step 1 is to run AutoD3PDMaker (a Ganga-based framework automating the job babysitting works for D3PDMaking).
2. step 2 is to run ama_ganga.gpi.py that will submit a second batch of jobs to the grid to generate StructNtuple from AMAEvents that was created by first step. This step can be ignored if StructNtuple generation was combined in the first step.

Renaming StructNtuple

Once the StructNtuples are generated, they are named in an non-intuitive way ...