Difference between revisions of "LHCb Software At Nikhef: Prepare for Workshop 22-24 June 2009"

From LHCb Wiki
Jump to navigation Jump to search
 
(102 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
== Introduction ==
 
== Introduction ==
  
The B-Fys workshop will not take place at a high energy physics lab. For these reasons, it is recommended that people install a virtual machine on their laptops in order to be able to run LHCb software locally, without necessity of an internet connection. Whereas this is not a "standard" way of performing analysis at the moment, it might be beneficial to set this up anyway. In what follows, there are explanations on how to run either with an internet connection to nikhef, or with a local virtual machine.
+
The Bfys workshop will not take place at a high energy physics lab.  
 +
Wireless connection is available, but for the hands-on work, we will not rely on that.
 +
Instead we provide two solutions:
  
== Running at Nikhef ==
+
1) Connect to a computer/server.
 +
(See: [http://www.nikhef.nl/pub/experiments/bfyswiki/index.php?title=LHCb_Software_At_Nikhef:_Prepare_for_Workshop_22-24_June_2009#Running_on_Workshop_Server Running on Workshop Server] )
  
''' Logon at Nikhef '''
+
2) Install and configure virtual machine on your laptop, "CernVM".
 +
[http://www.nikhef.nl/pub/experiments/bfyswiki/index.php/How_to_install_and_configure_Cern_Virtual_Machine_(CernVM)_on_your_laptop (How to install)]
  
<pre>
+
(This will give the possibility to run LHCb software locally,
ssh user@ribble.nikhef.nl
+
very much as if you logon to a computer inside your laptop...)
ssh stbc-16  (on stoomboot)
 
</pre>
 
  
''' Set the DaVinci environment '''
 
Here we set the environment for DaVinci, but it could also another LHCb application, like Bender or Panoramix.
 
<pre>
 
mkdir cmtuser
 
cd cmtuser
 
source /project/bfys/lhcb/sw/setup.sh  # for sh, bash, ksh, zsh
 
source /project/bfys/lhcb/sw/setup.csh  # for csh, tcsh
 
SetupProject --build-env DaVinci v23r1 # makes a DaVinci_v23r1 directory if one doesn't already exist
 
SetupProject DaVinci v23r1 # exports environment variables to the shell
 
</pre>
 
  
<pre>
 
getpack -p anonymous Ex/MicroDSTExample v2r5
 
</pre>
 
(NB: zsh-users should add 'setopt SH_WORD_SPLIT' to their .zshrc )
 
  
== Running on a virtual machine on your very own laptop ==
+
We ask you to try to run an example job on the server '''using your laptop''', before the workshop starts on Monday.
 +
(See [http://www.nikhef.nl/pub/experiments/bfyswiki/index.php?title=LHCb_Software_At_Nikhef:_Prepare_for_Workshop_22-24_June_2009#Running_on_Workshop_Server Running on Workshop Server])
  
== Running over some events on a DST or MicroDST ==
 
After the environment has been set (either by remote login to Nikhef or CERN, or in the local CernVM on your laptop, we can start some simple analysis:
 
  
''' Get a nice package full of example scripts (new Bender Examples package exists too...)
 
<pre>
 
cd ~/cmtuser/DaVinci_v23r1/Ex/MicroDSTExample/scripts
 
python -i MicroDSTReadingExample.py --input /data/bfys/h71/Workshop/SeqDC06selBs2JpsiPhiWithMC_50.0_Kevt_NoPVReFit.dst
 
</pre>
 
  
''' Plot some of the histograms made by the script '''
+
In addition - to familiarize with the LHCb software jargon - we ask you to read through the  
 +
following pages too:
  
<pre>
+
1) Python [http://www.astro.ufl.edu/~warner/prog/python.html Basics]
>>> massPlots[443].plot()
 
>>> massPlots[531].plot()
 
>>> massResPlots[531].plot()
 
>>> propTimePlots[531].plot()
 
>>> Ctrl-D
 
</pre>
 
  
== Introduction to Python ==
+
2) Bender [https://twiki.cern.ch/twiki/bin/view/LHCb/BenderTutorial Tutorial (twiki)]
 +
[http://www.nikhef.nl/pub/experiments/bfys/lhcb/computing/20080522_NIKHEF_Bender-Tutorial.pdf Tutorial (pdf)]
  
It is important that attendees make themselves familiar with the python programming language, at least at the level of writing simple scripts.
+
3) Gaudi [https://twiki.cern.ch/twiki/bin/view/LHCb/LHCbSoftwareTrainingBasics Basics (twiki)]
  
Extensive tutorial: [http://doc.python.org/tutorial Tutorial]
+
== Running on Workshop Server ==
  
Cheat Sheet: [http://www.yukoncollege.yk.ca/~ttopper/COMP118/rCheatSheet.html Cheat Sheet]
+
=== Login ===
  
 +
Linux or mac:
 
<pre>
 
<pre>
 +
ssh -Y u01@192.16.186.110    (# Available usernames: u01 - u15)
 +
Password: (LHCb psswrd)
 +
</pre>
  
/user/h71> python
+
Windows:
>>> hello = "Hello world"
+
<pre>
>>> print hello
+
Make sure you have Exceed running
Hello world
+
ssh to 192.16.186.110      (# Available usernames: u01 - u15)
>>> Ctrl-D
+
Password: (LHCb psswrd)
 
</pre>
 
</pre>
  
== Install a Virtual Machine on your laptop: CernVM ==
+
'''NB: In Egmond use: 192.168.1.1 '''
  
See talk by Pere Mato, slide 14:
+
=== Environment ===
[http://indico.cern.ch/conferenceDisplay.py?confId=52399 17 March 2009]
 
  
CernVM beginner's guide:
+
At startup, the script /opt/LHCb/LbLogin.sh is executed for all users. It sets up the LHCb environment. After this, follow the steps described in [http://www.nikhef.nl/pub/experiments/bfyswiki/index.php/LHCb_Software_At_Nikhef:_Prepare_for_Workshop_22-24_June_2009#Setting_the_Environment]
[https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMGuideDetail CernVM beginner's guide]
 
  
 +
=== Run ===
  
=== 1) Windows ===
+
Run over events as described [http://www.nikhef.nl/pub/experiments/bfyswiki/index.php/LHCb_Software_At_Nikhef:_Prepare_for_Workshop_22-24_June_2009#Running_over_some_events_on_a_DST_or_MicroDST here].
  
'''1a) Download vmware player [http://www.vmware.com/download/player/ here] '''
+
== Running on a Cern Virtual Machine (CernVM) ==
<pre>
 
Unzip and put CernVM in fixed place. For example C:\Program Files\CernVM
 
Start vmplayer from desktop
 
Open CernVM from C:\Program Files\CernVM
 
Open in browser: http://192.168.204.128:8004/
 
                (your number might be different!)
 
Create account. For example: use lxplus username and password
 
Virtual organization: lhcb
 
Change memory allocation in vmware player
 
  -> troubleshoot
 
    -> change memory allocation
 
        -> 1024 MB
 
</pre>
 
  
 +
This assumes you have successfully installed a CernVM on your computer. The installation varies between Windows, Mac OS and Linux, so see section further down for instructions on how to set it up. It also assumes you have set up basic login scripts, and installed AFS, X11R6, CVS and SVN if necessary, as indicated [http://www.nikhef.nl/pub/experiments/bfyswiki/index.php/How_to_install_and_configure_Cern_Virtual_Machine_(CernVM)_on_your_laptop#Configuration_of_CernVM here]. Also note that you '''have to run an application like DaVinci, Bender or Panoramix at least once while connected to the network in order to cache the necessary software'''.
  
 +
===Login===
  
 +
This could mean starting the virtual machine and CernVM, starting it and logging in "remotely" from a terminal (Linux, Mac) or Exceed (Windows).
  
''' 1b) Start CernVM '''
+
==== Linux/Mac ====
  
Follow CernVM beginner's guide step by step:
+
<pre>
[https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMGuideDetail CernVM beginner's guide]
+
ssh -Y -p 2222 CernVMUserName@localhost
 +
</pre>
  
''' 1c) Start Exceed + SSH '''
+
====Windows====
  
 
<pre>
 
<pre>
Line 110: Line 82:
 
</pre>
 
</pre>
  
''' 1d) Configure CernVM '''
+
=== Set the environment ===
  
 +
If you have followed the instructions for [http://www.nikhef.nl/pub/experiments/bfyswiki/index.php/How_to_install_and_configure_Cern_Virtual_Machine_(CernVM)_on_your_laptop#Configuration_of_CernVM setting up CernVM] and added the necessary group login scripts to your own login scripts, the LHCb environment should be set. If this is not the case,
 +
<pre>
 +
# for sh, bash, ksh, zsh
 +
. /opt/lhcb/group_login.sh
 +
# for csh, tcsh
 +
source /lhcb/group_login.csh
 +
</pre>
 +
Then follow the steps outlined in [http://www.nikhef.nl/pub/experiments/bfyswiki/index.php/LHCb_Software_At_Nikhef:_Prepare_for_Workshop_22-24_June_2009#Setting_the_Environment Setting the Environment]. If you can't find the required version of DaVinci or Bender, try refreshing the CernVM cache. '''You need an internet connection for this!'''
 
<pre>
 
<pre>
(once for configuring cernvm:)
+
sudo /etc/init.d/cernvm restart
back to instructions afs
 
sudo conary update openafs=1.4.8-1-1
 
sudo conary update openafs-client=1.4.8-1-1
 
sudo conary update openafs-kernel=1.4.8-1-1
 
sudo /etc/init.d/afs start
 
 
</pre>
 
</pre>
  
''' 1e) CernVM session on ssh '''
+
The main difference in setting the application environments here is that if you do not have the necessary software installed, CernVM will pull it and cache it for you. So the first time you do this you should have an interned connection.
 +
 
 +
== Setting the Environment ==
  
 +
We use the LHCb script [https://twiki.cern.ch/twiki/bin/view/LHCb/SetupProject SetupProject]. We don't need to care about details, other that when invoked with '''--build-env''' it doesn't set any environment variables in the shell, and it creates a folder
 
<pre>
 
<pre>
Never do: ls /opt/ !! (It could start caching all atlas software...)
+
~/cmtuser/ProjectName_vXrY
 
 
kinit user@CERN.CH (capitals!)
 
cd /opt/lhcb
 
source /opt/lhcb/LbLogin.csh
 
SetupProject Gaudi v21r0
 
python $GAUDIEXAMPLESROOT/python/HistoEx.py
 
SetupProject Bender v12r0
 
cd $BENDEREXAMPLEROOT/python/BenderExample/
 
./Phi.py
 
 
</pre>
 
</pre>
 
+
where all the software should be built. We will be using the Bender v12r1 and DaVinci v23r1p1 environments, but since Bender used DaVinci, the environment necessary for the latter is provided by the former, so we will set the Bender environment.
''' 1f) Create shared data folder on your laptop '''
 
  
 
<pre>
 
<pre>
Make folder C:\Data
+
# this line in case ~/cmtuser/Bender_v12r1 doesn't exist yet
Change properties of C:\Data
+
SetupProject Bender v12r1 --build-env
  -> sharing
+
# this exports environment variables to the shell. Some people might not like doing this
    -> network sharing without wizard
+
SetupProject Bender v12r1
      V share
 
          name: Data
 
      V allow
 
 
</pre>
 
</pre>
  
<pre>
+
This procedure is common to working at CERN, Nikhef, CernVM and the workshop server, provided the LHCb scripts have been set up correctly, be it automatically in the login scripts, or explicitly by sourcing some script. Details are in the subsections for each platform.
Edit the 2kB VMware configuration file (in C:\Program Files\CernVM\ ) and add:
 
  
sharedFolder.maxNum = "1"
+
== Running over some events on a DST or MicroDST ==
  
sharedFolder0.present = "TRUE"
+
After the environment has been set (either by remote login to Nikhef or CERN, or in the local CernVM on your laptop,) we can start some simple analysis:
sharedFolder0.enabled = "TRUE"
 
sharedFolder0.readAccess = "TRUE"
 
sharedFolder0.writeAccess = "TRUE"
 
sharedFolder0.hostPath = "C:\Data"
 
sharedFolder0.guestName = "data"
 
sharedFolder0.expiration = "session"
 
  
usb.autoConnect.device1 = ""
+
=== Get Tutorial packages Tutorial/BenderTutor, Tutorial/GaudiPythonTutor if not already there ===
  
NB: after editing the configuration file, restart VMware Player
+
If you haven't already got the tutorial packages in your build area, get them, as you will need to perform modifications to the exercise scripts. This has to be done when you have access to the internet.
</pre>
 
  
 
<pre>
 
<pre>
In VMware:
+
cd ~/cmtuser/Bender_v12r1
  -> Share folders
+
 
    -> o always enabled
+
# if the package is not there, do:
 +
# cvs co -r Tutorial/GaudiPythonTutor
 +
# cvs co -r v12r1 Tutorial/BenderTutor
 +
 
 +
# re-sets the $GAUDIPYTHONTUTORROOT and $BENDERTUTOR environment variables
 +
SetupProject Bender v12r1
 +
cd $GAUDIPYTHONTUTOR/cmt/
 +
cmt config
 +
make
 +
cd $BENDERTUTORROOT/cmt
 +
cmt config
 +
make
 
</pre>
 
</pre>
 +
 +
=== Run over some events and make plots using a script from the release area ===
 +
Don't worry, we'll go through exercises that will gradually allow us to write such a script. For now, just run it and stare, awe-struck, at the amazing plots.
 +
 +
'''Run over events'''
  
 
<pre>
 
<pre>
Add the following line to /etc/fstab:
+
python -i $MICRODSTEXAMPLEROOT/scripts/MicroDSTReadingExample.py --input <filename>
sudo emacs -nw /etc/fstab
+
 
    .host:/ /mnt/hgfs vmhgfs defaults 0 0
+
# <filename> can take one of the following values depending on context:
  
sudo ln -s /usr/sbin/mount.vmhgfs /sbin
+
# Workshop Server: /data/bfys/lhcb/workshop/Bs2JpsiPhiWithMC_1.mdst
 +
# Windows CernVM:  /mnt/hgfs/data/Bs2JpsiPhiWithMC_1.mdst
 +
# Mac CernVM:      /data/lhcb/Bs2JpsiPhiWithMC_1.mdst
  
I edited Phi.py and put input data location: /mnt/hgfs/data/Bs2DsK_1.dst )
 
Data can be found here:
 
/afs/cern.ch/lhcb/group/calo/ecal/vol10/DATA
 
(or ask Niels or Vanya for a copy...)
 
 
</pre>
 
</pre>
  
''' Once everything is installed... Start CernVM '''
+
''' Plot some of the histograms made by the script '''
  
 
<pre>
 
<pre>
Start VMware Player (don't select 'Open', but 'Recent')
+
>>> massPlots[443].plot()
 +
>>> massPlots[531].plot()
 +
>>> massResPlots[531].plot()
 +
>>> propTimePlot.plot()
 +
>>> propTimeResPlot.plot()
 +
>>> refitPropTimeResPlot.plot()
 +
>>> Ctrl-D
 +
</pre>
  
Login to your CernVM with your SSH.
+
Some more examples, using Gaudi or Bender environments and example scripts. Note we run them directly without getpacking them.
  
source /opt/lhcb/LbLogin.csh
+
<pre>
 +
SetupProject Gaudi v21r0
 +
python $GAUDIEXAMPLESROOT/python/HistoEx.py
 
SetupProject Bender v12r1
 
SetupProject Bender v12r1
 +
cd $BENDEREXAMPLEROOT/python/BenderExample/
 
./Phi.py
 
./Phi.py
 
root
 
TBrowser f
 
.q
 
 
</pre>
 
</pre>
  
=== 2) MAC ===
+
== Introduction to Python ==
  
I have succesfully tried using VirtualBox, from Sun Microsystems. It is free, it works.
+
It is important that attendees make themselves familiar with the python programming language, at least at the level of writing simple scripts.
Follow more or less the instructions in [[ATLAS CernVM wiki][https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMGuideDetail]]
 
  
'''Download and untar CernVM 1.20'''
+
Python [http://www.astro.ufl.edu/~warner/prog/python.html Basics]
Get the one for VirtualBox from [https://twiki.cern.ch/twiki/bin/view/Atlas/CernVM#Download_CernVM]
 
Unzip and untar it. And figure out where in your file system it is. This will be the boot image "disk" for the operating system so VirtualBox will need it for booting.
 
This must be done *before* the VirtualBox step.
 
  
'''Get VirtualBox'''
+
Extensive tutorial: [http://doc.python.org/tutorial Tutorial]
http://www.virtualbox.org/wiki/Downloads
 
Install it following instructions
 
run it and follow https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMGuideVirtualBox
 
I went for ~1542MB memory
 
<pre>
 
Name:                  CernVM
 
OS Type:              Linux 2.6
 
Base Memory:      1532 MB
 
Video Memory:    6 MB
 
  
Hard Disks
+
Cheat Sheet: [http://www.yukoncollege.yk.ca/~ttopper/COMP118/rCheatSheet.html Cheat Sheet]
IDE Primary Master: cernvm-1.2.0-x86.vmdk (Normal, 5.24 GB)
 
</pre>
 
I had some problems getting it to find the disk image. I did lots of
 
clicking around and it worked in the end. I cannot replicate the
 
"locate the vmdk file" snapshots in the
 
[https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMGuideVirtualBox wiki]
 
exactly. Maybe I'll try again to see.
 
  
For network setup, I used NAT following [https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMGuideVirtualBox#NAT_Networking these instructions]
+
Example: [http://www.nikhef.nl/~h71/Workshop/Instructions/myfirst.py myfirst.py ]
  
Then I followed the rest of the steps [https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMGuideVirtualBox here]
+
<pre>
 +
Execute:
 +
python myfirst.py
  
After starting a session, it is necessary to source the LHCb login
 
script:
 
<pre>
 
source /opt/lhcb/LbLogin.csh
 
</pre>
 
(check what shell you have set)
 
I added it to my login script.
 
Then, to cache lots of software,
 
<pre>
 
SetupProject DaVinci v23r1 (no v23r1p1 yet)
 
SetupProject Panoramix v17r0 (no v17r1 yet)
 
 
</pre>
 
</pre>
Hubert has given me instructions on how to set the login scripts properly, so I'll try them and update this document later.
 
 
Juan
 
=== 3) Linux ===
 

Latest revision as of 12:03, 25 June 2009

Introduction

The Bfys workshop will not take place at a high energy physics lab. Wireless connection is available, but for the hands-on work, we will not rely on that. Instead we provide two solutions:

1) Connect to a computer/server. (See: Running on Workshop Server )

2) Install and configure virtual machine on your laptop, "CernVM". (How to install)

(This will give the possibility to run LHCb software locally, very much as if you logon to a computer inside your laptop...)


We ask you to try to run an example job on the server using your laptop, before the workshop starts on Monday. (See Running on Workshop Server)


In addition - to familiarize with the LHCb software jargon - we ask you to read through the following pages too:

1) Python Basics

2) Bender Tutorial (twiki) Tutorial (pdf)

3) Gaudi Basics (twiki)

Running on Workshop Server

Login

Linux or mac:

ssh -Y u01@192.16.186.110    (# Available usernames: u01 - u15)
Password: (LHCb psswrd)

Windows:

Make sure you have Exceed running
ssh to 192.16.186.110       (# Available usernames: u01 - u15)
Password: (LHCb psswrd)

NB: In Egmond use: 192.168.1.1

Environment

At startup, the script /opt/LHCb/LbLogin.sh is executed for all users. It sets up the LHCb environment. After this, follow the steps described in [1]

Run

Run over events as described here.

Running on a Cern Virtual Machine (CernVM)

This assumes you have successfully installed a CernVM on your computer. The installation varies between Windows, Mac OS and Linux, so see section further down for instructions on how to set it up. It also assumes you have set up basic login scripts, and installed AFS, X11R6, CVS and SVN if necessary, as indicated here. Also note that you have to run an application like DaVinci, Bender or Panoramix at least once while connected to the network in order to cache the necessary software.

Login

This could mean starting the virtual machine and CernVM, starting it and logging in "remotely" from a terminal (Linux, Mac) or Exceed (Windows).

Linux/Mac

ssh -Y -p 2222 CernVMUserName@localhost

Windows

Start exceed
Start ssh
Connect to vm with ssh to 192.168.204.128
     (Your number might be different!)
     (fink tunnel x11 connections in ssh!)

Set the environment

If you have followed the instructions for setting up CernVM and added the necessary group login scripts to your own login scripts, the LHCb environment should be set. If this is not the case,

# for sh, bash, ksh, zsh
. /opt/lhcb/group_login.sh
# for csh, tcsh
source /lhcb/group_login.csh

Then follow the steps outlined in Setting the Environment. If you can't find the required version of DaVinci or Bender, try refreshing the CernVM cache. You need an internet connection for this!

sudo /etc/init.d/cernvm restart

The main difference in setting the application environments here is that if you do not have the necessary software installed, CernVM will pull it and cache it for you. So the first time you do this you should have an interned connection.

Setting the Environment

We use the LHCb script SetupProject. We don't need to care about details, other that when invoked with --build-env it doesn't set any environment variables in the shell, and it creates a folder

~/cmtuser/ProjectName_vXrY

where all the software should be built. We will be using the Bender v12r1 and DaVinci v23r1p1 environments, but since Bender used DaVinci, the environment necessary for the latter is provided by the former, so we will set the Bender environment.

# this line in case ~/cmtuser/Bender_v12r1 doesn't exist yet
SetupProject Bender v12r1 --build-env
# this exports environment variables to the shell. Some people might not like doing this
SetupProject Bender v12r1

This procedure is common to working at CERN, Nikhef, CernVM and the workshop server, provided the LHCb scripts have been set up correctly, be it automatically in the login scripts, or explicitly by sourcing some script. Details are in the subsections for each platform.

Running over some events on a DST or MicroDST

After the environment has been set (either by remote login to Nikhef or CERN, or in the local CernVM on your laptop,) we can start some simple analysis:

Get Tutorial packages Tutorial/BenderTutor, Tutorial/GaudiPythonTutor if not already there

If you haven't already got the tutorial packages in your build area, get them, as you will need to perform modifications to the exercise scripts. This has to be done when you have access to the internet.

cd ~/cmtuser/Bender_v12r1

# if the package is not there, do:
# cvs co -r Tutorial/GaudiPythonTutor
# cvs co -r v12r1 Tutorial/BenderTutor

# re-sets the $GAUDIPYTHONTUTORROOT and $BENDERTUTOR environment variables
SetupProject Bender v12r1
cd $GAUDIPYTHONTUTOR/cmt/
cmt config
make
cd $BENDERTUTORROOT/cmt
cmt config
make

Run over some events and make plots using a script from the release area

Don't worry, we'll go through exercises that will gradually allow us to write such a script. For now, just run it and stare, awe-struck, at the amazing plots.

Run over events

python -i $MICRODSTEXAMPLEROOT/scripts/MicroDSTReadingExample.py --input <filename>

# <filename> can take one of the following values depending on context:

# Workshop Server: /data/bfys/lhcb/workshop/Bs2JpsiPhiWithMC_1.mdst
# Windows CernVM:  /mnt/hgfs/data/Bs2JpsiPhiWithMC_1.mdst
# Mac CernVM:      /data/lhcb/Bs2JpsiPhiWithMC_1.mdst

Plot some of the histograms made by the script

>>> massPlots[443].plot()
>>> massPlots[531].plot()
>>> massResPlots[531].plot()
>>> propTimePlot.plot()
>>> propTimeResPlot.plot()
>>> refitPropTimeResPlot.plot()
>>> Ctrl-D

Some more examples, using Gaudi or Bender environments and example scripts. Note we run them directly without getpacking them.

SetupProject Gaudi v21r0
python $GAUDIEXAMPLESROOT/python/HistoEx.py
SetupProject Bender v12r1
cd $BENDEREXAMPLEROOT/python/BenderExample/
./Phi.py

Introduction to Python

It is important that attendees make themselves familiar with the python programming language, at least at the level of writing simple scripts.

Python Basics

Extensive tutorial: Tutorial

Cheat Sheet: Cheat Sheet

Example: myfirst.py

Execute:
python myfirst.py