Difference between revisions of "Modern C Compiler, ROOT, Geant"

From PDP/Grid Wiki
Jump to navigationJump to search
(→‎Figure out your platform: Add different options (e.g. needed for Ganymede/Visar))
 
(11 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
=== Introduction ===
 
=== Introduction ===
  
−
So you want a modern C compiler and/or some other tool.  As of this writing (2 November 2016) the normal desktop machines have  
+
So you want a modern C compiler and/or some other tool.  As of this writing (28 November 2017) the stoomboot interactive nodes have  
−
  gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
+
  gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
 
as the default compiler.  This is not recent enough for much of the HEP code ecosystem.
 
as the default compiler.  This is not recent enough for much of the HEP code ecosystem.
  
Line 22: Line 22:
  
 
Today on one of the stoomboot worker nodes, it says
 
Today on one of the stoomboot worker nodes, it says
−
  Scientific Linux CERN SLC release 6.8 (Carbon)
+
  Scientific Linux CERN SLC release 6.9 (Carbon)
 
  Kernel \r on an \m
 
  Kernel \r on an \m
  
 
Remember that first line, that tells you the platform, you'll need it shortly.
 
Remember that first line, that tells you the platform, you'll need it shortly.
  
−
=== Poke around in SFT and find the relevant release and compiler ===
 
  
−
stbc-i2:~> cd /cvmfs/sft.cern.ch/lcg/releases
+
Other ways to find which platform you're on include
−
stbc-i2:releases> ls -d LCG_*  
+
* cat /proc/version
−
LCG_71     LCG_81c     LCG_82rootaas4  LCG_84swan3  LCG_86
+
* lsb_release -a
−
LCG_71root6  LCG_81d     LCG_82rootaas5  LCG_84swan4  LCG_87
+
* uname -a
−
LCG_72a      LCG_81e     LCG_82rootaas6  LCG_85   LCG_latest
+
* ls /etc/*release* /etc/*version*
−
LCG_75root6  LCG_81f     LCG_83     LCG_85a   LCG_rootext20161010
 
−
LCG_78root6  LCG_82     LCG_84     LCG_85b   LCG_rootext20161129
 
−
LCG_79     LCG_82rootaas1  LCG_84a     LCG_85swan1
 
−
LCG_80     LCG_82rootaas2  LCG_84swan1    LCG_85swan2
 
−
LCG_81b      LCG_82rootaas3  LCG_84swan2    LCG_85swan3
 
−
Note the LCG87 is the highest number ... this is generally the latest full release.  Take a look at [http://lcgsoft.web.cern.ch/lcgsoft/ this page] for some documentation on what the various releases include.
 
  
−
Note I've replaced below, "LCG_87" with "LCG_XXX" in commands to help avoid mistakes made via blind copy-and-pasting.  Change the XXX to whatever you find the most recent release to be!
+
=== Poke around in SFT and find the relevant release and compiler ===
  
−
  stbc-022:releases> cd /cvmfs/sft.cern.ch/lcg/releases/LCG_XXX
+
  stbc-i2:lcg> . /cvmfs/sft.cern.ch/lcg/views/setupViews.sh
−
  stbc-i2:LCG_XXX> ls
+
  Usage:
−
  4suite gmp       LCG_externals_x86_64-ubuntu1604-gcc54-dbg.txt   pacparser        requests
+
  setupViews <LCG_directory> <arch-os-complier>
−
AIDA gperftools       LCG_externals_x86_64-ubuntu1604-gcc54-opt.txt   pandas       ROOT
+
  where <LCG_directory> can be any of these
−
astroid graphviz       LCG_generators_x86_64-centos7-gcc62-dbg.txt    pathlib2       root_numpy
+
   
−
autoconf Grid       LCG_generators_x86_64-centos7-gcc62-opt.txt    pathos       rootpy
+
   LCG_71
−
automake GSL       LCG_generators_x86_64-slc6-gcc49-dbg.txt       pcre       rpy2
+
   LCG_71root6
−
backports gtest       LCG_generators_x86_64-slc6-gcc49-opt.txt       pexpect       scikitlearn
+
  LCG_72a
−
blas hadoop       LCG_generators_x86_64-slc6-gcc62-dbg.txt       pickleshare      scipy
+
  LCG_79
−
Boost hepdata_converter       LCG_generators_x86_64-slc6-gcc62-opt.txt       pip       setuptools
+
  LCG_81c
−
castor hepdata_validator       LCG_generators_x86_64-ubuntu1604-gcc54-dbg.txt  pkg_config      simplegeneric
+
  LCG_81d
−
ccache HepMC       LCG_generators_x86_64-ubuntu1604-gcc54-opt.txt  pox       simplejson
+
  LCG_81e
−
certifi HepPDT       lcov       ppft       singledispatch
+
  LCG_81f
−
clhep hive       lhapdfsets       prettytable      sip
+
  LCG_82rootaas2
−
CMake igprof       libaio       processing      six
+
  LCG_82rootaas3
−
cmaketools ipykernel       libgit2       prompt_toolkit   sollya
+
  LCG_82rootaas4
−
cmmnbuild ipython       libsvm       protobuf       soqt
+
  LCG_82rootaas5
−
cmt ipython_genutils       libtool       psutil       spark
+
  LCG_82rootaas6
−
coin3d ipywidgets       libunwind       ptyprocess      sqlalchemy
+
   LCG_83
−
configparser java       libxml2       py       sqlite
+
  LCG_84
−
COOL jemalloc       libxslt       py2neo       stomppy
+
  LCG_84swan1
−
CORAL Jinja2       logilabcommon       py4j       storm
+
  LCG_84swan2
−
CouchDB joblib       lxml       pyanalysis      subprocess32
+
  LCG_84swan3
−
coverage jpype       m4       pyapigitlab      swig
+
  LCG_84swan4
−
cppgsl jsonc       MarkupSafe       pydot       sympy
+
  LCG_85
−
CppUnit jsoncpp       matplotlib       pydot_ng       tbb
+
  LCG_85a
−
curl jsonschema       maven       pygments       terminado
+
  LCG_85b
−
cx_oracle jupyter       MCGenerators       pygraphics      theano
+
  LCG_85swan1
−
cycler jupyter_client       messaging       pygsi       tornado
+
  LCG_85swan2
−
cython jupyter_console       metakernel       pylint       traitlets
+
  LCG_85swan3
−
Davix jupyter_contrib_core       mistune       pyminuit       urllib3
+
  LCG_86
−
DD4hep jupyter_contrib_nbextensions       mock       pyparsing        uuid
+
  LCG_87
−
decorator jupyter_core       mpfi       pyqt       valgrind
+
  LCG_88
−
dill jupyter_nbextensions_configurator       mpfr       pyqt5       vdt
+
  LCG_88Py3
−
distribute keras       mpich2       pytest       VecGeom
+
  LCG_89
−
doxygen lapack       multiprocess       Python       vectorclass
+
  LCG_89python3
−
eigen LCGCMT       multiprocessing       python_dateutil  wcwidth
+
  LCG_90
−
elasticsearch LCG_contrib_x86_64-centos7-gcc62-dbg.txt    mysql       python_gitlab    wheel
+
  LCG_90a
−
entrypoints LCG_contrib_x86_64-centos7-gcc62-opt.txt    mysql_python       pytimber       widgetsnbextension
+
  LCG_90python3
−
expat LCG_contrib_x86_64-slc6-gcc49-dbg.txt       nbconvert       pytools       xapian
+
  LCG_91
−
fastjet LCG_contrib_x86_64-slc6-gcc49-opt.txt       nbformat       pytz       XercesC
+
  LCG_91python3
−
fftw3 LCG_contrib_x86_64-slc6-gcc62-dbg.txt       networkx       pyxml       xqilla
+
  LCG_gaudiext20170622
−
fjcontrib LCG_contrib_x86_64-slc6-gcc62-opt.txt       neurobayes       pyyaml       xrootd
+
  LCG_latest
−
fplll LCG_contrib_x86_64-ubuntu1604-gcc54-dbg.txt  neurobayes_expert       pyzmq       xrootd_python
+
  LCG_rootext20161010
−
freetype LCG_contrib_x86_64-ubuntu1604-gcc54-opt.txt  ninja       QMtest       xz
+
  LCG_rootext20161129
−
frontier_client  lcgenv       nose       qt       yamlcpp
+
  LCG_rootext20170508
−
ftjam LCG_externals_x86_64-centos7-gcc62-dbg.txt   notebook       qt5       zeromq
+
  LCG_rootext20170929
−
future LCG_externals_x86_64-centos7-gcc62-opt.txt   numexpr       qtconsole        zlib
+
  LCG_rootext20171122
−
futures LCG_externals_x86_64-slc6-gcc49-dbg.txt      numpy       qwt
+
  dev3
−
gcc LCG_externals_x86_64-slc6-gcc49-opt.txt      omniorb       R
+
   dev3arch
−
Geant4 LCG_externals_x86_64-slc6-gcc62-dbg.txt      openssl       rangev3
+
   dev3python3
−
genshi LCG_externals_x86_64-slc6-gcc62-opt.txt      oracle       RELAX
+
  dev4
 +
  devgeantv
 +
  geantv
 +
  ml/1.0
 
   
 
   
−
There is a lot of stuff in here that might be interesting to you (eg ROOT and Geant4). See that gcc is also listed, list that directory to see the options:
+
  and <arch-os-complier> depends on what is available in <LCG_directory>
−
  stbc-i2:LCG_87> ls gcc
+
−
4.9.1 4.9.3  6.2.0
+
  stbc-i2:lcg>
−
stbc-i2:LCG_87> ls gcc/4.9.3
+
Each of these LCG_* tags is a release of a bunch of software such as Tensorflow, Python, ROOT, scikitlearn, etc.  To find out what is available inside each release (including versions) you can take a look at
−
x86_64-slc6
+
[http://lcginfo.cern.ch/ this page].
  
−
So gcc 4.9.3 is there, along with two other versions.  Now the /etc/issue business above becomes relevant: it said "Scientific Linux CERN SLC release 6.8" which corresponds to the "slc6" here, so this one will work for you:
+
The release exists in several versions, each one a specific combination of unix platform (red hat, ubuntu, etc) and C compiler version, as you can only be certain that software will work together if it's all been compiled on the same platform with the same C compiler. The views script will tell you which platform is available, if you select a LCG release:
−
  stbc-i2:LCG_87> ls gcc/4.9.3/x86_64-slc6
+
  stbc-i2:lcg> . /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_91
−
bin  include  lib  lib64  libexec  setup.csh  setup.csh~  setup.sh  setup.sh~  share
+
Available <arch-os-complier> for LCG_91 :
 +
  x86_64-centos7-gcc62-dbg
 +
  x86_64-centos7-gcc62-opt
 +
  x86_64-centos7-gcc7-dbg
 +
  x86_64-centos7-gcc7-opt
 +
  x86_64-slc6-gcc62-dbg
 +
  x86_64-slc6-gcc62-opt
 +
  x86_64-slc6-gcc7-dbg
 +
  x86_64-slc6-gcc7-opt
 +
  x86_64-ubuntu1604-gcc54-dbg
 +
  x86_64-ubuntu1604-gcc54-opt
 +
Going back to the previous section, we had CERN SLC 6.9 so this is the "slc6" platform ... C compilers available are gcc 6.2 or 7 ... and you can either have the software compiled with debugging enabled (dbg) or with optimisation turned on (opt).
  
 
=== Run the setup script ===
 
=== Run the setup script ===
 +
The views script will set things up for you.  Note that as of this writing, it doesn't work correctly with zsh but does with bash.
  
−
You want that setup.sh file ..
+
$ . /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_91 x86_64-slc6-gcc7-opt
−
 
+
After running that command, you can check that things are set up:
−
source /cvmfs/sft.cern.ch/lcg/releases/LCG_XXX/gcc/X.Y.Z/x86_64-platform/setup.sh
+
  $ gcc --version
−
 
+
  gcc (GCC) 7.1.0
−
Change XXX to the right LCG release tag, X.Y.Z to the right gcc version you found, and "platform" to the right platform you found.
+
  Copyright (C) 2017 Free Software Foundation, Inc.
−
 
 
−
'''note: csh users need the setup.csh file'''
 
−
We (PDP) assume that the csh setup works correctly, none of us use csh, hence we don't know for sure ;-)
 
−
 
 
−
=== Compile your code ===
 
−
 
 
−
Now you'll have the right stuff in your path, etc to use this version of gcc:
 
−
  stbc-022:x86_64-slc6> gcc --version
 
−
  gcc (GCC) 4.9.3
 
−
  Copyright (C) 2015 Free Software Foundation, Inc.
 
 
  This is free software; see the source for copying conditions.  There is NO
 
  This is free software; see the source for copying conditions.  There is NO
 
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 +
$ root -b
 +
  ------------------------------------------------------------
 +
  | Welcome to ROOT 6.10/06                http://root.cern.ch |
 +
  |                              (c) 1995-2017, The ROOT Team |
 +
  | Built for linuxx8664gcc                                    |
 +
  | From tag v6-10-06, 19 September 2017                      |
 +
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q' |
 +
  ------------------------------------------------------------
 
   
 
   
−
  stbc-022:x86_64-slc6> g++ --version
+
  root [0] .q
−
  g++ (GCC) 4.9.3
+
$ python
−
  Copyright (C) 2015 Free Software Foundation, Inc.
+
Python 2.7.13 (default, May  9 2017, 16:41:38)
−
  This is free software; see the source for copying conditions. There is NO
+
  [GCC 7.1.0] on linux2
−
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+
  Type "help", "copyright", "credits" or "license" for more information.
 +
  >>>
 +
  $
 +
 
 +
=== Compile your code ===
  
 +
Now you'll have the right stuff in your path, etc to use that version of gcc.
 
Enjoy that prize.  Don't forget to mention the PDP group in your speech.
 
Enjoy that prize.  Don't forget to mention the PDP group in your speech.

Latest revision as of 11:47, 25 March 2022

Introduction

So you want a modern C compiler and/or some other tool. As of this writing (28 November 2017) the stoomboot interactive nodes have 

gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)

as the default compiler. This is not recent enough for much of the HEP code ecosystem.

Modern C compilers can be found in a repository called "SFT" which is maintained by CERN, and distributed via something called CVMFS which is installed on stoomboot machines and most of the desktop machines (older ones may not have it). You can check whether you have it with: 

ls /cvmfs/sft.cern.ch

if you don't get an error (and it says "lcg") then you're good. What you'll need to do to set up the compiler is:

  1. figure out what kind of platform you'll be running on
  2. poke around in SFT to find out what the current release is
  3. run the setup script
  4. compile your code and win the Nobel Prize.

Figure out your platform

You'll generally need to compile for a specific platform; go (one of) the machine(s) on which you'll be running your code and type 

cat /etc/issue

Today on one of the stoomboot worker nodes, it says 

Scientific Linux CERN SLC release 6.9 (Carbon)
Kernel \r on an \m

Remember that first line, that tells you the platform, you'll need it shortly.


Other ways to find which platform you're on include

  • cat /proc/version
  • lsb_release -a
  • uname -a
  • ls /etc/*release* /etc/*version*

Poke around in SFT and find the relevant release and compiler

stbc-i2:lcg> . /cvmfs/sft.cern.ch/lcg/views/setupViews.sh
Usage:
 setupViews <LCG_directory> <arch-os-complier>
  where <LCG_directory> can be any of these

 LCG_71
 LCG_71root6
 LCG_72a
 LCG_79
 LCG_81c
 LCG_81d
 LCG_81e
 LCG_81f
 LCG_82rootaas2
 LCG_82rootaas3
 LCG_82rootaas4
 LCG_82rootaas5
 LCG_82rootaas6
 LCG_83
 LCG_84
 LCG_84swan1
 LCG_84swan2
 LCG_84swan3
 LCG_84swan4
 LCG_85
 LCG_85a
 LCG_85b
 LCG_85swan1
 LCG_85swan2
 LCG_85swan3
 LCG_86
 LCG_87
 LCG_88
 LCG_88Py3
 LCG_89
 LCG_89python3
 LCG_90
 LCG_90a
 LCG_90python3
 LCG_91
 LCG_91python3
 LCG_gaudiext20170622
 LCG_latest
 LCG_rootext20161010
 LCG_rootext20161129
 LCG_rootext20170508
 LCG_rootext20170929
 LCG_rootext20171122
 dev3
 dev3arch
 dev3python3
 dev4
 devgeantv
 geantv
 ml/1.0

 and <arch-os-complier> depends on what is available in <LCG_directory>

stbc-i2:lcg>

Each of these LCG_* tags is a release of a bunch of software such as Tensorflow, Python, ROOT, scikitlearn, etc. To find out what is available inside each release (including versions) you can take a look at this page.

The release exists in several versions, each one a specific combination of unix platform (red hat, ubuntu, etc) and C compiler version, as you can only be certain that software will work together if it's all been compiled on the same platform with the same C compiler. The views script will tell you which platform is available, if you select a LCG release: 

stbc-i2:lcg> . /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_91
Available <arch-os-complier> for LCG_91 :
 x86_64-centos7-gcc62-dbg
 x86_64-centos7-gcc62-opt
 x86_64-centos7-gcc7-dbg
 x86_64-centos7-gcc7-opt
 x86_64-slc6-gcc62-dbg
 x86_64-slc6-gcc62-opt
 x86_64-slc6-gcc7-dbg
 x86_64-slc6-gcc7-opt
 x86_64-ubuntu1604-gcc54-dbg
 x86_64-ubuntu1604-gcc54-opt

Going back to the previous section, we had CERN SLC 6.9 so this is the "slc6" platform ... C compilers available are gcc 6.2 or 7 ... and you can either have the software compiled with debugging enabled (dbg) or with optimisation turned on (opt).

Run the setup script

The views script will set things up for you. Note that as of this writing, it doesn't work correctly with zsh but does with bash. 

$ . /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_91 x86_64-slc6-gcc7-opt
After running that command, you can check that things are set up:
$ gcc --version
gcc (GCC) 7.1.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ root -b
  ------------------------------------------------------------
 | Welcome to ROOT 6.10/06                http://root.cern.ch |
 |                               (c) 1995-2017, The ROOT Team |
 | Built for linuxx8664gcc                                    |
 | From tag v6-10-06, 19 September 2017                       |
 | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q' |
  ------------------------------------------------------------

root [0] .q
$ python
Python 2.7.13 (default, May  9 2017, 16:41:38)
[GCC 7.1.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
$

Compile your code

Now you'll have the right stuff in your path, etc to use that version of gcc. Enjoy that prize. Don't forget to mention the PDP group in your speech.

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=Modern_C_Compiler,_ROOT,_Geant&oldid=10922"