Difference between revisions of "Modern C Compiler, ROOT, Geant"

From PDP/Grid Wiki
Jump to navigationJump to search
Line 29: Line 29:
 
=== Poke around in SFT and find the relevant release and compiler ===
 
=== Poke around in SFT and find the relevant release and compiler ===
  
  stbc-022:lcg> ls /cvmfs/sft.cern.ch/lcg
+
  stbc-i2:~> cd /cvmfs/sft.cern.ch/lcg/releases
  app dev external LCG85b   lcgjenkins mapfile.txt     mapfile.txt.05042016 mapfile.txt.16092016 nightlies  releases-javier   updatemapfile.py
+
stbc-i2:releases> ls -d LCG_*
  contrib etc hepsoft lcgcmake mapfile.ts mapfile.txt.01052016 mapfile.txt.13042016 mapfile.txt.18082016 releases  root-training-setup.sh views
+
  LCG_71     LCG_81c     LCG_82rootaas4  LCG_84swan3  LCG_86
Note the LCG85b ... this is generally a listing of the latest full release; when there is a new release, the tag gets updated, so you might see LCG91, LCG93c, etc instead. Whatever it is, this is the latest release (there are many older releases available btw).
+
LCG_71root6  LCG_81d     LCG_82rootaas5  LCG_84swan4  LCG_87
 +
  LCG_72a      LCG_81e     LCG_82rootaas6 LCG_85   LCG_latest
 +
  LCG_75root6 LCG_81f     LCG_83     LCG_85a   LCG_rootext20161010
 +
  LCG_78root6 LCG_82     LCG_84     LCG_85b   LCG_rootext20161129
 +
  LCG_79     LCG_82rootaas1 LCG_84a     LCG_85swan1
 +
  LCG_80     LCG_82rootaas2 LCG_84swan1    LCG_85swan2
 +
  LCG_81b      LCG_82rootaas3 LCG_84swan2    LCG_85swan3
 +
Note the LCG87 is the highest number ... this is generally the latest full release.  Take a look at [http://lcgsoft.web.cern.ch/lcgsoft/ this page] for some documentation on what the various releases include.
  
You can search for gcc in that LCG85b file, but it's easier to look at the file system.  Note I've replaced below, "LCG_85b" with "LCG_XXX" in commands to help avoid mistakes made via blind copy-and-pasting.  Change the XXX to whatever you find the most recent release to be!
+
Note I've replaced below, "LCG_87" with "LCG_XXX" in commands to help avoid mistakes made via blind copy-and-pasting.  Change the XXX to whatever you find the most recent release to be!
  
  stbc-022:LCG_85b> cd /cvmfs/sft.cern.ch/lcg/releases/LCG_XXX
+
  stbc-022:releases> cd /cvmfs/sft.cern.ch/lcg/releases/LCG_XXX
  stbc-022:LCG_85b> ls  
+
  stbc-i2:LCG_XXX> ls
  4suite       curl       GSL   jupyter_console       m4   oracle   pylint   root_numpy   urllib3
+
  4suite gmp       LCG_externals_x86_64-ubuntu1604-gcc54-dbg.txt  pacparser       requests
  AIDA       cx_oracle       gtest   jupyter_core       MarkupSafe   pacparser   pyminuit   rootpy   uuid
+
  AIDA gperftools       LCG_externals_x86_64-ubuntu1604-gcc54-opt.txt  pandas       ROOT
  astroid       cycler       hepdata_converter  lapack       matplotlib   pandas   pyparsing   rpy2   valgrind
+
  astroid graphviz       LCG_generators_x86_64-centos7-gcc62-dbg.txt    pathlib2       root_numpy
  autoconf     Davix       hepdata_validator  LCGCMT       MCGenerators   pathlib2   pyqt   scikitlearn   vdt
+
  autoconf Grid       LCG_generators_x86_64-centos7-gcc62-opt.txt    pathos       rootpy
  automake     decorator        HepMC   lcgenv       messaging   pathos   pyqt5   scipy   vectorclass
+
  automake GSL       LCG_generators_x86_64-slc6-gcc49-dbg.txt       pcre       rpy2
  backports    distribute      hepmc3   LCG_externals_x86_64-centos7-gcc49-dbg.txt   metakernel   pcre   pytest   setuptools   wcwidth
+
  backports gtest       LCG_generators_x86_64-slc6-gcc49-opt.txt       pexpect       scikitlearn
  blas       doxygen       HepPDT   LCG_externals_x86_64-centos7-gcc49-opt.txt   mistune   pexpect   Python   simplegeneric  wheel
+
  blas hadoop       LCG_generators_x86_64-slc6-gcc62-dbg.txt       pickleshare     scipy
  Boost       eigen       igprof   LCG_externals_x86_64-slc6-gcc49-dbg.txt      mock   pickleshare   python_dateutil  singledispatch  widgetsnbextension
+
  Boost hepdata_converter       LCG_generators_x86_64-slc6-gcc62-opt.txt       pip       setuptools
  castor       elasticsearch    ipykernel   LCG_externals_x86_64-slc6-gcc49-opt.txt     mpich2   pip   pytimber   sip   xapian
+
  castor hepdata_validator       LCG_generators_x86_64-ubuntu1604-gcc54-dbg.txt  pkg_config       simplegeneric
  ccache       entrypoints      ipython   LCG_generators_x86_64-centos7-gcc49-dbg.txt  multiprocessing   pkg_config   pytools   six   XercesC
+
  ccache HepMC       LCG_generators_x86_64-ubuntu1604-gcc54-opt.txt  pox       simplejson
  certifi      expat       ipython_genutils  LCG_generators_x86_64-centos7-gcc49-opt.txt  mysql   prettytable   pytz   soqt   xqilla
+
certifi HepPDT       lcov       ppft       singledispatch
  clhep       fastjet       ipywidgets   LCG_generators_x86_64-slc6-gcc49-dbg.txt    mysql_python   processing   pyxml   spark   xrootd
+
clhep hive       lhapdfsets       prettytable     sip
  CMake       fftw3       java   LCG_generators_x86_64-slc6-gcc49-opt.txt     nbconvert   prompt_toolkit pyyaml   sqlalchemy   xrootd_python
+
CMake igprof       libaio       processing      six
  cmaketools    fjcontrib        jemalloc   lcov       nbformat   ptyprocess   pyzmq   sqlite   yamlcpp
+
cmaketools ipykernel       libgit2       prompt_toolkit   sollya
  cmmnbuild    freetype       Jinja2   lhapdfsets       networkx   py   QMtest   stomppy   zeromq
+
cmmnbuild ipython       libsvm       protobuf       soqt
  cmt       frontier_client  joblib   libaio       neurobayes   py2neo   qt   storm   zlib
+
  cmt ipython_genutils       libtool       psutil       spark
  coin3d       gcc       jpype   libsvm       neurobayes_expert  py4j   qt5   subprocess32
+
coin3d ipywidgets       libunwind       ptyprocess      sqlalchemy
  configparser  Geant4       json   libtool       ninja   pyanalysis   qtconsole   swig
+
configparser java       libxml2       py       sqlite
  COOL       genshi       jsonc   libunwind       nose   pyapigitlab   qwt   sympy
+
COOL jemalloc       libxslt       py2neo       stomppy
  CORAL       gmp       jsoncpp   libxml2       notebook   pydot   R   tbb
+
CORAL Jinja2       logilabcommon       py4j       storm
  CouchDB      gperftools      jsonschema   libxslt       numexpr   pygments   RELAX   terminado
+
CouchDB joblib       lxml       pyanalysis      subprocess32
  coverage     graphviz       jupyter   logilabcommon       numpy   pygraphics   requests   tornado
+
coverage jpype       m4       pyapigitlab      swig
CppUnit      Grid       jupyter_client   lxml       openssl   pygsi   ROOT   traitlets
+
cppgsl jsonc       MarkupSafe       pydot       sympy
 +
CppUnit jsoncpp       matplotlib       pydot_ng       tbb
 +
curl jsonschema       maven       pygments       terminado
 +
cx_oracle jupyter       MCGenerators       pygraphics      theano
 +
cycler jupyter_client       messaging       pygsi       tornado
 +
cython jupyter_console       metakernel       pylint       traitlets
 +
Davix jupyter_contrib_core       mistune       pyminuit       urllib3
 +
DD4hep jupyter_contrib_nbextensions       mock       pyparsing        uuid
 +
decorator jupyter_core       mpfi       pyqt       valgrind
 +
dill jupyter_nbextensions_configurator       mpfr       pyqt5       vdt
 +
distribute keras       mpich2       pytest       VecGeom
 +
doxygen lapack       multiprocess       Python       vectorclass
 +
eigen LCGCMT       multiprocessing       python_dateutil  wcwidth
 +
elasticsearch LCG_contrib_x86_64-centos7-gcc62-dbg.txt    mysql       python_gitlab    wheel
 +
entrypoints LCG_contrib_x86_64-centos7-gcc62-opt.txt    mysql_python       pytimber       widgetsnbextension
 +
  expat LCG_contrib_x86_64-slc6-gcc49-dbg.txt       nbconvert       pytools       xapian
 +
fastjet LCG_contrib_x86_64-slc6-gcc49-opt.txt       nbformat       pytz       XercesC
 +
  fftw3 LCG_contrib_x86_64-slc6-gcc62-dbg.txt       networkx       pyxml       xqilla
 +
  fjcontrib LCG_contrib_x86_64-slc6-gcc62-opt.txt       neurobayes       pyyaml       xrootd
 +
fplll LCG_contrib_x86_64-ubuntu1604-gcc54-dbg.txt  neurobayes_expert       pyzmq       xrootd_python
 +
  freetype LCG_contrib_x86_64-ubuntu1604-gcc54-opt.txt  ninja       QMtest       xz
 +
  frontier_client  lcgenv       nose       qt       yamlcpp
 +
  ftjam LCG_externals_x86_64-centos7-gcc62-dbg.txt  notebook       qt5       zeromq
 +
  future LCG_externals_x86_64-centos7-gcc62-opt.txt   numexpr       qtconsole       zlib
 +
  futures LCG_externals_x86_64-slc6-gcc49-dbg.txt      numpy       qwt
 +
  gcc LCG_externals_x86_64-slc6-gcc49-opt.txt      omniorb       R
 +
  Geant4 LCG_externals_x86_64-slc6-gcc62-dbg.txt      openssl       rangev3
 +
  genshi LCG_externals_x86_64-slc6-gcc62-opt.txt     oracle       RELAX
 
   
 
   
 
There is a lot of stuff in here that might be interesting to you (eg ROOT and Geant4). See that gcc is also listed, list that directory to see the options:
 
There is a lot of stuff in here that might be interesting to you (eg ROOT and Geant4). See that gcc is also listed, list that directory to see the options:
  stbc-022:LCG_85b> ls gcc
+
  stbc-i2:LCG_87> ls gcc
  4.9.3
+
4.9.1 4.9.3 6.2.0
  stbc-022:LCG_85b> ls gcc/4.9.3
+
  stbc-i2:LCG_87> ls gcc/4.9.3
  x86_64-centos7 x86_64-slc6
+
  x86_64-slc6
  
So gcc 4.9.3 is there, and there are two versions.  Now the /etc/issue business above becomes relevant: it said "Scientific Linux CERN SLC release 6.8" which corresponds to the "slc6" version here, so that's the one you want to use in this example.  
+
So gcc 4.9.3 is there, along with two other versions.  Now the /etc/issue business above becomes relevant: it said "Scientific Linux CERN SLC release 6.8" which corresponds to the "slc6" here, so this one will work for you:  
  stbc-022:LCG_85b> ls gcc/4.9.3/x86_64-slc6
+
  stbc-i2:LCG_87> ls gcc/4.9.3/x86_64-slc6
 
  bin  include  lib  lib64  libexec  setup.csh  setup.csh~  setup.sh  setup.sh~  share
 
  bin  include  lib  lib64  libexec  setup.csh  setup.csh~  setup.sh  setup.sh~  share
  

Revision as of 15:43, 21 December 2016

Introduction

So you want a modern C compiler and/or some other tool. As of this writing (2 November 2016) the normal desktop machines have

gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)

as the default compiler. This is not recent enough for much of the HEP code ecosystem.

Modern C compilers can be found in a repository called "SFT" which is maintained by CERN, and distributed via something called CVMFS which is installed on stoomboot machines and most of the desktop machines (older ones may not have it). You can check whether you have it with:

ls /cvmfs/sft.cern.ch

if you don't get an error (and it says "lcg") then you're good. What you'll need to do to set up the compiler is:

  1. figure out what kind of platform you'll be running on
  2. poke around in SFT to find out what the current release is
  3. run the setup script
  4. compile your code and win the Nobel Prize.

Figure out your platform

You'll generally need to compile for a specific platform; go (one of) the machine(s) on which you'll be running your code and type

cat /etc/issue

Today on one of the stoomboot worker nodes, it says

Scientific Linux CERN SLC release 6.8 (Carbon)
Kernel \r on an \m

Remember that first line, that tells you the platform, you'll need it shortly.

Poke around in SFT and find the relevant release and compiler

stbc-i2:~> cd /cvmfs/sft.cern.ch/lcg/releases
stbc-i2:releases> ls -d LCG_* 
LCG_71	     LCG_81c	     LCG_82rootaas4  LCG_84swan3  LCG_86
LCG_71root6  LCG_81d	     LCG_82rootaas5  LCG_84swan4  LCG_87
LCG_72a      LCG_81e	     LCG_82rootaas6  LCG_85	  LCG_latest
LCG_75root6  LCG_81f	     LCG_83	     LCG_85a	  LCG_rootext20161010
LCG_78root6  LCG_82	     LCG_84	     LCG_85b	  LCG_rootext20161129
LCG_79	     LCG_82rootaas1  LCG_84a	     LCG_85swan1
LCG_80	     LCG_82rootaas2  LCG_84swan1     LCG_85swan2
LCG_81b      LCG_82rootaas3  LCG_84swan2     LCG_85swan3

Note the LCG87 is the highest number ... this is generally the latest full release. Take a look at this page for some documentation on what the various releases include.

Note I've replaced below, "LCG_87" with "LCG_XXX" in commands to help avoid mistakes made via blind copy-and-pasting. Change the XXX to whatever you find the most recent release to be!

stbc-022:releases> cd /cvmfs/sft.cern.ch/lcg/releases/LCG_XXX
stbc-i2:LCG_XXX> ls
4suite		 gmp					      LCG_externals_x86_64-ubuntu1604-gcc54-dbg.txt   pacparser        requests
AIDA		 gperftools				      LCG_externals_x86_64-ubuntu1604-gcc54-opt.txt   pandas	       ROOT
astroid		 graphviz				      LCG_generators_x86_64-centos7-gcc62-dbg.txt     pathlib2	       root_numpy
autoconf	 Grid					      LCG_generators_x86_64-centos7-gcc62-opt.txt     pathos	       rootpy
automake	 GSL					      LCG_generators_x86_64-slc6-gcc49-dbg.txt	      pcre	       rpy2
backports	 gtest					      LCG_generators_x86_64-slc6-gcc49-opt.txt	      pexpect	       scikitlearn
blas		 hadoop					      LCG_generators_x86_64-slc6-gcc62-dbg.txt	      pickleshare      scipy
Boost		 hepdata_converter			      LCG_generators_x86_64-slc6-gcc62-opt.txt	      pip	       setuptools
castor		 hepdata_validator			      LCG_generators_x86_64-ubuntu1604-gcc54-dbg.txt  pkg_config       simplegeneric
ccache		 HepMC					      LCG_generators_x86_64-ubuntu1604-gcc54-opt.txt  pox	       simplejson
certifi		 HepPDT					      lcov					      ppft	       singledispatch
clhep		 hive					      lhapdfsets				      prettytable      sip
CMake		 igprof					      libaio					      processing       six
cmaketools	 ipykernel				      libgit2					      prompt_toolkit   sollya
cmmnbuild	 ipython				      libsvm					      protobuf	       soqt
cmt		 ipython_genutils			      libtool					      psutil	       spark
coin3d		 ipywidgets				      libunwind					      ptyprocess       sqlalchemy
configparser	 java					      libxml2					      py	       sqlite
COOL		 jemalloc				      libxslt					      py2neo	       stomppy
CORAL		 Jinja2					      logilabcommon				      py4j	       storm
CouchDB		 joblib					      lxml					      pyanalysis       subprocess32
coverage	 jpype					      m4					      pyapigitlab      swig
cppgsl		 jsonc					      MarkupSafe				      pydot	       sympy
CppUnit		 jsoncpp				      matplotlib				      pydot_ng	       tbb
curl		 jsonschema				      maven					      pygments	       terminado
cx_oracle	 jupyter				      MCGenerators				      pygraphics       theano
cycler		 jupyter_client				      messaging					      pygsi	       tornado
cython		 jupyter_console			      metakernel				      pylint	       traitlets
Davix		 jupyter_contrib_core			      mistune					      pyminuit	       urllib3
DD4hep		 jupyter_contrib_nbextensions		      mock					      pyparsing        uuid
decorator	 jupyter_core				      mpfi					      pyqt	       valgrind
dill		 jupyter_nbextensions_configurator	      mpfr					      pyqt5	       vdt
distribute	 keras					      mpich2					      pytest	       VecGeom
doxygen		 lapack					      multiprocess				      Python	       vectorclass
eigen		 LCGCMT					      multiprocessing				      python_dateutil  wcwidth
elasticsearch	 LCG_contrib_x86_64-centos7-gcc62-dbg.txt     mysql					      python_gitlab    wheel
entrypoints	 LCG_contrib_x86_64-centos7-gcc62-opt.txt     mysql_python				      pytimber	       widgetsnbextension
expat		 LCG_contrib_x86_64-slc6-gcc49-dbg.txt	      nbconvert					      pytools	       xapian
fastjet		 LCG_contrib_x86_64-slc6-gcc49-opt.txt	      nbformat					      pytz	       XercesC
fftw3		 LCG_contrib_x86_64-slc6-gcc62-dbg.txt	      networkx					      pyxml	       xqilla
fjcontrib	 LCG_contrib_x86_64-slc6-gcc62-opt.txt	      neurobayes				      pyyaml	       xrootd
fplll		 LCG_contrib_x86_64-ubuntu1604-gcc54-dbg.txt  neurobayes_expert				      pyzmq	       xrootd_python
freetype	 LCG_contrib_x86_64-ubuntu1604-gcc54-opt.txt  ninja					      QMtest	       xz
frontier_client  lcgenv					      nose					      qt	       yamlcpp
ftjam		 LCG_externals_x86_64-centos7-gcc62-dbg.txt   notebook					      qt5	       zeromq
future		 LCG_externals_x86_64-centos7-gcc62-opt.txt   numexpr					      qtconsole        zlib
futures		 LCG_externals_x86_64-slc6-gcc49-dbg.txt      numpy					      qwt
gcc		 LCG_externals_x86_64-slc6-gcc49-opt.txt      omniorb					      R
Geant4		 LCG_externals_x86_64-slc6-gcc62-dbg.txt      openssl					      rangev3
genshi		 LCG_externals_x86_64-slc6-gcc62-opt.txt      oracle					      RELAX

There is a lot of stuff in here that might be interesting to you (eg ROOT and Geant4). See that gcc is also listed, list that directory to see the options:

stbc-i2:LCG_87> ls gcc
4.9.1  4.9.3  6.2.0
stbc-i2:LCG_87> ls gcc/4.9.3
x86_64-slc6

So gcc 4.9.3 is there, along with two other versions. Now the /etc/issue business above becomes relevant: it said "Scientific Linux CERN SLC release 6.8" which corresponds to the "slc6" here, so this one will work for you:

stbc-i2:LCG_87> ls gcc/4.9.3/x86_64-slc6
bin  include  lib  lib64  libexec  setup.csh  setup.csh~  setup.sh  setup.sh~  share

Run the setup script

You want that setup.sh file ..

source /cvmfs/sft.cern.ch/lcg/releases/LCG_XXX/gcc/X.Y.Z/x86_64-platform/setup.sh

Change XXX to the right LCG release tag, X.Y.Z to the right gcc version you found, and "platform" to the right platform you found.

note: csh users need the setup.csh file We (PDP) assume that the csh setup works correctly, none of us use csh, hence we don't know for sure ;-)

Compile your code

Now you'll have the right stuff in your path, etc to use this version of gcc:

stbc-022:x86_64-slc6> gcc --version
gcc (GCC) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

stbc-022:x86_64-slc6> g++ --version
g++ (GCC) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Enjoy that prize. Don't forget to mention the PDP group in your speech.