View source for Instructions to run Trajectum at Nikhef

'''<u><big>NOTE: This page is still work in progress!</big></u>'''

Before you start running your jobs at Nikhef, please make sure you join the stbc-mattermost channel using the following [https://mattermost.nikhef.nl/nikhef-members/channels/stbc-users link].  

==Pre-installed packages==
The packages needed to configure trajectum are stored at: <code>/project/alice/trajectum_packages</code> 

In this directory you find:

*<code>LHAPDF-6.5.4</code>

*<code>fastjet-3.4.2</code>
*<code>gemini_2024.8.8</code>
*<code>pythia8309</code>
*<code>smash</code>
*<code>autoconf-archive</code>

All users read and execute. 

==Access to Trajectum source code==
In order to work on Trajectum you need to be part of the trajectum repository at [https://codeberg.org codeberg.org] 

If you want you can request acces by creating an account, with your usename including: FirstnameLastname

''Then send an email to ... including your username to get access.'' 

==Setting up Trajectum==
The following steps can be used alongside the description of setting up Trajectum in the README file of the source code. 

Set up the environment in bash with the following function:
 setup_trajectum() { 
 source /cvmfs/alice.cern.ch/etc/login.sh
 source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_107 x86_64-el9-gcc14-opt
 export PKG_CONFIG=/usr/bin/pkg-config
 export LD_LIBRARY_PATH=/project/alice/trajectum_packages/pythia8309/lib/:/project/alice/trajectum_packages/lib/:$LD_LIBRARY_PATH
 export GINPUT=/project/alice/trajectum_packages/share/gemini/
 }
Alternatively, one can execute the commands inside the function.

Go to <code>/data/alice/<yourNikhefusername></code> (or any other preferred directory)

Clone the Trajectum source code: 
 git clone https://codeberg.org/Trajectum/Trajectum.git   
This will create a Trajectum directory. 
In this directory execute: 
  autoreconf --install -I /project/alice/trajectum_packages/autoconf-archive/m4/ 
which will create the configure executable with which you can run
 ./configure --with-lhapdf-prefix=/project/alice/trajectum_packages --with-fastjet-prefix=/project/alice/trajectum_packages --with-pythia-prefix=/project/alice/trajectum_packages/pythia8309 --with-gemini-prefix=/project/alice/trajectum_packages
Then execute (still in the Trajectum directory) 
 make

==Executing collide==
In order to run collide you need to set: (no need to do this if you set up your environment with <code>setup_trajectum</code>)
 export LD_LIBRARY_PATH=/project/alice/trajectum_packages/pythia8309/lib/:/project/alice/trajectum_packages/lib/:$LD_LIBRARY_PATH
 export GINPUT=/project/alice/trajectum_packages/share/gemini/
Within the Trajectum directory change to <code>src/</code> 

In the src directory execute 
 ./collide ../parfiles/collisionpp.par
''collisionpp.par'' is an example file with parameter setting for a pp collision. You can use all the example parameter files in <code>parfiles/</code> 

This command should output 5 files starting with pp.

== Compiling smash (next section more detailed) ==
To compile smash one first needs to setup the environment by selecting the proper gcc and rivet versions (note that everything i.e. pythia, collisions, needs to be compiled again):

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt
source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh
export PKG_CONFIG=/usr/bin/pkg-config
export LD_LIBRARY_PATH=/data/alice/pchrist/Hydro/pythia8309/lib/:/project/alice/trajectum_packages/lib/:$LD_LIBRARY_PATH
export GINPUT=/project/alice/trajectum_packages/share/gemini/
unset CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=/data/alice/pchrist/Hydro/pythia8309

You then have to configure and compile pythia:

 cd pythia8309
 ./configure
 make

Go to the directory where smash is cloned, create the build directory and go inside:

 cd smash
 mkdir build
 cd build

Then make and install:

 cmake  -DPythia_CONFIG_EXECUTABLE=/data/alice/pchrist/Hydro/pythia8309/bin/pythia8-config   -DPYTHIA8_DIR=/data/alice/pchrist/Hydro/pythia8309   -DTRY_USE_ROOT=OFF   -DBUILD_TESTING=OFF -DCMAKE_CXX_STANDARD=17  -DCMAKE_INSTALL_PREFIX=/data/alice/pchrist/Hydro/smash -S ../   -B /data/alice/pchrist/Hydro/Trajectum/smash/build

 make install


== Executing smash ==
I first used <code>setup_trajectum</code> to setup the environment. 

First, inside your own directory in <code>/data/alice</code>, run: 
 mkdir smash 
 git clone <nowiki>https://github.com/smash-transport/smash.git</nowiki> 
 cd smash
 git checkout SMASH-3.0
 mkdir build
 cd build
Now you are in the build directory inside the smash directory of your own directory. 

We need to copy 5 header files that have been modified, from Panos' directory into your own, as such:
 cp CMakeLists.txt /data/alice/<yourname>/smash
 cp src/include/smash/particletype.h /data/alice/<yourname>/smash/src/include/smash
 cp src/include/smash/pdgcode.h /data/alice/<yourname>/smash/src/include/smash
 cp src/include/smash/pdgcode_constants.h /data/alice/<yourname>/smash/src/include/smash
 cp src/include/smash/stringfunctions.h /data/alice/<yourname>/smash/src/include/smash
You can check if this worked by running <code>git status</code> which should output a list of the 5 header files that were modified. We are still in the <code>smash/build</code> directory.

We can now run cmake:
 cmake  -DPythia_CONFIG_EXECUTABLE=/data/alice/pchrist/Hydro/pythia8309/bin/pythia8-config   -DPYTHIA8_DIR=/data/alice/pchrist/Hydro/pythia8309   -DTRY_USE_ROOT=OFF   -DBUILD_TESTING=OFF -DCMAKE_CXX_STANDARD=17  -DCMAKE_INSTALL_PREFIX=/data/Alice/<yourname>/smash -S /data/Alice/<yourname>/smash   -B /data/alice/<yourname>/smash/build
Then run   
 make install 
No more error! (I hope)

Then we can use the output of the <code>collide</code> executable, which should be a set of 5 files: <code>pp.main, pp.bypass, pp.extra, pp.geometry,</code> and <code>pp.input</code> by first
 cd /data/alice/<yourname>/trajectum/src
Note: make sure the output file, specifically pp.main, are located in that directory

Then rename <code>pp.main</code> and run <code>smash</code>
 mv pp.main filename0  
 /data/alice/<yourname>/smash/build/smash -i /data/alice/<yourname>/Trajectum/parfiles/smash.yaml
This should feed smash with the right particle list file (<code>filename0</code>)  
Note: make sure that the environment is set up, as indicated above (in the <code>setup_trajectum</code> function). 
==Known issues==
If you already (before following the steps above) executed <code>make</code>  in your Trajectum directory within a slightly different environment as setup in "Setting up Trajectum". 

<code>./collide ../parfiles/collisionpp.par</code>  can crash with a segmentation fault giving multiple errors containing: <code>./collide: Symbol `<name>' has different size in shared object, consider re-linking</code>

If you get these errors, before executing <code>make</code> , execute <code>make clean</code> before following the rest of the remaining steps.

==Run your Trajectum jobs on the Stoomboot cluster==
For general information about the CPU Batch system at Nikhef and general guidelines on how to submit jobs, visit [https://kb.nikhef.nl/ct/Batch_jobs.html CT wiki] and the [https://htcondor.readthedocs.io/en/latest/users-manual/submitting-a-job.html HTCondor manual]. 

To submit your jobs to the Nikhef cluster you need a submission script <code><filename>.sub</code> and an executable <code><filename>.sh</code> that contains the code you want to execute. 

An example submission script: 
 executable              = testJob.sh
 log                     = test.log
 output                  = outfile.txt
 error                   = errors.txt
 
 +UseOS                  = "el9"
 +JobCategory            = "short"
 
 request_cpus            = 4
 request_memory          = 8G
 
 max_idle                = 50
 queue
If you need to pass arguments to your executable you can add them by adding the following line to your submission script:
 arguments               = "var1 var2 var3"
The workflow in your <code>testJob.sh</code> could contain the following
 #!/bin/bash
 
 source /cvmfs/alice.cern.ch/etc/login.sh
 source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_107 x86_64-el9-gcc14-opt
 
 outdir=/path/to/directory/where/you/want/to/store/your/output/files
 export TRAJECTUM_DIR=/path/to/Trajectum/directory
 
 export PKG_CONFIG=/usr/bin/pkg-config
 export LD_LIBRARY_PATH=/project/alice/trajectum_packages/pythia8309/lib/:/project/alice/trajectum_packages/lib/:$LD_LIBRARY_PATH
 export GINPUT=/project/alice/trajectum_packages/share/gemini
 
 #create output directory if it does not exist yet
 if [! -d ${outdir} ]
    then
  mkdir -p ${outdir}
 fi
 
 cd ${TMPDIR}
 pwd
 
 ${TRAJECTUM_DIR}/src/collide ${TRAJECTUM_DIR}/parfiles/collisionpp.par
 
 ls
 mv pp.* ${outdir}
 
 echo "Anything left?"
 ls
Make sure you point <code>outdir</code> and <code>TRAJECTUM_DIR</code> to the right directories.

Before submitting the submission file make sure that you make your shell script an executable by running <code>chmod +x testJob.sh</code> otherwise the system considers it to be just a text file and it will not run. If you submit the script like this, the log, output and error files will be created in the same directory as the submission file, add a different path if you'd like to save it elsewhere. Make sure you’ve completed all the steps on this wiki up to and including the “Executing Collide” section — specifically, up to successfully running <code>make</code> in your cloned ''Trajectum'' directory. 

To submit your workflow run:
 condor_submit <filename>.sub 
You can check the status of your job using <code>condor_q</code> for more advanced commands please check the [https://htcondor.readthedocs.io/en/latest/ HTCondor manual].

And check your output, error & log files to see if everything works. 

'''TIP:'''  First test your executable on the interactive nodes to debug any issues, if it runs without problems submit it to the cluster!