Instructions to run Trajectum at Nikhef

NOTE: This page is still work in progress!

Before you start running your jobs at Nikhef, please make sure you join the stbc-mattermost channel using the following link.

Pre-installed packages

In order to run, Trajectum relies on several packages that are listed below.

These packages have already been installed on the Nikhef cluster, and are stored at: /project/alice/trajectum_packages_new

In this directory you find:

• LHAPDF-6.5.4

• fastjet-3.4.2
• gemini_2024.8.8
• pythia8309
• smash
• autoconf-archive

All users read and execute. You therefore need to link to these external libraries when configuring Trajectum in your own directory (this will be explained in section "set up Trajectum").

Below are the commands followed to install the packages, for the purpose of documentation (and in case the packages need to be configured again!). Please skip to the next section to set up Trajectum.

How to install packages

All packages were uploaded and unpacked at /project/alice/trajectum_packages_new using the wget and tar -xvf commands. The guidelines are taken from the README file of the Trajectum repository on codeberg.org.

First step was to set up the environment. It was decided that gcc12 would be used consistently when configuring the packages.

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt
source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh
export PKG_CONFIG=/usr/bin/pkg-config

LHAPDF

cd /project/alice/trajectum_packages_new/LHAPDF-6.5.4
mkdir build
./configure --prefix=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build
make
make install

FastJet

cd /project/alice/trajectum_packages_new/fastjet-3.4.2
mkdir build
./configure --enable-limited-thread-safety --prefix=/project/alice/trajectum_packages_new/fastjet-3.4.2/build
make
make install

Pythia

cd /project/alice/trajectum_packages_new/pythia8309
./configure --cxx-common='-std=c++17 -march=native -mfpmath=sse -O3 -fPIC -pthread' --with-lhapdf6=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build --with-fastjet3=/project/alice/trajectum_packages_new/fastjet-3.4.2/build
make

Gemini

cd /project/alice/trajectum_packages_new/gemini_2024.8.8
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/project/alice/trajectum_packages_new/gemini_2024.8.8/build
make
make install

SMASH

export CMAKE_PREFIX_PATH=/project/alice/trajectum_packages_new/pythia8309
git clone https://github.com/smash-transport/smash.git
cd smash
git checkout SMASH-3.0

# Copy 5 header files from /data/alice/pchrist/Hydro/Trajectum/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/CMakeLists.txt /project/alice/trajectum_packages_new/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/particletype.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/pdgcode.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/pdgcode_constants.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/stringfunctions.h /project/alice/trajectum_packages_new/smash/src/include/smash

mkdir build
cd build
cmake  -DPythia_CONFIG_EXECUTABLE=/project/alice/trajectum_packages_new/pythia8309/bin/pythia8-config   -DPYTHIA8_DIR=/project/alice/trajectum_packages_new/pythia8309   -DTRY_USE_ROOT=OFF   -DBUILD_TESTING=OFF -DCMAKE_CXX_STANDARD=17  -DCMAKE_INSTALL_PREFIX=/project/alice/trajectum_packages_new/smash -S ../   -B /project/alice/trajectum_packages_new/smash/build
make

Trajectum source code

In order to work on Trajectum you need to be part of the Trajectum repository at codeberg.org

You can request access by creating an account, with your usename including: FirstnameLastname

Then send an email to ... including your username to get access.

Once access to the Trajectum repository is granted to you, follow the next section to copy the source code and configure Trajectum in your own directory, while linking to the packages at /project/alice/trajectum_packages_new. Then, the 4 following sections each corresponds to an executable: collide, smash (afterburner), analyze, collect.

Setting up Trajectum

The following steps can be used alongside the description of setting up Trajectum in the README file of the source code.

Set up the environment in bash with the following commands:

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt
source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh
export PKG_CONFIG=/usr/bin/pkg-config

Please note that it is important that gcc12 is sourced as this is the version of the compiler that was used to configured the packages Trajectum needs to work properly.

Go to /data/alice/<yourNikhefusername> (or any other preferred directory)

Clone the Trajectum source code:

git clone https://codeberg.org/Trajectum/Trajectum.git   

This will create a Trajectum directory.

In this directory execute:

 autoreconf --install -I /project/alice/trajectum_packages_new/autoconf-archive-2024.10.16/m4/

which will create the configure executable with which you can run

./configure --with-lhapdf-prefix=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build --with-fastjet-prefix=/project/alice/trajectum_packages_new/fastjet-3.4.2/build --with-pythia-prefix=/project/alice/trajectum_packages_new/pythia8309 --with-gemini-prefix=/project/alice/trajectum_packages_new/gemini_2024.8.8/build

This was the command linking to the packages installed on the Nikhef cluster.

Then execute (still in the Trajectum directory)

make

Executing collide

The executable collide uses a parameter file to generate events until the produced particles reach the freeze-out surface.

In order to run collide you need to set:

export LD_LIBRARY_PATH=/project/alice/trajectum_packages_new/LHAPDF6.5.4/build/lib/:/project/alice/trajectum_packages_new/pythia8309/lib/:/project/alice/trajectum_packages_new/gemini_2024.8.8/build/lib/:/project/alice/trajectum_packages_new/fastjet-3.4.2/build/lib/:$LD_LIBRARY_PATH
export GINPUT=/project/alice/trajectum_packages_new/gemini_2024.8.8/build/share/gemini/

Within the Trajectum directory change to src/.

To run collide, execute the following command (still in the src directory):

./collide ../parfiles/collisionpp.par

where collisionpp.par is an example file with parameter setting for a pp collision. You can use other example parameter files in parfiles/ or have your own parameter file (in that case, make sure to modify the path in the above command).

This command should output 5 files starting with pp in the directory you are in (a 6th file is produced if there are errors).

Here is a short description of what each file contains:

• pp.input: summary of input parameters, can be viewed in Mathematica
• pp.main: particle positions, momenta, identities, to be fed to SMASHexecutable (see next section)
• pp.bypass: same as filename. main but for particles that bypass hadron cascade (e.g. heavy quark hadrons)
• pp.geometry: geometrical properties of plasma
• pp.extra: initial state parameters, and event weights (needed for analyze executable)
• pp.error: only produced if error!

Compiling smash (next section more detailed)

To compile smash one first needs to setup the environment by selecting the proper gcc and rivet versions (note that everything i.e. pythia, collisions, needs to be compiled again):

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt
source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh
export PKG_CONFIG=/usr/bin/pkg-config
export LD_LIBRARY_PATH=/data/alice/pchrist/Hydro/pythia8309/lib/:/project/alice/trajectum_packages/lib/:$LD_LIBRARY_PATH
export GINPUT=/project/alice/trajectum_packages/share/gemini/
unset CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=/data/alice/pchrist/Hydro/pythia8309

You then have to configure and compile pythia:

cd pythia8309
./configure
make

Go to the directory where smash is cloned, create the build directory and go inside:

cd smash
mkdir build
cd build

Then make and install:

cmake  -DPythia_CONFIG_EXECUTABLE=/data/alice/pchrist/Hydro/pythia8309/bin/pythia8-config   -DPYTHIA8_DIR=/data/alice/pchrist/Hydro/pythia8309   -DTRY_USE_ROOT=OFF   -DBUILD_TESTING=OFF -DCMAKE_CXX_STANDARD=17  -DCMAKE_INSTALL_PREFIX=/data/alice/pchrist/Hydro/smash -S ../   -B /data/alice/pchrist/Hydro/Trajectum/smash/build

make install

Executing SMASH

The executable smash uses the output of collide (the pp.main file) to generate a hadron cascade. It's called the afterburner.

I first used setup_trajectum to setup the environment.

First, inside your own directory in /data/alice, run:

mkdir smash 
git clone https://github.com/smash-transport/smash.git 
cd smash
git checkout SMASH-3.0
mkdir build
cd build

Now you are in the build directory inside the smash directory of your own directory.

We need to copy 5 header files that have been modified, from Panos' directory into your own, as such:

cp CMakeLists.txt /data/alice/<yourname>/smash
cp src/include/smash/particletype.h /data/alice/<yourname>/smash/src/include/smash
cp src/include/smash/pdgcode.h /data/alice/<yourname>/smash/src/include/smash
cp src/include/smash/pdgcode_constants.h /data/alice/<yourname>/smash/src/include/smash
cp src/include/smash/stringfunctions.h /data/alice/<yourname>/smash/src/include/smash

You can check if this worked by running git status which should output a list of the 5 header files that were modified.

Still in the smash/build directory, run

cmake  -DPythia_CONFIG_EXECUTABLE=/data/alice/pchrist/Hydro/pythia8309/bin/pythia8-config   -DPYTHIA8_DIR=/data/alice/pchrist/Hydro/pythia8309   -DTRY_USE_ROOT=OFF   -DBUILD_TESTING=OFF -DCMAKE_CXX_STANDARD=17  -DCMAKE_INSTALL_PREFIX=/data/Alice/<yourname>/smash -S /data/Alice/<yourname>/smash   -B /data/alice/<yourname>/smash/build

Then run

make install 

No more error! (I hope)

Then we can use the output of the collide executable, which should be a set of 5 files (description above): pp.main, pp.bypass, pp.extra, pp.geometry, and pp.input by first

cd /data/alice/<yourname>/trajectum/src

Note: make sure the output file, specifically pp.main, are located in that directory

Then rename pp.main and run smash

mv pp.main filename0  
/data/alice/<yourname>/smash/build/smash -i /data/alice/<yourname>/Trajectum/parfiles/smash.yaml

This should feed smash with the right particle list file (filename0)

Note: make sure that the environment is set up, as indicated above (in the setup_trajectum function).

Known issues

If you already (before following the steps above) executed make in your Trajectum directory within a slightly different environment as setup in "Setting up Trajectum".

./collide ../parfiles/collisionpp.par can crash with a segmentation fault giving multiple errors containing: ./collide: Symbol `<name>' has different size in shared object, consider re-linking

If you get these errors, before executing make , execute make clean before following the rest of the remaining steps.

Run your Trajectum jobs on the Stoomboot cluster

For general information about the CPU Batch system at Nikhef and general guidelines on how to submit jobs, visit CT wiki and the HTCondor manual.

To submit your jobs to the Nikhef cluster you need a submission script <filename>.sub and an executable <filename>.sh that contains the code you want to execute.

An example submission script:

executable              = testJob.sh
log                     = test.log
output                  = outfile.txt
error                   = errors.txt

+UseOS                  = "el9"
+JobCategory            = "short"

request_cpus            = 4
request_memory          = 8G

max_idle                = 50
queue

If you need to pass arguments to your executable you can add them by adding the following line to your submission script:

arguments               = "var1 var2 var3"

The workflow in your testJob.sh could contain the following

#!/bin/bash

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_107 x86_64-el9-gcc14-opt

outdir=/path/to/directory/where/you/want/to/store/your/output/files
export TRAJECTUM_DIR=/path/to/Trajectum/directory

export PKG_CONFIG=/usr/bin/pkg-config
export LD_LIBRARY_PATH=/project/alice/trajectum_packages/pythia8309/lib/:/project/alice/trajectum_packages/lib/:$LD_LIBRARY_PATH
export GINPUT=/project/alice/trajectum_packages/share/gemini

#create output directory if it does not exist yet
if [! -d ${outdir} ]
   then
 mkdir -p ${outdir}
fi

cd ${TMPDIR}
pwd

${TRAJECTUM_DIR}/src/collide ${TRAJECTUM_DIR}/parfiles/collisionpp.par

ls
mv pp.* ${outdir}

echo "Anything left?"
ls

Make sure you point outdir and TRAJECTUM_DIR to the right directories.

Before submitting the submission file make sure that you make your shell script an executable by running chmod +x testJob.sh otherwise the system considers it to be just a text file and it will not run. If you submit the script like this, the log, output and error files will be created in the same directory as the submission file, add a different path if you'd like to save it elsewhere. Make sure you’ve completed all the steps on this wiki up to and including the “Executing Collide” section — specifically, up to successfully running make in your cloned Trajectum directory.

To submit your workflow run:

condor_submit <filename>.sub 

You can check the status of your job using condor_q for more advanced commands please check the HTCondor manual.

And check your output, error & log files to see if everything works.

TIP: First test your executable on the interactive nodes to debug any issues, if it runs without problems submit it to the cluster!

Re-compiling from scratch

These are the steps we followed on 26/09/2025, recompiling all packages with gcc12.

The packages are now all located in /project/alice/trajectum_packages_new. To set up Trajectum in your own directory using these packages, follow these steps.

First:

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt
export PKG_CONFIG=/usr/bin/pkg-config 

Then (similar to the "setting up Trajectum" section above, with the new paths), go to /data/alice/<yourNikhefusername> (or any other preferred directory)

Clone the Trajectum source code:

git clone https://codeberg.org/Trajectum/Trajectum.git   

This will create a Trajectum directory. In this directory execute:

 autoreconf --install -I /project/alice/trajectum_packages_new/autoconf-archive-2024.10.16/m4/

which will create the configure executable with which you can run

./configure --with-lhapdf-prefix=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build --with-fastjet-prefix=/project/alice/trajectum_packages_new/fastjet-3.4.2/build --with-pythia-prefix=/project/alice/trajectum_packages_new/pythia8309 --with-gemini-prefix=/project/alice/trajectum_packages_new/gemini_2024.8.8/build

Then execute (still in the Trajectum directory)

make

Next (again, with the new paths):

export LD_LIBRARY_PATH=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build/lib/:/project/alice/trajectum_packages_new/pythia8309/lib/:/project/alice/trajectum_packages_new/gemini_2024.8.8/build/lib/:/project/alice/trajectum_packages_new/fastjet-3.4.2/build/lib/:$LD_LIBRARY_PATH
export GINPUT=/project/alice/trajectum_packages_new/gemini_2024.8.8/build/share/gemini/ 

Going into /data/alice/<yourNikhefusername>/Trajectum/src, you should be able to run the collide executable as such

./collide ../parfiles/collisionpp.par

where collisionpp.par is an example file with parameter setting for a pp collision. This should take about 6 minutes and should output 5 files starting with pp in the directory you are in (a 6th file is produced if there are errors).

The Problem

Next is to run SMASH, but we encounter errors when cmake/make install it in /project/alice/trajectum_packages_new/smash/build. Here is what we did so far.

First we ran

export CMAKE_PREFIX_PATH=/project/alice/trajectum_packages_new/pythia8309
source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh

AND copied 5 header files from Panos' directory into my own, like this:

cp /data/alice/pchrist/Hydro/Trajectum/smash/CMakeLists.txt /project/alice/trajectum_packages_new/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/particletype.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/pdgcode.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/pdgcode_constants.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/stringfunctions.h /project/alice/trajectum_packages_new/smash/src/include/smash

Now going to /project/alice/trajectum_packages_new/smash/build we ran

cmake  -DPythia_CONFIG_EXECUTABLE=/project/alice/trajectum_packages_new/pythia8309/bin/pythia8-config   -DPYTHIA8_DIR=/project/alice/trajectum_packages_new/pythia8309   -DTRY_USE_ROOT=OFF   -DBUILD_TESTING=OFF -DCMAKE_CXX_STANDARD=17  -DCMAKE_INSTALL_PREFIX=/project/alice/trajectum_packages_new/smash -S ../   -B /project/alice/trajectum_packages_new/smash/build

which works fine. However when executing make install or make the process reaches 97% before crashing when trying to "Linking CXX executable ../smash". Seems like it has to do with CMakeFiles. I remember we had it before and it seems like sourcing the rivet and copying the header files soved it last time. Maybe source rivet before compiling all packages??