Instructions to run Trajectum at Nikhef

From ALICE Wiki
Jump to navigation Jump to search

NOTE: This page is still work in progress!

Before you start running your jobs at Nikhef, please make sure you join the stbc-mattermost channel using the following link.

For an overview of Trajectum, and to install and run it yourself, you need to be part of the Trajectum repository at codeberg.org (more on this later).

To see what simulations are run with Trajectum at Nikhef, ask Oza Passot (oza.passot@nikhef.nl) for access to the Github repository where you can find submission scripts, example runs with parameter files, analysis, and results.

Pre-installed packages

In order to run, Trajectum relies on several packages that are listed below.

These packages have already been installed on the Nikhef cluster, and are stored at: /project/alice/trajectum_codes

In this directory you find the following packages (last updated April 2026):

  • LHAPDF-6.5.5
  • fastjet-3.5.1
  • gemini_2026.3.13
  • pythia8315
  • smash
  • ghudi.3.11.0
  • CGAL-6.1

All users read and execute. You therefore need to link to these external libraries when configuring Trajectum in your own directory (this will be explained in section "set up Trajectum"), but don't need to install them yourself as this is already done.

If you do want to install the packages yourself, for example in another location (e.g. locally), please refer to the last section of this Wiki for instructions.

Please note: the packages are meant to be updated often such that newest version of Trajectum can be used.

Setting up Trajectum

In order to work on Trajectum you need to be part of the Trajectum repository at codeberg.org

You can request access by creating an account, with your usename including: FirstnameLastname. Then send an email to ... including your username to get access.

Once access to the Trajectum repository is granted to you, you need to copy the source code and configure Trajectum in your own directory, while linking to the above packages at /project/alice/trajectum_codes.

Note that the following steps can be used alongside the description of setting up Trajectum in the README file of the source code.

Also, it is important that gcc13 is sourced as this is the version of the compiler that was used to configured the packages Trajectum needs to work properly. Relevant sourcing is included in the shell script that compiles Trajectum.

First-time setup (to do once only)

Go to /data/alice/<yourNikhefusername> (or any other preferred directory)

Clone the Trajectum source code: 

git clone https://codeberg.org/Trajectum/Trajectum.git   

This will create a Trajectum directory.

Now we can run the git commands to initialise Trajectum as a git repository and be able to get the latest version while keeping our work intact. 

cd Trajectum
git init
git remote add origin ssh://git@codeberg.org/<your_codeberg_username>/Trajectum.git
git remote add upstream ssh://git@codeberg.org/Trajectum/Trajectum.git

Here origin refers to your own fork on codeberg, while upstream is the official Trajectum repository.

Updating to a new release

Follow this section if you would like to work with a specific version of Trajectum or want to update to the latest master branch, otherwise you can skip to the next subsection "Compiling Trajectum"

First you need to fetch the new release from upstream by running 

git fetch upstream

You can check which release branch are available 

git branch -r

Select the desired version by executing 

git merge upstream/<release_x.x>

Now you can continue with the next section to rebuild that version of Trajectum.

Compiling Trajectum

Still in the Trajectum directory, execute 

autoreconf --install -I /project/alice/trajectum_packages_new/autoconf-archive-2024.10.16/m4/

to create the configure executable.

Then copy this shell script and run it, as such: 

cp /project/alice/trajectum_codes/compiletrajectum.sh .
sh compiletrajectum.sh

In the compiletrajectum.sh shell script , configure and make take place. If the new release changes library versions in /project/alice/trajectum_codes/, update the codesdir paths accordingly.

If you run into errors at this point, refer to our later section on "Known issues".

Check that it works!

To check that Trajectum was properly set up, go to /data/alice/<yourNikhefusername>, and verify that the 3 following executables exist: 

Trajectum/src/analyze
Trajectum/src/collide
Trajectum/src/collect

Congratulations, Trajectum is ready to use!

Running Trajectum

You have several options to run simulations with Trajectum. You can execute commands on a stbc-ix interactive node on your terminal directly or you can submit jobs to the cluster using the HTCondor system.

In the 4 following subsections we guide you to run executables interactively. There are 4 executables in Trajectum: collide, smash (afterburner, optional), analyze, collect, which we will run for an example Pb-Pb collision at collision energy 2.76 TeV.

To submit jobs to the cluster, go to the later section about that.

Executing collide

The executable collide uses a parameter file to generate events until the produced particles reach the freeze-out surface.

In order to run collide you need to set: 

source /cvmfs/sft.cern.ch/lcg/releases/LCG_108/GSL/2.8/x86_64-el9-gcc13-opt/GSL-env.sh
source /cvmfs/sft.cern.ch/lcg/releases/LCG_108/hdf5/1.14.6/x86_64-el9-gcc13-opt/hdf5-env.sh
export PKG_CONFIG=/usr/bin/pkg-config
export LD_LIBRARY_PATH=/project/alice/trajectum_codes/copytonode:$LD_LIBRARY_PATH
export PATH=/project/alice/trajectum_codes/parallel/bin:$PATH

Within the Trajectum directory change to src/.

To run collide, execute the following command (still in the src directory): 

./collide ../parfiles/collisionPbPb2760.par

where collisionPbPb2760.par is an example file with parameter setting for a Pb-Pb collision at collision energy of 2.76 TeV. You can use other example parameter files in parfiles/ or have your own parameter file (in that case, make sure to modify the path in the above command).

This command should output 5 files starting with PbPb in the directory you are in (a 6th file is produced if there are errors).

Here is a short description of what each file contains:

  • PbPb.input: summary of input parameters, can be viewed in Mathematica
  • PbPb.main: particle positions, momenta, identities, to be fed to SMASHexecutable (see next section)
  • PbPb.bypass: same as filename. main but for particles that bypass hadron cascade (e.g. heavy quark hadrons)
  • PbPb.geometry: geometrical properties of plasma
  • PbPb.extra: initial state parameters, and event weights (needed for analyze executable)
  • PbPb.error: only produced if error!

Executing SMASH

The executable smash uses the output of collide (the PbPb.main file) to generate a hadron cascade. It's called the afterburner.

Before we can run smash, make sure the output files, especially PbPb.main, are located in the following directory: 

cd /data/alice/<yourname>/Trajectum/src

Sourcing is needed again 

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/releases/LCG_108/GSL/2.8/x86_64-el9-gcc13-opt/GSL-env.sh
source /cvmfs/sft.cern.ch/lcg/releases/LCG_108/hdf5/1.14.6/x86_64-el9-gcc13-opt/hdf5-env.sh
export PKG_CONFIG=/usr/bin/pkg-config
export LD_LIBRARY_PATH=/project/alice/trajectum_codes/copytonode:$LD_LIBRARY_PATH

(used to have export GINPUT=/project/alice/trajectum_packages_new/gemini_2024.8.8/build/share/gemini/ as well)

Then rename PbPb.main and run smash 

mv PbPb.main filename0  
/project/alice/trajectum_codes/smash/build/smash -I ../parfiles/smash.yaml

This should feed smash with the right particle list file (what we have called filename0 but can be renamed to what you want in the smash.yaml file).

Note that smash.yaml is the input file used here, and you can (and should) modify its parameters accordingly. For example, Nevents can be changed, as well as File_Directory and File_Prefix because filename0 was the name used by default by smash.yaml.

Executing analyze

Open a new terminal with a clean environment.

Source gcc and rivet versions, and shared libraries again 

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/releases/LCG_108/GSL/2.8/x86_64-el9-gcc13-opt/GSL-env.sh
source /cvmfs/sft.cern.ch/lcg/releases/LCG_108/hdf5/1.14.6/x86_64-el9-gcc13-opt/hdf5-env.sh
export PKG_CONFIG=/usr/bin/pkg-config
export LD_LIBRARY_PATH=/project/alice/trajectum_codes/copytonode:$LD_LIBRARY_PATH

(used to have export GINPUT=/project/alice/trajectum_packages_new/gemini_2024.8.8/build/share/gemini/ as well)

Still in the /data/alice/<yourname>/Trajectum/src directory, rename and move the output from smash 

mv data/0/particle_lists.oscar PbPb.afterburned

and run analyze 

./analyze ../parfiles/analysisPbPb.par

Executing collect

Following command 

./collect ../parfiles/collectPbPb.par

which should generate the final output file needed for analysis called PbPb.h5

Known issues

  • When configuring Trajectum
    • if configure cannot find eigen3, you can try to change the PKG path in the shell script to export PKG_CONFIG_PATH =/ usr / share / pkgconfig : $PKG_CONFIG_PATH
    • if configure cannot find HDF5, it might be a mismatch between the gcc version in your environment and in the HDF5 pkg-config file. You can make sure the correct gcc version is sourced by running source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_108 x86_64-el9-gcc13-opt
    • if you still get an error libhdf5.so: error adding symbols: DSO missing from command line you can try to force the library path by running make LIBS ="-lhdf5" instead of make
  • If you already (before following the steps above) executed make in your Trajectum directory within a slightly different environment as setup in "Setting up Trajectum", ./collide ../parfiles/collisionpp.par can crash with a segmentation fault giving multiple errors containing: ./collide: Symbol `<name>' has different size in shared object, consider re-linking.If you get these errors, before executing make , execute make clean before following the rest of the remaining steps.
  • When running analyze, you can get an error inconsistent number of events Nevents. To solve this, change Nevents in the file smash.yaml to be consistent with output of collide. Don't forget to set up the environment properly, as indicated at the beginning of every executable section of this Wiki page.

Run your Trajectum jobs on the Stoomboot cluster

This section outlines how to run your Trajectum simulation as jobs on the Nikhef Stoomboot cluster. For example job scripts, with parameter files and results, send an email to Oza Passot (oza.passot@nikhef.nl) to get access to a private Github repository with jobs scripts ready to run on the Stoomboot cluster.

For general information about the CPU Batch system at Nikhef and general guidelines on how to submit jobs, visit CT wiki and the HTCondor manual.

To submit your jobs to the Nikhef cluster you need a submission script <filename>.sub and an executable <filename>.sh that contains the code you want to execute.

An example submission script: 

executable              = testJob.sh
log                     = test.log
output                  = outfile.txt
error                   = errors.txt

+UseOS                  = "el9"
+JobCategory            = "short"

request_cpus            = 4
request_memory          = 8G

max_idle                = 50
queue

You can change the job category, numbers of requested CPUs, and requested memory according to the need of the job (note that collide typically needs a lot of CPUs and memory)

If you need to pass arguments to your executable you can add them by adding the following line to your submission script: 

arguments               = "var1 var2 var3"

The workflow in your testJob.sh could contain the following 

#!/bin/bash

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_107 x86_64-el9-gcc14-opt

outdir=/path/to/directory/where/you/want/to/store/your/output/files
export TRAJECTUM_DIR=/path/to/Trajectum/directory

export PKG_CONFIG=/usr/bin/pkg-config
export LD_LIBRARY_PATH=/project/alice/trajectum_packages/pythia8309/lib/:/project/alice/trajectum_packages/lib/:$LD_LIBRARY_PATH
export GINPUT=/project/alice/trajectum_packages/share/gemini

#create output directory if it does not exist yet
if [! -d ${outdir} ]
   then
 mkdir -p ${outdir}
fi

cd ${TMPDIR}
pwd

${TRAJECTUM_DIR}/src/collide ${TRAJECTUM_DIR}/parfiles/collisionpp.par

ls
mv pp.* ${outdir}

echo "Anything left?"
ls

Make sure you point outdir and TRAJECTUM_DIR to the right directories.

Before submitting the submission file make sure that you make your shell script an executable by running chmod +x testJob.sh otherwise the system considers it to be just a text file and it will not run. If you submit the script like this, the log, output and error files will be created in the same directory as the submission file, add a different path if you'd like to save it elsewhere. Make sure you’ve completed all the steps on this wiki up to and including the “Executing Collide” section — specifically, up to successfully running make in your cloned Trajectum directory.

To submit your workflow run: 

condor_submit <filename>.sub

You can check the status of your job using condor_q for more advanced commands please check the HTCondor manual.

And check your output, error & log files to see if everything works.

TIP: First test your executable on the interactive nodes to debug any issues, if it runs without problems submit it to the cluster!

How to install packages

Below are the commands followed to install the packages, for the purpose of documentation, and in case the packages need to be configured again (elsewhere)!.

All packages were uploaded and unpacked at /project/alice/trajectum_packages_new using the wget and tar -xvf commands. The guidelines are taken from the README file of the Trajectum repository on codeberg.org.

First step was to set up the environment. It was decided that gcc12 would be used consistently when configuring the packages. 

source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt
source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh
export PKG_CONFIG=/usr/bin/pkg-config

LHAPDF

cd /project/alice/trajectum_packages_new/LHAPDF-6.5.4
mkdir build
./configure --prefix=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build
make
make install
# Unpack 2 files (if not yet done), as specified in README

FastJet

cd /project/alice/trajectum_packages_new/fastjet-3.4.2
mkdir build
./configure --enable-limited-thread-safety --prefix=/project/alice/trajectum_packages_new/fastjet-3.4.2/build
make
make install

Pythia

cd /project/alice/trajectum_packages_new/pythia8309
./configure --cxx-common='-std=c++17 -march=native -mfpmath=sse -O3 -fPIC -pthread' --with-lhapdf6=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build --with-fastjet3=/project/alice/trajectum_packages_new/fastjet-3.4.2/build
make

Gemini

cd /project/alice/trajectum_packages_new/gemini_2024.8.8
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/project/alice/trajectum_packages_new/gemini_2024.8.8/build
make
make install

SMASH

export CMAKE_PREFIX_PATH=/project/alice/trajectum_packages_new/pythia8309
git clone https://github.com/smash-transport/smash.git
cd smash
git checkout SMASH-3.0

# Copy 5 header files from /data/alice/pchrist/Hydro/Trajectum/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/CMakeLists.txt /project/alice/trajectum_packages_new/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/particletype.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/pdgcode.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/pdgcode_constants.h /project/alice/trajectum_packages_new/smash/src/include/smash
cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/stringfunctions.h /project/alice/trajectum_packages_new/smash/src/include/smash

mkdir build
cd build
cmake  -DPythia_CONFIG_EXECUTABLE=/project/alice/trajectum_packages_new/pythia8309/bin/pythia8-config   -DPYTHIA8_DIR=/project/alice/trajectum_packages_new/pythia8309   -DTRY_USE_ROOT=OFF   -DBUILD_TESTING=OFF -DCMAKE_CXX_STANDARD=17  -DCMAKE_INSTALL_PREFIX=/project/alice/trajectum_packages_new/smash -S ../   -B /project/alice/trajectum_packages_new/smash/build
make
Retrieved from "https://wiki.nikhef.nl/alice/index.php?title=Instructions_to_run_Trajectum_at_Nikhef&oldid=552"