Instructions to run Trajectum at Nikhef
NOTE: This page is still work in progress!
Before you start running your jobs at Nikhef, please make sure you join the stbc-mattermost channel using the following link.
Pre-installed packages
In order to run, Trajectum relies on several packages that are listed below.
These packages have already been installed on the Nikhef cluster, and are stored at: /project/alice/trajectum_packages_new
In this directory you find:
LHAPDF-6.5.4
fastjet-3.4.2gemini_2024.8.8pythia8309smashautoconf-archive
All users read and execute. You therefore need to link to these external libraries when configuring Trajectum in your own directory (this will be explained in section "set up Trajectum"), but don't need to install them yourself as this is already done.
If you do want to install the packages yourself, for example in another location (e.g. locally), please refer to the last section of this Wiki for instructions.
Trajectum source code
In order to work on Trajectum you need to be part of the Trajectum repository at codeberg.org
You can request access by creating an account, with your usename including: FirstnameLastname
Then send an email to ... including your username to get access.
Once access to the Trajectum repository is granted to you, follow the next section to copy the source code and configure Trajectum in your own directory, while linking to the packages at /project/alice/trajectum_packages_new.
Then, the 4 following sections each corresponds to an executable: collide, smash (afterburner), analyze, collect.
Setting up Trajectum
The following steps can be used alongside the description of setting up Trajectum in the README file of the source code.
Set up the environment in bash with the following commands:
source /cvmfs/alice.cern.ch/etc/login.sh source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh export PKG_CONFIG=/usr/bin/pkg-config
Please note that it is important that gcc12 is sourced as this is the version of the compiler that was used to configured the packages Trajectum needs to work properly.
Go to /data/alice/<yourNikhefusername> (or any other preferred directory)
Clone the Trajectum source code:
git clone https://codeberg.org/Trajectum/Trajectum.git
This will create a Trajectum directory.
In this directory execute:
autoreconf --install -I /project/alice/trajectum_packages_new/autoconf-archive-2024.10.16/m4/
which will create the configure executable with which you can run
./configure --with-lhapdf-prefix=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build --with-fastjet-prefix=/project/alice/trajectum_packages_new/fastjet-3.4.2/build --with-pythia-prefix=/project/alice/trajectum_packages_new/pythia8309 --with-gemini-prefix=/project/alice/trajectum_packages_new/gemini_2024.8.8/build
This was the command linking to the packages installed on the Nikhef cluster.
Then execute (still in the Trajectum directory)
make
Executing collide
The executable collide uses a parameter file to generate events until the produced particles reach the freeze-out surface.
In order to run collide you need to set:
export LD_LIBRARY_PATH=/project/alice/trajectum_packages_new/LHAPDF6.5.4/build/lib/:/project/alice/trajectum_packages_new/pythia8309/lib/:/project/alice/trajectum_packages_new/gemini_2024.8.8/build/lib/:/project/alice/trajectum_packages_new/fastjet-3.4.2/build/lib/:$LD_LIBRARY_PATH export GINPUT=/project/alice/trajectum_packages_new/gemini_2024.8.8/build/share/gemini/
Within the Trajectum directory change to src/.
To run collide, execute the following command (still in the src directory):
./collide ../parfiles/collisionpp.par
where collisionpp.par is an example file with parameter setting for a pp collision. You can use other example parameter files in parfiles/ or have your own parameter file (in that case, make sure to modify the path in the above command).
This command should output 5 files starting with pp in the directory you are in (a 6th file is produced if there are errors).
Here is a short description of what each file contains:
pp.input: summary of input parameters, can be viewed in Mathematicapp.main: particle positions, momenta, identities, to be fed toSMASHexecutable (see next section)pp.bypass: same as filename. main but for particles that bypass hadron cascade (e.g. heavy quark hadrons)pp.geometry: geometrical properties of plasmapp.extra: initial state parameters, and event weights (needed foranalyzeexecutable)pp.error: only produced if error!
Executing SMASH
The executable smash uses the output of collide (the pp.main file) to generate a hadron cascade. It's called the afterburner.
Before we can run smash, make sure the output files, especially pp.main, are located in the following directory:
cd /data/alice/<yourname>/Trajectum/src
Sourcing is needed again
source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt
Then rename pp.main and run smash
mv pp.main filename0 /project/alice/trajectum_packages_new/smash/build/smash -i /data/alice/<yourname>/Trajectum/parfiles/smash.yaml
This should feed smash with the right particle list file (what we have called filename0).
Note that smash.yaml is the input file used here, and you can (and should) modify its parameters accordingly. For example, Nevents can be changed, as well as File_Directory and File_Prefix because filename0 was the name used by default by smash.yaml.
Note: make sure that the environment is set up, as indicated above
Executing analyze
Open a new terminal with a clean environment.
Source gcc and rivet versions, and shared libraries again
source /cvmfs/alice.cern.ch/etc/login.sh source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh export PKG_CONFIG=/usr/bin/pkg-config export LD_LIBRARY_PATH=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build/lib/:/project/alice/trajectum_packages_new/pythia8309/lib/:/project/alice/trajectum_packages_new/gemini_2024.8.8/build/lib/:/project/alice/trajectum_packages_new/fastjet-3.4.2/build/lib/:$LD_LIBRARY_PATH export GINPUT=/project/alice/trajectum_packages_new/gemini_2024.8.8/build/share/gemini/
Still in the /data/alice/<yourname>/Trajectum/src directory, rename and move the output from smash
mv data/0/particle_lists.oscar pp.afterburned
and run analyze
./analyze /data/alice/<yourname>/Trajectum/parfiles/analysispp.par
Executing collect
Following command
./collect /data/alice/<yourname>/Trajectum_new/Trajectum/parfiles/collectpp.par
So far, Nevents is too high so that step is taking forever. We are working on it.
Known issues
- If you already (before following the steps above) executed
makein your Trajectum directory within a slightly different environment as setup in "Setting up Trajectum",./collide ../parfiles/collisionpp.parcan crash with a segmentation fault giving multiple errors containing:./collide: Symbol `<name>' has different size in shared object, consider re-linking.If you get these errors, before executingmake, executemake cleanbefore following the rest of the remaining steps. - When running
analyze, you can get an errorinconsistent number of events Nevents. To solve this, changeNeventsin the filesmash.yamlto be consistent with output ofcollide. Don't forget to set up the environment properly, as indicated at the beginning of every executable section of this Wiki page.
Run your Trajectum jobs on the Stoomboot cluster
For general information about the CPU Batch system at Nikhef and general guidelines on how to submit jobs, visit CT wiki and the HTCondor manual.
To submit your jobs to the Nikhef cluster you need a submission script <filename>.sub and an executable <filename>.sh that contains the code you want to execute.
An example submission script:
executable = testJob.sh log = test.log output = outfile.txt error = errors.txt +UseOS = "el9" +JobCategory = "short" request_cpus = 4 request_memory = 8G max_idle = 50 queue
If you need to pass arguments to your executable you can add them by adding the following line to your submission script:
arguments = "var1 var2 var3"
The workflow in your testJob.sh could contain the following
#!/bin/bash
source /cvmfs/alice.cern.ch/etc/login.sh
source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_107 x86_64-el9-gcc14-opt
outdir=/path/to/directory/where/you/want/to/store/your/output/files
export TRAJECTUM_DIR=/path/to/Trajectum/directory
export PKG_CONFIG=/usr/bin/pkg-config
export LD_LIBRARY_PATH=/project/alice/trajectum_packages/pythia8309/lib/:/project/alice/trajectum_packages/lib/:$LD_LIBRARY_PATH
export GINPUT=/project/alice/trajectum_packages/share/gemini
#create output directory if it does not exist yet
if [! -d ${outdir} ]
then
mkdir -p ${outdir}
fi
cd ${TMPDIR}
pwd
${TRAJECTUM_DIR}/src/collide ${TRAJECTUM_DIR}/parfiles/collisionpp.par
ls
mv pp.* ${outdir}
echo "Anything left?"
ls
Make sure you point outdir and TRAJECTUM_DIR to the right directories.
Before submitting the submission file make sure that you make your shell script an executable by running chmod +x testJob.sh otherwise the system considers it to be just a text file and it will not run. If you submit the script like this, the log, output and error files will be created in the same directory as the submission file, add a different path if you'd like to save it elsewhere. Make sure you’ve completed all the steps on this wiki up to and including the “Executing Collide” section — specifically, up to successfully running make in your cloned Trajectum directory.
To submit your workflow run:
condor_submit <filename>.sub
You can check the status of your job using condor_q for more advanced commands please check the HTCondor manual.
And check your output, error & log files to see if everything works.
TIP: First test your executable on the interactive nodes to debug any issues, if it runs without problems submit it to the cluster!
How to install packages
Below are the commands followed to install the packages, for the purpose of documentation, and in case the packages need to be configured again (elsewhere)!.
All packages were uploaded and unpacked at /project/alice/trajectum_packages_new using the wget and tar -xvf commands. The guidelines are taken from the README file of the Trajectum repository on codeberg.org.
First step was to set up the environment. It was decided that gcc12 would be used consistently when configuring the packages.
source /cvmfs/alice.cern.ch/etc/login.sh source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_104 x86_64-el9-gcc12-opt source /cvmfs/sft.cern.ch/lcg/releases/MCGenerators/rivet/3.1.8-79cbb/x86_64-el9-gcc12-opt/rivetenv.sh export PKG_CONFIG=/usr/bin/pkg-config
LHAPDF
cd /project/alice/trajectum_packages_new/LHAPDF-6.5.4 mkdir build ./configure --prefix=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build make make install # Unpack 2 files (if not yet done), as specified in README
FastJet
cd /project/alice/trajectum_packages_new/fastjet-3.4.2 mkdir build ./configure --enable-limited-thread-safety --prefix=/project/alice/trajectum_packages_new/fastjet-3.4.2/build make make install
Pythia
cd /project/alice/trajectum_packages_new/pythia8309 ./configure --cxx-common='-std=c++17 -march=native -mfpmath=sse -O3 -fPIC -pthread' --with-lhapdf6=/project/alice/trajectum_packages_new/LHAPDF-6.5.4/build --with-fastjet3=/project/alice/trajectum_packages_new/fastjet-3.4.2/build make
Gemini
cd /project/alice/trajectum_packages_new/gemini_2024.8.8 mkdir build cd build cmake .. -DCMAKE_INSTALL_PREFIX=/project/alice/trajectum_packages_new/gemini_2024.8.8/build make make install
SMASH
export CMAKE_PREFIX_PATH=/project/alice/trajectum_packages_new/pythia8309 git clone https://github.com/smash-transport/smash.git cd smash git checkout SMASH-3.0
# Copy 5 header files from /data/alice/pchrist/Hydro/Trajectum/smash cp /data/alice/pchrist/Hydro/Trajectum/smash/CMakeLists.txt /project/alice/trajectum_packages_new/smash cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/particletype.h /project/alice/trajectum_packages_new/smash/src/include/smash cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/pdgcode.h /project/alice/trajectum_packages_new/smash/src/include/smash cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/pdgcode_constants.h /project/alice/trajectum_packages_new/smash/src/include/smash cp /data/alice/pchrist/Hydro/Trajectum/smash/src/include/smash/stringfunctions.h /project/alice/trajectum_packages_new/smash/src/include/smash
mkdir build cd build cmake -DPythia_CONFIG_EXECUTABLE=/project/alice/trajectum_packages_new/pythia8309/bin/pythia8-config -DPYTHIA8_DIR=/project/alice/trajectum_packages_new/pythia8309 -DTRY_USE_ROOT=OFF -DBUILD_TESTING=OFF -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=/project/alice/trajectum_packages_new/smash -S ../ -B /project/alice/trajectum_packages_new/smash/build make