RunningSPEC
Note: some steps can only be completed successfully by DavidG, since he owns the licenses needed for the Intel Compiler Suite and for SmartHeap. There is only one Intel license (commercial) available for doing this work, as well as only a single SmartHeap license (still on order). But, in principle, anyone at Nikhef is entitled to run the SPEC benchmark with their favourite plain old dirty gcc compiler and with conventional heap management. Access to the SPEC suites is limited to the NDPF systemAdministrator group (with the NDPF login) only to make sure that the code does not leak out, and that anyone running the benchmark does it in a compliant way.
Prepare the machine
- Install a machine with CentOS5 x86-64, e.g. via quattor or manually. Make sure it's a physical machine, and that it will be dedicated to the benchmark running for at least as long as you need it. Make sure numactl and taskset (from util-linux) are available on the machine.
Using the Intel compiler
If you want to get proper performace from the benchmark, and for vendor acceptance tests, you must use the Intel compiler:
- As root, install the Intel compiler suite (version 11.0) from https://www.nikhef.nl/grid/ndpf/files/local/intel/. Install both the i386 and the x86_64 varieties.
- As root, run the script intel-Compiler-postinstall.sh to set the installation destination into the generated scripts.
- As root, install the license file to /opt/intel/licenses/icc-davidg.lic and chown it to davidg:users, mode 0600.
- Add to your profile:
PATH=$PATH:/opt/intel/Compiler/11.0/081/bin/intel64/ export PATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/Compiler/11.0/081/lib/intel64 export LD_LIBRARY_PATH LANG=C export LANG
Using SmartHeap
Install the SmartHeap library:
- obtain the appropriate distribution and license from MicroQuill. The current 64 bit version is called sh8linux64b.tar.gz, and the 32-bit version sh81linux.tar.gz.
- unpack the tar-balls, e.g. to the location "/opt/SmartHeap8-x86_64/", so that the "lib/" directory is right below that one. Make the directory readable only for the user owning the license.
- Add SmartHeap to your library path and setup (bashrc is the best). For the 64 bits version:
LD_LIBRARY_PATH=/opt/SmartHeap8-x86_64/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH
- download a config file and flags file to enable the library. The flags file is essential, or you'll get an 'invalid run' complaint from Spec2006
Installing and running SPEC2006
- From the installation directory retrieve the SPEC2006 sources cpu2006.tar.bz2 (you'll need your NDPF username to retrieve the file, and be a member of the systemAdministrators group: for SPEC2006 and SPEC2000 Nikhef has a site license).
- Unpack the spec2006 sources in a dedicated directory (to prevent pollution):
mkdir -p benchmarking && ( cd benchmarking ; bzcat ../cpu2006.tar.bz2 | tar xf - )
- change to the benchmark directory
cd benchmarking
- 'install' the suite by running in the benchmarking/ directory:
./install.sh
and agreeing that the source and installation directory are the same.
- copy the proper configuration file from the private repository, for example:
- cpu2006-icc11-noSH-rate-nikhef-ppfB0.cfg
- for a configuration without SmartHeap but assuming some special stack settings in the notes. Put it in config/cpu2006-icc11-noSH-rate-nikhef-ppfB0.cfg
- cpu2006-icc11-SH8-x86_64-rate-nikhef-ppfB1.cfg
- for a configuration with SmartHeap 8.0 x86-64, and similar SMP stack settings. Use this with the additional flags file at http://www.nikhef.nl/grid/ndpf/files/local/spec2006/smartheap_flags.xml
- For good measure, download the flags files for icc and smartheap from http://www.spec.org/cpu2006/flags/Intel-ic11.0-int-linux64-revE.xml http://www.nikhef.nl/grid/ndpf/files/local/spec2006/smartheap_flags.xml and and put it in the config/ directory as well.
- Rename the config file, and make sure the hardware specification notes are correct. Also update test date and OS environment specifications, please.
- set the SMP affinity in the config file to the number of available cores. This directly affect the rate measurement!
- Set the environment correctly by SOURCING the setstack.sh script:
export KMP_STACK_SIZE=64M ulimit -s unlimited
- source the SPEC environment by
. shrc
- run the benchmark, making sure to redirect logging to a file for analysis:
runspec --rate 8 -c cpu2006-icc11-noSH-rate-nikhef-ppfB0.cfg --machine=stoakleydp8cores -T base -o all int > run.log 2>&1 & disown %1
- wait for the test to complete, but of course don't load the machine with queries. A full, valid, run takes approx. 6-12 hours on a modern 8-core system.
Verifying the output
The result of your SPEC run in left in numbered files in the results/ directory in the benchmarking working directory. The files are called CINT2006.001.ref.*. For a successful run, please copy all of these files to our persistent storage area on the web server at login.nikhef.nl:/www/grid/ndpf/files/nikhef-only/spec2006-nikhef/results/CONFIGFILENAME/.
The past results can be found at https://www.nikhef.nl/grid/ndpf/files/nikhef-only/spec2006-nikhef/results/.
If ever the CINT2006.XXX.ref.txt file start with this:
############################################################################## # INVALID RUN -- INVALID RUN -- INVALID RUN -- INVALID RUN -- INVALID RUN # # # # Your run was marked invalid because ...
you know you've messed up and you MUST NOT use the results. Please review the log file and error messages and try again.
If your result is very far form the ones published on http://www.spec.org/cpu2006/results/, then also review your setup. Was the Intel compiler used? Was SmartHeap correctly installed? Etc. Of course, if you used gcc than your performance will be down by 30-50% anyway.
Typical numbers for the different compiler scenarios are, on an 8-core decent and modern system of 2008:
Condition | Performance metric |
Vendor provided public value | 107 |
GCC v3.4 | 63.2 |
ICC11.0, no SH | 93.7 |
ICC11.0, with SH 8.0 x86-64 | 103 |
Other benchmarks
The LCG community prefers to use a lousy compiler and doesnt want to spend time to optimize the code and make it portable, so the normal benchmarks are not representative of the relative performance of these applications. They have therefore decided to run a representative set of benchmarks (cpp_all), but with compiler settings and flags that reflect their way of working. These tests are described at https://twiki.cern.ch/twiki/bin/view/FIOgroup/TsiBenchHEPSPECWlcg. The same machine as used above yields 69.11 on this 'hep-spec' benchmark.