User:Dennisvd@nikhef.nl/VASP

Compiling Openmpi on stoomboot

• Make sure the settings for the Intel compiler are correct. I've added this snippet to my .bash_profile:

if [ -f /opt/intel/composer_xe_2013.5.192/bin/compilervars.sh ]; then
    . /opt/intel/composer_xe_2013.5.192/bin/compilervars.sh intel64
fi

• Download the latest stable version of OpenMPI.

wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.5.tar.bz2

• Unpack and configure

tar xfj openmpi-1.6.5.tar.bz2
cd openmpi-1.6.5
mkdir build
cd build
../configure --disable-shared --enable-static --prefix=/project/detrd/vasp/openmpi-1.6.5 CC=icc CXX=icpc F77=ifort FC=ifort

• Build and deploy

make all
make install

Note: I'm not sure if openmpi still depends on torque-devel for tm.h. I've installed torque-devel on stbc-i1. (It appears this is still required.) The 'shared' and 'static' options are to make sure we do not end up with a run-time dependency on the Intel libraries which are not on the worker nodes.

The following snippet sets the paths for openmpi use.

if [ -f /project/detrd/vasp/openmpi-1.6.5 ]; then
    export LD_LIBRARY_PATH="/project/detrd/vasp/openmpi-1.6.5/lib:$LD_LIBRARY_PATH"
    export PATH="/project/detrd/vasp/openmpi-1.6.5/bin:$PATH"
fi

Test an openmpi program. Examples are found in the examples directory. I use ring_f90.

The compilation warns about missing functionality:

/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail

But this can probably be ignored (VASP doesn't use these C99 construct AFAIK).

A Torque submit script testrun-openmpi.pbs:

#!/bin/sh
#PBS -N vasp-openmpi-test
#PBS -l nodes=2:ppn=1
mpirun ${HOME}/vasp/ring_f90

And then:

qsub testrun-openmpi.pbs

Building VASP with OpenMPI support

I've copied the sources to a subdirectory mpi-build and did a

make clean

to start fresh. I copied the vasp.5.lib dir as-is from /project/detrd/vasp/src/vasp.5.lib/.

The Makefile needs two small edits compared to the original. In the MPI section:

FC=mpif90
FCL=$(FC)
CPP += -DMPI

and include support for parallel FFT:

FFT3D   = fftmpi.o fftmpi_map.o fft3dfurth.o fft3dlib.o 

The alternate relative path to the lib dir requires another change:

LIB  = -L./vasp.5.lib -ldmy \
     ./vasp.5.lib/linpack_double.o $(LAPACK) \
     $(BLAS)

The compiler crashed on one file, but replacing -O3 by -O2 worked around that:

ifort -FR -names lowercase -assume byterecl -heap-arrays 64 -O3 -ip -parallel \
-I/opt/intel/composer_xe_2013.5.192/mkl/include \
-I/opt/intel/composer_xe_2013.5.192/mkl/include/fftw  -c local_field.f90
: catastrophic error: **Internal compiler error: segmentation violation signal raised** \
Please report this error along with the circumstances in which it occurred in a Software \
Problem Report.  Note: File and line given may not be explicit cause of this error.
compilation aborted for local_field.f90 (code 1)