View source for User:Dennisvd@nikhef.nl/VASP

= Compiling Openmpi on stoomboot =

* Make sure the settings for the Intel compiler are correct. I've added this snippet to my <code>.bash_profile</code>:
 if [ -f /opt/intel/composer_xe_2013.5.192/bin/compilervars.sh ]; then
     . /opt/intel/composer_xe_2013.5.192/bin/compilervars.sh intel64
 fi
* Download the latest [http://www.open-mpi.org/software/ompi/v1.6/ stable version] of OpenMPI.
 wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.5.tar.bz2
* Unpack and configure
 tar xfj openmpi-1.6.5.tar.bz2
 cd openmpi-1.6.5
 mkdir build
 cd build
 ../configure --disable-shared --enable-static --prefix=/project/detrd/vasp/openmpi-1.6.5 --with-tm CC=icc CXX=icpc F77=ifort FC=ifort
* Build and  deploy
 make all
 make install

The torque-devel package should be installed so integration with the torque batch system will work.
The ''shared'' and ''static'' options are to make sure we do not end up with a run-time dependency on the Intel libraries which are not on the worker nodes. Unfortunately this makes the install base of openmpi ''very large'' (�600 MB).

The following snippet sets the paths for openmpi use.
 if [ -e /project/detrd/vasp/openmpi-1.6.5 ]; then
     export LD_LIBRARY_PATH="/project/detrd/vasp/openmpi-1.6.5/lib:$LD_LIBRARY_PATH"
     export PATH="/project/detrd/vasp/openmpi-1.6.5/bin:$PATH"
 fi


Test an openmpi program. Examples are found in the examples directory. I use ring_f90.

The compilation warns about missing functionality:
 /opt/intel/composer_xe_2013.5.192/compiler/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail
But this can probably be ignored (VASP doesn't use these C99 construct AFAIK).

A Torque submit script testrun-openmpi.pbs:
 #!/bin/sh
 #PBS -N vasp-openmpi-test
 #PBS -l nodes=2:ppn=1
 LD_LIBRARY_PATH="/project/detrd/vasp/openmpi-1.6.5/lib:$LD_LIBRARY_PATH"
 export LD_LIBRARY_PATH
 PATH="/project/detrd/vasp/openmpi-1.6.5/bin:$PATH"
 export PATH
 mpirun ${HOME}/vasp/ring_f90

And then:
 qsub testrun-openmpi.pbs

= Building VASP with OpenMPI support =

I've copied the sources to a subdirectory mpi-build and did a
 make clean
to start fresh. I copied the vasp.5.lib dir as-is from /project/detrd/vasp/src/vasp.5.lib/.

To use the Intel MKL FFT wrapper library, I had to build it myself (in the same directory):
 make -f /opt/intel/composer_xe_2013.5.192/mkl/interfaces/fftw3xf/makefile libintel64 

The Makefile needs some edits compared to the original. In the MPI section:
 FC=mpif90
 FCL=$(FC)
 CPP += -DMPI

and include support for parallel FFT with the Intel wrapper:
 FFT3D   = fftmpiw.o fftmpi_map.o  fftw3d.o  fft3dlib.o   libfftw3xf_intel.a
 INCS = -I$(MKLROOT)/include/fftw 

The prepocessor needs to be set to use NGZhalf:
 CPP     = $(CPP_)  -DHOST=\"LinuxIFC\" \
           -DCACHE_SIZE=12000 -DPGF90 -Davoidalloc -DNGZhalf \


The alternate relative path to the lib dir requires another change:
 LIB  = -L./vasp.5.lib -ldmy \
      ./vasp.5.lib/linpack_double.o $(LAPACK) \
      $(BLAS)

The compiler '''crashed''' on one file, but replacing -O3 by -O2 worked around that:

 ifort -FR -names lowercase -assume byterecl -heap-arrays 64 -O3 -ip -parallel \
 -I/opt/intel/composer_xe_2013.5.192/mkl/include \
 -I/opt/intel/composer_xe_2013.5.192/mkl/include/fftw  -c local_field.f90
 : catastrophic error: **Internal compiler error: segmentation violation signal raised** \
 Please report this error along with the circumstances in which it occurred in a Software \
 Problem Report.  Note: File and line given may not be explicit cause of this error.
 compilation aborted for local_field.f90 (code 1)

A line added to the makefile:
 local_field.o: OFLAG=-O2 -ip -parallel

Other lines for individual files were present that turned optimization down in a different way.


== Building VASP 5.4.1 ==

(Jan 2016)

New version has a better layout of the source tree, so more straightforward to build.

Newer Intel compiler installation in /opt/intel (actually /public/public_linux/intel but symlinked because the original path was /opt/intel and that's embedded in nearly every script).

I've made a few changes to the VASP makefile.include:

 # Precompiler options
 CPP_OPTIONS= -DMPI -DHOST=\"IFC91_ompi\" -DIFC \
              -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc \
              -DMPI_BLOCK=8000 -DscaLAPACK -Duse_collective \
              -DnoAugXCmeta -Duse_bse_te \
              -Duse_shmem -Dtbdyn
 
 CPP        = fpp -f_com=no -free -w0  $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
 
 FC         = mpif90
 FCL        = mpif90 -mkl
 
 FREE       = -free -names lowercase
 
 FFLAGS     = -assume byterecl
 OFLAG      = -O2
 OFLAG_IN   = $(OFLAG)
 DEBUG      = -O0
 
 MKL_PATH   = $(MKLROOT)/lib/intel64
 BLAS       =
 LAPACK     =
 BLACS      = -lmkl_blacs_custom_lp64
 SCALAPACK  = $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)
 
 OBJECTS    = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o \
              libfftw3xf_intel.a
 INCS       =-I$(MKLROOT)/include/fftw
 
 LLIBS      = $(SCALAPACK) $(LAPACK) $(BLAS)
 
 OBJECTS_O1 += fft3dfurth.o fftw3d.o fftmpi.o fftmpiw.o
 OBJECTS_O2 += fft3dlib.o
 
 # For what used to be vasp.5.lib
 CPP_LIB    = $(CPP)
 FC_LIB     = $(FC)
 CC_LIB     = icc
 CFLAGS_LIB = -O
 FFLAGS_LIB = -O1
 FREE_LIB   = $(FREE)
 
 OBJECTS_LIB= linpack_double.o getshmem.o
 
 # Normally no need to change this
 SRCDIR     = ../../src
 BINDIR     = ../../bin


The blacs_custom is my own openmpi based build

 /opt/intel/mkl/lib/intel64_lin/libmkl_blacs_custom_lp64.so

with the help of Intel's support pages. (And Ton helped install it along with the other blacs libraries.)