Difference between revisions of "User:Dennisvd@nikhef.nl/VASP"

From PDP/Grid Wiki
Jump to navigationJump to search
(Created page with "= Compiling Openmpi on stoomboot = * Make sure the settings for the Intel compiler are correct. I've added this snippet to my <code>.bash_profile</code>: if [ -f /opt/intel/...")
 
 
(18 intermediate revisions by the same user not shown)
Line 12: Line 12:
 
  mkdir build
 
  mkdir build
 
  cd build
 
  cd build
−
  ../configure --prefix=/project/detrd/vasp/openmpi-1.6.5 CC=icc CXX=icpc F77=ifort FC=ifort
+
  ../configure --disable-shared --enable-static --prefix=/project/detrd/vasp/openmpi-1.6.5 --with-tm CC=icc CXX=icpc F77=ifort FC=ifort
 
* Build and  deploy
 
* Build and  deploy
 
  make all
 
  make all
 
  make install
 
  make install
  
−
Note: I'm not sure if openmpi still depends on torque-devel for tm.h. I've installed torque-devel on stbc-i1.
+
The torque-devel package should be installed so integration with the torque batch system will work.
 +
The ''shared'' and ''static'' options are to make sure we do not end up with a run-time dependency on the Intel libraries which are not on the worker nodes. Unfortunately this makes the install base of openmpi ''very large'' (�600 MB).
 +
 
 +
The following snippet sets the paths for openmpi use.
 +
if [ -e /project/detrd/vasp/openmpi-1.6.5 ]; then
 +
    export LD_LIBRARY_PATH="/project/detrd/vasp/openmpi-1.6.5/lib:$LD_LIBRARY_PATH"
 +
    export PATH="/project/detrd/vasp/openmpi-1.6.5/bin:$PATH"
 +
fi
 +
 
 +
 
 +
Test an openmpi program. Examples are found in the examples directory. I use ring_f90.
 +
 
 +
The compilation warns about missing functionality:
 +
/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail
 +
But this can probably be ignored (VASP doesn't use these C99 construct AFAIK).
 +
 
 +
A Torque submit script testrun-openmpi.pbs:
 +
#!/bin/sh
 +
#PBS -N vasp-openmpi-test
 +
#PBS -l nodes=2:ppn=1
 +
LD_LIBRARY_PATH="/project/detrd/vasp/openmpi-1.6.5/lib:$LD_LIBRARY_PATH"
 +
export LD_LIBRARY_PATH
 +
PATH="/project/detrd/vasp/openmpi-1.6.5/bin:$PATH"
 +
export PATH
 +
mpirun ${HOME}/vasp/ring_f90
 +
 
 +
And then:
 +
qsub testrun-openmpi.pbs
 +
 
 +
= Building VASP with OpenMPI support =
 +
 
 +
I've copied the sources to a subdirectory mpi-build and did a
 +
make clean
 +
to start fresh. I copied the vasp.5.lib dir as-is from /project/detrd/vasp/src/vasp.5.lib/.
 +
 
 +
To use the Intel MKL FFT wrapper library, I had to build it myself (in the same directory):
 +
make -f /opt/intel/composer_xe_2013.5.192/mkl/interfaces/fftw3xf/makefile libintel64
 +
 
 +
The Makefile needs some edits compared to the original. In the MPI section:
 +
FC=mpif90
 +
FCL=$(FC)
 +
CPP += -DMPI
 +
 
 +
and include support for parallel FFT with the Intel wrapper:
 +
FFT3D  = fftmpiw.o fftmpi_map.o  fftw3d.o  fft3dlib.o  libfftw3xf_intel.a
 +
INCS = -I$(MKLROOT)/include/fftw
 +
 
 +
The prepocessor needs to be set to use NGZhalf:
 +
CPP    = $(CPP_)  -DHOST=\"LinuxIFC\" \
 +
          -DCACHE_SIZE=12000 -DPGF90 -Davoidalloc -DNGZhalf \
 +
 
 +
 
 +
The alternate relative path to the lib dir requires another change:
 +
LIB  = -L./vasp.5.lib -ldmy \
 +
      ./vasp.5.lib/linpack_double.o $(LAPACK) \
 +
      $(BLAS)
 +
 
 +
The compiler '''crashed''' on one file, but replacing -O3 by -O2 worked around that:
 +
 
 +
ifort -FR -names lowercase -assume byterecl -heap-arrays 64 -O3 -ip -parallel \
 +
-I/opt/intel/composer_xe_2013.5.192/mkl/include \
 +
-I/opt/intel/composer_xe_2013.5.192/mkl/include/fftw  -c local_field.f90
 +
: catastrophic error: **Internal compiler error: segmentation violation signal raised** \
 +
Please report this error along with the circumstances in which it occurred in a Software \
 +
Problem Report.  Note: File and line given may not be explicit cause of this error.
 +
compilation aborted for local_field.f90 (code 1)
 +
 
 +
A line added to the makefile:
 +
local_field.o: OFLAG=-O2 -ip -parallel
 +
 
 +
Other lines for individual files were present that turned optimization down in a different way.
 +
 
 +
 
 +
== Building VASP 5.4.1 ==
 +
 
 +
(Jan 2016)
 +
 
 +
New version has a better layout of the source tree, so more straightforward to build.
 +
 
 +
Newer Intel compiler installation in /opt/intel (actually /public/public_linux/intel but symlinked because the original path was /opt/intel and that's embedded in nearly every script).
 +
 
 +
I've made a few changes to the VASP makefile.include:
 +
 
 +
# Precompiler options
 +
CPP_OPTIONS= -DMPI -DHOST=\"IFC91_ompi\" -DIFC \
 +
              -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc \
 +
              -DMPI_BLOCK=8000 -DscaLAPACK -Duse_collective \
 +
              -DnoAugXCmeta -Duse_bse_te \
 +
              -Duse_shmem -Dtbdyn
 +
 +
CPP        = fpp -f_com=no -free -w0  $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
 +
 +
FC        = mpif90
 +
FCL        = mpif90 -mkl
 +
 +
FREE      = -free -names lowercase
 +
 +
FFLAGS    = -assume byterecl
 +
OFLAG      = -O2
 +
OFLAG_IN  = $(OFLAG)
 +
DEBUG      = -O0
 +
 +
MKL_PATH  = $(MKLROOT)/lib/intel64
 +
BLAS      =
 +
LAPACK    =
 +
BLACS      = -lmkl_blacs_custom_lp64
 +
SCALAPACK  = $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)
 +
 +
OBJECTS    = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o \
 +
              libfftw3xf_intel.a
 +
INCS      =-I$(MKLROOT)/include/fftw
 +
 +
LLIBS      = $(SCALAPACK) $(LAPACK) $(BLAS)
 +
 +
OBJECTS_O1 += fft3dfurth.o fftw3d.o fftmpi.o fftmpiw.o
 +
OBJECTS_O2 += fft3dlib.o
 +
 +
# For what used to be vasp.5.lib
 +
CPP_LIB    = $(CPP)
 +
FC_LIB    = $(FC)
 +
CC_LIB    = icc
 +
CFLAGS_LIB = -O
 +
FFLAGS_LIB = -O1
 +
FREE_LIB  = $(FREE)
 +
 +
OBJECTS_LIB= linpack_double.o getshmem.o
 +
 +
# Normally no need to change this
 +
SRCDIR    = ../../src
 +
BINDIR    = ../../bin
 +
 
 +
 
 +
The blacs_custom is my own openmpi based build
 +
 
 +
/opt/intel/mkl/lib/intel64_lin/libmkl_blacs_custom_lp64.so
 +
 
 +
with the help of Intel's support pages. (And Ton helped install it along with the other blacs libraries.)

Latest revision as of 14:32, 28 January 2016

Retrieved from "https://wiki.nikhef.nl/grid/index.php?title=User:Dennisvd@nikhef.nl/VASP&oldid=9902"