Difference between revisions of "User:Dennisvd@nikhef.nl/VASP"
From PDP/Grid Wiki
Jump to navigationJump to search| (11 intermediate revisions by the same user not shown) | |||
| Line 12: | Line 12: | ||
mkdir build | mkdir build | ||
cd build | cd build | ||
| − | ../configure --disable-shared --enable-static --prefix=/project/detrd/vasp/openmpi-1.6.5 CC=icc CXX=icpc F77=ifort FC=ifort | + | ../configure --disable-shared --enable-static --prefix=/project/detrd/vasp/openmpi-1.6.5 --with-tm CC=icc CXX=icpc F77=ifort FC=ifort |
* Build and deploy | * Build and deploy | ||
make all | make all | ||
make install | make install | ||
| − | + | The torque-devel package should be installed so integration with the torque batch system will work. | |
| − | The 'shared' and 'static' options are to make sure we do not end up with a run-time dependency on the Intel libraries which are not on the worker nodes. | + | The ''shared'' and ''static'' options are to make sure we do not end up with a run-time dependency on the Intel libraries which are not on the worker nodes. Unfortunately this makes the install base of openmpi ''very large'' (�600 MB). |
The following snippet sets the paths for openmpi use. | The following snippet sets the paths for openmpi use. | ||
| − | if [ - | + | if [ -e /project/detrd/vasp/openmpi-1.6.5 ]; then |
export LD_LIBRARY_PATH="/project/detrd/vasp/openmpi-1.6.5/lib:$LD_LIBRARY_PATH" | export LD_LIBRARY_PATH="/project/detrd/vasp/openmpi-1.6.5/lib:$LD_LIBRARY_PATH" | ||
export PATH="/project/detrd/vasp/openmpi-1.6.5/bin:$PATH" | export PATH="/project/detrd/vasp/openmpi-1.6.5/bin:$PATH" | ||
| Line 37: | Line 37: | ||
#PBS -N vasp-openmpi-test | #PBS -N vasp-openmpi-test | ||
#PBS -l nodes=2:ppn=1 | #PBS -l nodes=2:ppn=1 | ||
| + | LD_LIBRARY_PATH="/project/detrd/vasp/openmpi-1.6.5/lib:$LD_LIBRARY_PATH" | ||
| + | export LD_LIBRARY_PATH | ||
| + | PATH="/project/detrd/vasp/openmpi-1.6.5/bin:$PATH" | ||
| + | export PATH | ||
mpirun ${HOME}/vasp/ring_f90 | mpirun ${HOME}/vasp/ring_f90 | ||
And then: | And then: | ||
qsub testrun-openmpi.pbs | qsub testrun-openmpi.pbs | ||
| − | |||
= Building VASP with OpenMPI support = | = Building VASP with OpenMPI support = | ||
| Line 49: | Line 52: | ||
to start fresh. I copied the vasp.5.lib dir as-is from /project/detrd/vasp/src/vasp.5.lib/. | to start fresh. I copied the vasp.5.lib dir as-is from /project/detrd/vasp/src/vasp.5.lib/. | ||
| − | The Makefile needs | + | To use the Intel MKL FFT wrapper library, I had to build it myself (in the same directory): |
| + | make -f /opt/intel/composer_xe_2013.5.192/mkl/interfaces/fftw3xf/makefile libintel64 | ||
| + | |||
| + | The Makefile needs some edits compared to the original. In the MPI section: | ||
FC=mpif90 | FC=mpif90 | ||
FCL=$(FC) | FCL=$(FC) | ||
CPP += -DMPI | CPP += -DMPI | ||
| − | and include support for parallel FFT: | + | and include support for parallel FFT with the Intel wrapper: |
| − | FFT3D = | + | FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o libfftw3xf_intel.a |
| + | INCS = -I$(MKLROOT)/include/fftw | ||
| + | |||
| + | The prepocessor needs to be set to use NGZhalf: | ||
| + | CPP = $(CPP_) -DHOST=\"LinuxIFC\" \ | ||
| + | -DCACHE_SIZE=12000 -DPGF90 -Davoidalloc -DNGZhalf \ | ||
| + | |||
The alternate relative path to the lib dir requires another change: | The alternate relative path to the lib dir requires another change: | ||
| Line 64: | Line 76: | ||
The compiler '''crashed''' on one file, but replacing -O3 by -O2 worked around that: | The compiler '''crashed''' on one file, but replacing -O3 by -O2 worked around that: | ||
| − | ifort -FR -names lowercase -assume byterecl -heap-arrays 64 -O3 -ip -parallel -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/intel/composer_xe_2013.5.192/mkl/include/fftw -c local_field.f90 | + | ifort -FR -names lowercase -assume byterecl -heap-arrays 64 -O3 -ip -parallel \ |
| − | : catastrophic error: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report. Note: File and line given may not be explicit cause of this error. | + | -I/opt/intel/composer_xe_2013.5.192/mkl/include \ |
| + | -I/opt/intel/composer_xe_2013.5.192/mkl/include/fftw -c local_field.f90 | ||
| + | : catastrophic error: **Internal compiler error: segmentation violation signal raised** \ | ||
| + | Please report this error along with the circumstances in which it occurred in a Software \ | ||
| + | Problem Report. Note: File and line given may not be explicit cause of this error. | ||
compilation aborted for local_field.f90 (code 1) | compilation aborted for local_field.f90 (code 1) | ||
| + | |||
| + | A line added to the makefile: | ||
| + | local_field.o: OFLAG=-O2 -ip -parallel | ||
| + | |||
| + | Other lines for individual files were present that turned optimization down in a different way. | ||
| + | |||
| + | |||
| + | == Building VASP 5.4.1 == | ||
| + | |||
| + | (Jan 2016) | ||
| + | |||
| + | New version has a better layout of the source tree, so more straightforward to build. | ||
| + | |||
| + | Newer Intel compiler installation in /opt/intel (actually /public/public_linux/intel but symlinked because the original path was /opt/intel and that's embedded in nearly every script). | ||
| + | |||
| + | I've made a few changes to the VASP makefile.include: | ||
| + | |||
| + | # Precompiler options | ||
| + | CPP_OPTIONS= -DMPI -DHOST=\"IFC91_ompi\" -DIFC \ | ||
| + | -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc \ | ||
| + | -DMPI_BLOCK=8000 -DscaLAPACK -Duse_collective \ | ||
| + | -DnoAugXCmeta -Duse_bse_te \ | ||
| + | -Duse_shmem -Dtbdyn | ||
| + | |||
| + | CPP = fpp -f_com=no -free -w0 $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS) | ||
| + | |||
| + | FC = mpif90 | ||
| + | FCL = mpif90 -mkl | ||
| + | |||
| + | FREE = -free -names lowercase | ||
| + | |||
| + | FFLAGS = -assume byterecl | ||
| + | OFLAG = -O2 | ||
| + | OFLAG_IN = $(OFLAG) | ||
| + | DEBUG = -O0 | ||
| + | |||
| + | MKL_PATH = $(MKLROOT)/lib/intel64 | ||
| + | BLAS = | ||
| + | LAPACK = | ||
| + | BLACS = -lmkl_blacs_custom_lp64 | ||
| + | SCALAPACK = $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS) | ||
| + | |||
| + | OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o \ | ||
| + | libfftw3xf_intel.a | ||
| + | INCS =-I$(MKLROOT)/include/fftw | ||
| + | |||
| + | LLIBS = $(SCALAPACK) $(LAPACK) $(BLAS) | ||
| + | |||
| + | OBJECTS_O1 += fft3dfurth.o fftw3d.o fftmpi.o fftmpiw.o | ||
| + | OBJECTS_O2 += fft3dlib.o | ||
| + | |||
| + | # For what used to be vasp.5.lib | ||
| + | CPP_LIB = $(CPP) | ||
| + | FC_LIB = $(FC) | ||
| + | CC_LIB = icc | ||
| + | CFLAGS_LIB = -O | ||
| + | FFLAGS_LIB = -O1 | ||
| + | FREE_LIB = $(FREE) | ||
| + | |||
| + | OBJECTS_LIB= linpack_double.o getshmem.o | ||
| + | |||
| + | # Normally no need to change this | ||
| + | SRCDIR = ../../src | ||
| + | BINDIR = ../../bin | ||
| + | |||
| + | |||
| + | The blacs_custom is my own openmpi based build | ||
| + | |||
| + | /opt/intel/mkl/lib/intel64_lin/libmkl_blacs_custom_lp64.so | ||
| + | |||
| + | with the help of Intel's support pages. (And Ton helped install it along with the other blacs libraries.) | ||
