VASP#
Requesting Access to Use VASP on Quest#
VASP requires a paid license, so access to VASP modules on Quest must be reviewed and approved for specific users.
Quest users with a VASP license should email quest-help@northwestern.edu with the e-mail associated with your VASP access/license and a request to access the VASP modules on Quest. Quest admins will then verify that this email has a valid VASP license. Depending on your license, you may have access to either the VASP 5.X modules or the VASP 6.X modules or both.
Note that even after you are granted access to VASP, you will continue to see the following warning message when you try to use one of the VASP modules.
$ module purge
$ module load vasp/5.4.4-openmpi-intel
WARNING: Because VASP is licensed software, we need to give users explicit
access to the VASP modules on Quest. VASP provides Quest admins with access
to a portal where we can use the e-mail associated with your VASP access/license
to verify that you can use VASP on Quest. Please send an e-mail to quest-help@northwestern.edu
stating both that you would like to use VASP on Quest and the e-mail associated with your VASP access/license.
This message can be ignored once you have received confirmation confirming your access.
Attention
Please note that only the builds of VASP 6.X have CUDA support enabled.
Running VASP on Quest#
Once you have received confirmation that you have been given access to use the VASP modules on Quest, you can start to run VASP simulations. Please find an example of running VASP 5.4.4 on Quest below.
submit-vasp-5.4.4.sh
#!/bin/bash
#SBATCH --account=<pXXXXX>
#SBATCH --partition=<partition>
#SBATCH --job-name=vasp-openmpi-slurm
#SBATCH --ntasks=<ncores>
#SBATCH --time=01:00:00
#SBATCH --mem-per-cpu=2G
module purge all
module load vasp/5.4.4-openmpi-intel
export OMP_NUM_THREADS=1
mpirun -np ${SLURM_NTASKS} vasp_std
Note that this submission script is designed so that however many cores are requested from the scheduler via #SBATCH --ntasks=<ncores> will automatically translate to mpirun -np through the the
environmental variable ${SLURM_NTASKS}.
How to build your own copy of VASP#
You may find that you or your lab wants to have your own build of VASP on Quest. To help with this, we provide a Bash build script which will load the appropriate modules for building VASP and an appropriate makefile.include for building VASP on Quest.
Attention
Please note that we are not including any build templates for compiling VASP with beef, vtst, etc. If you need help compiling your own copy of VASP on Quest with any of these additional libraries/extensions, you can reach out to us at quest-help@northwestern.edu for support.
VASP 5.4.4#
The following Bash script (install_vasp_5.4.4.sh) can be used to build VASP in combination with makefile.include.vasp.5.4.4 below. Before running the contents of install_vasp_5.4.4.sh, please take the content of makefile.include.vasp.5.4.4 and place it in a file called makefile.include in the VASP root directory. Executing the content of install_vasp_5.4.4.sh will produce the following programs in the folder VASPROOT/bin: vasp_gam, vasp_ncl, and vasp_std.
install_vasp_5.4.4.sh
module purge
module load intel-oneapi-compilers/2021.4.0-gcc
module load intel-oneapi-mkl/2021.4.0-intel
module load mpi/openmpi-4.1.4-intel
make -j 1
makefile.include.vasp.5.4.4
# Precompiler options
CPP_OPTIONS= -DHOST=\"LinuxIFC\"\
-DMPI -DMPI_BLOCK=8000 \
-Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Duse_bse_te \
-Dtbdyn \
-Duse_shmem
CPP = fpp -f_com=no -free -w0 $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
FC = mpifort
FCL = mpifort -mkl=sequential -lstdc++
FREE = -free -names lowercase
FFLAGS = -assume byterecl -w -axCORE-AVX512,CORE-AVX2,AVX
OFLAG = -O3
OFLAG_IN = $(OFLAG)
DEBUG = -O0
MKL_PATH = $(MKLROOT)/lib/intel64
BLAS =
LAPACK =
BLACS = -lmkl_blacs_openmpi_lp64
SCALAPACK = $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)
OBJECTS = fftmpiw.o fftmpi_map.o fft3dlib.o fftw3d.o
INCS =-I$(MKLROOT)/include/fftw
LLIBS = $(SCALAPACK) $(LAPACK) $(BLAS)
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o
# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = $(FC)
CC_LIB = icc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB = $(FREE)
OBJECTS_LIB= linpack_double.o getshmem.o
# For the parser library
CXX_PARS = icpc
LIBS += parser
LLIBS += -Lparser -lparser -lstdc++
# # Normally no need to change this
SRCDIR = ../../src
BINDIR = ../../bin
VASP 6.2#
The following Bash script (install_vasp_6.2.X.sh) can be used to build VASP in combination with makefile.include.vasp.6.2.X below. Before running the contents of install_vasp_6.2.X.sh, please take the content of makefile.include.vasp.6.2.X and place it in a file called makefile.include in the VASP root directory. Executing the content of install_vasp_6.2.X.sh will produce the following programs in the folder VASPROOT/bin: vasp_gam, vasp_ncl, vasp_std, vasp_gpu_ncl, and vasp_gpu.
install_vasp_6.2.X.sh
$ module purge
$ module load intel-oneapi-compilers/2021.4.0-gcc
$ module load cuda/11.4.0-intel
$ module load intel-oneapi-mkl/2021.4.0-intel
$ module load mpi/openmpi-4.1.4-intel
$ make -j 1
makefile.include.vasp.6.2.X
# Precompiler options
CPP_OPTIONS= -DHOST=\"LinuxIFC\"\
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dfock_dblbuf
CPP = fpp -f_com=no -free -w0 $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
FC = mpifort
FCL = mpifort -mkl=sequential
FREE = -free -names lowercase
FFLAGS = -assume byterecl -w -axCORE-AVX512,CORE-AVX2,AVX
OFLAG = -O2
OFLAG_IN = $(OFLAG)
DEBUG = -O0
MKL_PATH = $(MKLROOT)/lib/intel64
BLAS =
LAPACK =
BLACS = -lmkl_blacs_openmpi_lp64
SCALAPACK = $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)
OBJECTS = fftmpiw.o fftmpi_map.o fft3dlib.o fftw3d.o
INCS =-I$(MKLROOT)/include/fftw
LLIBS = $(SCALAPACK) $(LAPACK) $(BLAS)
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o
# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = $(FC)
CC_LIB = icc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB = $(FREE)
OBJECTS_LIB= linpack_double.o getshmem.o
# For the parser library
CXX_PARS = icpc
LLIBS += -lstdc++
# Normally no need to change this
SRCDIR = ../../src
BINDIR = ../../bin
#================================================
# GPU Stuff
CPP_GPU = -DCUDA_GPU -DRPROMU_CPROJ_OVERLAP -DUSE_PINNED_MEMORY -DCUFFT_MIN=28 -UscaLAPACK -Ufock_dblbuf
OBJECTS_GPU= fftmpiw.o fftmpi_map.o fft3dlib.o fftw3d_gpu.o fftmpiw_gpu.o
CC = icc
CXX = icpc
CFLAGS = -fPIC -DADD_ -Wall -qopenmp -DMAGMA_WITH_MKL -DMAGMA_SETAFFINITY -DGPUSHMEM=300 -DHAVE_CUBLAS
# Minimal requirement is CUDA >= 10.X. For "sm_80" you need CUDA >= 11.X.
CUDA_ROOT ?= $(CUDA_HOME)
NVCC := $(CUDA_ROOT)/bin/nvcc -ccbin=icc -allow-unsupported-compiler
CUDA_LIB := -L$(CUDA_ROOT)/lib64 -L$(CUDA_ROOT)/lib64/stubs -lnvToolsExt -lcudart -lcuda -lcufft -lcublas
GENCODE_ARCH := -gencode=arch=compute_80,code=\"sm_80,compute_80\"
## For all legacy Intel MPI versions (before 2021)
#MPI_INC = $(I_MPI_ROOT)/intel64/include/
# Or when you are using the Intel oneAPI compiler suite
MPI_INC = /hpc/software/spack_v17d2/spack/opt/spack/linux-rhel7-x86_64/intel-2021.4.0/openmpi-4.1.4-ztg2tque7onlwhrv7t3kplcbd5ck5lwe/include
VASP 6.4#
The following Bash script (install_vasp_6.4.X.sh) can be used to build VASP with HDF5 enabled in combination with makefile.include.vasp.6.4.X below. Before running the contents of install_vasp_6.4.X.sh, please take the content of makefile.include.vasp.6.4.X and place it in a file called makefile.include in the VASP root directory. Executing the content of install_vasp_6.4.X.sh will produce the following programs in the folder VASPROOT/bin: vasp_gam, vasp_ncl, and vasp_std.
install_vasp_6.4.X.sh
module purge
module load intel-oneapi-compilers/2021.4.0-gcc
module load intel-oneapi-mkl/2021.4.0-intel
module load mpi/openmpi-4.1.4-intel
module load hdf5/1.10.7-openmpi-4.1.4-intel
make -j 1
makefile.include.vasp.6.4.X
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxIFC\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dfock_dblbuf
CPP = fpp -f_com=no -free -w0 $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
FC = mpif90
FCL = mpif90
FREE = -free -names lowercase
FFLAGS = -assume byterecl -w
OFLAG = -O2
OFLAG_IN = $(OFLAG)
DEBUG = -O0
OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o
# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = $(FC)
CC_LIB = icc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB = $(FREE)
OBJECTS_LIB = linpack_double.o
# For the parser library
CXX_PARS = icpc
LLIBS = -lstdc++
##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##
# When compiling on the target machine itself, change this to the
# relevant target when cross-compiling for another architecture
VASP_TARGET_CPU ?= -axCORE-AVX512,CORE-AVX2,AVX
FFLAGS += $(VASP_TARGET_CPU)
# Intel MKL for FFTW, BLAS, LAPACK, and scaLAPACK
# (Note: for Intel Parallel Studio's MKL use -mkl instead of -qmkl)
FCL += -qmkl=sequential
MKL_PATH = $(MKLROOT)/lib/intel64
BLACS = -lmkl_blacs_openmpi_lp64
SCALAPACK = $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)
INCS =-I$(MKLROOT)/include/fftw
LLIBS += $(SCALAPACK)
# Use a separate scaLAPACK installation (optional but recommended in combination with OpenMPI)
# Comment out the two lines below if you want to use scaLAPACK from MKL instead
#SCALAPACK_ROOT ?= /path/to/your/scalapack/installation
#LLIBS += -L${SCALAPACK_ROOT}/lib -lscalapack
# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT ?= /hpc/software/spack_v17d2/spack/opt/spack/linux-rhel7-x86_64/intel-2021.4.0/hdf5-1.10.7-mvbrxowc73gzgvfjxydytlxc5j6xxixi/
LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS += -I$(HDF5_ROOT)/include
# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier
# For the fftlib library (hardly any benefit in combination with MKL's FFTs)
#CPP_OPTION += -Dsysv
#FCL = mpif90 fftlib.o -qmkl
#CXX_FFTLIB = icpc -qopenmp -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib