Difference between revisions of "LAMMPS"
(8 intermediate revisions by the same user not shown) | |||
Line 11: | Line 11: | ||
|[http://lammps.sandia.gov/download.html#git git repo] | |[http://lammps.sandia.gov/download.html#git git repo] | ||
|Keeneland | |Keeneland | ||
− | | | + | |May 2012 |
|} | |} | ||
Line 18: | Line 18: | ||
To start copy over the file [[Media:Makefile.keeneland | Makefile.keeneland]] to the src/MAKE directory. Then do, | To start copy over the file [[Media:Makefile.keeneland | Makefile.keeneland]] to the src/MAKE directory. Then do, | ||
+ | |||
+ | configure TAU: | ||
+ | |||
+ | ./configure -mpi -cuda=<cuda dir> -pdt=<pdt dir> | ||
+ | make install | ||
+ | |||
+ | build LAMMPS: | ||
cd src | cd src | ||
Line 26: | Line 33: | ||
export TAU_MAKEFILE=<path to TAU>/x86_64/lib/Makefile.tau-icpc-mpi-pdt | export TAU_MAKEFILE=<path to TAU>/x86_64/lib/Makefile.tau-icpc-mpi-pdt | ||
export TAU_OPTIONS='-optVerbose -optShared' | export TAU_OPTIONS='-optVerbose -optShared' | ||
− | make keeneland CC | + | make keeneland CC=tau_cc.sh CXX=tau_cxx.sh F90=tau_f90.sh LINK=tau_cc.sh |
− | |||
− | |||
running: | running: | ||
cd examples/USER/cuda | cd examples/USER/cuda | ||
− | mpirun -np 3 tau_exec -T mpi,cupti, | + | |
+ | Then choose either method depending on your choice of MPI implementation: | ||
+ | |||
+ | |||
+ | === OPENMPI === | ||
+ | |||
+ | Running with MPI on Keeneland by doing: | ||
+ | qlogin -V -l nodes=2:ppn=3 | ||
+ | mpirun --mca btl_openib_flags 1 --mca mpi_leave_pinned 0 --mca \ | ||
+ | btl_openib_warn_default_gid_prefix 0 -npernode 3 -np $NPROCS_3 --hostfile ${PBS_NODEFILE} \ | ||
+ | tau_exec -T mpi,cupti,icpc -cupti \ | ||
+ | /lustre/medusa/biersdor/mylammps/src/lmp_keeneland_tau_wgpu -cuda on -sf cuda < in.melt_2.5.cuda | ||
+ | |||
+ | |||
+ | === MPICH2 === | ||
+ | As of this writing the USER/CUDA package requires mpich2 to run on more than one node. Here are the commands to get started: | ||
+ | |||
+ | qlogin -V -l nodes=2:ppn=3 | ||
+ | mpdboot --totalnum=2 --ncpus=3 -f mpich_machines.txt | ||
+ | cat mpich_machines.txt | ||
+ | kid008.nics.utk.edu:3 | ||
+ | kid009.nics.utk.edu:3 | ||
+ | mpirun -np 6 tau_exec -T icpc,mpi,pdt,cupti,mpich2 -cupti /lustre/medusa/biersdor/mylammps/src/lmp_keeneland_tau_mpich2 -sf cuda -cuda on < in.melt_2.5.cuda | ||
+ | |||
+ | == Performance results == | ||
+ | |||
+ | [[Image:lammps_profile.png|750px]] | ||
+ | [[Image:lammps_3gpu.ppk]] | ||
+ | |||
+ | [[Image:lammps_trace.png|750px]] |
Latest revision as of 19:33, 21 May 2012
Link | Code Version | Machine | Date |
---|---|---|---|
Sandia website | git repo | Keeneland | May 2012 |
Building with TAU
To start copy over the file Makefile.keeneland to the src/MAKE directory. Then do,
configure TAU:
./configure -mpi -cuda=<cuda dir> -pdt=<pdt dir> make install
build LAMMPS:
cd src make yes-user-cuda cd ../lib/cuda make lib cd ../../src export TAU_MAKEFILE=<path to TAU>/x86_64/lib/Makefile.tau-icpc-mpi-pdt export TAU_OPTIONS='-optVerbose -optShared' make keeneland CC=tau_cc.sh CXX=tau_cxx.sh F90=tau_f90.sh LINK=tau_cc.sh
running:
cd examples/USER/cuda
Then choose either method depending on your choice of MPI implementation:
OPENMPI
Running with MPI on Keeneland by doing:
qlogin -V -l nodes=2:ppn=3 mpirun --mca btl_openib_flags 1 --mca mpi_leave_pinned 0 --mca \ btl_openib_warn_default_gid_prefix 0 -npernode 3 -np $NPROCS_3 --hostfile ${PBS_NODEFILE} \ tau_exec -T mpi,cupti,icpc -cupti \ /lustre/medusa/biersdor/mylammps/src/lmp_keeneland_tau_wgpu -cuda on -sf cuda < in.melt_2.5.cuda
MPICH2
As of this writing the USER/CUDA package requires mpich2 to run on more than one node. Here are the commands to get started:
qlogin -V -l nodes=2:ppn=3 mpdboot --totalnum=2 --ncpus=3 -f mpich_machines.txt cat mpich_machines.txt kid008.nics.utk.edu:3 kid009.nics.utk.edu:3 mpirun -np 6 tau_exec -T icpc,mpi,pdt,cupti,mpich2 -cupti /lustre/medusa/biersdor/mylammps/src/lmp_keeneland_tau_mpich2 -sf cuda -cuda on < in.melt_2.5.cuda