Spp-benchmarks
Contents
Introduction
The Blue Waters SPP Benchmark overview is located here: spp-methodology. Download and build/run instructions for the individual packages are located here: spp-benchmarks
The build and run instructions provided above are specific to the Blue Waters platform. The modifications outlined for the individual benchmarks below cover building and execution with and without TAU instrumentation on the NERSC Cori system. Unload the darshan module before using TAU. The Program Environment suggested by the individual benchmarks will be used where possible. If the environment used in the Blue Waters default build is unavailable the PrgEnv-intel module will be used unless otherwise noted. TAU instrumentation and sampling is done with a configuration like:
"./configure -arch=craycnl -useropt=-g\ -O2 -pdt=/global/homes/w/wspear/bin/pdtoolkit-3.24 -mpi -bfd=download -iowrapper -unwind=download"
Resulting in a TAU_MAKEFILE like: /global/homes/w/wspear/bin/tau2/craycnl/lib/Makefile.tau-intel-mpi-pdt
TAU_MAKEFILE tags will vary depending on the compiler selected.
Some system specific optimizations from the Blue Waters build/run instructions may have been removed without replacement. Compilation options will generally have -dynamic added to allow sampling with tau_exec. Slurm batch scripts are modified versions of the Blue Waters PBS scripts and may include PBS style commands where they are still supported by Slurm.
AWP-ODC
Use this makefile to build on Cori. Substitute ftn and cc for tau_f90.sh and tau_cc.sh to build uninstrumented.
OFLAGS = tau_f90.sh -O3 -dynamic -mp1 -c -extend_source -assume byterecl -o PFLAGS = tau_f90.sh -O3 -dynamic -mp1 -extend_source -assume byterecl -o LIB = -lmpich BIN = ../bin OBJECTS = memory.o bound.o relax.o swap.o io.o source.o \ custompi.o station.o structure.o operator.o \ viscoop.o pml.o cerjan.o pmcl3d.o md5.o set_names.o \ sgsndyna.o sgsnswap.o pmcl3d: $(OBJECTS) $(PFLAGS) pmcl3d $(OBJECTS) $(LIB) cp pmcl3d $(BIN) md5.o: md5.c tau_cc.sh -O2 -c -g -o $@ md5.c pmcl3d.o: pmcl3d.f $(OFLAGS) pmcl3d.o pmcl3d.f sgsndyna.o: sgsndyna.f $(OFLAGS) sgsndyna.o sgsndyna.f sgsnswap.o: sgsnswap.f $(OFLAGS) sgsnswap.o sgsnswap.f memory.o: memory.f $(OFLAGS) memory.o memory.f relax.o: relax.f $(OFLAGS) relax.o relax.f bound.o: bound.f $(OFLAGS) bound.o bound.f swap.o: swap.f $(OFLAGS) swap.o swap.f io.o: io.f $(OFLAGS) io.o io.f structure.o: structure.f $(OFLAGS) structure.o structure.f operator.o: operator.f $(OFLAGS) operator.o operator.f viscoop.o: viscoop.f $(OFLAGS) viscoop.o viscoop.f pml.o: pml.f $(OFLAGS) pml.o pml.f cerjan.o: cerjan.f $(OFLAGS) cerjan.o cerjan.f source.o: source.f $(OFLAGS) source.o source.f station.o: station.f $(OFLAGS) station.o station.f custompi.o: custompi.f $(OFLAGS) custompi.o custompi.f set_names.o: set_names.f $(OFLAGS) set_names.o set_names.f clean: rm -f pmcl3d $(BIN)/pmcl3d *.o parstat* cleanmore: rm fort.* core.* CHK* SS* V* SRCT*
Use this SLURM batch script instead of the provided scripts to run a small test.
#!/bin/bash -login #PBS -N awp-odc_cpu_small #PBS -l walltime=00:03:00,nodes=1 #:ppn=8:xe #PBS -j eo #SBATCH --constraint=haswell #SBATCH --partition=debug #module swap $( module list | grep -o PrgEnv-.*$ ) PrgEnv-cray mkdir $SLURM_SUBMIT_DIR/$SLURM_JOBID cd $SLURM_SUBMIT_DIR/$SLURM_JOBID #cd $PBS_O_WORKDIR/$PBS_JOBID bash ../pre-run #export TAU_VERBOSE=1 unlink IN3D ln -s ../IN3D_small IN3D start_time=$(date +%s) #srun -n 8 tau_exec -T INTEL,PDT -ebs ../../../../src-v1.1.2/pmcl3d srun -n 8 ../../../../src-v1.1.2/pmcl3d end_time=$(date +%s) echo Completed in $(( end_time - start_time )) seconds
Cactus
In progress. The BlueWaters package (uninstrumented) is not yet building on Cori.