HYCOM
Using TAU with Hycom can be a bit tricky. It has .F files that need to be pre-processed. The source calls xcstop('msg') to end the program. So, we must use -optPreProcess to compile Hycom. I changed config/xt3.tau_mpi:
### Tau stub makefiles: ## Profiling: include /spin/proj/perc/TOOLS/tau-2.15.6b1/xt3/lib/Makefile.tau-multiplecounters-mpi-papi-pdt-pgi ## Tracing: #include /spin/proj/perc/TOOLS/tau-2.15.6b1/xt3/lib/Makefile.tau-multiplecounters-mpi-papi-pdt-trace-pgi OPTS_TAU = -optPdtGnuFortranParser -optVerbose -optPreProcess -optTauSelectFile=select.tau FCOMPILER = $(TAU_COMPILER) $(OPTS_TAU) ftn CCOMPILER = $(TAU_COMPILER) $(OPTS_TAU) cc
on Jaguar. To compile the code, place select.tau in the source directory:
GLBA0.08/src_2.1.35_26_mpi> pwd /lustre/scratch/sameer/hycom/GLBA0.08/src_2.1.35_26_mpi GLBA0.08/src_2.1.35_26_mpi> cat select.tau BEGIN_EXCLUDE_LIST DPUDPVJ MXKPRFBIJ MXKPRFCIJU MXKPRFCIJV POFLAT PROFILE_LAT ROFLAT TRIDCOF TRIDMAT TRIDRHS WSCALE END_EXCLUDE_LIST GLBA0.08/src_2.1.35_26_mpi>
and to run the program:
% cd EXAMPLE; % ./GLB_mpi0385.DC.TAU.com
This will create a symlink from hycom.tau in the source directory to the run directory. Remember to edit this file and set the correct paths for:
setenv V 2.1.35 setenv P hycom/GLBA0.08 setenv S /lustre/scratch/sameer/$P/expt_99.7/${OS}/data_${NMPI}x${NOMP}/DC/TAU setenv D /lustre/scratch/sameer/$P/expt_99.7/DATA
in this script. This sets up the data properly. To run the code:
DC/TAU> pwd /lustre/scratch/sameer/hycom/GLBA0.08/expt_99.7/XT3/data_385x0/DC/TAU DC/TAU> cat run.jaguar #!/bin/tcsh #PBS -N hycom.expt_99.7.DC.TAU #PBS -A CSC023 #PBS -m e #PBS -j oe #PBS -q batch #PBS -l walltime=1:30:00 #PBS -l size=193 cd $PBS_O_WORKDIR yod -sz 385 ./hycom.tau DC/TAU> > qsub run.jaguar
NOTE: Remember to remove any ovr* files from this directory prior to running. On p655-4, I have a run script for data_15x1 directory where we do:
> pwd /mnt/netapp/home/users/sameer/apps/hycom/GLBA0.72/expt_99.8/AIX/data_15x1 > cat run #!/bin/sh /bin/rm -f ar* flx* hycom ovr* patch.input sum* ln -s patch.input_0015 patch.input cp /mnt/sonic/home/users/sameer/apps/hycom/GLBA0.72/src_2.1.27B_26_mpi/hycom . export TAU_THROTTLE=1 poe ./hycom -procs 15
On AIX, my config/sp4_mpi file looks like:
include /home/users/sameer/TAU/tau-2.15.4/ibm64/lib/Makefile.tau-mpi-pdt OPT = -optVerbose -optTauSelectFile=select.tau -optNoRevert -optKeepFiles FC = $(TAU_COMPILER) $(OPT) $(TAU_F90)