Keeneland

Guide for using TAU on Keeneland

Setting up environment

We'll have a module for TAU setup shortly but for now, setup your environment this way:

   %> export PATH=/nics/c/home/biersdor/tau2/x86_64/bin/:$PATH
   %> export LD_LIBRARY_PATH=/nics/c/home/biersdor/tau2/x86_64/lib/:$LD_LIBRARY_PATH

Compiling with TAU

Details at: TAU's userguide

But the basic idea is to replace your compiler with the TAU wrapper compilers, ie gcc => tau_cc.sh, gfortran => tau_f90.sh etc...

Running with tau_exec

For a quick profile/trace try the tau_exec script.

Running CUDA applications

Both CUDA and OpenCL are instrumented dynamically through library preloading, use the tau_exec script to run the CUDA application:

   %> tau_exec -T serial -cuda ./Stencil2D

The -T serial specifies with TAU configuration to use, you can omit this for MPI applications and run:

Trouble-shooting

• CPU side looks fine but no GPU profile/trace generated.

This is likely because there is no cudaThreadExit() call at the end the application. By placing one there this will signal TAU that the applications CUDA accelerated section is finished and it can go ahead and write out the profile/trace.

Fix: Place cudaThreadExit() at the end of the application.

• Receiving Error calculating kernel event [start|stop], error #: 33. during execution.

This means that CUDA could not retrieve the event object at synchronization. Try placing the synchronize event right after the kernel is launched. In some cases no configuration of kernel launches/synchronization points will suffice, and although this one kernel could not be tracked any other ones taking place in the application should be tracked correctly.

Fix: Try placing a synchronization called right after the kernel launch.

Running OpenCL applications

Use tau_exec as well:

   %> tau_exec -T serial -opencl ./SGEMM 

CUpti and PAPI

Coming soon...