[Manual] Using Intel MPI Trace Analyzer and Collector with Devito

First, source the necessary stuff

source /opt/intel/oneapi/compiler/latest/env/vars.sh intel64
source /opt/intel/oneapi/mpi/latest/env/vars.sh
source /opt/intel/oneapi/itac/latest/env/vars.sh

By default, Trace Collector will produce a large number of files along with the .stf file, which is the primary collection database. However, Trace Collector can consolidate the entire database into a single .stf file by using the following environment variable:

Then, to include everything in one file, instead of producing several files

export VT_LOGFILE_FORMAT=STFSINGLE
export VT_LOGFILE_NAME=myfile.stf

To report OpenMP time as well (needs to compiler with icc)

export INTEL_LIBITTNOTIFY64=/opt/intel/oneapi/itac/latest/slib/libVT.so
export KMP_FORKJOIN_FRAMES_MODE=0

DEVITO_PROFILING=advanced2 DEVITO_ARCH=icc OMP_PLACES=threads DEVITO_LOGGING=DEBUG DEVITO_LANGUAGE=openmp DEVITO_MPI=basic mpiexec.hydra -trace "libVT.so libmpi.so" -n 2 python3 examples/seismic/acoustic/acoustic_example.py -d 100 100 100 --tn 500

Sources: [1] https://docs.nersc.gov/tools/performance/itac/#capturing-openmp-behavior

Adding vtune in the game

You may need to:

[0] vtune: Error: Cannot start data collection because the scope of ptrace system call is limited. To enable profiling, please set /proc/sys/kernel/yama/ptrace_scope to 0. To make this change permanent, set kernel.yama.ptrace_scope to 0 in /etc/sysctl.d/10-ptrace.conf and reboot the machine.
[0] vtune: Warning: Microarchitecture performance insights will not be available. Make sure the sampling driver is installed and enabled on your system.

source /opt/intel/oneapi/vtune/latest/env/vars.sh
DEVITO_JIT_BACKDOOR=0 DEVITO_LOGGING=DEBUG DEVITO_LANGUAGE=openmp DEVITO_MPI=full mpiexec.hydra -trace "libVT.so libmpi.so" -n 2 -l vtune -quiet -collect hotspots -trace-mpi -result-dir r0_norm_ranks2_edit numactl --cpunodebind=0 --membind=0 python3 toy.py -d 200 200 400 -nt 500

export DEVITO_JIT_BACKDOOR=1; export VT_LOGFILE_NAME=default8_pokey.stf; DEVITO_PROFILING=advanced2 DEVITO_ARCH=icc OMP_PLACES=threads DEVITO_LOGGING=DEBUG DEVITO_MPI=full mpiexec.hydra -trace "libVT.so libmpi.so" -n 8 python3 examples/seismic/acoustic/acoustic_example.py -d 400 400 400 --tn 100