Using Intel Advisor with Devito

This page explains how to use Intel Advisor's command line interface to profile a Devito program and generate a roofline model showing its performance. We will be using benchmarks/user/benchmark.py from the repository as an example during this tutorial.

Generating Profiling Data Through Intel Advisor CLI

This section goes through the commands to use to generate roofline data from a Devito program.

First of all, make sure that Intel Advisor is installed correctly on the machine by using the command:

advixe-cl --version

Here, advixe-cl is the generic command for Intel Advisor and --version specifies its version. If Intel Advisor is not installed, it can be installed as a part of Intel Parallel Studio or through Intel oneAPI. Follow the instructions on the websites if you haven't already done so.

Now, cd into your project's main directory. For the tutorial, this will be the main directory of the repository: devito/ (NB: not devito/devito/). Obtaining a roofline of a Devito program will be done in two phases. The first is the Survey, done to obtain generic performance data about the program. To obtain the Survey data for the program being profiled run the following:

advixe-cl --collect=survey --project-dir=prof/ -- python benchmarks/user/benchmark.py run -P acoustic -d 512 512 512 -so 4 --tn 100 --autotune off

As before, advixe-cl is the command to access Intel Advisor, then --collect=survey specifies that we wish to collect Survey data form the program and --project-dir=prof/ tells Advisor to generate profiling data within the prof/ directory. Note that in the case of this tutorial, the directory prof/ did not exist a-priori, so Advisor has generated it for us. For this reason, make sure to use a unique directory name to collect Advisor data. Finally, -- python benchmarks/user/benchmark.py run -P acoustic -d 512 512 512 -so 12 --tn 100 --autotune off specifies that we wish to profile python benchmarks/user/benchmark.py run with all the following Devito command line arguments (run -P acoustic -d 512 512 512 -so 4 --tn 100 --autotune off).

Now, Advisor should start and you should see something starting with the following lines:

Intel(R) Advisor Command Line Tool Copyright (C) 2009-2020 Intel Corporation. All rights reserved. advixe: Collection started. To stop the collection,...

Await its termination.

Once collecting Survey data has finished, the following phase collects the FLOPS of the program. To collect this information, run the command:

advixe-cl --collect=tripcounts -enable-cache-simulation -flop --project-dir=prof/ -- python benchmarks/user/benchmark.py run -P acoustic -so 4 --tn 100 --autotune off

Here, --collect=tripcounts tells Advisor to collect data regarding the number of times loops are executed within the program.-enable-cache-simulation specifies that Advisor should simulate hits, misses and cache evictions on all cache levels in the cache hierarchy during the execution of the program to generate a roofline for each cache level. To read more about this, consult this page. Finally, the -flop flag tells Advisor to collect information about floating point and integer operations during the trip count collection. The --project-dir=prof/ flag and -- python benchmarks/... are as before.

Run the command. You should see the same kind of messages as before. Once the collection stops successfully, the project directory (e.g. prof/) will now contain all the data needed for exporting and visualising the results of the program's profiling.

Exporting Data to Visualise Roofline Plot

This section explains how to produce a snapshot of generated profiling data and use an external Advisor GUI to visualise the roofline plot produced.

Now that the data has been collected, cd into the Advisor project's directory (prof/ in our case). To collect generated data ready for export, Advisor provides a tool to create a "snapshot" of the program's performance. To obtain this snapshot, run the following command:

advixe-cl --snapshot

In the Advisor project's main directory, you should now see a file with a name like snapshot000.advixeexpz. This file contains all the information needed and can now be copied to a desired path on the machine where the results have to be visualised. From that same machine, run the following command to open Intel Advisor with its GUI:

advisor-gui

Once the GUI is open, press the "Open Result" button to look for the previously created snapshot. The button is an open, yellow folder sitting on the top navigation bar. Now navigate to the snapshot000.advixeexpz file and press "Open". Advisor should now elaborate the data for a few seconds. Then, navigate to the "Survey & Roofline" tab and press "Roofline" to finally visualise the data as a roofline model.

Exporting Profiling Raw Data