Skip to content

Commit

Permalink
Merge pull request #64 from lanl/spatter-fixes
Browse files Browse the repository at this point in the history 
Spatter fixes
  • Loading branch information
@JDTruj2018
JDTruj2018 authored Oct 18, 2023
2 parents 467354e + e4afd58 commit 1a66178
Show file tree
Hide file tree
Showing 34 changed files with 151 additions and 490 deletions.
184 changes: 55 additions & 129 deletions doc/sphinx/09_Microbenchmarks/M2_SPATTER/SPATTER.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,20 +39,38 @@ Accessing the benchmark, memory access patterns, and scaling scripts [LANL-Spatt
cd <path to benchmarks>
git submodule update --init --recursive
cd spatter
cd microbenchmarks/spatter
..
Set-up:

The setup script will initialize your CPU configuration file (scripts/cpu_config.sh) with CTS-1 defaults and the GPU configurationo file (scripts/gpu_config.sh) with V100/A100 defaults, and will buid Spatter for CPU and GPU. See the Spatter documentation and other build scripts (scripts/build_cpu.sh and scripts/build_cuda.sh) for further instructions for building with different compilers or for GPUs.
The setup script will initialize your CPU configuration file (scripts/cpu_config.sh) with ATS-3 defaults and the GPU configurationo file (scripts/gpu_config.sh) with V100/A100 defaults, and will buid Spatter for CPU and GPU. See the Spatter documentation and other build scripts (scripts/build_cpu.sh and scripts/build_cuda.sh) for further instructions for building with different compilers or for GPUs.

The scripts/setup.sh scripts has the following options

* c: Toggle CPU Build (default: off)
* g: Toggle GPU Build (default: off)
* h: Print usage message

To setup and build for only the CPU, run the following:

.. code-block:: bash
bash scripts/setup.sh -c
..

Or to build only for the GPU, run:

.. code-block:: bash
bash scripts/setup.sh -g
..

To setup and build for both the CPU and GPU, run the following:

.. code-block:: bash
Expand All @@ -73,27 +91,29 @@ This setup script performs the following:

* patterns/xrage/asteroid/spatter.json

#. Extracts patterns from patterns/xrage/asteroid/spatter.json to separate JSON files located at patterns/xrage/asteroid/spatter{1-9}.json

#. Generates default module files located in modules/cpu.mod and modules/gpu.mood

* Contains generic module load statements for CPU and GPU dependencies

* Assumes you are utilizing the module load system to configure environment. Change as needed (i.e. changes to PATH etc.) if you utilize a different system.

#. Populates the configuration file (scripts/cpu_config.sh) with reasonable defaults for a CTS-1 system
#. Populates the configuration file (scripts/cpu_config.sh) with reasonable defaults for a ATS-3 system

* HOMEDIR is set to the directory this repository sits in

* MODULEFILE is set to modules/cpu.mod

* SPATTER is set to path of the Spatter CPU executable

* ranklist is set to sweep from 1-36 threads/ranks respectively for a CTS-1 type system
* ranklist is set to sweep from 1-112 ranks respectively for a ATS-3 type system

* boundarylist is set to reasonable defaults for scaling experiments (specifies the maximum value of a pattern index, limiting the size of the data array)

* sizelist is set to reasonable defaults for strong scaling experiments (specifies the size of the pattern to truncate at)

#. Poopulates the GPU configuration file (scripts/gpu_config.sh) with reasonable defaults for single-GPU throughput experiments on a V100 or A100 system
#. Populates the GPU configuration file (scripts/gpu_config.sh) with reasonable defaults for single-GPU throughput experiments on a V100 or A100 system

* HOMEDIR is set to the directory this repository sits in

Expand Down Expand Up @@ -141,6 +161,7 @@ Running
=======

Running a Scaling Experiment

This will perform a weak scaling experiment

The scripts/scaling.sh script has the following options (a scripts/mpirunscaling.sh script with identical options has been provided if required to use mpirun rather than srun):
Expand All @@ -162,83 +183,68 @@ The Application name, Problem name, and Pattern name each correspond to subdirec

All Figures use solid lines for Gathers and dashed lines for Scatters.


V100
Crossroads
------------

Strong-Scaling throughput experiment with plotting enabled. Results will be found in spatter.strongscaling/V100/flag/static_2d/001 and Figures will be found in figures/spatter.strongscaling/V100/flag/static_2d/001.


Flag Static 2D 001
~~~~~~~~~~~~~~~~~~

Throughput experiment for the 8 patterns in patterns/flag/static_2d/001.json on a single GPU with plotting enabled. Results will be found in spatter.strongscaling/V100/flag/static_2d/001/ and Figures will be found in figures/spatter.strongscaling/V100/flag/static_2d/001/
These weak-scaling experiements were ran on 1, 2, 4, 8, 16, 32, 56, 64, 96, and 112 ranks with a single Crossroads node.

.. code-block:: bash
These experiments were ran with core-bidning turned on and plotting enabled. All were built with gcc 9.4.0 and openmpi 3.1.6

bash scripts/scaling.sh -a flag -p static_2d -f 001 -n V100 -g -t
xRAGE Asteroid Spatter Pattern 5
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

..
Weak-scaling experiment for the pattern in patterns/xrage/asteroid/spatter5.json. Results will be found in spatter.weakscaling/Crossroads/xrage/asteroid/spatter5/ and Figures will be found in figures/spatter.weakscaling/Crossroads/xrage/asteroid/spatter5

.. csv-table:: Spatter Throughput (MB/s) on V100 Flag Static 2D 001 Patterns
:file: v100_throughput_001.csv
:align: center
:widths: 5, 8, 8, 8, 8, 8, 8, 8, 8
:header-rows: 1
This pattern is a Gather (dest[i] = src[pattern[i]]) with a length of 8,368,968 elements (pattern[i]) with a target vector (src[i]) length of 1,120,524.

.. figure:: v100_throughput_001.png
:align: center
:scale: 50%
:alt: Spatter Throughput on V100 Flag Static 2D 001 Patterns

Spatter Throughput on V100 Flag Static 2D 001 Patterns


Flag Static 2D 001.FP
~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash
bash scripts/scaling.sh -a flag -p static_2d -f 001.fp -n V100 -g -t
bash scripts/scaling.sh -a xrage -p asteroid -f spatter5 -n Crossroads -c -w
..
.. csv-table:: Spatter Throughput (MB/s) on V100 Flag Static 2D 001 FP Patterns
:file: v100_throughput_001fp.csv
.. csv-table:: Spatter Weak Scaling Performance for xRAGE Spatter Pattern 5 on Crossroads
:file: ats3_weak_average_asteroid_5.csv
:align: center
:widths: 5, 8, 8, 8, 8
:widths: 5, 8
:header-rows: 1

.. figure:: v100_throughput_001fp.png
.. figure:: ats3_weak_average_asteroid_5.png
:align: center
:scale: 50%
:alt: Spatter Throughput on V100 Flag Static 2D 001 FP Patterns
:alt: Spatter Weak Scaling Performance for xRAGE Spatter Pattern 5 on Crossroads

Spatter Throughput on V100 Flag Static 2D 001 FP Patterns

xRAGE Asteroid Spatter Pattern 9
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Weak-scaling experiment for the pattern in patterns/xrage/asteroid/spatter5.json. Results will be found in spatter.weakscaling/Crossroads/xrage/asteroid/spatter9/ and Figures will be found in figures/spatter.weakscaling/Crossroads/xrage/asteroid/spatter9

Flag Static 2D 001.NONFP
~~~~~~~~~~~~~~~~~~~~~~~~
This pattern is a Scatter (dest[pattern[i]] = src[i]) with a length of 6,664,304 elements (pattern[i]) with a target vector (dest[i]) length of 2,051,100.

.. code-block:: bash
bash scripts/scaling.sh -a flag -p static_2d -f 001.nonfp -n V100 -g -t
bash scripts/scaling.sh -a xrage -p asteroid -f spatter9 -n Crossroads -c -w
..
.. csv-table:: Spatter Throughput (MB/s) on V100 Flag Static 2D 001 Non-FP Patterns
:file: v100_throughput_001nonfp.csv
.. csv-table:: Spatter Weak Scaling Performance for xRAGE Spatter Pattern 9 on Crossroads
:file: ats3_weak_average_asteroid_9.csv
:align: center
:widths: 5, 8, 8, 8, 8, 8, 8, 8, 8
:widths: 5, 8
:header-rows: 1

.. figure:: v100_throughput_001nonfp.png
.. figure:: ats3_weak_average_asteroid_9.png
:align: center
:scale: 50%
:alt: Spatter Throughput on V100 Flag Static 2D 001 Non-FP Patterns
:alt: Spatter Weak Scaling Performance for xRAGE Spatter Pattern 9 on Crossroads

Spatter Throughput on V100 Flag Static 2D 001 Non-FP Patterns

V100
------------

Strong-Scaling throughput experiments with plotting enabled.


xRAGE Asteroid
Expand Down Expand Up @@ -270,87 +276,7 @@ Throughput experiment for the 9 patterns in patterns/xrage/asteroid/spatter.json
A100
------------

Strong-Scaling throughput experiment with plotting enabled. Results will be found in spatter.strongscaling/A100/flag/static_2d/001 and Figures will be found in figures/spatter.strongscaling/A100/flag/static_2d/001.

.. code-block:: bash
cd spatter
bash scripts/scaling.sh -a flag -p static_2d -f 001 -n A100 -g -t
..
Flag Static 2D 001
~~~~~~~~~~~~~~~~~~

Throughput experiment for the 8 patterns in patterns/flag/static_2d/001.json on a single GPU with plotting enabled. Results will be found in spatter.strongscaling/A100/flag/static_2d/001/ and Figures will be found in figures/spatter.strongscaling/A100/flag/static_2d/001/

.. code-block:: bash
bash scripts/scaling.sh -a flag -p static_2d -f 001 -n A100 -g -t
..
.. csv-table:: Spatter Throughput (MB/s) on A100 Flag Static 2D 001 Patterns
:file: a100_throughput_001.csv
:align: center
:widths: 5, 8, 8, 8, 8, 8, 8, 8, 8
:header-rows: 1

.. figure:: a100_throughput_001.png
:align: center
:scale: 50%
:alt: Spatter Throughput on A100 Flag Static 2D 001 Patterns

Spatter Throughput ono A100 Flag Static 2D 001 Patterns


Flag Static 2D 001.FP
~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash
bash scripts/scaling.sh -a flag -p static_2d -f 001.fp -n A100 -g -t
..
.. csv-table:: Spatter Throughput (MB/s) on A100 Flag Static 2D 001 FP Patterns
:file: a100_throughput_001fp.csv
:align: center
:widths: 5, 8, 8, 8, 8
:header-rows: 1

.. figure:: a100_throughput_001fp.png
:align: center
:scale: 50%
:alt: Spatter Throughput on A100 Flag Static 2D 001 FP Patterns

Spatter Throughput on A100 Flag Static 2D 001 FP Patterns



Flag Static 2D 001.NONFP
~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash
bash scripts/scaling.sh -a flag -p static_2d -f 001.nonfp -n A100 -g -t
..
.. csv-table:: Spatter Throughput (MB/s) on A100 Flag Static 2D 001 Non-FP Patterns
:file: a100_throughput_001nonfp.csv
:align: center
:widths: 5, 8, 8, 8, 8, 8, 8, 8, 8
:header-rows: 1

.. figure:: a100_throughput_001nonfp.png
:align: center
:scale: 50%
:alt: Spatter Throughput on A100 Flag Static 2D 001 Non-FP Patterns

Spatter Throughput on A100 Flag Static 2D 001 Non-FP Patterns

Strong-Scaling throughput experiment with plotting enabled.

xRAGE Asteroid
~~~~~~~~~~~~~~
Expand Down
17 changes: 1 addition & 16 deletions doc/sphinx/09_Microbenchmarks/M2_SPATTER/a100.gp
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/gnuplot
set terminal pngcairo enhanced size 1024, 768 dashed font 'Helvetica,18'
set output "a100_throughput_001.png"
set output "a100_throughput_asteroid.png"

set title "Spatter Throughput on A100, Flag Static 2D Patterns" font "serif,22"
set xlabel "No. Gathers/Scatters"
Expand All @@ -25,21 +25,6 @@ do for [i=11:15] {
set style line i linewidth 3 dashtype 2 pointsize 1.5
}

plot "a100_throughput_001.csv" using 1:2 with linespoints linestyle 1, "" using 1:3 with linespoints linestyle 2, "" using 1:4 with linespoints linestyle 3, "" using 1:5 with linespoints linestyle 4, "" using 1:6 with linespoints linestyle 5, "" using 1:7 with linespoints linestyle 11, "" using 1:8 with linespoints linestyle 12, "" using 1:9 with linespoints linestyle 13

set output "a100_throughput_001fp.png"
set title "Spatter Throughput on A100, Flag Static 2D FP Patterns" font "serif,22"
set ylabel "Figure of Merit (Total Bandwidth MB/s)"
plot "a100_throughput_001fp.csv" using 1:2 with linespoints linestyle 1, "" using 1:3 with linespoints linestyle 2, "" using 1:4 with linespoints linestyle 3, "" using 1:5 with linespoints linestyle 4

set output "a100_throughput_001nonfp.png"
set title "Spatter Throughput on A100, Flag Static 2D Non-FP Patterns" font "serif,22"
set ylabel "Figure of Merit (Total Bandwidth MB/s)"
plot "a100_throughput_001nonfp.csv" using 1:2 with linespoints linestyle 1, "" using 1:3 with linespoints linestyle 2, "" using 1:4 with linespoints linestyle 3, "" using 1:5 with linespoints linestyle 4, "" using 1:6 with linespoints linestyle 5, "" using 1:7 with linespoints linestyle 11, "" using 1:8 with linespoints linestyle 12, "" using 1:9 with linespoints linestyle 13

set output "a100_throughput_asteroid.png"
set title "Spatter Throughput on A100, xRAGE Asteroid Patterns" font "serif,22"
set ylabel "Figure of Merit (Total Bandwidth MB/s)"
plot "a100_throughput_asteroid.csv" using 1:2 with linespoints linestyle 1, "" using 1:3 with linespoints linestyle 2, "" using 1:4 with linespoints linestyle 3, "" using 1:5 with linespoints linestyle 4, "" using 1:6 with linespoints linestyle 5, "" using 1:7 with linespoints linestyle 11, "" using 1:8 with linespoints linestyle 12, "" using 1:9 with linespoints linestyle 13, "" using 1:10 with linespoints linestyle 14


9 changes: 0 additions & 9 deletions doc/sphinx/09_Microbenchmarks/M2_SPATTER/a100_throughput_001.csv
View file

This file was deleted.

9 changes: 0 additions & 9 deletions doc/sphinx/09_Microbenchmarks/M2_SPATTER/a100_throughput_001fp.csv
View file

This file was deleted.

9 changes: 0 additions & 9 deletions doc/sphinx/09_Microbenchmarks/M2_SPATTER/a100_throughput_001nonfp.csv
View file

This file was deleted.

40 changes: 40 additions & 0 deletions doc/sphinx/09_Microbenchmarks/M2_SPATTER/ats3.gp
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/usr/bin/gnuplot
set terminal pngcairo enhanced size 1024, 768 dashed font 'Helvetica,18'
set output "ats3_weak_average_asteroid_5.png"

set xlabel "No. Processing Elements"
set ylabel "Figure of Merit (Avg. Bandwidth per rank MB/s)"

set xrange [1:128]
set key outside

set logscale x 2

set grid
show grid

set datafile separator comma
set key autotitle columnheader

do for [i=1:10] {
set style line i linewidth 3 pointsize 1.5
}

do for [i=11:15] {
set style line i linewidth 3 dashtype 2 pointsize 1.5
}


plot "ats3_weak_average_asteroid_5.csv" using 1:2 with linespoints linestyle 1

set output "ats3_weak_total_asteroid_5.png"
set ylabel "Figure of Merit (Total Bandwidth MB/s)"
plot "ats3_weak_total_asteroid_5.csv" using 1:2 with linespoints linestyle 1

set output "ats3_weak_average_asteroid_9.png"
set ylabel "Figure of Merit (Total Bandwidth MB/s)"
plot "ats3_weak_average_asteroid_5.csv" using 1:2 with linespoints linestyle 1

set output "ats3_weak_total_asteroid_9.png"
set ylabel "Figure of Merit (Total Bandwidth MB/s)"
plot "ats3_weak_total_asteroid_9.csv" using 1:2 with linespoints linestyle 1
11 changes: 11 additions & 0 deletions doc/sphinx/09_Microbenchmarks/M2_SPATTER/ats3_weak_average_asteroid_5.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
No. Cores,Pattern
1,7427.1
2,7400.44
4,7080.54
8,6595.21
16,5480.66
32,5533.13
56,4884.83
64,4820.64
96,4430.0
112,4406.79
Loading

0 comments on commit 1a66178

Please sign in to comment.