forked from HPCToolkit/hpctoolkit
-
Notifications
You must be signed in to change notification settings - Fork 2
/
README.Install
1089 lines (833 loc) · 42.5 KB
/
README.Install
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Installing HPCToolkit with Spack
================================
1 Introduction
2 Prerequisites
3 Spack Notation
4 Clone Spack and HPCToolkit
5 Config.yaml
6 Packages.yaml
6.1 External Packages
6.2 Micro-Architecture Targets
7 Compilers and compilers.yaml
8 Spack Install
9 Manual Install
10 Advanced Options
10.1 CUDA
10.2 MPI
10.3 PAPI vs Perfmon
11 Platform Specific Notes
11.1 Blue Gene
11.2 Cray
12 Hpcviewer and Hpctraceviewer
12.1 Spack Install
12.2 Manual Install
13 Building a New Compiler
13.1 Using the New Compiler
13.2 Bootstrapping Environment Modules
14 Spack Mirrors
15 Common Problems
15.1 Unable to fetch tar file
15.2 New releases break the build
15.3 Failure to load modules
1 Introduction
==============
These notes describe how to build and install HPCToolkit and hpcviewer
and their prerequisites with Spack. HPCToolkit proper (hpcrun,
hpcstruct and hpcprof) is used to measure and analyze an application's
performance and then produce a database for hpcviewer. HPCToolkit is
supported on the following platforms.
1. Linux (64-bit) on x86_64, big and little-endian powerpc (power7, 8
and 9) and ARM (aarch64).
2. Cray XT/XE/XK/XC on x86_64 and Compute Node Linux.
3. IBM Blue Gene/Q on ppc64 (now deprecated).
We provide binary distributions for hpcviewer and hpctraceviewer on
Linux (x86_64 and powerpc), Windows and MacOS. HPCToolkit databases are
platform-independent and it is common to run hpcrun on one machine and
then view the results on another machine.
We build HPCToolkit and its prerequisite libraries from source.
HPCToolkit has some 15-20 prerequisites and we now use spack to build
them. It is possible to use spack to install all of hpctoolkit or build
just the prerequisites and then build hpctoolkit with the traditional
'configure ; make ; make install' method from autotools. Developers
will probably want to run 'configure' and 'make' manually, but both
methods are supported.
Note: the old method of using hpctoolkit-externals to build the
prerequisite libraries is now superseded by spack. Although externals
will continue to work for a while, we have stopped updating externals
and now add newer versions of packages only in spack.
These notes are written mostly from the view of using spack to build
hpctoolkit and its dependencies. If you are a more experienced spack
user, especially if you want to use spack to build hpctoolkit plus
several other packages, then you will want to adapt these directions to
your own needs.
Spack documentation is available at:
<https://spack.readthedocs.io/en/latest/index.html>
The current status of using Spack for HPCToolkit is at:
<http://hpctoolkit.org/spack-issues.html>
Last revised: June 17, 2020.
2 Prerequisites
===============
Building HPCToolkit requires the following prerequisites.
1. The GNU gcc and g++ compilers version 5.x or later. On systems
with older compilers, you can use spack to build a later version of
gcc.
2. GNU glibc version 2.16 or later. Note: Red Hat 6.x uses glibc 2.12
which is too old.
3. Basic build tools: make, ld, ar, objcopy, nm, etc, and shell
utilities: bash, sed, awk, grep, etc. Most Linux systems have
these tools, or else you couldn't compile anything.
4. Cmake version 3.4 or later, perl version 5.x, and python version
2.7 or 3.4 or later. On systems that are missing these tools or
have versions that are too old, you can use spack to build a later
version.
5. Git and curl for downloading files.
6. (optional) Environment (TCL) or LUA (Lmod) modules, if you want to
make HPCToolkit available as a module. Again, spack can install
these packages if they are missing from your system.
Hpcviewer and hpctraceviewer require Java version 8. On Linux, the
viewers also require GTK+ version 2.x.
3 Spack Notation
================
Spack uses a special notation for specifying the version, variants,
compilers and dependencies when describing how to build a package. This
combination of version, variants, etc is called a 'spec' and is used
both on the command line and in config files.
1. '@' specifies the package version. 'spack info <package>' shows
the available versions and variants for a package. In most cases,
spaces are optional between elements of a spec. For example:
[email protected] dyninst @10.1.0 hpctoolkit @master
2. '+', '-', '~' specify boolean (on/off) variants. Note: '-' (dash)
and '~' (tilde) both mean 'off'. Use dash after a space and tilde
after a non-space. For example:
elfutils+bzip2~nls elfutils +bzip2 -nls [email protected] +bzip2~nls
3. 'name=value' specifies a non-boolean variant, for example:
dyninst+openmp build_type=RelWithDebInfo [email protected] transcoder=iconv
4. '%' specifies the build compiler and its version, for example:
hpctoolkit@master %[email protected]
5. 'cflags', 'cxxflags', 'fflags', 'cppflags', 'ldflags' and 'ldlibs'
are special name/value variants for compiler flags. These are
normally not needed, but if you do need to add a flag to the build,
then one example might be:
amg2013 cflags='-O2 -mavx512pf'
6. '^' represents a dependency spec. The spec for a dependency
package is a full spec and may include its own version, variants,
etc. For example:
hpctoolkit@master ^[email protected]+openmp
7. 'arch', 'platform', 'os' and 'target' are special options for the
system architecture and machine type. Platform is normally
'linux', or else 'cray' or 'bgq' (or even 'windows' or 'darwin').
OS is the Linux distribution, 'rhel7', 'sles15', etc, and target is
the machine type, 'x86_64', 'ppc64le', etc. Arch is a triple of
platform, os and target separated by dashes.
Normally, a system has only one arch type and you don't need to
specify this. However, for systems with separate front and
back-end types, the default is the back end. For example, if you
wanted to build python for the front end on Cray, then you might
use something like this.
[email protected] arch=cray-sles12-x86_64
Also, now that spack has implemented microarchitecture targets
(haswell, ivybridge, etc), you can use 'target' to build for a
generic x86_64 or a specific CPU type. For example:
amg2013 target=x86_64 lulesh target=ivybridge
When writing a spec (for 'spack install' or other), spack will fully
resolve all possible choices for the package and all of its dependencies
and create a unique hash value for that exact configuration. This
process is called 'concretization.' To see how spack would concretize a
spec, use 'spack spec <partial-spec>'.
<https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies>
4 Clone Spack and HPCToolkit
============================
Spack is available via git clone from GitHub. This includes the core
spack machinery and recipes for building more than 4,200 packages (and
growing). You should also clone HPCToolkit for the 'packages.yaml' file
which is used to configure the spack build.
git clone https://github.com/spack/spack.git
git clone https://github.com/hpctoolkit/hpctoolkit.git
After cloning, add the 'spack/bin' directory to your PATH, or else
source the spack 'setup-env' script.
(bash) . /path/to/spack/share/spack/setup-env.sh
(csh) setenv SPACK_ROOT /path/to/spack/root
source $SPACK_ROOT/share/spack/setup-env.csh
It suffices to add 'spack/bin' to your PATH (or even symlink the spack
launch script). Sourcing the 'setup-env' script adds extra support for
modules built by spack.
5 Config.yaml
=============
'config.yaml' is the top-level spack config file. This specifies the
directory layout for installed files and the top-level spack parameters.
There are two fields that you normally want to set, especially if you
want to install modules for hpctoolkit in a public directory. For a
module to be available, both the 'install_tree' and 'module_roots'
directories must be accessible.
1. 'install_tree' - by default, spack installs packages inside the
spack repository at 'spack/opt/spack'. If you want to use another
location, then set this.
2. 'module_roots' - by default, spack installs module files inside the
spack repository at 'spack/share/spack'. If you want to use
another location, then set this.
There are two other fields that you may want to set for you local
system.
3. 'connect_timeout' - some download sites, especially sourceforge are
often slow to connect. If you find that connections are timing
out, then increase this time to 30 or 60 seconds (default is 10
seconds).
4. 'build_jobs' - by default, spack uses all available hardware
threads for parallel make, up to a limit of 16. If you're using a
shared machine and don't want to use the entire machine, then set
this to a smaller number.
There are also parameters for the locations of the build directories,
the cache of downloaded tar files, etc, which you may wish to set.
The default 'config.yaml' file is in the spack repository at
'spack/etc/spack/defaults'. The simplest solution is to copy this file
one directory up and then edit the copy (don't edit the default file
directly).
cd spack/etc/spack
cp defaults/config.yaml .
vi config.yaml
Alternatively, you could put this file in a separate directory, outside
of the spack repository and then use '-C dir' on the spack command line.
(The '-C' option goes before the spack command name.)
spack -C dir install ...
Note: if you put 'config.yaml' in 'spack/etc/spack', then it will apply
to every spack command for that repository (and you won't forget).
Putting it in a separate directory is more flexible because you can
support multiple configurations from the same repository. But then you
must use '-C dir' with every spack command or else you will get
inconsistent results.
You can view the current configuration with 'spack config'.
spack [-C dir] config get config
See the spack docs on 'Configuration Files' and 'Basic Settings'.
<https://spack.readthedocs.io/en/latest/configuration.html>
<https://spack.readthedocs.io/en/latest/config_yaml.html>
6 Packages.yaml
===============
The 'packages.yaml' file specifies the versions and variants for the
packages that spack installs and serves as a common reference point for
HPCToolkit's prerequisites. This file also specifies the paths or
modules for system build tools (cmake, python, etc) to avoid rebuilding
them. Put this file in the same directory as 'config.yaml'. A sample
'packages.yaml' file is available in the 'spack' directory of the
hpctoolkit repository.
There are two main sections to 'packages.yaml'. The first specifies
the versions and variants for hpctoolkit's prereqs. By default, spack
will choose the latest version of each package (plus any constraints
from hpctoolkit's 'package.py' file). In most cases, this will work,
but not always. If you need to specify a different version or variant,
then set this in 'packages.yaml'.
Note: the versions and variants specified in hpctoolkit's
'package.py' file are hard constraints and should not be changed.
Variants in 'packages.yaml' are preferences that may be modified for
your local system. (But don't report a bug until you have first tried
the versions from 'packages.yaml' that we supply.)
There are at least two packages with a variant that you may need to
change depending on your system. But always check the current
'packages.yaml' file to see if any more have been added.
1. 'intel-tbb' - for very old Intel or AMD systems that don't support
transactional memory, change '+tm' to '~tm'. (This option has no
effect on non-x86 systems.)
2. 'libmonitor' - on Blue Gene/Q, add '+bgq'.
6.1 External Packages
---------------------
The second section in 'packages.yaml' specifies a path or module for
system build tools. Building hpctoolkit's prerequisites requires cmake
3.4 or later, perl 5.x and python 2.7 or 3.4 or later. There are three
ways to satisfy these requirements: a system installed version (eg,
/usr), a pre-built module or build from scratch.
By default, spack will rebuild these from scratch, even if your local
version is perfectly fine. If you already have an installed version and
prefer to use that instead, then you can specify this in
'packages.yaml'. Note that these are only build tools. Hpctoolkit does
not link with any of their libraries.
For example, this entry says that cmake 3.7.2 is available from
module 'CMake/3.7.2' and that spack should use this instead of building
its own copy.
cmake:
modules:
[email protected]: CMake/3.7.2
buildable: False
This example says that perl v5.16.3 is installed at '/usr/bin/perl'.
Note that the 'paths:' entry is the parent directory of 'bin', not the
bin directory itself (similar to prefix).
perl:
paths:
[email protected]: /usr
buildable: False
6.2 Micro-Architecture Targets
------------------------------
Spack implements a hierarchy of micro-architecture targets, where
'target' is a specific architecture (eg, haswell, ivybridge) instead of
a generic family (x86_64). This allows the compiler to optimize code
for the specific target.
You will notice this choice in two main places: the 'spack spec' and
the path for the install directory. For example, 'linux-rhel7-x86_64'
might become 'linux-rhel7-broadwell'. You can use 'spack arch' to see
the list of generic families and micro-architecture targets.
spack arch --known-targets
If you prefer a generic install, you can use the 'target' option to
specify a generic family instead of a micro-architecture target. This
would be useful for a shared install that needs to work across multiple
machines with different micro-arch types. For example:
spack install hpctoolkit ... target=x86_64
If you want to use this target for all spack builds, then you can
specify this in 'packages.yaml'. The generic families are: x86_64,
ppc64, ppc64le and aarch64. For example:
packages:
all:
target: [x86_64]
See the spack docs on 'Build Customization' and 'Specs and
Dependencies'.
<https://spack.readthedocs.io/en/latest/build_settings.html>
<https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies>
7 Compilers and compilers.yaml
==============================
Building HPCToolkit requires GNU gcc/g++ version 5.x or later. By
default, spack uses the latest available version of gcc, but you can
specify a different compiler, if one is available.
Spack uses a separate file, 'compilers.yaml' to store information
about available compilers. This file is normally in your home directory
at '~/.spack/platform' where 'platform' is normally 'linux' (or else
'cray' or 'bgq').
The first time you use spack, or after adding a new compiler, you
should run 'spack compiler find' to have spack search your system for
available compilers. If a compiler is provided as a module, then you
should load the module before running 'find'. Normally, you only need
to run 'find' once, unless you want to add or delete a compiler. You
can also run 'spack compiler list' and 'spack compiler info' to see what
compilers spack knows about.
For example, on one power8 system running RedHat 7.3, /usr/bin/gcc is
version 4.8.5, but gcc 6.4.0 is available as module 'GCC/6.4.0'.
module load GCC/6.4.0
spack compiler find
==> Added 2 new compilers to /home/krentel/.spack/linux/compilers.yaml
==> Compilers are defined in the following files:
/home/krentel/.spack/linux/compilers.yaml
spack compiler list
==> Available compilers
-- gcc rhel7-ppc64le --------------------------------------------
spack compiler info [email protected]
paths:
cc = /opt/apps/software/Core/GCCcore/6.4.0/bin/gcc
cxx = /opt/apps/software/Core/GCCcore/6.4.0/bin/g++
f77 = /opt/apps/software/Core/GCCcore/6.4.0/bin/gfortran
fc = /opt/apps/software/Core/GCCcore/6.4.0/bin/gfortran
modules = ['GCC/6.4.0']
operating system = rhel7
Note: for compilers from modules, spack does not fill in the 'modules:'
field in the 'compilers.yaml' file. You need to do this manually. In
the above example, after running 'find', I edited 'compilers.yaml' to
add 'GCC/6.4.0' to the 'modules:' field as below. This is important to
how spack manipulates the build environment.
- compiler:
modules: [GCC/6.4.0]
operating_system: rhel7
spec: [email protected]
...
Spack uses '%' syntax to specify the build compiler and '@' syntax to
specify the version. For example, suppose you had gcc versions 7.3.1,
6.4.0 and 5.4.0 available and you wanted to use 6.4.0. You could write
this as:
spack install package %[email protected]
See the spack docs on 'Compiler Configuration'.
<https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration>
8 Spack Install
===============
First, set up your 'config.yaml', 'packages.yaml' and 'compilers.yaml'
files as above and edit them for your system. You can see how spack
will build hpctoolkit with 'spack spec' and 'spack graph'.
spack spec hpctoolkit
spack graph hpctoolkit
Then, the "one button" method uses spack to install everything.
spack install hpctoolkit
Tip: Spack fetch is somewhat fragile and often has transient problems
downloading files. You can use 'spack fetch -D' to pre-fetch all of the
tar files and resolve any downloading problems before starting the full
install.
spack fetch -D hpctoolkit
9 Manual Install
================
The manual method uses spack to build hpctoolkit's prerequisites and
then uses the traditional autotools 'configure && make && make install'
to install hpctoolkit. This method is primarily for developers who want
to compile hpctoolkit, edit the source code and recompile, etc. This
method is also useful if you need some configure option that is not
available through spack.
First, use spack to build hpctoolkit's prerequisites as above. You
can either build some version of hpctoolkit as before (which will pull
in the prerequisites), or else use '--only dependencies' to avoid
building hpctoolkit itself.
spack install --only dependencies hpctoolkit
Then, configure and build hpctoolkit as follows. Hpctoolkit uses
automake and so allows for parallel make.
configure \
--prefix=/path/to/hpctoolkit/install/prefix \
--with-spack=/path/to/spack/install_tree/linux-fedora26-x86_64/gcc-7.3.1 \
...
make -j <num>
make install
The argument to '--with-spack' should be the directory containing all of
the individual package directories, normally two directories down from
the top-level 'install_tree' and named by the platform and compiler.
This option replaces the old '--with-externals'. The following are
other options that may be useful. For the full list of options, see
'configure -h'.
1. '--enable-all-static' - build hpcprof-mpi statically linked for the
compute nodes.
2. '--enable-bgq' - configure for Blue Gene/Q back end.
3. '--enable-develop' - compile with optimization turned off for
debugging.
4. '--with-package=path' - specify the install prefix for some
prerequisite package, mostly for developers who want to use a
custom, non-spack version of some package.
5. 'MPICXX=compiler' - specify the MPI C++ compiler for hpcprof-mpi,
may be a compiler name or full path.
Note: if your spack install tree has multiple versions or variants for
the same package, then '--with-spack' will select the one with the most
recent directory time stamp (and issue a warning). If this is not what
you want, then you will need to specify the correct version with a
'--with-package' option.
10 Advanced Options
===================
10.1 CUDA
---------
Beginning with the 2020.03.01 version, HPCToolkit now supports profiling
CUDA binaries (nVidia only). For best results, use CUDA version 10.1 or
later and Dyninst 10.1 or later. Note: in addition to a CUDA
installation, you also need the CUDA system drivers installed. This
normally requires root access and is outside the scope of spack.
For a spack install with CUDA, use the '+cuda' variant.
spack install hpctoolkit +cuda
For a manual install, either download and install CUDA or use an
existing module, and then use the '--with-cuda' configure option.
configure \
--prefix=/path/to/hpctoolkit/install/prefix \
--with-spack=/path/to/spack/install/dir \
--with-cuda=/path/to/cuda/install/prefix \
...
If you installed CUDA with spack in the same directory as the rest of
the prerequisites, then the '--with-spack' option should find it
automatically (but check the summary at the end of the configure
output). If you are using CUDA from a separate system module, then you
will need the '--with-cuda' option.
10.2 MPI
--------
HPCToolkit always supports profiling MPI applications. For hpctoolkit,
the spack variant '+mpi' is for building hpcprof-mpi, the MPI version of
hpcprof. If you want to build hpcprof-mpi, then you need to supply an
installation of MPI.
Normally, for systems with compute nodes, you should use an existing
MPI module that was built for the correct interconnect for your system
and add this to 'packages.yaml'. The MPI module should be built with
the same version of GNU gcc/g++ used to build hpctoolkit (to keep the
C++ libraries in sync).
10.3 PAPI vs Perfmon
--------------------
HPCToolkit can access the Hardware Performance Counters with either PAPI
or Perfmon (libpfm4). By default, the hpctoolkit package uses perfmon.
If you want to use PAPI instead, then build hpctoolkit with '+papi'.
However, you can't use both due to a potential conflict in their header
files.
PAPI runs on top of the perfmon library, but PAPI uses its own,
internal copy of perfmon. Prior to version 5.6.0, PAPI did not install
the perfmon header files, so it was impossible to access the perfmon
events through PAPI.
However, starting with version 5.6.0, PAPI now installs both the
perfmon library and its header files. Hpctoolkit configure will
automatically detect this, so if you build hpctoolkit with a recent
enough version of PAPI, then both the PAPI and perfmon interfaces will
be available.
11 Platform Specific Notes
==========================
11.1 Blue Gene
--------------
Support for Blue Gene/Q is now deprecated and versions 2020.03.01 and
later will no longer build. If you want to install HPCToolkit on Blue
Gene, then use spack version 2019.12.28 or git commit 0d91fb34edb2 with
gcc 4.8.x and follow the directions below. These directions will also
work (without Blue Gene) for Red Hat 6.x (glibc 2.12 is too old to
support current hpctoolkit).
Blue Gene systems are being phased out, without a next generation
replacement, but are still supported by hpctoolkit (for now). Blue Gene
normally comes with a Red Hat Enterprise Linux (RHEL) 6.x front end and
the GNU gcc/g++ 4.4 compilers with are too old to support the latest
version of hpctoolkit.
Spack supports two architecture types on Blue Gene. The front-end
arch is 'bgq-rhel6-ppc64' and the back-end arch is 'bgq-cnk-ppc64'.
Normally, it would matter whether we built for the front end or back
end. But for the GNU compilers on Blue Gene, both compilers are
/usr/bin/gcc. So, we just build for the default back end.
There are two ways to build hpctoolkit for Blue Gene, depending on
the version of the compiler and Dyninst. The simple method uses the
default GCC 4.4 compiler and Dyninst 9.3.2. Edit these two entries in
the 'packages.yaml' file.
1. Set the dyninst version to '9.3.2'.
2. Add '+bgq' to the libmonitor variants.
Then, build hpctoolkit with '+bgq' turned on. The '+bgq' variant adds
MPI and hpcprof-mpi and supersedes the '+mpi' variant.
spack install hpctoolkit +bgq
The advanced method involves building a new compiler (gcc 4.8 or 4.9)
but allows using Dyninst 10.x. This method requires TCL or Lmod modules
and an MPI C++ compiler built for a compatible version of g++. Mira and
vesta at ANL have softenv modules for +mpiwrapper-gcc and
+bgqtoolchain-gcc484 but don't provide TCL or Lmod modules.
First, use spack to build a new compiler. Dyninst 10.x requires
gcc 4.8 or later, but libmonitor requires the true back-end
'powerpc64-bgq-linux-gcc' compiler which is version 4.4. We settle on
gcc 4.8.5 as a compromise between these two constraints. Edit
'config.yaml' to set 'module_roots' to a directory in which to install
the modules and then build gcc.
spack install gcc @4.8.5
Assuming you have modules installed, then load the new gcc module and
rerun 'spack compiler find'. This should add the new compiler to your
'compilers.yaml' file (twice, for front and back end). For example, on
vulcan at LLNL,
module use /path/to/modules/bgq-cnk-ppc64
module load gcc-4.8.5-gcc-4.4.7-nzvjwva
spack compiler find
==> Added 2 new compilers to /g/g21/krentel1/.spack/bgq/compilers.yaml
Note: after running 'spack compiler find', you still need to add the
modules field to the compiler entry in 'compilers.yaml' as in the
section on Compilers.
Note: in the above example, we used gcc 4.4.7 to build gcc 4.8.5 (so
the gcc-4.8.5 package is in the gcc-4.4.7 directory). You could then
use gcc 4.8.5 to rebuild 4.8.5, if you like, but this is not necessary.
Finally, use the new compiler to build hpctoolkit with the new
compiler. If you have modified the dyninst entry in 'packages.yaml',
then reset the dyninst version to 10.1.0 (or later).
spack install hpctoolkit +bgq %[email protected]
If you don't have modules installed, then use 'spack bootstrap' to build
the environment-modules package. Then, source the bash or csh script in
the 'Modules/init' directory to add the 'module' function to your
environment. For example,
spack bootstrap
cd /path/to/environment-modules-4.2.4-rwtvlanuss35fxd3xuyldbfopt2m4ecs/init
(bash) . ./bash
(csh) source ./csh
Alternatively, you could try adding the module environment variables
(PATH and LD_LIBRARY_PATH) manually, but then you may have to run spack
install with '--dirty'. The '--dirty' option tells spack not to erase
LD_LIBRARY_PATH while building packages.
spack install --dirty hpctoolkit +bgq %[email protected]
For developers, if you are building hpctoolkit directly (outside of
spack) but using spack prerequisites, then use a configure line similar
to the following.
configure \
--prefix=/path/to/install/prefix \
--with-spack=/path/to/bgq-cnk-ppc64/gcc-4.8.5 \
--enable-bgq \
--enable-all-static \
MPICXX=mpicxx
11.2 Cray
---------
Cray systems, like Blue Gene, have separate front and back-end
architecture types. For example, on theta at ANL, the front-end arch is
'cray-sles12-haswell' (SuSE Linux version 12 for haswell) and the
back-end is 'cray-cnl6-mic_knl' (Compute Node Linux for KNL).
Hpctoolkit needs to be built with the GNU Programming Environment and
the front-end x86_64 compilers, plus the 'CC' MPI C++ wrapper. Switch
to the 'PrgEnv-gnu' module and unload the darshan module. Darshan is a
profiling tool that monitors an application's use of I/O, but it
conflicts with hpctoolkit.
module swap PrgEnv-intel PrgEnv-gnu
module unload darshan
By default, spack does not correctly identify the gcc compiler modules
as front-end compilers. Start by loading a recent gcc module (eg, 7.x
or 8.x) and try running 'spack compiler find' and 'spack compiler list'.
For example on theta at ANL,
module load gcc/8.3.0
spack compiler find /opt/gcc/8.3.0
spack compiler list
==> Available compilers
-- gcc cnl6-any -------------------------------------------------
-- gcc sles12-x86_64 --------------------------------------------
If spack identifies the compiler (as above), then great. Note that we
need spack to accept this as a front-end compiler (sles12-x86_64), not
just for the back end (cnl6-any). If spack doesn't find the compiler
automatically, then you need to add it manually. Edit
'~/.spack/cray/compilers.yaml' and add an entry similer to the following
(your versions may differ).
- compiler:
environment: {}
flags: {}
modules:
- PrgEnv-gnu/6.0.5
- gcc/8.3.0
- craype/2.6.1
- cray-mpich/7.7.10
operating_system: sles12
paths:
cc: /opt/gcc/8.3.0/bin/gcc
cxx: /opt/gcc/8.3.0/bin/g++
f77: /opt/gcc/8.3.0/bin/gfortran
fc: /opt/gcc/8.3.0/bin/gfortran
spec: [email protected]
target: x86_64
Regardless of whether spack created the entry or you added it, you still
need to add the 'modules:' field manually (be sure to include all four
modules). Although it takes several steps to create the correct entry,
the good news is that it should continue to work as long as the
underlying module exists.
Next, review your 'packages.yaml' file. On Cray systems with Xeon
Phi back-end nodes (KNL, KNH, etc), add '~tm' to intel-tbb to disable
transactional memory.
Finally, build hpctoolkit with the front-end arch type (eg,
cray-sles12-x86_64) and option '+cray'. Normally, you can use OS type
'fe' (front-end) in place of 'sles12'. As with Blue Gene, the '+cray'
option adds MPI and hpcprof-mpi and supersedes the '+mpi' variant.
spack install hpctoolkit +cray arch=cray-fe-x86_64
On some systems, spack fails to properly load the compiler modules
during the build and fails during hpctoolkit configure with an error
about, "MPICXX is not a valid compiler." In this case, make sure that
you have the PrgEnv-gnu and gcc modules loaded and retry the install
with '--dirty'.
spack install --dirty hpctoolkit +cray arch=cray-fe-x86_64
For developers, if you are building hpctoolkit directly (outside of
spack) but using spack prerequisites, then use a configure line similar
to the following.
configure \
--prefix=/path/to/install/prefix \
--with-spack=/path/to/cray-sles12-x86_64/gcc-8.3.0 \
--enable-all-static \
MPICXX=CC
12 Hpcviewer and Hpctraceviewer
===============================
We provide binary distributions for hpcviewer and hpctraceviewer on
Linux (x86_64 and powerpc), Windows and MacOS. HPCToolkit databases are
platform-independent and it is common to run hpcrun on one machine and
then view the results on another machine.
All versions of the viewers require Java version 8 (not 9 or later).
The Linux versions also require GTK+ version 2.x.
12.1 Spack Install
------------------
The spack install is available on Linux x86_64, big-endian power7
(ppc64) and little-endian power8 and 9 (ppc64le). This installs both
hpcviewer and hpctraceviewer and includes the Java 8 prerequisite.
On x86_64, use either jdk or openjdk. The latest 1.8 jdk package,
1.8.0_202 has trouble downloading due to licensing issues, however
1.8.0_141-b15 seems to work. (Or, you could manually download jdk
1.8.0_202 and copy it to the spack file cache.) For openjdk, use
version 1.8.0_202-b08.
spack install hpcviewer ^[email protected]_141-b15
spack install hpcviewer ^[email protected]_202-b08
On powerpc (big and little-endian), use the IBM version of Java.
spack install hpcviewer ^ibm-java
12.2 Manual Install
-------------------
Binary distributions of the viewers for all supported platforms are
available at:
<http://hpctoolkit.org/download/hpcviewer>
On Linux, download the 'linux.gtk' versions of hpcviewer and
hpctraceviewer, unpack the tar files and run the install scripts (for
both viewers) with the path to the desired install prefix.
./install /path/to/install/directory
On Windows and MacOS, download the 'win32' or 'macosx.cocoa' versions
and unpack the zip files in the desired directory. Due to Apple's
security precautions, on MacOS, you will need to use curl or wget
instead of a web browser.
13 Building a New Compiler
==========================
Some systems may have compilers that are too old for building HPCToolkit
or other packages. For example, RedHat 7.x comes with gcc 4.8.x which
is very old. If your system doesn't already have modules for later
compilers, then you may need to build a new compiler yourself.
First, pick a directory in which to install the modules and make
subdirectories for the spack packages and module files. In this
example, I'm using '/opt/spack/Modules' as my top-level directory and
subdirectories 'packages' and 'modules'. Edit 'config.yaml' to add
these paths.
config:
install_tree: /opt/spack/Modules/packages
# just use one of these
module_roots:
tcl: /opt/spack/Modules/modules
lmod: /opt/spack/Modules/modules
Determine if your system uses TCL (environment) or Lmod modules.
Normally, the 'module' command is a shell function. TCL modules use
'modulecmd' and Lmod modules eval 'LMOD_CMD'. Edit 'modules.yaml' to
enable the module type for your system. Again, you only need one of
these (and don't use dotkit unless you work at LLNL).
modules:
enable:
- tcl
- lmod
Then, choose a version of gcc and use the default compiler (normally
'/usr/bin/gcc') to build the newer version. Currently, gcc 7.x is a
good choice that builds robustly, has enough modern features but is not
too new to cause problems for some packages. For example,
spack install [email protected]
Note: it is not necessary to rebuild the new compiler with itself.
13.1 Using the New Compiler
---------------------------
After building a new compiler, then you need to tell spack how to find
it. First, use 'module use' and 'module load' to load the module. For
TCL modules, the module files are in a subdirectory of 'module_roots'
named after the system architecture. For example,
module use /opt/spack/Modules/modules/linux-rhel7-x86_64
module load gcc-7.4.0-gcc-4.8.5-qemsqrc
For Lmod modules, the module directory is one level below that and the
module names are a little different.
module use /opt/spack/Modules/modules/linux-rhel7-x86_64/Core
module load gcc/7.4.0-dan4vbm
For both TCL and Lmod modules, it's best to put the 'module use' command
in your shell's startup scripts so that 'module avail' and 'module load'
will know where to find them. After loading the module, run 'spack
compiler find'.
$ spack compiler find
==> Added 1 new compiler to /home/krentel/.spack/linux/compilers.yaml
Finally, always check the new entry in 'compilers.yaml' and add the name
of the module to the 'modules:' field.
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules:
- gcc-7.4.0-gcc-4.8.5-qemsqrc
operating_system: rhel7
paths:
cc: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gcc
cxx: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/g++
f77: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
fc: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
spec: [email protected]
target: x86_64
Note: as long as the spack packages and modules directories remain
intact and you don't remove the 'compilers.yaml' entry, then this
compiler will always be available from within spack. You can also use
this compiler outside of spack by using 'module load'. If you want to
make this your default compiler for all spack builds, then you can
specify this in 'packages.yaml'. For example,
packages:
all:
compiler: [[email protected]]
Also, when using the compiler from within spack, it doesn't matter if
you have the module loaded or not. Spack will erase your environment
and re-add the appropriate modules automatically.
<https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration>
13.2 Bootstrapping Environment Modules
--------------------------------------
If your system does not support modules, then you will have to add it.
If you have root access, the easiest solution is to install a system
package for modules. If not, then use spack to install the
environment-modules package. Source the bash or csh script in the
'init' directory to add the 'module' function to your environment. For
example,
spack install environment-modules
cd /path/to/environment-modules-4.3.1-ism7cdy4xverxywj27jvjstqwk5oxe2v/init
(bash) . ./bash
(csh) source ./csh
Again, add the setup command to your shell's startup scripts.
14 Spack Mirrors
================
A spack mirror allows you to download and save a source tar file in
advance. This is useful if your system is behind a firewall, or if you
need to manually agree to a license, or if you just don't want to keep
downloading the same file over and over.
A mirror has a simple directory structure and is easy to set up.
Create a top-level directory with subdirectories named after the spack
packages and copy the tar files into their package's subdirectory. For
example,
my-mirror/
boost/
boost-1.66.0.tar.bz2 (from boost_1_66_0.tar.bz2)
boost-1.70.0.tar.bz2
dyninst/
dyninst-10.1.0.tar.gz (from git checkout)
ibm-java/
ibm-java-8.0.5.30.None (from ibm-java-sdk-8.0-5.30-ppc64le-archive.bin)
intel-xed/
intel-xed-2019.03.01.tar.gz (from git checkout)
mbuild-2019.03.01.tar.gz (resource from git checkout)
jdk/
jdk-1.8.0_202.tar.gz (from jdk-8u202-linux-x64.tar.gz)
Note: the names of the files in the spack mirror always follow the same,
specific format, regardless of the actual name of the tar file. Version
is the spack name for the version (from 'spack info'), and extension is
the same extension as the tar file ('tar.gz', 'tar.bz2', etc) or else
'None' for other types of files.
<package-name> - <version> . <extension>
For example, the boost 1.66.0 tar file is actually named
'boost_1_66_0.tar.bz2' but is stored in the mirror as
'boost-1.66.0.tar.bz2' and 'jdk-8u202-linux-x64.tar.gz' is renamed to
'jdk-1.8.0_202.tar.gz'.
For packages that use a snapshot from a git repository (tag or commit
hash), clone the repository, checkout the desired version, make a tar
file and gzip the file. (You should exclude the '.git' subdirectory.)
But note that spack refuses to use a cached file for the head of a
branch because it is a moving target.