doit.org

HDF5 I/O Test Log Book

Global Configuration

(setq myvars (make-hash-table :test 'equal))
(puthash "default" "no match found" myvars)
(puthash "greetings" "Hello, World!" myvars)

(gethash key myvars)

echo <<myx(key="greetings")>>

(message <<myx(key="greetings")>>)

Phase 1: Data Gathering

Environment

Emacs + Org

(princ (concat
        (format "Emacs version: %s\n"
                (emacs-version))
        (format "org version: %s\n"
                (org-version))))

File system

make -C src -f Makefile whats_mounted

make -C src -f Makefile block_devs

Versions

OS

make -C src -f Makefile os_info

GCC

make -C src -f Makefile cc_info

MPI

make -C src -f Makefile mpi_info

HDF5

make -C src -f Makefile hdf5_version

System performance

System specifications and acceptance test results

Fio

make -C src -f Makefile run_fio

IOR

h5perf

Application documentation

Goals and expectations

• [ ] Describe the application
• [ ] Document performance goals
• [ ] Perform back-of-the-envelope calculations
• [ ] Document qualitative behavior expectations

  make -C src -f Makefile veryclean all
      

Baseline

• [ ] Document the sequential application baseline
• [ ] Document the parallel application baseline(s)
  • Strong and weak scaling
• [ ] Capture the outputs of h5dump -pBH and h5stat

Sequential

cat <<EOF
[DEFAULT]
version = 0                ; Let's keep an open mind!
steps = 20                 ; Number of steps
arrays = 500               ; Number of arrays
rows = 100                 ; Total number of array rows for strong scaling
                           ; Number of array rows per block for weak scaling
columns = 200              ; Total number of array columns for strong scaling
                           ; Number of array columns per block for weak scaling
process-rows = 1           ; Number of MPI-process rows:
                           ; rows % proc-rows == 0 for strong scalingx
process-columns = 1        ; Number of MPI-process columns:
                           ; columns % proc-columns == 0 for strong scaling
scaling = weak             ; Scaling ([weak, strong])
dataset-rank = 4           ; Rank of the dataset(s) in the file ([2, 3, 4])
slowest-dimension = step   ; Slowest changing dimension ([step, array])
layout = contiguous        ; HDF5 dataset layout ([contiguous, chunked])
mpi-io = independent       ; MPI I/O mode ([independent, collective])
hdf5-file = hdf5_iotest.h5 ; HDF5 output file name
csv-file = hdf5_iotest.csv ; CSV results file name
EOF

make -C src -f Makefile baseline

make -C src -f Makefile h5dump

make -C src -f Makefile h5stat

Profiles

gperf

make -C src -f Makefile gprofile

Darshan

make -C src -f Makefile darshan

Recorder

Parallel

Symptoms

• [ ] Document user concerns and observations
• [ ] Confirm their reproducibility

Phase 2: Analysis / Diagnosis

• [ ] Visualize the data
• [ ] If the data does not confirm your expectations document what you didn’t expect.
• [ ] Look for telltale signs indicative of performance problems
  • gperf
  • Darshan
  • Recorder
• [ ] Document your observations

Phase 3: Performance Improvement Plan

Goal Setting

Low-hanging Fruit

• PFS Settings
• MPI-IO Hints
• HDF5 Settings

Hypothesis Testing

Phase 4: Execution

Traps and Pitfalls

Unobserved Variables

Complex Explanations

Occam’s Razor

Paths of Diminishing Returns

Stopping Criteria

Documentation

Conclusion

Appendix