WIP: Introducing the grcli development tool

[mabruzzo]

This PR is ~90% complete.

Overview/Motivation

This PR introduces a new development tool called grcli. It is a command-line tool, written in C++, that can be used to configure and execute fairly arbitrary grackle-sample problems. Currently, the primary purpose is to use it for benchmarking.

At some point in the near future (in a different PR), it is my intention to add support for debugging.¹

This tool makes use of Google's benchmark library (I'm not 100% sure this is the right tool for the job). The CMake build system is configured to automatically download and build that library as part of compiling grcli.

Description

The outputs of grcli --help are displayed down below (in the collapsible section

grcli description

NAME:

    grcli  - A command line interface for benchmarking and debugging Grackle

SYNOPSIS:

    grcli [--version] [--precision] [-h | --help] bench -S... PARAMETER_OPTIONS -O...

    grcli [--version] [--precision] [-h | --help] show-parameters PARAMETER_OPTIONS

    grcli [--version] [--precision] [-h | --help] show-initial-units

DESCRIPTION:
    A simple command-line tool intended for debugging and basic benchmarking of
    grackle.

    We summarize the subcommands down below. Be note that the current
    organization is experimental and is subject to change

      -> bench sets up an example problem and benchmarks it

      -> show-parameters is intended to display the configured parameters.

      -> show-initial-units is intended to display the initial units used
         during configuration of the grackle-solver. In the future, we may add
         support for customizing these units and using a different set of units
         during Grackle operations

OPTIONS:
    Some Flags are grouped into categories

    -S...
        Prefix for all "scenario arguments". The "scenario" is the algorithm
        for initializing all grackle fields that are used in a "test-problem."

        At the moment, this might include
          * -Sgrid.ax<I>=<quantity>,<seq>
        Where the quantities in brackets get replaced by various things. For
        example:
          * <I> is replaced with 0, 1, or 2
          * <quantity> is replaced with temperature, density, metallicity
          * <seq> is replaced by arguments that specify a sequence of 1 or more
            values. In practice, this is either a lone floating-point value OR
            geomspace,<start>,<end>,<num>

    -O...
        Prefix for all "operation arguments". Essentially, this denotes the
        operation that is executed in a given "test-problem." Currently, you
        can exclusively specify
          * -Ocalc-cooling-time  Computes the cooling time
          * -Ocalc-dust-temperature  Computes the dust-temperature
          * -Ocalc-pressure  Computes the pressure
          * -Ocalc-temperature  Computes the temperature
          * -Osolve-chemistry-dt=<dt-value>  Solves chemistry for the specified
            timestep (given in code-units).

    -h; --help
        Prints this message

    --version
        Prints the version of Grackle

    --precision
        Prints the floating-point precision selected while compiling Grackle

  PARAMETER OPTIONS

    --par-start
        Every "word" after this option (until the --par-stop option) is parsed
        as a key-value pair that is used as a grackle parameter.

    --par-stop
        The sentinel value that must follow --par-start

Here's an example invocation:

Currently, there isn't any way to adjust the code_units. But that's something that can be introduced

./build/src/grcli/grcli bench \
    --par-start metal_cooling=1 use_grackle=1 primordial_chemistry=0 \
    'grackle_data_file="input/CloudyData_UVB=HM2012.h5"' \
    --par-stop \
    -Sgrid.ax0=temperature,geomspace,1e4,1e6,128 \
    -Sgrid.ax1=metallicity,1.0 \
    -Sgrid.ax2=mass_density,geomspace,0.01,10.0,128 \
    -Osolve-chemistry-dt=1e-6

Here's the output on my Mac Book Pro with an M3 Pro processor (I think the output probably looks nicer and is more informative on Linux):

Unable to determine clock rate from sysctl: hw.cpufrequency: No such file or directory
This does not affect benchmark measurements, only the metadata output.
***WARNING*** Failed to set thread affinity. Estimated CPU frequency may be incorrect.
2024-08-07T23:51:08-04:00
Running ./build/src/grcli/grcli
Run on (12 X 24.0001 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB
  L1 Instruction 128 KiB
  L2 Unified 4096 KiB (x12)
Load Average: 1.76, 1.82, 1.98
-----------------------------------------------------
Benchmark           Time             CPU   Iterations
-----------------------------------------------------
scenario      4736382 ns      4723905 ns          148

This translates to a case where the timestep is ~1e-6 Myr.

The following collapsed section shows a snippets of the outputs for calculate_temperature and calculate_cooling_time.

other timings

Calculate cooling time:

Load Average: 1.53, 1.65, 1.81
-----------------------------------------------------
Benchmark           Time             CPU   Iterations
-----------------------------------------------------
scenario      4680909 ns      4674407 ns          150

Calculate temperature:

Load Average: 1.87, 1.82, 1.92
-----------------------------------------------------
Benchmark           Time             CPU   Iterations
-----------------------------------------------------
scenario      3330047 ns      3329395 ns          210

This suggests, that up to ~70% of the cooling-time calculation is spent computing temperature (for primordial_chemistry == 0)

ToDo List

There are essentially 3 main things left to do:

• explore compilation without the benchmark library
• improve integration with the benchmark library (allow us to properly forward arguments to the benchmark library)
• consider adding support for "chemistry-data presets" (so that we don't need to specify some really common arguments, like metal_cooling=1 use_grackle=1 every time we run a test-problem

Footnotes

1. In more detail, PR State-Dumping Debug Tool #197 seeks to add debugging tools to dump the state of grackle to disk. That PR introduces some python functions to read in the dumped data. The idea would be to add functionality to grcli to do the same. ↩