Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark GeoTIFF read speeds #22

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Benchmark GeoTIFF read speeds #22

wants to merge 8 commits into from

Conversation

weiji14
Copy link
Member

@weiji14 weiji14 commented Oct 8, 2024

Measuring how fast it takes to read GeoTIFF files (in terms of duration/time and throughput), setting up a baseline level of performance as we work towards addressing #5.

  • Using three 50MB GeoTIFF files from https://github.com/kokoalberti/geotiff-benchmark of dtype u8, i16 and f32
  • Set up CI to run the criterion.rs benchmarks that will be tracked on Codspeed over time
    • Note that the benchmarks will only run on PRs labelled run/benchmark, on the main branch, or on release.

To run locally:

# Download and extract GeoTIFF files
wget https://s3.us-east-2.amazonaws.com/geotiff-benchmark-sample-files/geotiff_sample_files.tar.gz -P resources
tar --extract --verbose --file resources/geotiff_sample_files.tar.gz

# Run benchmarks
cargo bench

Example output on my laptop:

     Running benches/read_geotiff.rs (target/release/deps/read_geotiff-adb36ae75c76df4d)
Gnuplot not found, using plotters backend
read_geotiff_50MB/u8_dtype
                        time:   [15.652 ms 15.730 ms 15.822 ms]
                        thrpt:  [3.1601 GB/s 3.1786 GB/s 3.1945 GB/s]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
read_geotiff_50MB/i16_dtype
                        time:   [15.104 ms 15.154 ms 15.207 ms]
                        thrpt:  [3.2879 GB/s 3.2995 GB/s 3.3103 GB/s]
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe
read_geotiff_50MB/f32_dtype
                        time:   [1.4039 ms 1.4068 ms 1.4099 ms]
                        thrpt:  [35.463 GB/s 35.541 GB/s 35.616 GB/s]
Found 18 outliers among 100 measurements (18.00%)
  4 (4.00%) low severe
  9 (9.00%) low mild
  3 (3.00%) high mild
  2 (2.00%) high severe

References:

Statistics-driven micro-benchmarking library! Repo at https://github.com/bheisler/criterion.rs
Based on https://docs.codspeed.io/benchmarks/rust#running-the-benchmarks-in-your-ci, adapted to have a trigger that only runs when a `run/benchmark` label is in place, and includes a download/extract step for the sample GeoTIFF files.
@weiji14 weiji14 added this to the v0.1.0 milestone Oct 8, 2024
@weiji14 weiji14 self-assigned this Oct 8, 2024
Codspeed action only supports Ubuntu 20.04 and 22.04 at the moment.
Set benchmark group name to `read_geotiff_50MB`, and benchmark function names to `u8_dtype`, `i16_dtype`, `f32_dtype`.
Specifically byte_50m.tif, float32_50m.tif, int16_50m.tif.
From downloading the tar archive, extracting, and running the benchmark via `cargo bench`.
@weiji14
Copy link
Member Author

weiji14 commented Oct 8, 2024

Ok, so the Codspeed Action (for tracking benchmarks over time) requires someone with admin permissions to install the CodspeedHQ GitHub App (see docs at https://docs.codspeed.io/importing-repositories). Have requested the install, and will re-run the benchmarks CI once that is enabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant