Where can I find benchmark results?

[marcinz]

I am wondering if the benchmarks in this repo (or some other cupy benchmarks) are run regularly somewhere. If they are, is there a place where I can find the results?

[emcastillo]

They aren't 😄

[kmaehashi]

This is in our TODO list for a long time... How can we make it public? I'm thinking of:

• For every release, run a benchmark against the wheel. Persist the raw benchmark results and rendered HTML to GCS.
• For every pull-requests, run the benchmark against the code, then compare with the raw benchmark results in GCS, then persist rendered HTML to GCS.

[leofang]

@kmaehashi FYI: https://github.com/spcl/npbench

[jakirkham]

Interesting hadn't heard of that one before.

Another popular one is airspeed velocity (asv), which NumPy & SciPy both use to run and show benchmarks. Here's NumPy's benchmark results.

[jakirkham]

cc @pentschev (in case you have thoughts here)

[pentschev]

I agree ASV is potentially the best candidate for a library like CuPy as it will provide a historic view on a per-commit basis, making it really useful to catch regressions too.