You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As mentioned in #59 and #57, it's currently difficult to identify changes that are leading to performance regression. I spoke to a labmate who suggested two possible packages for implementing some form of performance benchmarking:
However there is a major caveat which is that running performance benchmarks as a part of cloud based CI can lead to highly variable results and potential false positives just as a function of what the compute resources were at the time of benchmarking.
My suggestion would be to implement a pytest-benchmark routine that we run on CI, but also run locally when we are concerned about a particular change leading to performance differences.
I'll probably take a stab at implementing something next week when I work with Caroline. Let me know if you have any thoughts on this approach, esp @DragaDoncila@bentaculum
The text was updated successfully, but these errors were encountered:
As mentioned in #59 and #57, it's currently difficult to identify changes that are leading to performance regression. I spoke to a labmate who suggested two possible packages for implementing some form of performance benchmarking:
However there is a major caveat which is that running performance benchmarks as a part of cloud based CI can lead to highly variable results and potential false positives just as a function of what the compute resources were at the time of benchmarking.
My suggestion would be to implement a
pytest-benchmark
routine that we run on CI, but also run locally when we are concerned about a particular change leading to performance differences.I'll probably take a stab at implementing something next week when I work with Caroline. Let me know if you have any thoughts on this approach, esp @DragaDoncila @bentaculum
The text was updated successfully, but these errors were encountered: