Skip to content

JS/TS benchmarking solution to track performance regressions in CI

License

Notifications You must be signed in to change notification settings

ChainSafe/benchmark

 
 

Repository files navigation

Benchmark

Ensures that new code does not introduce performance regressions with CI. Tracks:

  • Do PR against the base branch include a performance regression?
  • Do new commits in the main branch include a performance regression?

This tooling provides both a easy to use runner for benchmarking and easy integrations to persist past benchmark data.

Quick start

Create a test mocha test file but use itBench instead of it

import {itBench, setBenchOpts} from "../../src";

describe("Sum array benchmark", () => {
  itBench("sum array with reduce", () => {
    arr.reduce((total, curr) => total + curr, 0);
  });
});

Then run the CLI, compatible with all mocha options.

benchmark 'test/perf/**/*.perf.ts' --local

Inspect benchmark results in the terminal

  Sum array benchmark
    ✔ sum array with reduce                                               826.0701 ops/s    1.210551 ms/op   x0.993        578 runs   1.21 s

How does it work?

This tool is a CLI wrapper around mocha, example usage:

benchmark 'test/perf/**/*.perf.ts' --s3

The above command will:

  • Read benchmark history from the specified provider (AWS S3)
  • Figure out the prev benchmark based on your option (defaults to latest commit in main branch)
  • Run benchmark comparing with previous
    • Runs mocha programatically against the file globs
    • Collect benchmark data in-memory while streaming results with a familiar mocha reporter
    • Note: also runs any test that would regularly be run with mocha
  • Add result to benchmark history and persist them to the specified provider (AWS S3)
  • If in CI, post a PR or commit comment with an expandable summary of the benchmark results comparision
  • If a performance regression was detected, exit 1

Track performance in CI

Below is a suggested Github action to run this tool with s3 history provider:

name: Benchmark
# Ensure a single benchmark is run at a time
concurrency: cd-benchmark-${{ github.ref }}

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  s3:
    if: always()
    runs-on: ubuntu-latest
    # Ensure both don't run at the same time
    needs:
      - local
      # - ga-cache

    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2-beta
      - run: yarn install --frozen-lockfile

      # Run benchmark with custom tooling and stores the output to a file
      - name: Run performance tests
        run: yarn benchmark --s3
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # S3 credentials
          S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
          S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
          S3_REGION: ${{ secrets.S3_REGION }}
          S3_BUCKET: ${{ secrets.S3_BUCKET }}
          S3_ENDPOINT: ${{ secrets.S3_ENDPOINT }}
          # Key prefix to separate benchmark data from multiple repositories
          S3_KEY_PREFIX: ${{ github.repository }}/${{ runner.os }}

Track performance locally

When working on optimizing a function you may want to know if your code is actually faster than the previous implementation.

To do that you can keep a benchmark history file locally and run the benchmark first against previous code

git checkout master
benchmark test/perf/func.perf.ts --local
  • Runs benchmark without comparing with previous
  • Writes single benchmark data to ./benchmark_data

Then measure performance with the new code

git checkout fix1
benchmark test/perf/func.perf.ts --local
  • Reads single benchmark data from ./benchmark_data
  • Run benchmark comparing with prev
  • Does not write benchmark data

Config

--defaultBranch

Provide the default branch of this repository to prevent fetching from Github

  • type: string
  • default:

--persistBranches

Choose what branches to persist benchmark data

  • type: array
  • default: default-branch

--benchmarksPerBranch

Limit number of benchmarks persisted per branch

  • type: number
  • default: Infinity

--threshold

Ratio of new average time per run vs previos time per run to consider a failure. Set to 'Infinity' to disable it.

  • type: number
  • default: 2

--compareBranch

Compare new benchmark data against the latest available benchmark in this branch

  • type: string
  • default: default-branch

--compareCommit

Compare new benchmark data against the benchmark data associated with a specific commit

  • type: string
  • default:

--prune

When persisting history, delete benchmark data associated with commits that are no longer in the current git history

  • type: boolean
  • default:

--persist

Force persisting benchmark data in history

  • type: boolean
  • default:

--noThrow

Exit cleanly even if a preformance regression was found

  • type: boolean
  • default:

--historyLocal, --local

Persist benchmark history locally. May specify just a boolean to use a default path, or provide a path

  • type: string
  • default: ./benchmark_data

--historyGaCache, --ga-cache

Persist benchmark history in Github Actions cache. Requires Github authentication. May specify just a boolean to use a default cache key or provide a custom key

  • type: string
  • default: benchmark_data

--historyS3, --s3

Persist benchmark history in an Amazon S3 bucket. Requires Github authentication

  • type: string
  • default:

Roadmap

Compare performance

bench compare --from <branch-name | commit-hash> --to <branch-name | commit-hash>
  • Retrieves benchmark history
  • Don't run benchmark
  • Print comparision of from benchmark with to benchmark

Add more providers

  • Github actions cache: Doesn't work due to Github's limitations
  • Commited in the repository (gh-pages, or data branch)

About

JS/TS benchmarking solution to track performance regressions in CI

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • TypeScript 94.0%
  • JavaScript 6.0%