The import language affect the benchmark result

[D-Sketon]

Describe the bug

I try the following code and run vitest bench

import { bench, describe } from "vitest";
import { after as afterToolkit } from "es-toolkit/function";

const after = afterToolkit;
const add = (a: number, b: number) => a + b;
const n = 10;
describe("after", () => {
  bench("es-toolkit/after - reassign", () => {
    after(n, add);
  });

  bench("es-toolkit/after", () => {
    afterToolkit(n, add);
  });
});

Here's what I got:

You can see that a direct import statement using import is significantly slower than reassigning values
However, when I try tinybench like this:

import { Bench } from "tinybench";
import { after as afterToolkit } from "es-toolkit";

const bench = new Bench({ time: 100 });
const add = (a: number, b: number) => a + b;
const n = 10;
const after = afterToolkit;
bench
  .add("es-toolkit/after - reassign", () => {
    after(n, add);
  })
  .add("es-toolkit/after", () => {
    afterToolkit(n, add);
  });

await bench.warmup();
await bench.run();

console.table(bench.table());

There is no difference between them:

Reproduction

Use the code above

System Info

System:
    OS: Windows 11 10.0.22631
    CPU: (16) x64 13th Gen Intel(R) Core(TM) i5-13400F
    Memory: 16.18 GB / 31.82 GB
  Binaries:
    Node: 20.13.1 - D:\nodejs\node.EXE
    Yarn: 1.22.22 - D:\nodejs\yarn.CMD
    npm: 10.5.2 - D:\nodejs\npm.CMD
    bun: 1.1.27 - D:\nodejs\bun.CMD
  Browsers:
    Edge: Chromium (127.0.2651.74)
    Internet Explorer: 11.0.22621.3527
  npmPackages:
    vitest: ^2.0.4 => 2.0.4

Used Package Manager

yarn

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[dayongkr]

The same issue arises when calling with the subpath as shown below, resulting in a performance difference.

Before

import { bench, describe } from 'vitest';
import { after as afterToolkit } from 'es-toolkit'; // This is difference.
import { after as afterLodash } from 'lodash';

describe('after', () => {
 bench('es-toolkit/after', () => {
   const add = (a: number, b: number) => a + b;
   const n = 10;
   const afterFn = afterToolkit(n, add);
   for (let i = 0; i < n + 1; i++) {
     afterFn(1, 2);
   }
 });

 bench('lodash/after', () => {
   const add = (a: number, b: number) => a + b;
   const n = 10;
   const afterFn = afterLodash(n, add);
   for (let i = 0; i < n + 1; i++) {
     afterFn(1, 2);
   }
 });
});

Bench Result

Using subpath

import { bench, describe } from 'vitest';
import { after as afterToolkit } from 'es-toolkit/function'; // This is difference.
import { after as afterLodash } from 'lodash';

describe('after', () => {
  bench('es-toolkit/after', () => {
    const add = (a: number, b: number) => a + b;
    const n = 10;
    const afterFn = afterToolkit(n, add);
    for (let i = 0; i < n + 1; i++) {
      afterFn(1, 2);
    }
  });

  bench('lodash/after', () => {
    const add = (a: number, b: number) => a + b;
    const n = 10;
    const afterFn = afterLodash(n, add);
    for (let i = 0; i < n + 1; i++) {
      afterFn(1, 2);
    }
  });
});

Bench Result

Additional

You can see our barrel files in here(src/index.ts, src/function/index.ts).

Thanks

[hi-ogawa]

This is likely due to how Vite/Vitest transform and import modules. You can roughly see that by VITE_NODE_DEBUG_DUMP=1 and import/export are transformed as Object.defineProperty, which might be causing perf hit when benchmarking simple enough functions.

For example, in this reproduction https://stackblitz.com/edit/vitest-dev-vitest-br64n3?file=src%2Fbasic.bench.ts,

// basic.ts
export const squared = (n: number) => n * n

// basic.bench.ts
import { squared } from './basic.js';

squared(2)

will be transformed to something like

const viteSsrExports = Object.create(null);

const viteSsrExportsSquared = (n: number) => n * n;

Object.defineProperty(viteSsrExports, 'squared', {
  enumerable: true,
  configurable: true,
  get() {
    return viteSsrExportsSquared;
  },
});

viteSsrExports.squared(2)

To avoid this, you need to benchmark without transforming, so you probably should try using pre-built version of es-toolkit.

[crimx]

Is there a workaround for this bug (without modifying code in src)? It appears to occur when importing anything from another file, whether from node_modules or local project files.

// a.ts

export const fn = () => {};

export const a = () => {
  fn();
}

// b.ts

import { fn } from "./a";

export const b = () => {
  fn();
}

// x.bench.ts

import { bench, describe } from "vitest"
import { a } from "./a"
import { b } from "./b"

describe("x", () => {
  bench("a", () => {
    a()
  })
  bench("a2", () => {
    a()
  })
  bench("b", () => {
    b()
  })
  bench("b2", () => {
    b()
  })
})

 RUN  v2.1.3

 ✓ src/x.bench.ts (4) 15487ms
   ✓ x (4) 15486ms
     name             hz     min     max    mean     p75     p99    p995    p999     rme   samples
   · a     24,697,176.57  0.0000  0.1943  0.0000  0.0000  0.0000  0.0000  0.0001  ±0.16%  12348589
   · a2    24,768,664.32  0.0000  0.1781  0.0000  0.0000  0.0000  0.0000  0.0001  ±0.18%  12384333   fastest
   · b     20,451,525.10  0.0000  0.1673  0.0000  0.0000  0.0001  0.0001  0.0001  ±0.18%  10225764
   · b2    20,424,575.59  0.0000  0.1577  0.0000  0.0000  0.0001  0.0001  0.0001  ±0.19%  10212288   slowest


 BENCH  Summary

  a2 - src/x.bench.ts > x
    1.00x faster than a
    1.21x faster than b
    1.21x faster than b2

[D-Sketon]

@crimx Here's how we handle it in es-toolkit:

1. Use deps.optimizer.ssr to bundle the source files, thus eliminating the import overhead inside the modules

deps: {
  optimizer: {
    ssr: {
      enabled: true,
      include: ['es-toolkit', 'es-toolkit/compat'],
    },
  },
}

2. Re-assign values to imported variables, thus eliminating import overhead in the bench file

import { bench, describe } from 'vitest';
import { chunk as chunkToolkit_ } from 'es-toolkit';
import { chunk as chunkCompatToolkit_ } from 'es-toolkit/compat';
import { chunk as chunkLodash_ } from 'lodash';

const chunkToolkit = chunkToolkit_;
const chunkCompatToolkit = chunkCompatToolkit_;
const chunkLodash = chunkLodash_;

describe('chunk', () => {
  bench('lodash/chunk', () => {
    chunkLodash([1, 2, 3, 4, 5, 6], 3);
  });

  bench('es-toolkit/chunk', () => {
    chunkToolkit([1, 2, 3, 4, 5, 6], 3);
  });

  bench('es-toolkit/compat/chunk', () => {
    chunkCompatToolkit([1, 2, 3, 4, 5, 6], 3);
  });
});

After all this effort, our results are not too far off from tinybench (although there are still some gaps)
before

   name                                hz     min     max    mean     p75     p99    p995    p999     rme  samples
   · lodash/chunk              7,786,900.20  0.0000  0.4533  0.0001  0.0001  0.0003  0.0003  0.0005  ±0.73%  3893461   slowest
   · es-toolkit/chunk         10,678,422.00  0.0000  0.9141  0.0001  0.0001  0.0002  0.0002  0.0003  ±0.46%  5339211   fastest
   · es-toolkit/compat/chunk   7,789,212.00  0.0000  1.2057  0.0001  0.0001  0.0003  0.0003  0.0005  ±0.52%  3894606

after

  name                                hz     min     max    mean     p75     p99    p995    p999     rme  samples
   · lodash/chunk             17,114,118.00  0.0000  1.3395  0.0001  0.0001  0.0001  0.0001  0.0003  ±1.21%  8557059   fastest
   · es-toolkit/chunk         15,336,056.93  0.0000  0.7054  0.0001  0.0001  0.0001  0.0002  0.0002  ±0.47%  7668030   slowest
   · es-toolkit/compat/chunk  15,599,688.00  0.0000  0.4493  0.0001  0.0001  0.0001  0.0002  0.0002  ±0.30%  7799844

tinybench

┌─────────┬─────────────────────┬──────────────┬────────────────────┬──────────┬─────────┐
│ (index) │ Task Name           │ ops/sec      │ Average Time (ns)  │ Margin   │ Samples │
├─────────┼─────────────────────┼──────────────┼────────────────────┼──────────┼─────────┤
│ 0       │ 'lodash'            │ '17,132,909' │ 58.36720090166528  │ '±2.20%' │ 1713291 │
│ 1       │ 'es-toolkit'        │ '16,746,723' │ 59.71317402670255  │ '±0.67%' │ 1674674 │
│ 2       │ 'es-toolkit/compat' │ '16,974,469' │ 58.912001376214626 │ '±0.54%' │ 1697447 │
└─────────┴─────────────────────┴──────────────┴────────────────────┴──────────┴─────────┘

We hope our efforts have inspired you.