Add support for inline tests for Vitest using includeSource attribute

[jacob-8]

Issue description or question

In my monorepo, I have a packages/parts/src/lib/foo.ts file which contains a function and an inline test:

export function foo(str: string) { return str+ 'world'; }

if (import.meta.vitest) {
  test('foo works', () => {
     expect(foo('Hello ')).toMatchInlineSnapshot(`Hello world`)
  });
}

My vite.config.js points Vitest to run this test and it does, successfully:

const config = {
    define: {
      'import.meta.vitest': false,
    },
    test: {
        globals: true,
        includeSource: ['src/**/*.ts'],
    },
};
export default config;

Wallaby doesn't find this test. However, if I change my filename to foo.test.ts if runs just fine. Am I missing something in the documentation? Wallaby has no trouble with running an inline test when the extension is test.ts, but rather doesn't act on the includeSource property of the vitest config to run tests found in files pointed to by includeSource.

Wallaby diagnostics report

{
  editorVersion: '1.73.1',
  pluginVersion: '1.0.348',
  editorType: 'VSCode',
  osVersion: 'win32 10.0.22621',
  nodeVersion: 'v18.12.1',
  coreVersion: '1.0.1346',
  checksum: 'MmZhNGFjNTI4OTQxZjk2ZDYyNGEwNTM5MTk1NzYwNWEsMTcwMDUyNDgwMDAwMCww',
  config: {
    diagnostics: {
      vitest: {
        file: {
          config: "import { sveltekit } from '@sveltejs/kit/vite';\r\n" +
            "// import { svelte } from '@sveltejs/vite-plugin-svelte';\r\n" +
            '\r\n' +
            "/** @type {import('vite').UserConfig} */\r\n" +
            'const config = {\r\n' +
            '\tplugins: [sveltekit()],\r\n' +
            '\t// plugins: [\r\n' +
            '  //   svelte({ hot: !process.env.VITEST }),\r\n' +
            '  // ],\r\n' +
            '\tdefine: {\r\n' +
            "\t\t'import.meta.vitest': false,\r\n" +
            '\t},\r\n' +
            '\ttest: {\r\n' +
            '    globals: true,\r\n' +
            "    includeSource: ['src/**/*.ts'],\r\n" +
            '\t\t\r\n' +
            '  },\r\n' +
            '};\r\n' +
            '\r\n' +
            'export default config;\r\n'
        },
        config: {
          allowOnly: true,
          watch: true,
          globals: true,
          environment: 'node',
          threads: true,
          clearMocks: false,
          restoreMocks: false,
          mockReset: false,
          include: [ '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}' ],
          exclude: [ '**/node_modules/**', '**/dist/**', '**/cypress/**', '**/.{idea,git,cache,output,temp}/**' ],
          testTimeout: 5000,
          hookTimeout: 10000,
          teardownTimeout: 1000,
          isolate: true,
          watchExclude: [ '**/node_modules/**', '**/dist/**' ],
          forceRerunTriggers: [ '**/package.json/**', '**/vitest.config.*/**', '**/vite.config.*/**' ],
          update: false,
          reporters: [ 'default' ],
          silent: false,
          ui: false,
          uiBase: '/__vitest__/',
          open: true,
          css: { include: [], modules: { classNameStrategy: 'stable' } },
          coverage: {
            provider: 'c8',
            enabled: false,
            clean: true,
            cleanOnRerun: false,
            reportsDirectory: './coverage',
            excludeNodeModules: true,
            exclude: [
              'coverage/**',
              'dist/**',
              'packages/*/test{,s}/**',
              '**/*.d.ts',
              'cypress/**',
              'test{,s}/**',
              'test{,-*}.{js,cjs,mjs,ts,tsx,jsx}',
              '**/*{.,-}test.{js,cjs,mjs,ts,tsx,jsx}',
              '**/*{.,-}spec.{js,cjs,mjs,ts,tsx,jsx}',
              '**/__tests__/**',
              '**/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress}.config.{js,cjs,mjs,ts}',
              '**/.{eslint,mocha,prettier}rc.{js,cjs,yml}'
            ],
            reporter: [ 'text', 'html', 'clover', 'json' ],
            allowExternal: false,
            extension: [
              '.js',  '.cjs',
              '.mjs', '.ts',
              '.tsx', '.jsx',
              '.vue', '.svelte'
            ]
          },
          fakeTimers: { loopLimit: 10000, shouldClearNativeTimers: true, toFake: [ 'setTimeout', 'clearTimeout', 'setInterval', 'clearInterval', 'setImmediate', 'clearImmediate', 'Date' ] },
          maxConcurrency: 5,
          dangerouslyIgnoreUnhandledErrors: false,
          includeSource: [ 'src/**/*.ts' ],
          defines: {},
          root: 'C:/dev/jia-you/packages/parts',
          mode: [],
          deps: {
            inline: [
              {},
              {},
              {},
              {},
              {},
              {},
              '@nuxt/test-utils',
              'svelte',
              {},
              'svelte-store-router',
              '@sveltejs/package',
              '@testing-library/svelte',
              'kitbook',
              'prettier-plugin-svelte',
              'svelte-pieces',
              'svelte2tsx'
            ],
            registerNodeLoader: false
          },
          snapshotOptions: { snapshotFormat: {}, updateSnapshot: 'new' },
          setupFiles: [],
          cache: { dir: 'C:/dev/jia-you/packages/parts/node_modules/.vitest' },
          sequence: {},
          package: {
            version: '0.24.3',
            urls: { hooks: 'file:///c:/Users/jacob/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.348/wallaby154d60/runners/node/hooks.mjs' },
            paths: {
              root: 'C:\\dev\\jia-you\\node_modules\\.pnpm\\[email protected][email protected]\\node_modules\\vitest',
              dist: 'C:\\dev\\jia-you\\node_modules\\.pnpm\\[email protected][email protected]\\node_modules\\vitest\\dist'
            }
          }
        }
      }
    },
    testFramework: { version: '[email protected]', configurator: '[email protected]', reporter: '[email protected]', starter: '[email protected]', autoDetected: true },
    preserveComments: false,
    extractComments: true,
    files: [
      { pattern: '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}', ignore: true, trigger: true, load: true, file: true },
      { pattern: '**/node_modules/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/dist/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/cypress/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/.{idea,git,cache,output,temp}/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/*.*', ignore: false, trigger: true, load: true, order: 1 }
    ],
    tests: [
      { pattern: '**/node_modules/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/dist/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/cypress/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/.{idea,git,cache,output,temp}/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}', ignore: false, trigger: true, load: true, test: true, order: 2 }
    ],
    workers: { initial: 1, regular: 1, recycle: false },
    filesWithNoCoverageCalculated: [],
    runAllTestsInAffectedTestFile: false,
    updateNoMoreThanOneSnapshotPerTestFileRun: false,
    compilers: {},
    logLimits: { inline: { depth: 5, elements: 5000 }, values: { default: { stringLength: 200 }, autoExpand: { elements: 5000, stringLength: 8192, depth: 10 } } },
    preprocessors: {},
    maxConsoleMessagesPerTest: 100,
    autoConsoleLog: true,
    delays: { run: 0, edit: 100, update: 0 },
    teardown: undefined,
    hints: {
      ignoreCoverage: '__REGEXP /ignore coverage|istanbul ignore/',
      ignoreCoverageForFile: '__REGEXP /ignore file coverage/',
      commentAutoLog: '?',
      testFileSelection: { include: '__REGEXP /file\\.only/', exclude: '__REGEXP /file\\.skip/' }
    },
    automaticTestFileSelection: true,
    runSelectedTestsOnly: false,
    mapConsoleMessagesStackTrace: false,
    extensions: {},
    env: {
      type: 'node',
      params: { runner: '--experimental-loader=file:///c:/Users/jacob/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.348/wallaby154d60/runners/node/hooks.mjs' },
      runner: 'C:\\Program Files\\nodejs\\node.exe',
      viewportSize: { width: 800, height: 600 },
      options: { width: 800, height: 600 },
      bundle: true
    },
    reportUnhandledPromises: true,
    slowTestThreshold: 75,
    lowCoverageThreshold: 80,
    runAllTestsWhenNoAffectedTests: true,
    symlinkNodeModules: true,
    configCode: 'auto.detect#1187933729'
  },
  packageJSON: {
    dependencies: { comlink: '^4.3.1', 'd3-dsv': '^3.0.1', 'svelte-store-router': '^2.0.10' },
    devDependencies: {
      '@iconify/json': '^2.1.126',
      '@sveltejs/adapter-auto': '1.0.0-next.84',
      '@sveltejs/kit': '1.0.0-next.525',
      '@sveltejs/package': '1.0.0-next.5',
      '@testing-library/svelte': '^3.2.2',
      '@unocss/preset-icons': '^0.45.29',
      'cross-env': '^7.0.3',
      'happy-dom': '^6.0.3',
      'jieba-wasm': '^0.0.2',
      kitbook: '^0.0.24',
      mdsvex: '^0.10.6',
      'node-fetch': '^3.3.0',
      svelte: '^3.52.0',
      'svelte-check': '^2.9.2',
      'svelte-pieces': '^1.0.50',
      'svelte-preprocess': '^4.10.7',
      svelte2tsx: '^0.5.20',
      'temp-s-p-u': '^0.0.7',
      tslib: '^2.4.0',
      tsx: '^3.11.0',
      typescript: '~4.8.4',
      unocss: '^0.45.30',
      vite: '^3.1.8',
      vitest: '^0.24.3',
      zh_cn_zh_tw: '^1.0.7'
    }
  },
  fs: { numberOfFiles: 104 },
  debug: []
}

[smcenlly]

Wallaby doesn't currently support inline tests in source files using if (import.meta.vitest) {.

Adding as a feature request.

[jacob-8]

Adding as a feature request.

Thanks!

Wallaby doesn't currently support inline tests in source files using if (import.meta.vitest) {.

Things work great however if I rename foo.ts to foo.test.ts, but that's kind of annoying (and breaking convention) to be naming files with helper functions using the .test addition. I've tried figuring things out from Vitest and Wallaby's docs myself but am not having luck. Do you have any tips for me on how to make Wallaby scan all my typescript files so I don't have to do crazy renaming hacks? I tried adding the following to my package.json with no luck:

"wallaby": {
  "tests": [
    "src/**/*.ts"
  ]
}

I think in combo with Smart Start this would work very well.

[smcenlly]

Things work great however if I rename foo.ts to foo.test.ts

At this point in time, we're not entirely sure how vitest works under the covers to run tests where you us if (import.meta.vitest) { so while renaming your files may work right now, there's no guarantee that this will continue to work in the future.

One approach you may like to try is to try overriding the files and tests configuration for your project (see our docs). You will need to make sure that any file with if (import.meta.vitest) { is explicitly excluded from files patterns and is explicitly included in tests patterns. You could start by hard-coding foo.ts and see if it works for you.

[martaver]

Also expressing my interest in this feature!

Many of the tests where inline tests are most useful are for small chunks of code where creating a separate test file is burdensome. These are also often the tests that wallaby is most handy for too.

[stagas]

Folks using vitest that want to use inline tests alongside regular tests, this wallaby.js config will do the trick:

import path from 'node:path'
import fm from 'file-matcher'

export default async () => {
  const root = 'src/'
  const ext = '{js,jsx,ts,tsx}'
  const filePattern = '**/*.' + ext
  const testFilePattern = '**/*.{spec,test}.' + ext
  const inlineTestPattern = filePattern

  const fileOptions = {
    path: root,
    recursiveSearch: true,
    fileFilter: {
      fileNamePattern: testFilePattern,
    }
  }

  const inlineOptions = {
    path: root,
    recursiveSearch: true,
    fileFilter: {
      fileNamePattern: inlineTestPattern,
      content: /import\.meta\.vitest/,
    }
  }

  const cwd = process.cwd()
  const relative = filename => path.relative(cwd, filename)

  const fileMatcher = new fm.FileMatcher()
  const fileTests = (await fileMatcher.find(fileOptions)).map(relative)
  const inlineTests = (await fileMatcher.find(inlineOptions)).map(relative)

  return {
    autoDetect: true,
    files: [
      root + filePattern,
      ...fileTests.map(file => `!${file}`),
      ...inlineTests.map(file => `!${file}`)
    ],
    tests: [
      ...fileTests,
      ...inlineTests,
    ],
  }
}

[smcenlly]

@stagas - thanks for sharing your configuration.

While this may partially work for some projects, it can also lead to errors. For example, if you have a file that imports from your includeSource mixed source/test file that is later tested by a separate test file, you will receive a Wallaby error similar to:

Test file 'src/my.spec.js' is importing or is causing import of another test file 'inlineTests/myFile.js'. 

This may cause tests in the 'inlineTests/myFile.js' file to be executed more than once (when the test file runs and every time it gets imported).

If the 'inlineTests/myFile.js' test file contains some test helper code, try refactoring the reused code into a file of its own and use it by importing the new file into 'src/my.spec.js' and 'inlineTests/myFile.js' test files. 

You will also find that Wallaby App no longer includes your inline test files in your project code coverage calculations.

If this works for you, please go ahead and use your configuration, but we wanted to highlight that there are some larger fundamental Wallaby changes that are required to properly support this scenario and this configuration does not add proper support for vitest inline testing.