feat(ignore): support disable directives in source code (#3072)

Add support `// Stryker disable` and `// Stryker restore` directives in source code.

Some examples:

```js
// Stryker disable all
const foo = 2 + 40; // mutants here are ignored
// Stryker restore all

// Stryker disable BooleanLiteral
const foo = true || false; // => BooleanLiteral mutant is ignored, but other mutants are tested.
// Stryker restore all

// Stryker disable next-line all
const foo = true || false; // => ignored mutants
const bar = true || false; // => mutants 

// Stryker disable next-line all: I don't care about this piece of code
const foo = bar || baz; // ignored mutants with reason "I don't care about this piece of code"
```

Ignored mutants _do end up in your mutation report_, but are not placed or tested and they don't count towards your mutation score.

![image](https://user-images.githubusercontent.com/1828233/131149477-cf91ce19-9d87-4005-8af5-68e2a4b920c8.png)

docs/configuration.md

@@ -208,7 +208,7 @@ Config file:  `"mutator": { "plugins": ["classProperties"], "excludedMutations":
 * `plugins`: allows you to override the default [babel plugins](https://babeljs.io/docs/en/plugins) to use for JavaScript files.
 By default, Stryker uses [a default list of babel plugins to parse your JS file](https://github.com/stryker-mutator/stryker-js/blob/master/packages/instrumenter/src/parsers/js-parser.ts#L8-L32). It also loads any plugins or presets you might have configured yourself with `.babelrc` or `babel.config.js` files.
 In the rare situation where the plugins Stryker loads conflict with your own local plugins (for example, when using the decorators and decorators-legacy plugins together), you can override the `plugins` here to `[]`.
-* `excludedMutations`: allow you to specify a [list of mutator names](https://github.com/stryker-mutator/stryker-handbook/blob/master/mutator-types.md#supported-mutators) to be excluded (`ignored`) from the test run.
+* `excludedMutations`: allow you to specify a [list of mutator names](https://github.com/stryker-mutator/stryker-handbook/blob/master/mutator-types.md#supported-mutators) to be excluded (`ignored`) from the test run. See [Disable mutants](./disable-mutants) for more options of how to disable specific mutants.
 
 _Note: prior to Stryker version 4, the mutator also needed a `name` (or be defined as `string`). This is removed in version 4. Stryker now supports mutating of JavaScript and friend files out of the box, without the need for a mutator plugin._
 

docs/disable-mutants.md

@@ -0,0 +1,181 @@
+---
+title: Disable mutants
+custom_edit_url: https://github.com/stryker-mutator/stryker-js/edit/master/docs/disable-mutants.md
+---
+
+During mutation testing, you might run into [equivalent mutants](../mutation-testing-elements/equivalent-mutants) or simply mutants that you are not interested in.
+
+## An example
+
+Given this code:
+
+```js
+function max(a, b) {
+  return a < b ? b : a;
+}
+```
+
+And these tests:
+
+```js
+describe('math', () => {
+  it('should return 4 for max(4, 3)', () => {
+    expect(max(4, 3)).eq(4);
+  });
+  it('should return 4 for max(3, 4)', () => {
+    expect(max(3, 4)).eq(4);
+  });
+});
+```
+
+Stryker will generate (amongst others) these mutants:
+
+```diff
+function max(a, b) {
+-  return a < b ? b : a;
++  return true ? b : a; // 👽 1
++  return false ? b : a; // 👽 2
++  return a >= b ? b : a; // 👽 3
+}
+```
+
+Mutant 1 and 2 are killed by the tests. However, mutant 3 isn't killed. In fact, mutant 3 _cannot be killed_ because the mutated code is equivalent to the original. It is therefore called _equivalent mutant_.
+
+![equivalent mutant](./images/disable-mutants-equivalent-mutant.png)
+
+## Disable mutants
+
+StrykerJS supports 2 ways to disable mutants.
+
+1. [Exclude the mutator](#exclude-the-mutator).
+2. [Using a `// Stryker disable` comment](#using-a--stryker-disable-comment).
+
+Disabled mutants will still end up in your report, but will get the `ignored` status. This means that they don't influence your mutation score, but are still visible if you want to look for them. This has no impact on the performance of mutation testing.
+
+## Exclude the mutator
+
+You can simply disable the mutator entirely. This is done by stating the mutator name in the `mutator.excludedMutations` array in your stryker configuration file:
+
+```json
+{
+    "mutator": {
+        "excludedMutations": ["EqualityOperator"]
+  }
+}
+```
+
+The mutator name can be found in the clear-text or html report. 
+
+If you've enabled the clear-text reporter (enabled by default), you can find the mutator name in your console:
+
+```
+#3. [Survived] EqualityOperator
+src/math.js:3:12
+-       return a < b ? b : a;
++       return a <= b ? b : a;
+Tests ran:
+    math should return 4 for max(4, 3)
+    math should return 4 for max(3, 4)
+```
+
+In the html report, you will need to select the mutant you want to ignore, the drawer at the bottom has the mutator name in its title.
+
+However, disable the mutator for all your files is kind of a shotgun approach. Sure it works, but the mutator is now also disabled for other files and places. You probably want to use a comment instead.
+
+## Using a `// Stryker disable` comment.
+
+_Available since Stryker 5.4_
+
+You can disable Stryker for a specific line of code using a comment.
+
+
+```js
+function max(a, b) {
+    // Stryker disable next-line all
+  return a < b ? b : a;
+}
+```
+
+After running Stryker again, the report looks like this:
+
+![disable all](./images/disable-mutants-disable-all.png)
+
+This works, but is not exactly what we want. As you can see, all mutants on line 4 are not disabled. 
+
+We can do better by specifying which mutator we want to ignore:
+
+```js
+function max(a, b) {
+    // Stryker disable next-line EqualityOperator
+  return a < b ? b : a;
+}
+```
+
+We can even provide a custom reason for disabling this mutator behind a colon (`:`). This reason will also end up in your report (drawer below)
+
+```js
+function max(a, b) {
+    // Stryker disable next-line EqualityOperator: The <= mutant results in an equivalent mutant
+  return a < b ? b : a;
+}
+```
+
+After running Stryker again, the report looks like this:
+
+![disable equality operator](./images/disable-mutants-disable-equality-operator.png)
+
+## Disable comment syntax
+
+_Available since Stryker 5.4_
+
+The disabled comment is pretty powerful. Some more examples:
+
+Disable an entire file:
+
+```js
+// Stryker disable all
+function max(a, b) {
+  return a < b ? b : a;
+}
+```
+
+Disable parts of a file:
+
+```js
+// Stryker disable all
+function max(a, b) {
+    return a < b ? b : a;
+}
+// Stryker restore all
+function min(a, b) {
+  return a < b ? b : a;
+}
+```
+
+Disable 2 mutators for an entire file with a custom reason:
+
+```js
+// Stryker disable EqualityOperator,ObjectLiteral: We'll implement tests for these next sprint
+function max(a, b) {
+  return a < b ? b : a;
+}
+```
+
+Disable all mutators for an entire file, but restore the EqualityOperator for 1 line:
+
+```js
+// Stryker disable all
+function max(a, b) {
+  // Stryker restore EqualityOperator
+  return a < b ? b : a;
+}
+```
+
+The syntax looks like this:
+
+```
+// Stryker [disable|restore] [next-line] *mutatorList*[: custom reason]
+```
+
+The comment always starts with `// Stryker`, followed by either `disable` or `restore`. Next, you can specify whether or not this comment targets the `next-line`, or all lines from this point on. The next part is the mutator list. This is either a comma separated list of mutators, or the "all" text signaling this comment targets all mutators. Last is an optional custom reason text, which follows the colon.
+

e2e/helpers.ts

@@ -54,7 +54,7 @@ export async function readMutationTestResult(eventResultDirectory = path.resolve
   return metricsResult;
 }
 
-async function readMutationTestingJsonResult(jsonReportFile = path.resolve('reports', 'mutation', 'mutation.json')) {
+export async function readMutationTestingJsonResult(jsonReportFile = path.resolve('reports', 'mutation', 'mutation.json')) {
   const mutationTestReportContent = await fsPromises.readFile(jsonReportFile, 'utf8');
   const report = JSON.parse(mutationTestReportContent) as mutationTestReportSchema.MutationTestResult;
   const metricsResult = calculateMetrics(report.files);
@@ -99,7 +99,7 @@ export async function expectMetrics(expectedMetrics: Partial<Metrics>) {
   expectActualMetrics(expectedMetrics, actualMetricsResult);
 }
 
-function expectActualMetrics(expectedMetrics: Partial<Metrics>, actualMetricsResult: MetricsResult) {
+export function expectActualMetrics(expectedMetrics: Partial<Metrics>, actualMetricsResult: MetricsResult) {
   const actualMetrics: Partial<Metrics> = {};
   Object.entries(expectedMetrics).forEach(([key]) => {
     if (key === 'mutationScore' || key === 'mutationScoreBasedOnCoveredCode') {

e2e/test/ignore-project/src/Add.js

@@ -1,26 +1,41 @@
-module.exports.add = function(num1, num2) {
+module.exports.add = function (num1, num2) {
   return num1 + num2;
 };
 
-module.exports.addOne = function(number) {
+module.exports.addOne = function (number) {
   number++;
   return number;
 };
 
-module.exports.negate = function(number) {
+module.exports.negate = function (number) {
   return -number;
 };
 
-module.exports.notCovered = function(number) {
+module.exports.notCovered = function (number) {
   return number > 10;
 };
 
-module.exports.isNegativeNumber = function(number) {
+module.exports.userNextLineIgnored = function (number) {
+  // Stryker disable next-line all: Ignoring this on purpose
+  return number > 10;
+};
+
+// Stryker disable all
+module.exports.blockUserIgnored = function (number) {
+  return number > 10;
+};
+// Stryker restore all
+
+module.exports.userNextLineSpecificMutator = function (number) {
+  // Stryker disable next-line BooleanLiteral, ConditionalExpression: Ignore boolean and conditions
+  return true && number > 10;
+};
+
+
+module.exports.isNegativeNumber = function (number) {
   var isNegative = false;
-  if(number < 0){
+  if (number < 0) {
     isNegative = true;
   }
   return isNegative;
 };
-
-

e2e/test/ignore-project/stryker.conf.json

@@ -9,6 +9,7 @@
   "reporters": [
     "clear-text",
     "html",
+    "json",
     "event-recorder"
   ],
   "plugins": [

e2e/test/ignore-project/verify/verify.ts

@@ -1,16 +1,52 @@
-import { expectMetrics } from '../../../helpers';
+import { expect } from 'chai';
+import { MutantStatus } from 'mutation-testing-report-schema';
+import { expectMetricsJson, readMutationTestingJsonResult  } from '../../../helpers';
 
 describe('After running stryker on jest-react project', () => {
   it('should report expected scores', async () => {
-    await expectMetrics({
+    await expectMetricsJson({
       killed: 8,
-      ignored: 13,
-      mutationScore: 66.67,
+      ignored: 29,
+      mutationScore: 53.33,
+    });
+  });
+
+  /*
+  -----------|---------|----------|-----------|------------|----------|---------|
+  File       | % score | # killed | # timeout | # survived | # no cov | # error |
+  -----------|---------|----------|-----------|------------|----------|---------|
+  All files  |   53.33 |        8 |         0 |          0 |        7 |       0 |*/
+
+
+  it('should report mutants that are disabled by a comment with correct ignore reason', async () => {
+    const actualMetricsResult = await readMutationTestingJsonResult();
+    const addResult = actualMetricsResult.childResults.find(file => file.name.endsWith('Add.js')).file!;
+    const mutantsAtLine31 =  addResult.mutants.filter(({ location }) => location.start.line === 31)
+    const booleanLiteralMutants = mutantsAtLine31.filter(({mutatorName}) => mutatorName === 'BooleanLiteral');
+    const conditionalExpressionMutants = mutantsAtLine31.filter(({mutatorName}) => mutatorName === 'ConditionalExpression');
+    const equalityOperatorMutants = mutantsAtLine31.filter(({mutatorName}) => mutatorName === 'EqualityOperator');
+    booleanLiteralMutants.forEach((booleanMutant) => {
+      expect(booleanMutant.status).eq(MutantStatus.Ignored);
+      expect(booleanMutant.statusReason).eq('Ignore boolean and conditions');
+    });
+    conditionalExpressionMutants.forEach((conditionalMutant) => {
+      expect(conditionalMutant.status).eq(MutantStatus.Ignored);
+      expect(conditionalMutant.statusReason).eq('Ignore boolean and conditions');
+    });
+
+    equalityOperatorMutants.forEach((equalityMutant) => {
+      expect(equalityMutant.status).eq(MutantStatus.NoCoverage);
+    });
+  });
+
+  it('should report mutants that result from excluded mutators with the correct ignore reason', async () => {
+    const actualMetricsResult = await readMutationTestingJsonResult();
+    const circleResult = actualMetricsResult.childResults.find(file => file.name.endsWith('Circle.js')).file!;
+    const mutantsAtLine3 =  circleResult.mutants.filter(({ location }) => location.start.line === 3)
+
+    mutantsAtLine3.forEach((mutant) => {
+      expect(mutant.status).eq(MutantStatus.Ignored);
+      expect(mutant.statusReason).eq('Ignored because of excluded mutation "ArithmeticOperator"');
     });
-    /*
------------|---------|----------|-----------|------------|----------|---------|
-File       | % score | # killed | # timeout | # survived | # no cov | # error |
------------|---------|----------|-----------|------------|----------|---------|
-All files  |   66.67 |        8 |         0 |          0 |        4 |       0 |*/
   });
 });

packages/instrumenter/.vscode/launch.json

@@ -4,7 +4,7 @@
     {
       "type": "node",
       "request": "launch",
-      "name": "Unit / Integration tests",
+      "name": "🎻 Unit / Integration tests",
       "program": "${workspaceRoot}/../../node_modules/mocha/bin/_mocha",
       "internalConsoleOptions": "openOnSessionStart",
       "outFiles": [

packages/instrumenter/src/transformers/babel-transformer.ts

@@ -11,6 +11,8 @@ import { ScriptFormat } from '../syntax';
 import { allMutantPlacers, MutantPlacer, throwPlacementError } from '../mutant-placers';
 import { Mutable, Mutant } from '../mutant';
 
+import { DirectiveBookkeeper } from './directive-bookkeeper';
+
 import { AstTransformer } from '.';
 
 interface MutantsPlacement<TNode extends types.Node> {
@@ -37,6 +39,9 @@ export const transformBabel: AstTransformer<ScriptFormat> = (
   // Create a placementMap for the mutation switching bookkeeping
   const placementMap: PlacementMap = new Map();
 
+  // Create the bookkeeper responsible for the // Stryker ... directives
+  const directiveBookkeeper = new DirectiveBookkeeper();
+
   // Now start the actual traversing of the AST
   //
   // On the way down:
@@ -52,6 +57,8 @@ export const transformBabel: AstTransformer<ScriptFormat> = (
   // eslint-disable-next-line @typescript-eslint/no-unsafe-argument
   traverse(file.ast, {
     enter(path) {
+      directiveBookkeeper.processStrykerDirectives(path.node);
+
       if (shouldSkip(path)) {
         path.skip();
       } else {
@@ -149,7 +156,11 @@ export const transformBabel: AstTransformer<ScriptFormat> = (
   function* mutate(node: NodePath): Iterable<Mutable> {
     for (const mutator of mutators) {
       for (const replacement of mutator.mutate(node)) {
-        yield { replacement, mutatorName: mutator.name, ignoreReason: formatIgnoreReason(mutator.name) };
+        yield {
+          replacement,
+          mutatorName: mutator.name,
+          ignoreReason: directiveBookkeeper.findIgnoreReason(node.node.loc!.start.line, mutator.name) ?? formatIgnoreReason(mutator.name),
+        };
       }
     }