Produce tests per line data

[cgewecke]

It has been proposed in JoranHonig/vertigo 20 that being able to identify exactly which tests exercise which line of Solidity would be useful data for mutation testing and fault-localization techniques.

The issue linked to above includes lots of discussion about how to achieve this. Would probably be a command separate from coverage & just consume some services available through the beta's new API.

Maybe even in it's own repo at sc-forks org? With SC as a dependency?

See also: #209

[yxliang01]

Cool! I think this issue moves solidity-coverage from a pure utility to become a more framework or library-like project :)

[JoranHonig]

It's been a while 😅. If you could point me in the right direction then I might be able to contribute this feature.

[cgewecke]

Hi @JoranHonig!

That sounds great. Am about to make a set of +/- large changes here as part of #541.

Could I get back to you next week on this? (Will have a better sense of how this might be possible then.)

---

Am copying over some of the design details from the Vertigo discussion.

The generated object might look something like this?

{
 File.sol: {
   8: [
     {
       test: 'it errors when called by alice',
       file: '/Users/area/code/tests/alice.js'
     }, ...
   ], ...
 },
 AnotherFile.sol: {...},
}

The test names would come from a custom mocha reporter attached before the test command is run.

Mocha details are:

reporter's 'test' hook (for it blocks)

runner.on("test", info => {
}

info (much abbreviated)

info: {
  suite: {
    title: 'Contract: EtherRouter Proxy',
    file: '/Users/cgewecke/code/eth-gas-reporter/mock/test/etherrouter.js' 
  },
  test: {
    "title": "Resolves methods routed through an EtherRouter proxy",
    "file": "/Users/cgewecke/code/eth-gas-reporter/mock/test/etherrouter.js",
  } 
},

[JoranHonig]

Thanks! Take your time with #541 👍.

The dataformat looks good to me! It'd be perfect for mutation testing.

[cgewecke]

Update: have a working version of this in #593 which will published as part of the new v.0.8 beta in the next couple days.

Am calling this data a "test matrix". It can be generated using the --matrix flag and get's written to testMatrix.json in the project's root repository. (The file name is configurable).

Output for Truffle's metacoin repository looks like:

{
 "contracts/MetaCoin.sol": {
  "17": [
   {
    "title": "should put 10000 MetaCoin in the first account",
    "file": "test/metacoin.js"
   }
  ],
  "21": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "22": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "23": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "24": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "25": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "29": [
   {
    "title": "should call a function that depends on a linked library",
    "file": "test/metacoin.js"
   }
  ],
  "33": [
   {
    "title": "should put 10000 MetaCoin in the first account",
    "file": "test/metacoin.js"
   },
   {
    "title": "should call a function that depends on a linked library",
    "file": "test/metacoin.js"
   },
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ]
 },
 "contracts/ConvertLib.sol": {
  "7": [
   {
    "title": "should call a function that depends on a linked library",
    "file": "test/metacoin.js"
   }
  ]
 }
}

[JoranHonig]

@cgewecke 🎉 This is awesome!

Thanks for this feature!

[JoranHonig]

@cgewecke I'm playing around with this right now (doing an implementation of the fault localisation algorithm you mentioned 😄).

Here is the implementation that I'm working on https://github.com/JoranHonig/tarantula

I think it might be necessary to use the fullTitle for each test, otherwise you might be able to get collisions (multiple tests with the same title in a file).

[JoranHonig]

Another question: I think mocha doesn't give a "file" for the tests that are actually written in solidity (if I'm using truffle). Is the "file" param also optional in solidity-coverage's results?

[cgewecke]

@JoranHonig Wow! Super cool.

multiple tests with the same title in a file

These are filtered out while generating the matrix. If a title occurs twice for the same line & test file it's skipped. Please let me know if you see these in the output though since they shouldn't be there.

tests that are actually written in solidity

Unfortunately solidity-coverage doesn't support Truffle's native solidity testing framework atm. (Issue #578) It's disabled because of a bug in the Truffle compilation pipeline. Will need to revisit this

[JoranHonig]

@cgewecke do you know a clean way of getting out the test results (passed/failed) when checking test coverage?

--matrix would output the coverage results to a file (super nice), but not this information right?

[cgewecke]

@JoranHonig It's possible to start collecting it - the mocha reporter has 'pass' and 'fail' hooks that could be repurposed for this.

Out of curiosity - is this data useful for the mutation tester or fault localization or something else?

One (possibly incorrect) premise I've had is that matrix reporter would be run once just to generate a map. And that the execution context is a test suite with probably all passing tests.

[cgewecke]

@JoranHonig How did you produce the test data in your tarantula unit test?

Perhaps the --matrix should also generate that?

[JoranHonig]

@cgewecke this data could actually be useful for both mutation testing and fault localization. But I'm asking specifically for fault localisation.

I would like to have some setup where people will run npm test and automatically generate the artifacts (coverage + test results) for fault localisation. An IDE plugin/ CLI can then pick up those artifacts and do the fault localisation logic.

Vertigo currently reads the standard output of truffle (after configuring it to use the json reporter) and parses it. This is super unclean bc truffle also outputs a bunch of other stuff over stdout, which Vertigo needs to filter.

@JoranHonig How did you produce the test data in your tarantula unit test?

The other testdata is the output of mocha's json reporter on Metacoin contract. If --matrix could output that, that'd be super helpful!

[cgewecke]

The other testdata is the output of mocha's json reporter on Metacoin contract. If --matrix could output that, that'd be super helpful!

Oh! Perfect, I'll add that rn.

[cgewecke]

Ok... from commit ac532d6, mocha json output is written separately to mochaOutput.json.

I ran the tarantula lib against sushiswap. All the tests pass so suspiciousness is either 1 or 0.

• gist of the output
• conventional code coverage report

I think further refinements may be required at solidity-coverage but I'm not sure what would be helpful...please just lmk the short-comings you see on your side.

One issue I noticed is that else statements are not being tracked as lines because of a parser-antlr quirk.

I adapted the mocha json reporter here. A reporter like that might be useful if you're programatically introducing failures.

[JoranHonig]

Hi @cgewecke, @tintinweb (and me a bit) built a VSCode plugin that does fault localisation.

https://marketplace.visualstudio.com/items?itemName=tintinweb.vscode-tarantula

It looks for mochaOutput.json and testMatrix.json & will automatically highlight (with opacity based on suspiciousness) lines, and give you a ranking.

💪 wouldn't have been possible without this coverage feature! Thanks!

[cgewecke]

@JoranHonig Oh great!! Looks super cool.

Thanks for doing all this Joran! Just incredible :)

[cgewecke]

v0.8.0