covimerage

Generates code coverage information for Vim scripts.

It parses the output from Vim's :profile command, and generates data compatible with Coverage.py.

Installation

You can install covimerage using pip:

pip install covimerage

Simple usage

You can use covimerage run to wrap the call to Neovim/Vim with necessary boilerplate:

covimerage run vim -Nu test/vimrc -c 'Vader! test/**'

This will write the file .coverage_covimerage by default (use --data-file to configure it), which is compatible with Coverage.py. A report is automatically generated (on stdout).

You can then call covimerage xml to create a coverage.xml file (Cobertura-compatible), which tools like Codecov's codecov tool can consume, e.g. via codecov -f coverage.xml.

Manual/advanced usage

1. Generate profile information for your Vim script(s)

You have to basically add the following to your tests vimrc:

profile start /tmp/vim-profile.txt
profile! file ./*

This makes Neovim/Vim then write a file with profiling information.

2. Call covimerage on the output file(s)

covimerage write_coverage /tmp/vim-profile.txt

This will create a file .coverage_covimerage (the default for --data-file), with entries marked for processing by a Coverage.py plugin (provided by covimerage)).

3. Include the covimerage plugin in .coveragerc

When using coverage on the generated output (data file), you need to add the covimerage plugin to the .coveragerc file (which Coverage.py uses). This is basically all the .coveragerc you will need, but you could use other settings here (for Coverage.py), e.g. to omit some files:

[run]
plugins = covimerage
data_file = .coverage_covimerage

4. Create the report(s)

You can now call e.g. coverage report -m, and you should be able to use coverage reporting platforms like https://codecov.io/ or https://coveralls.io, which are basically using coverage xml.

Reference implementation

• Neomake is the first adopter of this. It has an advanced test setup (including Docker based builds), and looking at tis setup could be helpful when setting up covimerage for your plugin/project.

  • Neomake's coverage report on codecov.io
  • PR/change to integrate it in Neomake (Neomake's test setup is rather advanced, so do not let that scare you!)

Caveats

Coverage information for a function may get wiped if the function is re-defined after testing it. This is a known limitation that does not depend on the implementation of covimerage, but on the vim profiler itself, since profiling data for a function, used to extract coverage data, is reset when the function is re-defined, even if the definition is identical.

This can be an issue if the source files being checked for coverage are reloaded with runtime between tests for some reasons, e.g. when mocking some function. To prevent coverage information from being lost, one solution is to write such tests in a different suite that is run separately, and combine the coverage results with covimerage run --append.

Links

• Discussion in Coverage.py's issue tracker: coverage issue 607

TODO

• Line hit counts: known to covimerage, but not supported by Coverage.py (https://bitbucket.org/ned/coveragepy/issues/607/#comment-40048034).