ytestrunner

Run YUI tests for node packages using "npm test" with support for saving test results, coverage and more. Supports YUITest as well as YUI3 style tests.

To use this module, declare it as a devDependency along with the YUI module that you are using for tests (yui for yui3 style tests or yuitest for standalone yui tests).

Usage

• Write some code (this module assumes, by default, that code will found at the source root and under lib/)
• Write tests using a supported format (this module assumes, by default, that tests will be found test/)
• Modify your package.json as follows

For tests that are implemented using yuitest:

{
    "devDependencies": {
        "ytestrunner": "*",
        "yuitest": "*"
    },
    "scripts": {
        "test": "ytestrunner <other-args>"
    }
}

For tests in YUI3 module format that are implemented using the Y.Test object from yui use:

{
    "devDependencies": {
        "ytestrunner": "*",
        "yui": "*"
    },
    "scripts": {
        "test": "ytestrunner --yui3 <other-args>"
    }
}

• To run your tests, just type:

$ npm test

in the folder that has your package.json file

The ytestrunner script will run all tests and store results and coverage, if requested.

Command line options

Usage: ytestrunner <options>

Basic options
  --root <dir>               : the source root (default: .)
  --yui3                     : flag that enables the YUI3 test harness rather than the YUITest harness
  --verbose, -v              : run tests and instrumentation in verbose mode
  --help, -h                 : print this message
  --tmp <dir>                : tmp directory under which instrumented files will be written
                               (default: /tmp)
  --colors                   : show colored output (use --no-colors to disable)

Tests and results
  --include <pattern>        : include file patterns for test files
                               (default: [test/**/*.js, tests/**/*.js])
  --exclude <pattern>        : exclude file patterns for test files
                               (default: [ empty ])
  --save-results             : save results to disk
  --results-format <format>  : results file format
  --results-file <name>      : basename of results file without extension
                               (default: results)

Instrumentation and coverage
  --coverage, -c             : enable instrumentation and coverage stats
  --istanbul                 : use istanbul instead of YUI coverage for instrumentation and coverage stats
  --fast-cover               : skip source coverage if a newer instrumented file found from previous run
  --cov-include <pattern>    : include file patterns for JS files to be instrumented
                               (default: [**/*.js])
  --cov-exclude <pattern>    : exclude file patterns for instrumentation
                               (default: [test/**/*.js, tests/**/*.js, **/.*, **/node_modules/**])
  --save-coverage            : save JSON coverage information to disk
  --coverage-file <name>     : basename of JSON coverage file without extension
                               (default: coverage/test-coverage)
  --coverage-report-format <format> : format of coverage report (default: lcov)


In addition, the environment variable [ytestopts] can be set with override options.
These take precedence over the options passed to command line.

Using this module effectively

The module is designed so that it can provide test results and coverage information without having to write any files under your source tree. Saving results and coverage information to disk must be asked for explicitly.

This allows you to have a simple command in your package.json (e.g. ytestrunner or ytestrunner -c for default coverage) and yet have an automated build set the ytestopts environment variable to control writing of results and coverage in supported formats.

Use of ytestopts is also useful during development. If you want to just run one test, say, foo.test.js and instrument only one file, say, foo.js you could set ytestopts thus:

$ export ytestopts="--include=test/foo.test.js --cov-include=lib/foo.js"
$ npm test

This strategy requires no changes to package.json and yet will only instrument your source under test and run only one test file.

Also, keep in mind that boolean options can be negated. So, if your test script says ytestrunner -c and you are not interested in coverage, you could set ytestopts to --no-coverage to disable coverage.

Examples

See the strawman examples in the directories yuitest-sample and yui3-test-sample. Run npm install and npm test in either of those directories.

A word on code coverage

Code coverage is implemented as follows:

• Create an instrumented version of all JS files matching the coverage filters somewhere under /tmp. This is done using the yuitest-coverage java module.
• Hook the module loader to serve the instrumented content at require time when the original file is requested. This step is skipped for a file if the coverage module reports an error when instrumenting it.
• Collect stats from the global coverage object created by the instrumented code at the end of all tests
• Save coverage information and write reports if requested using the yuitest-coverage-report java module

Debugging test execution

The -v option is your friend.

Caveats and known issues

• This module must not be globally installed. Even though it uses a YUI test library as part of its runtime execution, it does not declare dependencies on these. The caller is responsible for declaring this module as well as the test library used as devDependencies. This strategy ensures that this module uses the same instance of the testing library that your tests use.
• For the same reason, this module may not be npm-linked into your package either, since linked modules cannot "see" other dependencies under your node_modules/ folder.
• A modern java is required for instrumentation and coverage reports.
• Since instrumentation is done by transparently substituting the instrumented file for the original at require time, stack traces will refer to original file but the line numbers in the instrumented file and will not make sense when code coverage is turned on

License

ytestrunner is licensed under the MIT License.

Third-party libraries

The following third-party libraries are used by this module:

• async: https://github.com/caolan/async
• colors: https://github.com/Marak/colors.js
• fileset: https://github.com/mklabs/node-fileset
• mkdirp: https://github.com/substack/node-mkdirp
• nopt: https://github.com/isaacs/nopt
• resolve: https://github.com/substack/node-resolve
• nodeunit: https://github.com/caolan/nodeunit (dev dependency for unit tests of this module)

and enables running tests supported by these libraries:

• YUITest: https://github.com/yui/yuitest
• YUI: https://github.com/yui/

Credits

• mfncooper - for ideas on module loader hooks to support code coverage
• baechul - for ideas on standard test running using Y.Test.Runner
• nzakas - test loading strategy for YUITest tests heavily influenced by command line support in that module
• davglass - for review of this module