cypress

Cypress Testing

Table of Contents

• Before you begin (some known issues)
  • GUI and headless mode produce different snapshots
  • GUI mode saves snapshots in different locations
• Why Cypress?
• Structure
• Creating a test
• Running the tests
  • Command Line (Headless)
  • GUI (Headed)
• Debugging a test
  • Debugging locally
  • Debugging on CI
• Updating snapshots
  • CI snapshots
  • GUI snapshots

Before you begin (some known issues)

GUI and headless mode produce different snapshots

An open issue with Cypress cypress-io/cypress#3324 causes the cypress-image-snapshot plugin to capture snapshots at a smaller size when run with yarn cypress (cypress run) as opposed to yarn cypress:gui (cypress open).

To get around this issue, two sets of snapshots are generated and committed for every test:

• ci - Used by Travis (CI) and for headless browser testing using the Docker container. These are the important ones.
• local - Used by the headed browser when testing locally using the Cypress GUI.

See "Updating snapshots" for more information on generating both types of snapshots. Also see "Running the tests" for more information on how they're used.

GUI mode saves snapshots in different locations

When using the GUI mode (yarn cypress:gui), how you run the specs affects where the snapshots are saved:

1. Run All Specs: saves snapshots under <cypress dir>/snapshots/local/All Specs
2. Running an individual spec: saves snapshots under <cypress dir>/snapshots/local/<name-of-spec.spec>

Headless mode always saves snapshots under <cypress dir>/snapshots/ci/<name-of-spec.spec>. With this in mind we are .gitignoreing the All Specs directory.

Feel free to run specs individually during development. But be aware that the first run will generate the snapshots (do this before you make changes to the code!).

Why Cypress?

Cypress enables us to:

• Open a browser during development to visibly see failing tests.
• Produce pixel diffs of the VideoContext canvas before and after changes to code.
• Run headless browser testing as part of our Travis CI.

The Cypress GUI

A failing image diff

Structure

📁 html/index.html

The very simple html fixture we use for all tests. This is pretty much just a canvas element and some js to help inject VideoContext into our test code.

📂 assets/

A few video and image files used in the tests.

📁 support/commands.js

Where we register our custom test commands and configure the third party matchImageSnapshot command.

📂 integration/*.spec.js

The spec files containing the test code.

Creating a test

To create a new test, add a new my-test.spec.js file to the <cypress dir>/integration directory and write your test code.

We recommend reading through one of the existing specs first, to get an idea of how they should be structured and formatted.

If you're testing with snapshots, then you'll need to generate some base snapshots first to test against. See "Updating snapshots" for instructions.

Running the tests

There are two ways to run the tests:

Command line (Headless)

To run the tests in an environment most similar to the CI, use:

yarn cypress

This wil build and run a Docker container with Cypress installed. The tests will be run in a headless version of the Chrome browser using the snapshots in the <cypress dir>/snapshots/ci/ directory (see "GUI and headless mode produce different snapshots" for an explanation on why different snapshots are used).

Prerequisite: Docker must be installed on your system - download the Mac or Windows version.

GUI (Headed)

To see the tests running inside a browser, you can run the GUI version with:

yarn cypress:gui

This will serve a HTML page and open the Cypress GUI, where you can select to run all or individual specs.

These tests will use the snapshots from the <cypress dir>/snapshots/local directory (see "GUI and headless mode produce different snapshots" for an explanation on why different snapshots are used).

Debugging a test

Debugging locally

There are a couple of ways to find out what went wrong when a test fails.

1. Watch it fail

You can watch the test fail by opening the Cypress GUI and running the test:

yarn cypress:gui

2. Look at the image diff

Alternatively, you can inspect the difference between the accepted snapshots for your test and your most recent test run (assuming the test failed because a snapshot did not match).

First run your tests in either in headless or GUI mode then, once its failed, inspect the contents of:

<cypress dir>/screenshots/<name of your spec / All Specs>/diff_output/

Here you will find an image like this one:

A failing image diff

Debugging on CI

We have a cypress dashboard where you can view screenshots of the CI test runs:

https://dashboard.cypress.io/#/projects/mdase9/runs

Updating snapshots

If you add a new test or change the result of an existing test, then you will need to generate, commit and push an updated snapshot for both the CI and GUI version.

CI snapshots

Snapshots for the CI and headless environment can be generated using:

yarn cypress:update-snapshots

This wil build and run a Docker container with Cypress installed. It will run the tests through a headless version of the Chrome browser and update the snapshots in the <cypress dir>/snapshots/ci/ directory.

Prerequisite: Docker must be installed on your system - download the Mac or Windows version.

GUI snapshots

Snapshots for the local GUI version can be generated using:

yarn cypress:gui-update-snapshots

This will load the Cypress GUI and run through all the tests, updating the snapshots in the <cypress dir>/snapshots/local directory.