Add coverage reports to e2e tests

[kinow]

This is a small change with no associated Issue.

Let's see if we can compare what's being tested in unit tests, and what's being tested in e2e tests (and what's not tested yet).

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Already covered by existing tests.
• No change log entry required (why? e.g. invisible to users).
• No documentation update required.

[kinow]

Had to wait a few minutes for Codecov to show the correct percentage in the branch of my fork.

Master:

This branch:

Which I think is correct. There are no tests for Drawer.vue in unit tests (Codecov is hiding files not covered). I thought we would have tested Drawer.vue pretty close to 100% in e2e. But it's only 60% (better than nothing). And the graph package and its components are only unit-tested, as the offline mode doesn't have graph data.

[codecov-io]

Codecov Report

Merging #443 into master will increase coverage by 11.10%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master     #443       +/-   ##
===========================================
+ Coverage   45.24%   56.35%   +11.10%     
===========================================
  Files          37       50       +13     
  Lines         862      976      +114     
  Branches       67       75        +8     
===========================================
+ Hits          390      550      +160     
+ Misses        459      414       -45     
+ Partials       13       12        -1     

Flag	Coverage Δ
#e2e	40.40% <ø> (?)
#unittests	45.24% <ø> (?)

Impacted Files	Coverage Δ
src/components/cylc/graph/Graph.vue	0.00% <0.00%> (ø)
src/plugins/axios.js	30.00% <0.00%> (ø)
src/i18n/index.js	100.00% <0.00%> (ø)
src/components/cylc/Drawer.vue	60.00% <0.00%> (ø)
src/views/UserProfile.vue	100.00% <0.00%> (ø)
src/views/Dashboard.vue	62.50% <0.00%> (ø)
src/views/Workflow.vue	51.28% <0.00%> (ø)
src/services/mock/checkpoint.js	100.00% <0.00%> (ø)
src/services/mock/workflow.service.mock.js	100.00% <0.00%> (ø)
src/plugins/vuetify.js	100.00% <0.00%> (ø)
... and 17 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0cba68...8ff8921. Read the comment docs.

[kinow]

To review, just confirm GitHub workflow looks OK, look at the changes in the package.json and other files, and if you'd like have a look at the Codecov reports.

[kinow]

(Codecov seems confused as we have 2 reports uploaded. It's commenting only on the unit test report. But there is a second one for e2e that should increase the overall coverage.)

[kinow]

Will fix conflicts and rebase tomorrow morning 👍

[kinow]

(Codecov seems confused as we have 2 reports uploaded. It's commenting only on the unit test report. But there is a second one for e2e that should increase the overall coverage.)

Note, when I created the PR, Codecov added a comment with the unit tests coverage (~40%). Only a few moments later it removed the old comment, and replaced it by a new one with the correct coverage (above, of ~50%).

That makes sense I think. We have 2 jobs in the GitHub workflow. One for Ubuntu latest (default), and another with Ubuntu 16.04 due to a bug in Cypress. Codecov must be sending the first comment once the first job is done. Then there is a delay until the Cypress e2e job is done, and the coverage report is merged & updated in Codecov.

[kinow]

Rebased and fixed conflicts. Now waiting for GitHub to tell whether that worked OK or not 😬

[oliver-sanders]

This might not kick in until merged as secrets are not exposed to forks.

Annoyingly GitHub doesn't raise an error if a template variable is unset, it just gives you an empty string.

[kinow]

👍 I thought it was working for the other coverage report (I copied this block from the existing line for unit tests coverage)

[kinow]

Will resolve conflicts once #446 is in.

[oliver-sanders]

Looks ok, how do we know if it's working?

[kinow]

Ah, good question. Should have included how to review. Best way, IMO, would be looking at Codecov, and confirming it got a report from GitHub action.

https://codecov.io/gh/kinow/cylc-ui/branch/coverage-e2e (rebased some hours ago)
vs.
https://codecov.io/gh/kinow/cylc-ui/branch/master (I synced my master today too)

Or compare both: https://codecov.io/gh/kinow/cylc-ui/compare/master...coverage-e2e

The reports should show files that are not covered by unit tests, e.g. src/services/mock/workflow.service.mock.js (as we use the offline mode). The unit tests run with the test environment. While the e2e tests run using the offline environment.

[oliver-sanders]

Or compare both: https://codecov.io/gh/kinow/cylc-ui/compare/master...coverage-e2e

Nice!

[hjoliver]

LGTM