e2e tests with ever changing repository data

[BePo65]

Until now we have the problem that e2e tests often fail as the latest version in the repository is newer than the expected result in our code.

In reality our e2e tests are handling 2 questions:

1. will the program transform the data requested from the repository to the expected output (e..g. formatted as csv)
2. are there any changes in the server schema or in the communication between this program and the server

Wouldn't it be easier if we split these questions into 2 tests?

1. Use a mockup for the server data to test for the correctness of this program
2. use separate tests for verifying the communication with the server and the server schema?

This way the existing e2e tests only fail, if we broke the code and we get a detailed information, if the repository schema or the communication with the repository changed (what should rarely be the case).

[BePo65]

Update:
in the current version of license-report the e2e tests will break, even when the code of license-report is unmodified, when:

1. a new version of a package is installed
2. a new version of a package is available in the repository and we didn't update the installed version

I looked at the 'using_package_lock' branch;
if I get it right, using the information in the package-lock.json file we can solve the first point. I will work this out in detail.

But for the second point I am afraid that the solution is not so easy. We could

• require that all packages are updated when "point 2" breaks the test (good for security, but potentially not so easy if there are breaking changes in the updated packages)
• reuse data from 'getPackageReportData' to generate the expected values (but then we could not verify if this data is valid)
• mockup the repository (e.g. via nock) (but then we have to verify in another test that the repository api is still as we expect it to be)

So how will we go on with "point 2" (a new version of a package is available in the repository)?

[BePo65]

Update 2:
Sometimes I am thinking very slow :-)
In my current understanding we have 3 different versions:

1. the version in the (dev-) dependencies in package.json (that is what we defined, when adding a package to a project)
2. the version in package-lock.json (that is what gets installed, when we do npm i e.g. in a freshly pulled project - that is the latest version that respects the version range we defined for a package in package.json)
3. the version in npmjs (that is the latest version published)

By now license-report shows versions 1 and 3 (even when we call 1 "installedVersion", but that is my fault)

Perhaps we should show all 3 version (or use "2" as "installedVersion")