Use Phantomjs 2.0 for UI screenshot tests

[mattab]

At some point we should use Phantomjs 2.0 for our screenshot tests. If we are lucky (!) it may help us resolve some issues such as: #7147 #8034. Maybe it would also let us re-enable some of our disabled screenshot tests in #8142

Current blocker: http://phantomjs.org/download.html

Binary packages for Linux are still being prepared. There are still issues to be solved until a static build is available (see issue 12948 for more details).

If we didn't want to wait for official linux builds we could maybe use these PPA eg. https://launchpad.net/~tanguy-patte/+archive/ubuntu/phantomjs

[tsteur]

I just tried it with our ActionsDataTable specs. Tests run on my server 20% faster (43 seconds instead of 53 seconds) and screenshots look much nicer and is "closer" to a real world result. Eg on PhantomJS 1.9.X the sort arrow icons in dataTables are misplaced.

Used PhantomJS from here: https://launchpad.net/~tanguy-patte/+archive/ubuntu/phantomjs

Was installed quickly and worked out of the box so far. Not sure about other possible problems. It says it is 2.0.1-development version

[tsteur]

FYI: With PhantomJS 2 we still get the crashes / seg faults

[tsteur]

Related issue on Travis Github repository: travis-ci/travis-ci#3225

Adding PhantomJS 2.0 could be done like this: https://github.com/piwik/piwik/compare/phantomjs_2.0?expand=1

Problems:

• We're kinda dependent on https://s3.amazonaws.com/travis-phantomjs/phantomjs-2.0.0-ubuntu-12.04.tar.bz2 which was linked in the related travis issue. To reduce this dependency we could host it ourselves or keep a copy of it.
• This binary works for Ubuntu 12.04 (used on Travis), https://launchpad.net/~tanguy-patte/+archive/ubuntu/phantomjs works for Ubuntu 14.04. I'm not sure if there are binaries for other Linux versions which may restrict on which Linux versions the tests can be executed
• The PhantomJS version for Ubuntu 14.04 is 2.0.1-development while the one for Travis is 2.0.0. As the version number is close to 2.0.0 there could be differences in the captured screenshots between 2.0.0 and 2.0-1-development meaning we might not be able to generate the same screenshots in local dev environment vs travis. Not sure if this is a problem
• From what I know file uploads are broken with PhantomJS 2, not sure if we use it.

Improvements:

• Newer Webkit version which is faster but also has more features and is closer to latest Webkit versions that are used out there (one could also say this is a disadvantage as no longer test older version)
• Speed. At least on my local VM it seems to run about 20% faster, needs to be checked once all tests are green. As on Travis all tests are failing we cannot compare the difference
• Generated screenshots look better eg compare the red arrow and the font:
  PhantomJS 1.9.8:

PhantomJS 2.0.1-development:

[tsteur]

FYI: PhantomJS 2.0 executed in 31 minutes instead of 37 minutes even though all screenshots fail (it takes more time when screenshots fail usually). It's hard to compare it especially since it also depends on the Travis VM.

Problem: It failed at the end when generating the diff file like this: https://travis-ci.org/piwik/piwik/jobs/67782618#L6092 and https://travis-ci.org/piwik/piwik/jobs/67781956#L5203

Here's more output https://travis-ci.org/piwik/piwik/jobs/67929639#L7500 It fails when copying the expected screenshot here: https://github.com/piwik/piwik/blob/2.14.0-b7/tests/lib/screenshot-testing/support/diff-viewer.js#L56 from /home/travis/build/piwik/piwik/tests/lib/screenshot-testing/../../../tests/UI/./expected-ui-screenshots/UIIntegrationTest_referrers_campaigns.png to /home/travis/build/piwik/piwik/tests/UI/screenshot-diffs/UIIntegrationTest_referrers_campaigns.png. As all UI tests fail we might just spawn too many child processes?

Not sure how to continue with this issue. Is it worth to investigate this buffer overflow? Do we want to use PhantomJS 2 or not? (see previously mentioned problems/advantages)

Related issues: ariya/phantomjs#11703 ariya/phantomjs#11213

@mattab @diosmosis thoughts?

[tsteur]

Here's a diff http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7

for example: http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/EvolutionGraph_annotation_edit_submit http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/EvolutionGraph_initial http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/Login_login_fail http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/Menus_admin_changed http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/ActionsDataTable_column_sorted

[tsteur]

@diosmosis What do you reckon how much work it would be to remove PhantomJS backend and try using Electron? Not supporting different backends, just Electron? Did you have a look whether Electron provides all API's that we need? Eg inject CSS class, evaluate code, spawn processes (I presume we'd use node.js built-in here), ... Not sure re: electron/electron#228

Background: I just spent 2 or 3 hours trying to make this UI test work until I noticed PhantomJS just doesn't handle the floating correctly and I haven't found a way yet to make it work http://builds-artifacts.piwik.org/ui-tests.7822_3/13859.7/UIIntegrationTest_goals_overview

Also with PhantomJS 2 I don't get this test working. This morning I was also 2 or 3 hours busy trying to fix another UI test and I couldn't.

I really think it's time to try something else if it isn't too much work. I think we use too many PhantomJS API's

[tsteur]

I just installed electron and executed it and got the following message:

vagrant@piwik-trusty64:~/www/piwik$ electron
/usr/lib/node_modules/electron-prebuilt/dist/electron: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory

The x11 requirement might be a problem since one doesn't really want to install x11 on a server or on travis (could be also slow to install maybe?)

[ngotchac]

I am currently running a serie of CasperJS tests with SlimerJS (Gecko, natively not headless) on a Jenkins box that takes screenshots of my application. This is possible thanks to xvfb, and it works quite good (with some tweaks, as always...). So I guess running Electron should be also possible, I'm really looking into it right now.

I hope I helped!

[tsteur]

Thx for that! Do you get SlimerJS to run on Travis? From what I remember it's not headless?

[ngotchac]

Hey!
So we are running SlimerJS tests in a Jenkin build, but it just launches a task on a AWS box... Anyhow, on Travis you have several other options: http://docs.travis-ci.com/user/gui-and-headless-browsers/

All of these are pretty great, but I think Saucelab is the way to go for an OpenSource project, as it's free!

[diosmosis]

What do you reckon how much work it would be to remove PhantomJS backend and try using Electron? Not supporting different backends, just Electron?

I already managed to get it working, until right before the actual tests run. That is to say, fixtures can be run successfully, but I didn't convert page-renderer.js.

Did you have a look whether Electron provides all API's that we need? Eg inject CSS class, evaluate code, spawn processes (I presume we'd use node.js built-in here), ... Not sure re: electron/electron#228

I believe it has everything needed, except for simulating native events. I found the code in libchromium for it though (hopefully the browser tab still exists), so it can be done through a v8 extension or possibly through node-ffi (though this has to be done on the v8 side of electron, not the node side, so maybe not too easy). Or it could just be avoided, and jquery/javascript can be used (ie, $('...').click() instead of sendMouseEvent()). Tests will lose a little accuracy though (eg, if an animation is broken so the click won't work in a real browser, we won't notice because we just trigger the handler directly).

Btw, I don't know why "inject CSS class" would be a feature, it's just a evaluate JS use case.

Regarding lack of headless-ness, xvfb should work fine. We used it w/ phantomjs for a bit in travis while experimenting w/ slimerjs.

[tsteur]

FYI: I quickly tried to use slimerjs but a problem is that global variables do not work there see https://docs.slimerjs.org/current/differences-with-phantomjs.html

I will invest a few minutes to try to make it run but probably it'll be a bit more work.

[ngotchac]

I can try to help you, I migrated 2 or 3 times in the past month PhantomJS tests to SlimerJS, and the last time was yesterday :p

[tsteur]

I spent 40 minutes on it but everything in our screenshot testing lib is basically based on global variables and we use quite a lot of phantom.injectJS to have variables available globally which doesn't work and it is expected (laurentj/slimerjs#337). We'd have to rewrite to use modules.

Also require('fs').changeWorkingDirectory() did not work but that should be maybe possible via process. It was not needed for my test. I reckon it's about 1 day of work to make it run with SlimerJS. @mattab what do you think is it worth to give it a try?

@ngotchac Help is much appreciated. I'll definitely ping you when there are problems.

[mattab]

@tsteur if there is a chance we can make SlimerJS work in one day, +1 to try it

[tsteur]

After some more thoughts and discussion I think the long term best solution might be electron (from what it seems). It recently added support for webdriver which is pretty awesome see http://electron.atom.io/docs/v0.28.0/tutorial/using-selenium-and-webdriver/ and https://code.google.com/p/selenium/wiki/WebDriverJs

We can use all the feature of node.js, it is actually maintained and updated, it is stable and should be fast, bigger community, many tools around electron etc. The only downside is the need of X11 so far. Think I changed my mind and would rather prefer to give that one a try

[tsteur]

Just FYI: Today I spent again 4 hours trying to fix 5 segment selector tests (http://builds-artifacts.piwik.org/ui-tests.7822_3/13908.7/) that started to fail after this commit: 7b6132d

I couldn't reproduce this when opening the test URL directly and everything seems fine. I still don't know why this one would fail because of that change. I debugged the JavaScript using console.log in UI tests etc. What makes the test work seems to click() on the element directly (729af37 ). This would mean there is maybe an invisible div on top of the Save button but it is basically impossible to find it. I just presume it is a problem with PhantomJS

As soon as possible I'll have a look re an alternative

[tsteur]

I tried to make it work with Electron today but looks like we definitely need WebDriver, couldn't make it work without. Eg for clicking, reading content of page, manipulating page, ...

[tsteur]

It would be maybe something good to work on in Berlin or Poland with combined power, will see :)

[tsteur]

What makes electron very interesting is https://github.com/segmentio/nightmare which is like CasperJS but for electron.

[tsteur]

We likely update to PhantomJS 2. We're also testing https://github.com/webdriverio/webdrivercss + https://github.com/webdriverio/webdriverio but we need to wait for webdriverio 4 support in webdrivercss (refs webdriverio-boneyard/webdrivercss#123 webdriverio-boneyard/webdrivercss#151 )

[tsteur]

BTW: In future Puppeteer might be very interesting see https://github.com/GoogleChrome/puppeteer and https://ropig.com/blog/end-end-tests-dont-suck-puppeteer/

It looks like this could work quite nicely and easy to build / migrate etc.