Added an automated Selenium UI test for a small Zimit2 archive #1286

THEBOSS0369 · 2024-11-11T11:23:29Z

This PR Fixes #1205

Hello Everyone!

Description

In this PR i have added an automated Selenium UI test for a small Zimit2 archive (used tonedear.com_en_2024-09.zim)

The tests i have added are as follows :

Load Kiwix JS and verify title
Switch to Jquery or SW mode
Load Tonedear archive and verify content
Navigate from main page to Android & iOS section
Verify Android and iOS store images in Both Modes

Test

I have done all the necessary test

Checked in both "Restricted" and "ServiceWorker" Everything is working fine check the ss below.
Unit tests npm test no issue
End-to-end (e2e) tests-e2e-firefox-> All tests Passed Below is the ss
extension versions with production code tested
Browser Test -> Microsoft Edge, Chrome, Firefox and Brave

Screenshot for tests

THEBOSS0369 · 2024-11-11T11:27:06Z

@Jaifroid I have not added test for android & ios page because i didn't know what test should i do, your suggestion will be appreciated 😅😃 Thanks!
!

Jaifroid · 2024-11-13T07:19:57Z

Thank you @THEBOSS0369. That's very useful. I would suggest adding a test that switches to the page that shows the Android and iOS store images, and verifying that both images have loaded correctly:

In Restricted mode (aka JQuery), test that both images' src attribute has changed to a data URI (see screenshot), and assert that both data URIs match the beginning of the expected content (say 30 characters, so you get the beginning of the actual data, not just "data:img/png,base64,". If you want to be thorough, you could hash the string and test the hashes, but that may be overkill.
In ServiceWorker mode, you can do a basic image verification: check if the image has completed loading using JavaScript properties, and verify that the image has natural dimensions (width/height > 0). Use the computed naturalWidth and naturalHeight properties of the images to verify (see https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/naturalWidth). This will tell us that the image loaded correctly (after appropriate wait).

THEBOSS0369 · 2024-11-15T13:18:48Z

Hey @Jaifroid ! I have added the android and ios test as you asked me but there is a problem occurring when i am running the tests the restricted mode test passes well but in sw mode this error comes

i have tried by making the waititme 120000 , by removin cache but still this won't work. I don't understand what i am doing wrong i checked in the sw mode the image's alt is present and i have added the same in the code. Your help will be appreciated . Thanks!

Jaifroid · 2024-11-15T15:59:30Z

Thanks @THEBOSS0369. Sorry to hear about the error. It's tricky to get the assertions right with the timing. I'll take a look at the weekend, but in the meantime it's possibly the case that there's some error in selecting the images and checking their natural width. Adding some console logging to see what it's actually finding and what you are expecting it to find might help!

THEBOSS0369 · 2024-11-16T07:40:01Z

Yes sir. I'm doing that by further more progress i have found that there are in total 21 images showing 19 are icon_external_link one is gif and one is kiwix logo this script is still ain't able to find the store images. i will try to find more in the meanitime maybe i find something useful ,right now this is what's showing.

Note: I didn't pushed the code as it has become very messy so if you want to see the code just tell me i will push it. 😃🙂

Jaifroid · 2024-11-16T11:18:06Z

OK, I ran this myself (in the Edge runner, adding the appropriate commands), and while watching the browser locally it's clear that the test never actually loads the page "Android & iOS App". It remains stuck on "Ear Training Practice" (see screenshot). You're not going to be able to verify the images on the "Android & iOS App" page unless you actually switch to it first!

Jaifroid

Please see below for the steps to remedy the issue you're encountering.

Jaifroid · 2024-11-16T11:20:37Z

tests/e2e/runners/firefox/firefox.e2e.runner.js

@@ -27,6 +28,8 @@ async function loadFirefoxDriver () {
 // so we need to use the second one first
 const driver_for_gutenberg = await loadFirefoxDriver();
 const driver_for_ray_charles = await loadFirefoxDriver();
+const driver_for_tonedear = await loadFirefoxDriver();


As the comment says above this block, the drivers need to be listed in reverse test order here, so that when the user runs the tests locally we see the Ray Charles first (which will be on top of the other windows). So please put the tonedear test above gutenberg in this list, and update the comment to reflect that there are three tests

tests/e2e/spec/tonedear.e2e.spec.js

THEBOSS0369 · 2024-11-17T07:08:16Z

@Jaifroid Another problem occurring , when i try to navigate to Android Page it won't, i think this is happening because i didn't know the exact path for it at first i used this android-ios-ear-training-app it didn't worked i tried by adding /android-ios-ear-training-app & ../android-ios-ear-training-app and many more tries but still didn't worked

then i looked into the ray_charles code as it also have a navigation function i tried this which is
const appLink = await driver.wait(until.elementLocated(By.xpath("//ul[contains(@class, 'navbar-nav navbar-right')]/li[1]/a")), 20000);
similar to that present in ray chares
articleLink = await driver.findElement(By.xpath('/html/body/div/div/ul/li[77]/a[2]'));
but still not working.

Then i think i can do that by link but The link for the server remains same
http://localhost:5173/www/index.html?appCache=false
even when i open any other page but when i run the tests it have a new url
http://localhost:8080/dist/www/index.html?title=C/tonedear.com/android-ios-ear-training-app
for different pages i tried by clearing cache and using incognito mode still url remains same. I'm not sure if this is default or not.

Can you help me with this path problem if we get it fixed then every tests will pass correctly.

Jaifroid · 2024-11-17T07:15:26Z

@THEBOSS0369 Oh no, this app doesn't support navigating to a page by URL fragment. There are two ways to get to a particular page: either click on the link that you highlight in the screenshot (easiest), or search for it using the search bar. Also, remember that the links you're clicking on are in the iframe, not at the top level, so you have to switch to the iframe before finding the link and clicking it. Take a look at the code in the Ray Charles test to see how to switch to the iframe.

This explains why you said you were finding 21 images in the landing page when in fact there are none! You were finding these images in the app itself, as you hadn't switched to the iframe containing the content.

THEBOSS0369 · 2024-11-17T11:41:40Z

@Jaifroid Sirrr ! it worked now its navigating to andorid page although the images are still not found now i will try to fix that

THEBOSS0369 · 2024-11-17T12:39:22Z

@Jaifroid Sir all The tests have passed . Does any thing else needed to be added in the test? If yes then do let me know.

The comments in this image for tonedear tests will not be visible in the main tests i have commented it out.

Jaifroid · 2024-11-17T12:49:18Z

@THEBOSS0369 That's very good news! I don't think we need more tests. I'll review the new code as soon as I can, but in the meantime, you might want to clean it up, remove any excessive timeouts you previously added, etc. Once the spec is ready, you should add it to all the other runners in the folders under tests/e2e/runners. Then I'll need to run it on BrowserStack to be sure the tests pass on all browsers.

Oh, I see GitHub tests have failed. I'll re-run those to be sure it wasn't a timeout fluke.

THEBOSS0369 · 2024-11-17T12:51:21Z

@Jaifroid I think the tests are failing because when it navigate to android page it asks me to authorize the author and i used to press that maybe that's causing error.

THEBOSS0369 · 2024-11-17T12:55:03Z

@Jaifroid Yes this is happening due to the reason i told above i didn't clicked the trust source this time and tests failed. I don't know how to fix this one

Jaifroid · 2024-11-17T17:00:14Z

Yes this is happening due to the reason i told above i didn't clicked the trust source this time and tests failed. I don't know how to fix this one

This is because you erased https://github.com/kiwix/kiwix-js/blob/main/tests/e2e/spec/legacy-ray_charles.e2e.spec.js#L125 (the setting "noPrompts=true") from the URL when loading the app. This was in your previous spec, but then you did a hard reset of your branch, and you erased it, so it's no longer in the commit history.

That command prevents the security prompt from showing. All you need to do is copy the appropriate code block from the ray charles version of the test.

Also, in future, try to avoid rewriting history in your PRs. It makes things difficult to review as well, because when you do a force push, I have to reset my copy of your PR instead of simply pulling a new commit! (Don't worry, everyone makes these kinds of mistakes with PRs when learning.)

THEBOSS0369 · 2024-11-18T05:16:08Z

@Jaifroid Done Sir! Now all tests passes well without any issue.

I'm waiting for these tests to pass then i will add the tests in all the files and remove any unnecessary time waits.

Also, in future, try to avoid rewriting history in your PRs. It makes things difficult to review as well, because when you do a force push, I have to reset my copy of your PR instead of simply pulling a new commit! (Don't worry, everyone makes these kinds of mistakes with PRs when learning

My apologies Sir i didn't knew force push can do this much damage. I will be careful from next time 😅

THEBOSS0369 · 2024-11-18T05:38:34Z

I have increased the timewait maybe this can fix it

Edit (11:10): All the tests have been passed so i'm reducing time wait

Edit Again (12:30): Sir i have added the tests in all the files .

Signed-off-by: THEBOSS0369 <[email protected]>

THEBOSS0369 · 2024-11-18T11:25:09Z

@Jaifroid Sir, The tests are failing in chrome i am not sure why its happening i tested in firefox it passes well. I just did same steps in chrome's e2e test file as i did in the firefox's e2e test file

Jaifroid · 2024-11-18T14:59:14Z

Hmm looks like a timeout issue since it works on Windows in the same browsers. Let me re-run and see again.

Jaifroid · 2024-11-19T08:52:13Z

tests/e2e/spec/tonedear.e2e.spec.js

+                const androidLink = await driver.findElement(By.css('a[href="android-ios-ear-training-app"]'));
+
+                // Verify that the element is found
+                // console.log('Android & iOS App link found:', androidLink !== null);


It might be useful to uncomment this, to check if the link is found.

Jaifroid · 2024-11-19T08:52:52Z

tests/e2e/spec/tonedear.e2e.spec.js

+                // console.log('Android & iOS App link found:', androidLink !== null);
+
+                // Scroll the element into view and click it
+                await driver.executeScript('arguments[0].scrollIntoView(true);', androidLink);


I don't think the element needs to be scrolled into view. You could try without this (comment it out).

Jaifroid · 2024-11-19T08:54:01Z

tests/e2e/spec/tonedear.e2e.spec.js

+
+                // Scroll the element into view and click it
+                await driver.executeScript('arguments[0].scrollIntoView(true);', androidLink);
+                await driver.wait(until.elementIsVisible(androidLink), 10000); // Wait until it's visible


If you've already found the element (androidLink), it doesn't seem necessary to wait for it to be visible, as you wouldn't have found it if it weren't already visible. You could also try commenting this out.

Jaifroid · 2024-11-19T08:55:25Z

tests/e2e/spec/tonedear.e2e.spec.js

+                await driver.wait(until.elementIsVisible(androidLink), 10000); // Wait until it's visible
+                await androidLink.click();
+
+                // Take a screenshot after clicking for debugging


Problem with this is that it's impossible to see screenshots in GitHub Actions AFAIK!

Since we have videos taken on BrowserStack, I think this screenshotting probably isn't necessary. Best to remove it.

THEBOSS0369 · 2024-11-19T11:07:04Z

@Jaifroid ! This worked sir All the tests have passed.

But sir one doubt if the github tests wasn't able to recognize the images in the linux tests then how it was able to verify for the windows test? Like is it some compatible issue or anything else.

Jaifroid · 2024-11-19T13:31:17Z

Very good news @THEBOSS0369! I guess it was a timing issue of some kind, but I need to check the verification method to be sure there isn't an issue with the assertions.

The next step is for me to get the Browser Stack tests running against this PR (you can't do that, because you don't have access to the credentials). To do this, I'll make a dummy PR that has the same code as your PR, and this should then run on Browser Stack, and we can get feedback on whether it's passing also on old browsers. The fact that it's already passing on IE11 mode on Windows (Edge) implies that there shouldn't be any problem with other browsers, but it needs to be checked. I'll do this when I can in the next few days.

THEBOSS0369 · 2024-11-20T12:01:23Z

@Jaifroid Got it Sir! If my work's done here then shall i find any other issue to work on or wait till this PR completes all its checks?

Jaifroid · 2024-12-01T06:32:04Z

Apologies for slowness here -- things have been manic at work. I finally pushed a dummy branch to get the BrowserStack tests to run. It failed the Safari test, and you can see a video of the failed test run here (it starts around 00:50):

Tonedear Test Safari 14.1.2

For comparison, here is a video of a different ZIM passing on that browser:

Gutenberg RO test Safari 14.1.2

The issue, I think, is that this version of Safari can only run in Restricted mode on BrowserStack, and the app informs this to the user with a dialogue box, which blocks the ZIM from loading. You need to test for and dismiss that dialogue box if present. You could just copy the code for that from the Gutenberg or Ray Charles test.

Jaifroid · 2024-12-01T06:41:01Z

@THEBOSS0369 We should add your test to a few more BrowserStack browsers. I suggest adding it to:

Firefox 70 in SW mode only: https://github.com/kiwix/kiwix-js/blob/main/tests/e2e/runners/firefox/firefox70.bs.runner.js
Chrome 60 on macOS in Restricted and SW modes: https://github.com/kiwix/kiwix-js/blob/main/tests/e2e/runners/chrome/chrome60.bs.runner.js
Edge 18 on Win10 in Restricted and SW modes: https://github.com/kiwix/kiwix-js/blob/main/tests/e2e/runners/edge/edge18.bs.runner.js

The reason for adding it selectively and in selected modes is because otherwise tests take too long to run if we test every spec in every mode.

Jaifroid

Code is looking good so far. Please see some minor suggestions.

Jaifroid · 2024-12-01T06:46:10Z

tests/e2e/spec/tonedear.e2e.spec.js

+                }
+            });
+
+            it('Should navigate from main page to Android & iOS section', async function () {


I think it would be better just to say "Navigate from ...." (without "Should").

Jaifroid · 2024-12-01T06:47:51Z

tests/e2e/spec/tonedear.e2e.spec.js

+                const androidLink = await driver.findElement(By.css('a[href="android-ios-ear-training-app"]'));
+
+                // Verify that the element is found
+                console.log('Android & iOS App link found:', androidLink !== null);


Rather than using console.log here, make this into a test and assert that androidLink !== null. If it is null, then the test would fail. The formatting would be better in the log too.

Jaifroid · 2024-12-01T06:48:51Z

tests/e2e/spec/tonedear.e2e.spec.js

+                await driver.wait(until.elementIsVisible(androidLink), 10000); // Wait until it's visible
+                await androidLink.click();
+
+                // Take a screenshot after clicking for debugging


Since we have videos taken on BrowserStack, I think this screenshotting probably isn't necessary. Best to remove it.

…different browsers

Signed-off-by: THEBOSS0369 <[email protected]>

THEBOSS0369 · 2024-12-02T05:23:37Z

@Jaifroid ! Sir I have done all the changes as you asked, i have run the tests on firefox all test has passed but i am on windows so i can't install safari to test the tests on it 😓.

I have added the tests in all the files as you told me but in the Edge 18 file i added the tests in the previous commit can you please take a look at it and let me know if that's the correct way or not or anything missing. Thanks

THEBOSS0369 · 2024-12-02T05:34:25Z

Few Unnecessary files have come along with the commit I will remove them in few hours after coming back from college please don't mind them 😅😅

Edit : Sir i've removed those unnecessary files.

Selenium ui test

6859ab9

THEBOSS0369 mentioned this pull request Nov 11, 2024

Add an automated Selenium UI test for a small Zimit2 archive #1205

Open

Jaifroid assigned THEBOSS0369 Nov 13, 2024

Jaifroid added the tests label Nov 13, 2024

Jaifroid added this to the v4.2 milestone Nov 13, 2024

Added android test

f1c5ec1

Jaifroid requested changes Nov 16, 2024

View reviewed changes

New test

66aeef5

THEBOSS0369 force-pushed the selenium-test branch from aa8674e to 66aeef5 Compare November 17, 2024 06:57

Fixed all the tests

badde6f

Merge branch 'main' into selenium-test

b8543f0

THEBOSS0369 added 2 commits November 18, 2024 10:40

Fixed the failing tests

f0b17ae

Merge remote-tracking branch 'origin/selenium-test' into selenium-test

3cfc527

Update tonedear.e2e.spec.js

92727c3

THEBOSS0369 added 5 commits November 18, 2024 11:48

Reducing Time Wait

f7916ca

Adding tests in every files

ec085d0

Merge remote-tracking branch 'origin/selenium-test' into selenium-test

cb7f22b

Signed-off-by: THEBOSS0369 <[email protected]>

Fixing tests again

1bc38fd

Fixing tests again 2

d381a37

Increased Time Out for images verififcation

db01d15

Jaifroid reviewed Nov 19, 2024

View reviewed changes

THEBOSS0369 added 3 commits November 19, 2024 16:19

Fixing tests

5a6b02f

Merge remote-tracking branch 'origin/selenium-test' into selenium-test

f349db4

Update tonedear.e2e.spec.js

f2ebf5f

Jaifroid reviewed Dec 1, 2024

View reviewed changes

THEBOSS0369 added 2 commits December 2, 2024 10:44

Removing of Dialogue box which fails the tests & increasing tests on …

cb14eef

…different browsers

Merge remote-tracking branch 'origin/selenium-test' into selenium-test

1e5e6dd

Signed-off-by: THEBOSS0369 <[email protected]>

Delete tests/e2e/spec/tonedear.js

5a53ffe

THEBOSS0369 added 2 commits December 2, 2024 17:20

Remove unwanted image files

06c33dc

Merge remote-tracking branch 'origin/selenium-test' into selenium-test

088e354

Added an automated Selenium UI test for a small Zimit2 archive #1286

Are you sure you want to change the base?

Added an automated Selenium UI test for a small Zimit2 archive #1286

Conversation

THEBOSS0369 commented Nov 11, 2024 • edited Loading

Description

Test

Screenshot for tests

THEBOSS0369 commented Nov 11, 2024 • edited Loading

Jaifroid commented Nov 13, 2024 • edited Loading

THEBOSS0369 commented Nov 15, 2024 • edited Loading

Jaifroid commented Nov 15, 2024

THEBOSS0369 commented Nov 16, 2024 • edited Loading

Jaifroid commented Nov 16, 2024

Jaifroid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

THEBOSS0369 commented Nov 17, 2024 • edited Loading

Jaifroid commented Nov 17, 2024

THEBOSS0369 commented Nov 17, 2024

THEBOSS0369 commented Nov 17, 2024

Jaifroid commented Nov 17, 2024

THEBOSS0369 commented Nov 17, 2024 • edited Loading

THEBOSS0369 commented Nov 17, 2024 • edited Loading

Jaifroid commented Nov 17, 2024 • edited Loading

THEBOSS0369 commented Nov 18, 2024 • edited Loading

THEBOSS0369 commented Nov 18, 2024 • edited Loading

THEBOSS0369 commented Nov 18, 2024

Jaifroid commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

THEBOSS0369 commented Nov 19, 2024

Jaifroid commented Nov 19, 2024 • edited Loading

THEBOSS0369 commented Nov 20, 2024 • edited Loading

Jaifroid commented Dec 1, 2024

Jaifroid commented Dec 1, 2024

Jaifroid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

THEBOSS0369 commented Dec 2, 2024

THEBOSS0369 commented Dec 2, 2024 • edited Loading

THEBOSS0369 commented Nov 11, 2024 •

edited

Loading

THEBOSS0369 commented Nov 11, 2024 •

edited

Loading

Jaifroid commented Nov 13, 2024 •

edited

Loading

THEBOSS0369 commented Nov 15, 2024 •

edited

Loading

THEBOSS0369 commented Nov 16, 2024 •

edited

Loading

THEBOSS0369 commented Nov 17, 2024 •

edited

Loading

THEBOSS0369 commented Nov 17, 2024 •

edited

Loading

THEBOSS0369 commented Nov 17, 2024 •

edited

Loading

Jaifroid commented Nov 17, 2024 •

edited

Loading

THEBOSS0369 commented Nov 18, 2024 •

edited

Loading

THEBOSS0369 commented Nov 18, 2024 •

edited

Loading

Jaifroid commented Nov 19, 2024 •

edited

Loading

THEBOSS0369 commented Nov 20, 2024 •

edited

Loading

THEBOSS0369 commented Dec 2, 2024 •

edited

Loading