Add smoke-test workflow #103

uuf6429 · 2025-01-08T23:29:17Z

Purpose

The objective is to provide an extendable workflow for testing basic compatibility* multiple drivers in isolation.

* latest drivers and oldest supported dependencies

Third-party drivers can also add themselves, but will be treated a little bit differently, to avoid failures reflecting badly on our CI results (since GitHub doesn't allow us to ignore failures).

Current Driver Tests

✔️ BrowserKit (http client)
✔️ BrowserKit (http kernel)
🚧 Selenium2
- Inform developers about non-working "rightClick" on Selenium Server 3.x MinkSelenium2Driver#407
- Fix D&D especially on Selenium 3 MinkSelenium2Driver#408
🚧 Chrome
- many failures without a seemingly good reason (their CI passes fine though)
- (it's worth noting that their custom tests use data uris instead of a webserver)
✔️ Classic
♻️ Sahi / Zombie / Goutte - deprecated/abandoned
❓ Panther
❓ Electron

.github/workflows/smoke.yml

aik099 · 2025-01-09T08:04:23Z

Selenium2
right-click test failed, no idea why

That's why it fails: minkphp/MinkSelenium2Driver#381 . I have no idea how to fix it using https://github.com/instaclick/php-webdriver . That is the only blocker before that driver's release.

aik099 · 2025-01-09T09:15:57Z

Failures for the MinkBrowserKitDriver are fixed in minkphp/MinkBrowserKitDriver#183 .

stof · 2025-01-09T10:37:05Z

I suggest removing the job for the third-party driver. I don't think it makes sense to make our CI job fail based on their behavior which is out of our control, especially given that they fail many tests currently.

uuf6429 · 2025-01-09T22:51:29Z

I suggest removing the job for the third-party driver. I don't think it makes sense to make our CI job fail based on their behavior which is out of our control, especially given that they fail many tests currently.

@stof @aik099 I'm not strongly in favour of keeping it, but hear out my arguments:

we probably care that our changes do not impact them
there are numerous ways to configure the workflow in a way it does not reflect bad on us, eg:
- make the workflow as manually-triggered only so we only run it on demand (this was my original intention, but github doesn't let me run workflows manually before they have been merged to master, so I had to add the "pull_request" trigger)
- marking jobs as allowed-to-fail
we may request the vendor to maintain that part (with normal PRs) and if we see it doesn't happen, we can eventually just remove it
adding to point 3, we can explicitly ask vendors to participate, and if they don't, they won't be guaranteed backward compatibility (which makes maintenance easier for us)
the main objective of this workflow is to avoid making a local setup that is time consuming, non-reproducible and probably replicates other maintainers' efforts on their own machines.

Let me know if any of that convinces you and if still not convinced, I'll gladly remove this.

aik099 · 2025-01-10T08:25:47Z

We can talk to 3rd party driver manufacturers to see if they're willing for their driver to be compatible by our test suite changes.

Previously, GitHub Actions wasn't allowing ignoring build failures, which affected the CI badge in general on the README.md file. If you know how to allow a specific job within the build to fail without making the whole build fail but still showing that that job has failed (like in Travis CI times), then please do.

So let's try. Trying doesn't cost us anything.

uuf6429 · 2025-01-10T23:57:43Z

Previously, GitHub Actions wasn't allowing ignoring build failures, which affected the CI badge in general on the README.md file.

@aik099 apparently it's still not possible without some workarounds: actions/runner#2347 😞

Then the remaining option is (as said) a manual job not triggered by CI (or workaround in the linked issue).

aik099 · 2025-01-11T09:41:00Z

Previously, GitHub Actions wasn't allowing ignoring build failures, which affected the CI badge in general on the README.md file.

@aik099 apparently it's still not possible without some workarounds: actions/runner#2347 😞

Then the remaining option is (as said) a manual job not triggered by CI (or workaround in the linked issue).

@uuf6429 , then we'll do 2 jobs:

one job for drivers located in the minkphp organization, that will run automatically
another job for 3rd-party drivers (one per driver or maybe a combo job), that driver maintainers are welcome to adjust (we need to inform driver maintainers about such feature)

aik099 · 2025-01-11T11:48:41Z

It's strange that MinkSelenium2Driver job fails with Drag drop onto hidden itself test, when it doesn't fail on MinkSelenium2Driver repo test.

uuf6429 · 2025-01-11T12:22:08Z

I think I've seen that failure in the past...could it be transient/random? Unfortunately I can't retry that job without pushing something to run the whole workflow.

aik099 · 2025-01-11T14:02:21Z

... Unfortunately I can't retry that job without pushing something to run the whole workflow.

@uuf6429 , I've granted you the same permissions for this repository as on the webdriver-classic-driver repository. Now you should be able to retry builds, assign reviewers, merge, etc.

Enjoy.

uuf6429 · 2025-01-11T14:18:18Z

@uuf6429 , I've granted you the same permissions for this repository as on the webdriver-classic-driver repository. Now you should be able to retry builds, assign reviewers, merge, etc.

Awesome, I can stop that job that has been running for 5 hours 😅
https://github.com/minkphp/driver-testsuite/actions/runs/12722948823/job/35467568743

uuf6429 · 2025-01-17T19:35:01Z

.github/workflows/ci.yml

+              docker compose logs --no-color --no-log-prefix --timestamps selenium > ./logs/selenium.docker.log
+
+  Unofficial:
+    name: "✨️ ${{ matrix.name }}"


I just pushed a major update; it should work much better now and should be relatively easy to follow.

There's one major annoyance (on GitHub's side): they do not evaluate expressions in job.name, (that line above), when the job is skipped.

This causes some ugly UI:

But the impact is just that, so I think we can live with it.

https://github.com/orgs/community/discussions/13261

job.name expression not evaluated when job skipped actions/runner#1215

aik099 · 2025-01-18T11:23:05Z

.github/workflows/ci.yml

+          - name: "Selenium2"
+            driverRepoUrl: "https://github.com/minkphp/MinkSelenium2Driver.git"
+            php: "7.2"
+            # language=bash
+            setUpCmd: |
+              export SELENIUM_IMAGE=selenium/standalone-firefox:2.53.1
+              docker compose up --wait --quiet-pull
+              curl --retry 5 --retry-all-errors --retry-delay 1 --max-time 10 --head -X GET http://localhost:4444/wd/hub/status
+            # language=bash
+            testCmd: |
+              export WEB_FIXTURES_BROWSER=firefox
+              export DRIVER_MACHINE_BASE_PATH=/fixtures/
+              export WEB_FIXTURES_HOST=http://host.docker.internal:8002
+              ./vendor/bin/phpunit --colors=always --testdox
+            # language=bash
+            tearDownCmd: |
+              docker compose logs --no-color --no-log-prefix --timestamps selenium > ./logs/selenium.docker.log


Could you please update this part to reflect build script changes made in the minkphp/MinkSelenium2Driver#406?

I'm not sure what you mean, it's already using docker compose to extract logs (and the setUpCmd also uses it).

Ah, I thought that you'd copy-pasted code fragments from the driver's build script. Apparently you didn't. Anyway, there is now a docker-compose.yml file in the driver's repository that you could use.

I wonder how you're able to correctly start Selenium's Docker image with only docker compose up --wait --quiet-pull command instead of docker run --net host --name selenium --volume /dev/shm:/dev/shm --volume ./vendor/mink/driver-testsuite/web-fixtures:/fixtures --shm-size 2g selenium/standalone-firefox:${{ matrix.selenium_version }} &> ./logs/selenium.log & and still make fixtures discoverable from the Selenium ?

Ah, I thought that you'd copy-pasted code fragments from the driver's build script. Apparently you didn't.

Actually, I did. That's why before it was using docker run and then after your change I updated it to docker compose up.

I wonder how you're able to correctly start Selenium's Docker image with only...

All the parts you mentioned after docker run are in the docker-compose.yml file. So docker compose up is effectively doing the same thing as docker run.

Note that docker-compose up has been replaced with docker compose up (in case that's unclear).

Also keep in mind that this workflow works differently - normally in most CI workflows you git clone your repository, but it this workflow, I'm cloning the driver repository. So if the driver has its own docker-compose file or some special testing process, we can still call/use it in our workflow.

uuf6429 added 5 commits January 9, 2025 00:28

Add smoke-test workflow

e384263

Minor adjustments

c9568b6

Use oldest supported zombie driver

526cd83

Set node version

5156733

Use even older crap

85524bf

This comment was marked as resolved.

Sign in to view

Roll back 9 year old downgrade

22a2728

This comment was marked as resolved.

Sign in to view

Remove zombie driver

53c41cc

aik099 reviewed Jan 9, 2025

View reviewed changes

uuf6429 added 2 commits January 9, 2025 09:21

Various fixes

b5ce5bc

dvfgsafgnhdbgdbfndnfghz

6fa5925

aik099 mentioned this pull request Jan 9, 2025

PHPStan improvements #100

Merged

uuf6429 added 2 commits January 11, 2025 10:08

Limit minkse2driver to selenium 2

fbf3233

Fix infinite loops; fix selenium image

3827d67

uuf6429 added 2 commits January 11, 2025 11:09

Fix many problems

168e58d

Fix job name

719d090

Various fixes; merge workflows

d868dd1

uuf6429 commented Jan 17, 2025

View reviewed changes

aik099 reviewed Jan 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add smoke-test workflow #103

Add smoke-test workflow #103

uuf6429 commented Jan 8, 2025 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

aik099 commented Jan 9, 2025

aik099 commented Jan 9, 2025

stof commented Jan 9, 2025

uuf6429 commented Jan 9, 2025 •

edited

Loading

aik099 commented Jan 10, 2025

uuf6429 commented Jan 10, 2025 •

edited

Loading

aik099 commented Jan 11, 2025 •

edited

Loading

aik099 commented Jan 11, 2025

uuf6429 commented Jan 11, 2025

aik099 commented Jan 11, 2025

uuf6429 commented Jan 11, 2025 •

edited

Loading

uuf6429 Jan 17, 2025 •

edited

Loading

aik099 Jan 18, 2025

uuf6429 Jan 18, 2025

aik099 Jan 18, 2025

uuf6429 Jan 18, 2025 •

edited

Loading

Add smoke-test workflow #103

Are you sure you want to change the base?

Add smoke-test workflow #103

Conversation

uuf6429 commented Jan 8, 2025 • edited Loading

Purpose

Current Driver Tests

This comment was marked as resolved.

This comment was marked as resolved.

aik099 commented Jan 9, 2025

aik099 commented Jan 9, 2025

stof commented Jan 9, 2025

uuf6429 commented Jan 9, 2025 • edited Loading

aik099 commented Jan 10, 2025

uuf6429 commented Jan 10, 2025 • edited Loading

aik099 commented Jan 11, 2025 • edited Loading

aik099 commented Jan 11, 2025

uuf6429 commented Jan 11, 2025

aik099 commented Jan 11, 2025

uuf6429 commented Jan 11, 2025 • edited Loading

uuf6429 Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

aik099 Jan 18, 2025

Choose a reason for hiding this comment

uuf6429 Jan 18, 2025

Choose a reason for hiding this comment

aik099 Jan 18, 2025

Choose a reason for hiding this comment

uuf6429 Jan 18, 2025 • edited Loading

Choose a reason for hiding this comment

uuf6429 commented Jan 8, 2025 •

edited

Loading

uuf6429 commented Jan 9, 2025 •

edited

Loading

uuf6429 commented Jan 10, 2025 •

edited

Loading

aik099 commented Jan 11, 2025 •

edited

Loading

uuf6429 commented Jan 11, 2025 •

edited

Loading

uuf6429 Jan 17, 2025 •

edited

Loading

uuf6429 Jan 18, 2025 •

edited

Loading