Metamask approve doesnt work in GH actions

[conwayconstar]

Describe the bug
I am trying to run the tests in CI with github actions. It completes the tests up to the point where it needs to connect to meta mask.

Bellow is my code for the test.

const connectBtn = page.getByRole("button", { name: "Connect" }).first();

await expect(connectBtn).toBeVisible();
await expect(connectBtn).toBeEnabled();
await connectBtn.click();

const metaMaskBtn = page.getByTestId("rk-wallet-option-metaMask");
await expect(metaMaskBtn).toBeVisible();
await expect(metaMaskBtn).toBeEnabled();
await metaMaskBtn.click();
await wallet.approve();

The script i run in CI uses xvfb-run

{
 "scripts": {
    "test": "playwright test -c src/playwright.config.ts",
    "test:ci": "xvfb-run --auto-servernum yarn test",
    "typecheck": "tsc --noEmit",
    "format": "biome format . --write",
    "lint": "biome check ."
  },
}

My workflow file runs this task

 - name: Run tests
   env:
       ANVIL_BLOCK_NUMBER: ${{ secrets.ANVIL_BLOCK_NUMBER }}
       ANVIL_FORK_URL: ${{ secrets.ANVIL_FORK_URL }}
       SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
   run: yarn test:ci

Logs

Error: page.waitForSelector: Timeout 1000ms exceeded.
Call log:
  - waiting for locator('.mm-box--color-error-default') to be visible

Expected behavior
Its supposed to pass the test for accepting the request.

Screenshots
I have added

System:

• OS: Ubuntu
• OS version latest
• NodeJs version 18
• dAppwright version latest
• testing framework playwright
• testing framework version 1.40.1

Additional context
I have attached my traces view them with this link https://trace.playwright.dev
trace.zip

[conwayconstar]

All good, I started using the playwright container rather than installing it in the GH action worked perfectly!

[conwayconstar]

Reopening as it fails sometimes and its to do with this:

Error: page.waitForSelector: Timeout 1000ms exceeded.
Call log:
  - waiting for locator('.mm-box--color-error-default') to be visible

I believe its the getErrorMessage function

[osis]

oh interesting. Something to look into for sure.

[osis]

@conwayconstar there has been a couple of issues solved that may have fixed this. Still having the issue?

[marcinlesek]

@conwayconstar there has been a couple of issues solved that may have fixed this. Still having the issue?

I'm having it still in GitHub Actions. I'm installing Playwright directly in the actions container:

os: ubuntu
nodejs: 21.7.3
dappwright: 2.8.2
playwright: 1.43.1
metamask: 11.10.0

You can check out code in my PR and check out actions results.

Any tips on that one @osis @conwayconstar?

[sunwrobert]

I'm also noticing this issue breaking in CI where wallet.approve just doesn't work :/. It ends up just hanging on wallet for the new extension page. @osis any ideas?

I can upload my Playwright trace if that helps

[osis]

Yea, the GitHub actions behaviour has been a bit tricky to track down. There are some known behavioural issues with playwright. Are you saying that it's waiting for the pop up context?

[sunwrobert]

Yeah that seems to be the problem, hangs on waiting for the pop up context.

Are there currently any E2E tests in this Repo that run in GH actions?

[mbarr1111]

still cannot get it working on github actions, is someone working on this repo or is archived?

[osis]

still cannot get it working on github actions, is someone working on this repo or is archived?

Feel free to PR a fix if you know where the flow is breaking. Otherwise it's hard to debug without context/logs.

The flow shown above isn't necessarily what approve was meant for so would love to understand the use case so we can expand the test suite to cover this.

[impelcrypto]

Hi @osis
I'm having the same issue here, the test works on my local but not on CI with GitHub actions. Here I have created a demo app for you to test it out. Please help take a look and fix the issue. Thanks a lot!

Error log:
https://github.com/impelcrypto/portal-playground/actions/runs/9381909069/job/25832178357?pr=3

PR:
impelcrypto/portal-playground#3

Demo app repo:
https://github.com/impelcrypto/portal-playground/tree/feat/test

You can test it by:

1. Clone the repo and go to feat/test branch
2. yarn
3. yarn playwright test example.spec.ts --debug

[impelcrypto]

Hi @osis

I found out that CI works with this playwright.yml (it doesn't work if node-version: lts) and using Coinbase wallet instead.

name: Playwright Tests

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  playwright:
    name: "Playwright Tests"
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.44.1-jammy
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
      - name: Install dependencies
        run: npm install -g pnpm && pnpm install
      - name: Install Playwright Browsers
        run: pnpm exec playwright install --with-deps
      - name: Install Chrome
        run: pnpm exec playwright install chrome
      - name: Run Playwright tests
        run: xvfb-run --auto-servernum --server-args='-screen 0 1920x1080x24' pnpm exec playwright test
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30

[johanneskares]

I had the same issue: Unable to approve certain Metamask pop-ups. I created this lib, which creates a headless Browser Mock Wallet for testing: https://github.com/johanneskares/wallet-mock

[GetRaider]

Any fixes here?

[mbarr1111]

I thin can be dynamic addresses , just use a fixed wallet address

[GetRaider]

As far as I understood, it happens due to the headless mode on running on CI. As I found:
"Metamask might not work properly in headless mode (which is often the default in CI/CD environments like GitHub Actions)" - therefore, I tried to run with "headless: false" using the "xvfb-run". (the required command to run with headless on CI, the playwright doesn't allow running with "headless: false" without "xvfb".

I set the "headless: false" for dappwright.bootstrap calling and added "xvfb-run" for test running command.
My test job from CI config:
`jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.44.1-jammy
steps:
- name: Checkout
uses: actions/checkout@v4

  - name: Node install
    uses: actions/setup-node@v4
    with:
      cache: npm
      node-version: 18

  - name: Install dependencies
    run: npm ci

  - name: Run Playwright tests
    run: xvfb-run npx playwright test

  - uses: actions/upload-artifact@v4
    if: failure()
    with:
      name: playwright-report
      path: playwright-report/
      retention-days: 30`

My bootstrap call:
await dappwright.bootstrap("", { wallet: "metamask", version: MetaMaskWallet.recommendedVersion, seed: "yourSeed", headless: false, });

I hope it'll help somebody

[conwayconstar]

Hi guys long time no speak. Sorry a lot happened between then and now! I did however resolve this issue 10 months ago here was my solution!

name: Run tests
on: workflow_dispatch

jobs:
  tests:
    name: Tests
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.40.0-jammy
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install Node.js
        uses: actions/setup-node@v4
        with:
          cache: yarn
          node-version: '16'

      - name: Install dependencies
        run: yarn install --frozen-lockfile

      - name: Set up foundry
        uses: foundry-rs/foundry-toolchain@v1

      - name: Run tests
        env:
          ANVIL_BLOCK_NUMBER: ${{ secrets.ANVIL_BLOCK_NUMBER }}
          ANVIL_FORK_URL: ${{ secrets.ANVIL_FORK_URL }}
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
          GITHUB_TOKEN: ${{ github.token }}
          CI: true
        run: xvfb-run --auto-servernum yarn run test

      - name: Upload test screenshot
        uses: actions/upload-artifact@v3
        if: failure()
        with:
          name: test-results
          path: test-results
          retention-days: 30

[osis]

@conwayconstar thanks for sharing! Do you know what in particular in this example worked for you?

[GetRaider]

@osis As far as I see, it's
xvfb-run

Because another, typical way is not working now on the GitHub actions

[osis]

ah yes. I was hoping it was something different since we use this today our GH actions but I'm not a fan of the workaround. Will continue to dream of a better way or wait for a playwright fix around this. Thanks @GetRaider!