[🚫 Bug] "WebDriverError: unknown error: Chrome failed to start: exited abnormally" issue

[jeffb-sfdc]

I'm evaluating vscode-extension-tester, and it looks promising, but I'm running into a few issues. Namely, I can run (more or less) once, but subsequent attempts to run I get the error, "WebDriverError: unknown error: Chrome failed to start: exited abnormally".

I searched for this issue, and I see that it's been reported in the past, but the issues have been closed. I've done a lot of A-B testing to try and troubleshoot this and narrow down the issue, but I'm still stumped. Here's what I'm experiencing...

1. After running npm install --save-dev vscode-extension-tester and then cloning vscode-extension-tester,

2. I cd into vscode-extension-tester/sample-projects/

3. When I ran ls -ll, only helloworld-sample is listed (this is important later)

4. I then cd into helloworld-sample

5. I then run npm install

6. I then run npm run ui-test

7. At this point, the resources were dowloaded and the tests started to run

8. I'm then presented with:

 WARNING  LICENSE.md, LICENSE.txt or LICENSE not found
Do you want to continue? [y/N]

Issue 1: OK, this is a minor annoyance, but it would be nice if the repro had a LICENSE.md file, so I'm not prompted everytime I run.

I press Y and continue...

9. VS Code opens, and the tests run...

10. The "Get a view control and open its associated view" test passes, but...

11. Issue 2: the "Shows explorer view control (container)" test fails with the following error:

 1) Activity Bar Example Tests
       Shows explorer view control (container):
     StaleElementReferenceError: stale element reference: element is not attached to the page document
  (Session info: chrome=100.0.4896.160)
      at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:522:15)
      at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:589:13)
      at Executor.execute (node_modules/selenium-webdriver/lib/http.js:514:28)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Driver.execute (node_modules/selenium-webdriver/lib/webdriver.js:740:17)

OK, at this point what I see in steps 8 and 11 are minor annoyances, but now, here's the issue...

12. I then run npm run ui-test again...

Issue 3: This time, VS Code does not launch, and none of the tests run. In the console, I see this:

Launching browser...
  1) "before all" hook in "{root}"
  2) "after all" hook in "{root}"

  0 passing (778ms)
  2 failing

  1) "before all" hook in "{root}":
     WebDriverError: unknown error: Chrome failed to start: exited abnormally.
  (chrome not reachable)
  (The process started from chrome location /var/folders/rn/p50bsb8j3ml9tklgv2b2yx2w0000gp/T/extest-code/Contents/MacOS/Electron is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
      at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:522:15)
      at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:589:13)
      at Executor.execute (node_modules/selenium-webdriver/lib/http.js:514:28)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)

  2) "after all" hook in "{root}":
     TypeError: Cannot read properties of undefined (reading 'manage')
      at VSBrowser.<anonymous> (node_modules/vscode-extension-tester/out/browser.js:126:48)
      at Generator.next (<anonymous>)
      at /Users/my-user/src/github.com/redhat-developer/vscode-extension-tester/sample-projects/helloworld-sample/node_modules/vscode-extension-tester/out/browser.js:8:71
      at new Promise (<anonymous>)
      at __awaiter (node_modules/vscode-extension-tester/out/browser.js:4:12)
      at VSBrowser.quit (node_modules/vscode-extension-tester/out/browser.js:125:16)
      at Context.<anonymous> (node_modules/vscode-extension-tester/out/suite/runner.js:81:35)
      at Generator.next (<anonymous>)
      at /Users/my-user/src/github.com/redhat-developer/vscode-extension-tester/sample-projects/helloworld-sample/node_modules/vscode-extension-tester/out/suite/runner.js:8:71
      at new Promise (<anonymous>)

I spent a fair amount of time this afternoon trying to figure this out, and attempted things like deleting the files in test-resources, but what I found was this...

After ui-tests successfully runs (more or less) the first time, in vscode-extension-tester/sample-projects, if I run ls -ll, I get:

srwxr-xr-x   1 my-user  staff     0B Aug 17 16:12 helloworld-samp
drwxrwxr-x@ 18 my-user  staff   576B Aug 17 16:13 helloworld-sample

So there's an extra file, named "helloworld-samp" that's being created.

I've found is that if I...

1. run npm run ui-test (first time)
2. run rm ../helloworld-samp
3. then run npm run ui-test
   ...that it works (well, the "Shows explorer view control (container)" test still fails, but at least I'm able to run and I don't get the "WebDriverError: unknown error: Chrome failed to start: exited abnormally" error.

...and then I repeat #2 and #3

Curious as to what the "s" file attribute is, I read here (https://askubuntu.com/questions/919952/what-does-the-s-in-srwxr-xr-x-in-the-ls-output-mean) that the "s" stands for socket. So it looks like during the test, something is opening a socket called "helloworld-samp", saving it in vscode-extension-tester/sample-projects, and then not cleaning up.

If it matters, I'm running this locally on a mac OSX x64 machine.

[infiniteregrets]

@djelinek This is happening on every CI run for me when I update to 5.7.0, maybe that is related?

[djelinek]

@djelinek This is happening on every CI run for me when I update to 5.7.0, maybe that is related?

@infiniteregrets is it hapenning for you also on macOS as describes this issue?

[infiniteregrets]

@djelinek This is happening on every CI run for me when I update to 5.7.0, maybe that is related?

@infiniteregrets is it hapenning for you also on macOS as describes this issue?

oh oops, its on an ubuntu runner in the ci as described by this comment #214 (comment), do you want me to open another issue?

[djelinek]

@djelinek This is happening on every CI run for me when I update to 5.7.0, maybe that is related?

@infiniteregrets is it hapenning for you also on macOS as describes this issue?

oh oops, its on an ubuntu runner in the ci as described by this comment #214 (comment), do you want me to open another issue?

no necessarily, thanks. I think it will be somehow connected anyway.

[infiniteregrets]

@djelinek This is happening on every CI run for me when I update to 5.7.0, maybe that is related?

@infiniteregrets is it hapenning for you also on macOS as describes this issue?

oh oops, its on an ubuntu runner in the ci as described by this comment #214 (comment), do you want me to open another issue?

no necessarily, thanks. I think it will be somehow connected anyway.

hi @djelinek, is this issue planned for resolution any time soon? if not, let me know ill start looking for a fix as its making our ci very flaky, happens only 5.7.0 for me, so probably need to do git bisect there

[djelinek]

@infiniteregrets Hello, I can plan for NEXT release but anyway summer is coming and I cannot ensure when I will be able to investigate more and provide some solution. This problem is a bit tricky,

But I am a bit curious why it is happening so often to you because I did not any other feedback someone else is having problems, neither our own CI with tests.

Could you provide me a bit more info about extension and your CI environment in general? (into DM if you will feel more comfortable to share that information). Also what nodeJS are you using? Node 16? (which is the only supported currently)

Note: Could you please also test with latest vscode-extension-tester 5.8.0 ?

[infiniteregrets]

@infiniteregrets Hello, I can plan for NEXT release but anyway summer is coming and I cannot ensure when I will be able to investigate more and provide some solution. This problem is a bit tricky,

But I am a bit curious why it is happening so often to you because I did not any other feedback someone else is having problems, neither our own CI with tests.

Could you provide me a bit more info about extension and your CI environment in general? (into DM if you will feel more comfortable to share that information). Also what nodeJS are you using? Node 16? (which is the only supported currently)

Note: Could you please also test with latest vscode-extension-tester 5.8.0 ?

Hi @djelinek if you can plan it for the NEXT then it would be great! The extension is available here https://github.com/metalbear-co/mirrord-vscode a failed run with 5.8 https://github.com/metalbear-co/mirrord-vscode/actions/runs/5641704113/job/15280194281#step:8:137

[infiniteregrets]

Also I am trying to use my fork, but cant understand how to make it work, keep getting Cannot find module 'vscode-extension-tester' or its corresponding type declarations when I add "vscode-extension-tester": "github:infiniteregrets/vscode-extension-tester#input-box", to my dev dependencies, any ideas how to use my fork? perhaps any steps i am missing?

[nadiaivc]

I had the same problem on remote linux machine (I connected to it via vscode remote ssh ext). Don't be me and don't forget to connect to the remote machine via, for example, vncserver (it fixed the problem). Otherwise, how will the display be created if it has not been configured in advance...

[infiniteregrets]

@djelinek any updates on this?

[djelinek]

Hello, actually not much..

• I've found out that some workaround should be to define different folder for test resources then default one.. eg --storage tmp-resources.
• also I have tested that the latest NodeJS version can be now used 18.15.x which can maybe bring some bug fixes too
• we are now using latest version of selenium-webdriver which could bring some better stability too

but I did not have find out yet the invalid socket file is coming from...

[infiniteregrets]

Hello, actually not much..

• I've found out that some workaround should be to define different folder for test resources then default one.. eg --storage tmp-resources.
• also I have tested that the latest NodeJS version can be now used 18.15.x which can maybe bring some bug fixes too
• we are now using latest version of selenium-webdriver which could bring some better stability too

but I did not have find out yet the invalid socket file is coming from...

i have pretty much tried all of these, no progress.. were you able to run git bisect? since this happens somewhere in a commit bw 5.6.0 to 5.7.0

[djelinek]

Hello, actually not much..

• I've found out that some workaround should be to define different folder for test resources then default one.. eg --storage tmp-resources.
• also I have tested that the latest NodeJS version can be now used 18.15.x which can maybe bring some bug fixes too
• we are now using latest version of selenium-webdriver which could bring some better stability too

but I did not have find out yet the invalid socket file is coming from...

i have pretty much tried all of these, no progress.. were you able to run git bisect? since this happens somewhere in a commit bw 5.6.0 to 5.7.0

tbh I missed somehow this information that we know this issue started to occurs between some versions.. let me check properly what could change! thank you @infiniteregrets !

[djelinek]

no progress there. I have looked into changes between 5.6.0 and 5.7.0 releases and did not find related changes which would allow me to reproduce something.

I have opened bug on Selenium project, so you can track and comment also there - issue #12947

[djelinek]

okey... I have finally found something promising because for me it started to fail also for vscode unit tests with similar issue - see microsoft/vscode#86382

I have tested locally for our internal test/test-project and sample-projects/helloworld-sample which I was able to reproduce this failure and after I changed the resources folder to use shorter store PATH (it is an outcome of the linked issue above) it seems the invalid socket file is no more generated...

could you please verify @infiniteregrets if you will change the storage of your tests to e.g. --storage /tmp/test-resources will that fix the issue for you too?

[infiniteregrets]

@djelinek this is how I ran my tests:

import * as path from 'path';
import { ExTester, ReleaseQuality } from 'vscode-extension-tester';

// Note: we are using our fork https://github.com/infiniteregrets/vscode-extension-tester/tree/input-box 
// which allows to increase timeout for InputBox.create(), to be removed when
// https://github.com/redhat-developer/vscode-extension-tester/issues/485#issuecomment-1648050797 is fixed

async function main(): Promise<void> {
    const version = "latest";
    const testPath = path.join(__dirname, 'e2e.js');
    const storageFolder = "/tmp/tmp-resources";
    const extFolder = path.join(__dirname, '..', 'extensions');
    
    // required extension for debugging a python file
    const requiredExtension = "ms-python.python";

    try {
        console.log(`Running tests from ${testPath}`);
        const exTester = new ExTester(storageFolder, ReleaseQuality.Stable, extFolder);
        await exTester.downloadCode(version);
        await exTester.installVsix({ useYarn: false });
        await exTester.installFromMarketplace(requiredExtension);
        await exTester.downloadChromeDriver(version);
        const result = await exTester.runTests(testPath, {
            vscodeVersion: version,
            resources: [storageFolder],
        });

        process.exit(result);
    } catch (err) {
        console.log(err);
        process.exit(1);
    }
}

main();

changed the storage to "/tmp/tmp-resources"; and it did not make a difference. See the attached run: https://github.com/infiniteregrets/mirrord-vscode/actions/runs/6509189181/job/17680165875?pr=3#step:17:630

[djelinek]

@infiniteregrets just one thing before I will continue... could you please try to rerun you pipeline with NodeJS version 18.15.0 at max? because I saw your pipeline is running with 18.18.1 (latest LTS) which is currently not supported by vscode-extension-tester (neither selenium-webdriver)

[infiniteregrets]

@infiniteregrets just one thing before I will continue... could you please try to rerun you pipeline with NodeJS version 18.15.0 at max? because I saw your pipeline is running with 18.18.1 (latest LTS) which is currently not supported by vscode-extension-tester (neither selenium-webdriver)

on it!

[infiniteregrets]

@djelinek I switched to latest on the project and it seems to be working now. Thank you so much for your help.

also I have tested that the latest NodeJS version can be now used 18.15.x which can maybe bring some bug fixes too

Sorry I missed this one while testing earlier

[infiniteregrets]

@djelinek sorry I think it passed in one run but it failed in other:

https://github.com/infiniteregrets/mirrord-vscode/actions/runs/6541317891/job/17762663963?pr=4#step:17:601

i am using latest as my node version

[djelinek]

@djelinek sorry I think it passed in one run but it failed in other:

https://github.com/infiniteregrets/mirrord-vscode/actions/runs/6541317891/job/17762663963?pr=4#step:17:601

i am using latest as my node version

not the latest.. you need to use NodeJS v18.15.x

[swathisuja]

HI @djelinek

I am seeing this issue in Linux Ubuntu 22.04
Node 18.20.0 and also with 20.12.1
yarn 1.22.19

1. "before all" hook in "{root}"
2. "after all" hook in "{root}"

0 passing (243ms)
2 failing

1. "before all" hook in "{root}":
   SessionNotCreatedError: session not created: Chrome failed to start: exited normally.
   (session not created: DevToolsActivePort file doesn't exist)
   (The process started from chrome location /home/workspace/resources/VSCode-linux-x64/code is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
   at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:521:15)
   at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:514:13)
   at Executor.execute (node_modules/selenium-webdriver/lib/http.js:446:28)
   at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

2. "after all" hook in "{root}":
   TypeError: Cannot read properties of undefined (reading 'manage')
   at VSBrowser.quit (node_modules/vscode-extension-tester/out/browser.js:174:44)
   at Context. (node_modules/vscode-extension-tester/out/suite/runner.js:126:31)
   at process.processImmediate (node:internal/timers:478:21)

Unable to run tests as it fails at launching browser. Can you help fix this as we have this on jenkins pipeline for our project validation.

Thanks,
Swathi