README.md

Composite browser tests

This folder contains some hermetic and live browser tests for the composites. See the top-level testing documentation to understand how these tests fit into our overall testing strategy.

Folder structure

Tests for each composite are contained in their own folder. Further, hermetic and live tests for each composite are contained in separate sub-folders.

• ./browser/call - Tests for CallComposite
  • ./browser/call/app - Test application used for all CallComposite tests
  • ./browser/call/hermetic - Hermetic tests for CallComposite
  • ./browser/call/live - Live tests for CallComposite
• ./browser/chat - Tests for ChatComposite
  • ./browser/chat/app - Test application used for all ChatComposite tests
  • ./browser/chat/hermetic - Hermetic tests for ChatComposite
  • ./browser/chat/live - Live tests for ChatComposite
• ./browser/callwithchat - Tests for CallWithChatComposite
  • ./browser/callwithchat/app - Test application used for all CallWithChatComposite tests
  • ./browser/callwithchat/live - Live tests for CallWithChatComposite
  • ./browser/callwithchat/hermetic - Hermetic tests for CallWithChatComposite

Both hermetic and live tests use playwright test framework. Playwright is configered via

• ../playwright.config.hermetic.ts for hmermetic tests
• ../playwright.config.live.ts for live tests

Setup

• Create .env file under packages/react-composites/tests/browser and in this file, make sure you have defined:
  • CONNECTION_STRING
  • TEST_DIR
  • SNAPSHOT_DIR
  • PLAYWRIGHT_OUTPUT_DIR
• Build all dependencies. Inside packages/react-composite, run

  rush update; rush build -t .

  (if there were some build failures, run rush update -p)
• Build the test applications.

  rushx build:e2e

Live tests

In addition, live tests require credentials to make Azure Communication Services backend API calls. Update the connection string in ./browser/.env file (set CONNECTION_STRING).

Known issues

If you are having issues with playwright detecting a browser platform try the following commands

# Install with dependancies
npx playwright install --with-deps
# Install without dependancies
npx playwright install
# Install specific browser
npx playwright install chrome

Run tests

For local development, the preferred way to run browser tests is via ../scripts/runBrowswerTests.mjs.

In packages/react-composites, run

node scripts/runBrowserTests.mjs --help

for a full description of available options. The CLI documentation includes examples that are particularly applicable to local test development.

package.json contains several scripts that invoke this helper. Those scripts are used by GitHub workflows to run the tests, but you can use the helper script directly.

Stress testing

Browser tests are inherently more flakey than unit-tests, and this flakiness has an outsized impact on CI reliability and latency. It is very important to ensure browser tests are not flakey, particularly when adding new tests, modifying tests non-trivially, or when we detect flakey tests in the CI.

For this, the helper script includes an option to run tests multiple times and report number of failures.

For example, to stress test a particular CallComposite test, select just that test to run via test.only, and then stress test the CallComposite tests

node scripts/runBrowserTests.mjs -c call -s 10

Update snapshots

Snapshots must only be updated as part of the CI pipeline. This is because the build agents may run on a different OS and have different graphics setup that your local machine (so there will be slight pixel differences between locally generated snapshots and ones generated by the CI pipeline).

To update snapshots,

• Add the update_snapshots label to your PR.
• The update-snapshots.yml GitHub action will generate new snapshots and push commit to your branch.
  • The workflow will remove the update_snapshots label from your PR. If you need to update snapshots again, you must add the label once again.

Manual updates

As mentioned above, you should not need to update snapshots manually. This section describes the tooling available for snapshot update. This tooling is also used by the GitHub action mentioned above.

scripts/runBrowswerTests.mjs provides a flag -u to update tests snapshots for the various groups of tests. package.json also provides scripts that invoke the helper script to update snapshots. If you are updating snapshots locally, directly invoke the helper.

Test development

❗Live-test are costly to run and maintain

Live tests should include smoke tests that verify basic functionality of top-level API. For detailed tests, choose a more appropriate testing strategy.

When developing (hermetic or live) browswer tests, use the following workflow:

• If making changes to the UI library code (packages/react-composites/src/* or its dependencies) or the test application, recompile the app before rerunning the browswer tests.
  • You can recompile just the application you're testing by running rushx build:e2e:chat, rushx build:e2e:call or rushx build:e2e:callwithchat as needed.
• Run the tests with node scripts/runBrowserTests.mjs.
• Once all tests pass, make sure to stress test the affected test to avoid introducing flakiness into the CI.

Conditional Compilation

Just like the rest of the UI library code, the test applications and the browser tests use conditional compilation. For a primer on conditional compilation in this repository, see the top-level docs.

Most commonly, you may want to write a test that runs only when a conditionally compiled feature is enabled or disabled. A common pattern for this is to define a helper function that uses conditional compilation, and use test.skip():

const demoFeatureEnabled = (): boolean => {
  /* @conditional-compile-remove(demo) */
  return true;
  return false;
};

test('test demo feature', async () => {
  test.skip(!demoFeatureEnabled());

  // ...
});

test('test default behavior when demo feature is disabled', async () => {
  test.skip(demoFeatureEnabled());

  // ...
});

Mobile Only tests

If you are writing a test for only on Mobile make sure to add it to a test suite that is just for mobile. They will be marked with [Mobile Only] in the suite title. If there is not a suite for the page you are testing add one for that page with the [Mobile only] in the title.

Once you have added your test to the appropriate suite use the following call to make sure it is not run on the desktop project:

test('Your test name here', async ({ pages }, testInfo) => {
    // Mobile check
    test.skip(skipTestIfDesktop(testInfo));
    '...'

Local debugging

Use node scripts/runBrowswerTests.mjs -d to debug a test locally. In debug mode the script runs the test under Playwright Inspector. Additionally, some internal test timeouts are relaxed to allow you to single-step through the tests via the inspector.

This mode requires a display device because the tests need to run in a browser with a display. In particular, this means that you can not use this mode on GitHub Codespaces.

To debug a particular test,

• Select the test to run via test.only

  test.describe('Chat Composite E2E Tests', () => {
    test.only('participant can receive message', async ({ serverUrl, page }) => {
      const messageReader = DEFAULT_FAKE_CHAT_ADAPTER_ARGS.remoteParticipants[0];
      ...

• Run the relevant test suite, usually selecting a single project: node ./scripts/runBrowserTests.mjs -c chat -p desktop -d
• Single-step through the test, record a video etc through the Inspector.

Running just specific hermetic tests

When working on a specific hermetic test be sure to use node scripts/runBrowserTests.mjs -l. This should be used because all of the live tests will be run after even though you are using test.only(...). This happens because the wrapper script calls playwright multiple times depending on which flags are given.

As well you will need to also use the flags ... -c call and ... -c chat to run just the composite tests that you are writing tests for.

Troubleshooting

• react-use-draggable-scroll.js doesn't exist

  You need run rush update -p