Skip to content

Latest commit

 

History

History
151 lines (111 loc) · 5.96 KB

run_web_platform_tests.md

File metadata and controls

151 lines (111 loc) · 5.96 KB

Running Web Platform Tests with run_wpt_tests.py

run_web_tests.py runs web tests with content shell through protocol mode. See web_tests.md for details. run_wpt_tests.py is a WebDriver-based alternative that can run web platform tests with Chrome, headless shell, Chrome Android, and WebView. This document explains how to use run_wpt_tests.py in these scenarios.

[TOC]

Running Web Platform Tests for Desktop Platforms

On Linux, macOS, and Windows, run_wpt_tests.py supports testing with Chrome or headless shell. Chrome is closer to the binary Google ships to end users, but is generally slower. Headless shell is a lightweight alternative that suffices for testing features implemented entirely in Blink.

Running Tests Locally

First, you will need to build one of the following targets to get all needed binaries:

autoninja -C out/Default chrome_wpt_tests     # For testing with `chrome`
autoninja -C out/Default headless_shell_wpt   # For testing with `headless_shell`

Once the build is done, running tests is very similar to how you would run tests with run_web_tests.py. For example, to run all tests under external/wpt/html/dom, run:

third_party/blink/tools/run_wpt_tests.py --target=Default --product=headless_shell external/wpt/html/dom

--product (or -p) selects which browser to test with. Supported values are:

Also, consider using -v to get browser logs. It can be provided multiple times to increase verbosity.

run_wpt_tests.py --help shows a full description of run_wpt_tests.py's CLI, which resembles that of run_web_tests.py.

Running Tests in CQ/CI

To satisfy different testing requirements, WPT coverage in CQ/CI is partitioned between suites that target different //content embedders:

Suite Name Browser Under Test Harness Tests Run
blink_wpt_tests content_shell --run-web-tests run_web_tests.py Tests that depend on web test-specific features (e.g., internal WPTs that depend on nonstandard window.internals or window.testRunner APIs).
chrome_wpt_tests chrome --headless=new run_wpt_tests.py Tests that depend on the //chrome layer. Can be slow, so prefer headless_shell testing if possible.
headless_shell_wpt_tests headless_shell run_wpt_tests.py All other tests. Most WPTs should eventually run here.

To avoid redundant coverage, each WPT should run in exactly one suite listed above. The chrome.filter file lists tests that chrome_wpt_tests should run, and that headless_shell_wpt_tests and blink_wpt_tests should skip. headless_shell.filter works similarly. Tests not listed in either file run in blink_wpt_tests by default.

*** note Running tests in blink_wpt_tests is discouraged because run_web_tests.py doesn't drive tests through standard WebDriver endpoints. This can cause blink_wpt_tests results to diverge from the Chrome results published to wpt.fyi. You can help unblock the eventual deprecation of blink_wpt_tests by adding tests that you own to either filter file.

Test Expectations and Baselines

To suppress failures, run_wpt_tests.py uses the same *-expected.txt and TestExpectations files that run_web_tests.py uses.

Running webdriver tests with Chrome

wdspec tests are a subset of WPT that verifies conformance to the WebDriver specification. run_wpt_tests.py can run wdspec tests like any other WPT:

third_party/blink/tools/run_wpt_tests.py -t Default -p chrome \
  external/wpt/webdriver/tests/classic/find_element/find.py

On the bots, the webdriver_wpt_tests suite runs wdspec tests separately from the other WPT types. The linux-blink-rel builder can provide results for rebaselining.

Running Web Platform Tests on Android

See here for Android specific instructions.

Debugging Support

Text-Based Debuggers

To interactively debug WPTs, prefix the run_wpt_tests.py command with debug_renderer to attach a debugger to a desired renderer.

For other use cases, see these debugging tips.

FAQ

  • Do headless shell and Chrome support MojoJS bindings?
    • Yes. run_wpt_tests.py enables the MojoJS and MojoJSTest features and serves //out/<target>/gen/ as /gen/ in wptserve. However, in the public WPT suite, testdriver.js APIs must be backed by fully-specified testing APIs (preferably implemented with WebDriver or alternatively with MojoJS). Tests that rely on unspecified testing APIs cannot be put in WPT, but may live in chromium's own wpt_internal. See web-platform-tests/rfcs#172 for additional discussion.

Known Issues

The wptrunner-migration hostlist tracks test results where headless shell and content shell differ. For runner bugs and feature requests, please file an issue against Blink>Infra.