You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Use only first line of wpt report (servo runs wpt twice, second time only on unexpected results to detect flakes but they are logged in second wpt report in same json file; ideally we would reconcile it)
Also, when developing new webgpu features on servo I found useful to have (773f118) to only set new good expectations, or 039200b to set only those that are reported (if we do partial run of WPT tests)
The text was updated successfully, but these errors were encountered:
Thinking things through more, @sagudev, you might be interested in the workflow that we're undertaking at Mozilla. We recently added the implementation-status: backlog property to all test cases (see upstream docs on metadata), and have incrementally removed it as we've determined tests are worth blocking on in CI. We now use both tier 2 and tier 3 of Firefox CI.
I'm currently calling the migration of a test from a less stable tier to a more stable tier by removing implementation-status: backlog a "promotion". We do experiments in promoting tests according to heuristics like "promote permanently PASSing tests" (already implemented), "promote tests that aren't observed to FAIL or CRASH" (to be implemented in #109), with more to come. The workflow then becomes:
Run tests, and gather wptreport.json files.
Run update-expected as desired for backlogged tests, and create a commit.
Run update-backlog to tentatively promote tests. Commit and submit to CI as an experiment.
Check tentatively promoted tests are successful in the above experiment. Where they are not, demote them, preferably with bugs in Bugzilla.
We consider using implementation-status: backlog to be valuable because:
wptrunner has CLI support for filtering tests by implementation status, so the lift to adjust our CI was light.
It's a clear signal that a test is not yet considered valuable to run as a blocker in CI yet.
It's orthogonal to expected; sometimes, we want to model that a test should pass, but not block CI on it yet. This came up recently with bug 1897131.
Basic support for servo has landed in #92, but there are still some things we need before servo can move off #80 :
Also, when developing new webgpu features on servo I found useful to have (773f118) to only set new good expectations, or 039200b to set only those that are reported (if we do partial run of WPT tests)
The text was updated successfully, but these errors were encountered: