Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test,tools: update wpt weekly #50567

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

legendecas
Copy link
Member

@legendecas legendecas commented Nov 5, 2023

To keep the Web API behavior synchronized with the latest Web specs and
WPT tests, add a GitHub Action to automatically update WPT fixtures with
WPT weekly epoch. The status files are updated with the test results of
the main branch as well.

Example of automatic PR: legendecas#3

To keep the Web API behavior synchronized with the latest Web specs and
WPT tests, add a GitHub Action to automatically update WPT fixtures with
WPT weekly epoch. The status files are updated with the test results of
the main branch as well.
@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/actions
  • @nodejs/security-wg

@nodejs-github-bot nodejs-github-bot added build Issues and PRs related to build files or the CI. meta Issues and PRs related to the general management of the project. needs-ci PRs that need a full CI run. tools Issues and PRs related to the tools directory. labels Nov 5, 2023
@legendecas legendecas requested a review from panva November 6, 2023 09:18
Copy link
Member

@panva panva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am -1 on automated WPT fixture updates. They change the status json expectations without notice to the subsystem maintainers, making tests skipped and lowering our chance of finding regressions.

This could get bundled with automated issue creation for newly skipped tests but I think that would create too much noise resulting in unresolved and ignored bot issues anyway.


What about, instead of this, have the daily WPT job do something similar to what's done in nodejs/reliability and create / update a single issue with a list of unexpected passes & unexpected failures, that's effectively a link to this query

@jasnell
Copy link
Member

jasnell commented Nov 11, 2023

+1 to @panva's suggestion

Comment on lines +66 to +78
- uses: gr2m/create-or-update-pull-request-action@77596e3166f328b24613f7082ab30bf2d93079d5
# Creates a PR or update the Action's existing PR, or
# no-op if the base branch is already up-to-date.
env:
GITHUB_TOKEN: ${{ secrets.GH_USER_TOKEN }}
with:
author: Node.js GitHub Bot <[email protected]>
body: This is an automated patch update of wpt to https://github.com/web-platform-tests/wpt/commit/${{ env.WPT_REVISION }}.
branch: actions/update-wpt # Custom branch *just* for this Action.
commit-message: 'deps: update wpt to ${{ env.WPT_SHORT_REVISION }}'
labels: test
title: 'deps: update wpt to ${{ env.WPT_SHORT_REVISION }}'
update-pull-request-title-and-body: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this log the failed tests in the PR body?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need reviews on #54460

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Issues and PRs related to build files or the CI. meta Issues and PRs related to the general management of the project. needs-ci PRs that need a full CI run. tools Issues and PRs related to the tools directory.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants