Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce ARIA snapshot tests for all components #3271

Merged
merged 11 commits into from
Jan 16, 2025
Merged

Introduce ARIA snapshot tests for all components #3271

merged 11 commits into from
Jan 16, 2025

Conversation

khiga8
Copy link
Contributor

@khiga8 khiga8 commented Jan 16, 2025 •
edited
Loading

What are you trying to accomplish?

This PR:

  • upgrades to latest Playwright version.
  • introduces ARIA snapshot tests for all components, alongside existing visual regression tests. These ARIA snapshot tests store the role and accessible name exposed to assistive technologies.

Why is this helpful?

  • Right now, there's no great way to test accessible markup in PVC.
  • Capybara, the main testing library used in PVC, doesn't natively support any accessible name/ARIA assertions making it difficult to test the semantics of the markup.
  • Consequently, it's very easy to push a change that unintentionally ends up changing the underlying accessible markup, and breaks the experience for assistive technology users.
    • Here's an example bug (which.. I introduced 🤦‍♀️ ) where there's a regression in the accessible name.
  • By having ARIA snapshot tests in place, it should be much easier to detect changes in the accessibility markup, and become easier to catch regressions.

Consequences

  • Additional build time

Integration

List the issues that this change affects.

Closes # (type the GitHub issue number after #)

Risk Assessment

  • Low risk the change is small, highly observable, and easily rolled back.
  • Medium risk changes that are isolated, reduced in scope or could impact few users. The change will not impact library availability.
  • High risk changes are those that could impact customers and SLOs, low or no test coverage, low observability, or slow to rollback.

What approach did you choose and why?

Anything you want to highlight for special attention from reviewers?

Merge checklist

  • Added/updated tests
  • Added/updated documentation
  • Added/updated previews (Lookbook)
  • Tested in Chrome
  • Tested in Firefox
  • Tested in Safari
  • Tested in Edge

Take a look at the What we look for in reviews section of the contributing guidelines for more information on how we review PRs.

@changeset-bot changeset-bot
Copy link

changeset-bot bot commented Jan 16, 2025 •
edited
Loading

⚠️ No Changeset found

Latest commit: c20230c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@khiga8 khiga8 changed the title experiment - ariaSnapshots Experiment - Introduce ARIA snapshot tests Jan 16, 2025
@khiga8
Only generate snapshot for one theme, and add snapshot for interacted…
fe49c49 
… state

* The markup shouldn't change based on the theme so let's just generate the snapshot for one component on default theme.
* The markup may change based on interaction so add support for that.
@khiga8 khiga8 changed the title Experiment - Introduce ARIA snapshot tests Introduce ARIA snapshot tests for all components Jan 16, 2025
@khiga8 khiga8 added the skip changeset Pull requests that don't change the library output label Jan 16, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 16, 2025 •
edited
Loading

⚠️ Visual or Semantic markup differences found

Our visual and semantic markup comparison tests found UI differences. Please review the differences by viewing the files changed tab to ensure that the changes were intentional.

Review differences

kendallgassner
kendallgassner reviewed Jan 16, 2025
View reviewed changes
.playwright/screenshots/snapshots.test.ts-snapshots/primer/alpha/action_bar/default/default.yml Outdated
@@ -0,0 +1,63 @@
- toolbar:
Copy link
Contributor

@kendallgassner kendallgassner Jan 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it useful or possible to throw these snapshots into axe testing?

Copy link
Contributor

@kendallgassner kendallgassner Jan 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Essentially I am wondering if we can use these snapshots to validate that the markup is correct ... This might be a pointless question if we are already running axe on lookbook.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have Axe tests running against these previews and in interactions that take place in system tests 🎉 .

@khiga8 khiga8 force-pushed the kh-test-aria-snapshots branch from 7fbf826 to 3afd47a Compare January 16, 2025 17:17
@khiga8 khiga8 force-pushed the kh-test-aria-snapshots branch from 73c8b4b to d850a1c Compare January 16, 2025 17:38
camertron
camertron approved these changes Jan 16, 2025
View reviewed changes
Copy link
Contributor

@camertron camertron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Sounds good to me 👍

@khiga8 khiga8 marked this pull request as ready for review January 16, 2025 17:51
@khiga8 khiga8 requested review from a team as code owners January 16, 2025 17:51
@khiga8 khiga8 requested review from emilybrick and jonrohan January 16, 2025 17:51
@khiga8
Copy link
Contributor Author

khiga8 commented Jan 16, 2025

why aren't the checks kicking off 😠

@khiga8 khiga8 closed this Jan 16, 2025
@khiga8 khiga8 reopened this Jan 16, 2025
@khiga8
Copy link
Contributor Author

khiga8 commented Jan 16, 2025

I'm ignoring the broken "Demo Build & Deploy" check per #3258 (comment).

@khiga8 khiga8 merged commit 7c59b29 into main Jan 16, 2025
35 of 37 checks passed
@khiga8 khiga8 deleted the kh-test-aria-snapshots branch January 16, 2025 18:21
This was referenced Jan 16, 2025
Open
Merged
khiga8 added a commit that referenced this pull request Jan 16, 2025
@khiga8
Revert "Introduce ARIA snapshot tests for all components (#3271)"
0dbac03 
This reverts commit 7c59b29.
@khiga8 khiga8 mentioned this pull request Jan 16, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kendallgassner kendallgassner kendallgassner left review comments

@camertron camertron camertron approved these changes

@emilybrick emilybrick Awaiting requested review from emilybrick emilybrick is a code owner automatically assigned from primer/design-reviewers

@jonrohan jonrohan Awaiting requested review from jonrohan jonrohan is a code owner automatically assigned from primer/engineer-reviewers

Assignees
No one assigned
Labels
skip changeset Pull requests that don't change the library output
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@khiga8 @camertron @kendallgassner