Interop 2024 Accessibility Investigation

[cookiecrook]

Separate from the Interop 2024 Accessibility Focus Area, we'll also have an investigation area that doesn't contribute to the overall Interop 2024 score.

Scoring Criteria for Interop 2024 Accessibility Testing Investigation

[Significant Edit May 7, 2024] The Accessibility Investigation Group, in a meeting today and based on the culmination of discussion below, determined to abandon the Serialized Accessibility Tree Dump path in favor of a tree walker investigation using the Accessible Node direction. The 25% scoring criteria previously dedicated to Tree Dump is now repurposed into the Accessible Node investigation, including Tree Walking features.

• (25%) Accessible Node or similar API

  • (10%) Initial spec language agreement of Accessible Node (DRI @cookiecrook)
    • Computing the Accessible Backing Node, or a similar test-only interface WICG/aom#197
    • and/or
    • Explainer: WebDriver Extension for Accessible Nodes, etc. (potential solution for #197) WICG/aom#203
  • (10%) Initial implementation of Accessible Node (behind runtime flag?) in WebKit, Chromium, Gecko
    • (5%) First Implementation
    • (3%) Second Implementation
    • (2%) Third Implementation
  • (5%) Tree Walker investigation using Accessible Node
    • (3%) Write basic tree walker tests (tentative) that purposefully avoid all known tree implementation differences (scroll areas, pseudo elements and other generated content, etc.)
    • (2%) Author investigation tests for which parts of Node or Tree might be standardized in 2025 or later?
      • (1%) Purposefully write failing tests (tentative) that expose the areas where the internal accessibility tree differ between engines
      • (1%) File AOM or ARIA issues if it appears are some tree relationships that might be standardized as part of a 2025 Investigation

• (25%) Acacia AAM test exploration (DRI @spectranaut)

  • (5%) initial prototype
  • (5%) functional in implementations/APIs
    • not calling out all combos, but as examples:
      • Mac AX API in WebKit, Chromium, Gecko
      • Linux GTK in Chromium, Gecko
      • Windows UIA in Chromium, Gecko
      • Others?
        • Window MSAA? in Chromium, Gecko
        • Window IA2? in Chromium, Gecko
  • (5%) RFC consensus?
  • (5%) functional in WPT CI?
  • (5%) additional test writing?

• (30%) Documentation

  • (5%) Initial call-for-feedback in Google Docs draft
  • (25%) Accessibility test standardization, improvements and documentation #83 (DRI @rahimabdi)
    • (10%) Incorporate feedback from reviewers
    • (15%) Deliver/publish final v1 standalone documentation

• (20%) Miscellaneous

  • Review and possibly port Chromium accname tests into WPT #41 (@jnurthen)
  • add error handling for missing attrs in aria-utils.js #74 (@janewman)
  • Add SVG name fallback tests from aria attrs to host lang title etc #84
  • Tests for removal of aria-hidden=false #88 (@rahimabdi)
  • Test aria-hidden using label/role where possible #91
  • Remove the SVG-AAM image test until blocking issue resolved #96
  • Investigate keyboard operability testing #101 (@rahimabdi)
  • Add subtest count in test utils convenience methods to help avoid typos and other unexpected changes #105
  • Rewrite accessibility/crashtests/svg-mouse-listener.html to avoid timeout #106
  • Remove accname/name/comp_text_node.html tests that assume &nbsp trimming #115
  • Remove invalid test re: listitem namefrom contents, as it does not align with the spec. #118
  • Fix acc label expectations for td elements in display-contents-role-and-label.html #119
  • Review and either confirm, remove, or fix "recursive/traversal" tests in ./accname/name/comp_name_from_content.html due to outstanding AccName issues #123

[cookiecrook]

Note: I tweaked the numbers above a bit since the time of the call this morning to account for the two items now marked as 3% each. I also added all the "candidate" issues to the final 10% item.

[jcsteh]

I'd like to understand the reason we're looking at prototyping both tree dump and Accessible Node. It seems to me that the concept of Accessible Node is more flexible: you can implement a tree dump framework using Accessible Node (by walking all nodes from JS, likely packaged into a nice utility module), whereas you can't really implement Accessible Node using a tree dump. Tree dump also seems like it might be less flexible and more challenged by the tree structure differences between engines, which, while mostly inconsequential to users and AT, will be very troublesome in tests. Is there some context written down somewhere that could help me understand better? Thanks.

[cookiecrook]

As I understand it, Google & Bocoup wanted to prototype the tree dump for the dual purpose of 1. syncing WebDriver and CDP accessors to that feature, and 2. That it will allow some limited amount of WPT testing potentially more quickly the other path. For example, though the output trees may be slightly different, substring searches on leaf nodes could allow us to test things like the hidden state of certain nodes faster than the other path.

Since it's an investigation (not a Focus Area), I don't think there's any harm in exploring that path. Currently it's blocked because the investigators (@mzgoddard, @boazsender, etc) have not yet addressed the RFC guidance in this comment. If that happens, we can continue the exploration. If not, that's okay too. We would hit 100% We would NOT hit 100% of the investigation, but that's why it's an investigation.

[cookiecrook]

Minor note: we are at 1% or 2%, so we could get the investigation score off the ground, but I'm not really worried about updating it until we hit the 5% or 10% mark.

[zcorpan]

@cookiecrook you mean we would not hit 100%, right?

From Gecko's point of view, since we don't already have a tree dump API, we're reluctant to invest time into implementing two things instead of one.

[boazsender]

Responding to a few different threads in this issue:

Next steps for tree dumping:
After our chromium tree-dumping research last year, we think the next step should also be spec writing in the form of two standards:

1. a web driver bidi extension for tree dumping
2. a spec for a small part of the accessibility tree that each implementation can agree on the shape of, location TBD

We no longer think that landing the chromium prototype is the right next step. We did start an RFC conversation, which ultimately led to this new direction.

I think we should update the roadmap above to reflect this. Perhaps our 2024 scope should be the specs, and not the implementation.

@zcorpan would Gecko consider implementing tree dumping in 2025 if there were specs?

Next steps for accessible nodes:
For accessible nodes, the only comment we have is that we recommend specifying it as a bidi extension, rather than in webdriver classic.

Pursuing two testing paths:
I think the question of whether to pursue both accessible node and tree dumping strategies in the long term comes down to two things

1. whether or not we think that more testing surface would lead to more stable and interoperable ARIA rendering in the platform.
2. whether or not we think pursuing two paths would ultimately lead to a better single strategy

@jcsteh et al, do you think specing and testing parts of trees and along with accessible nodes would improve ARIA rendering interop?

[cookiecrook]

A few replies to @boazsender:

a web driver bidi extension for tree dumping

As Gecko doesn't support tree dump functionality and WebKit doesn't support webdriver bidi, this path may not enable near-term testing in anything besides Chromium. IMO, if this is to kept on the Investigation roadmap, at least one of those variables should change to support testing this year in at least two implementations.

a spec for a small part of the accessibility tree that each implementation can agree on the shape of

I think AOM #197 & AOM #203 may be a more likely path toward standardizing... I thought the goal of the tree dump was effectively a temporary stand-in API that might enable testing sooner than the longer-term interoperable web driver APIs.

• accessibility tree that each implementation can agree on the shape of, location TBD
• whether or not we think that more testing surface would lead to more stable and interoperable ARIA rendering in the platform.

AOM or ARIA I think... but isn't that what AOM #203 would standardize? We know there are tree differences, so aligning parent/child returns .tentative WPT files using webdriver ~accessiblenode could be the way to detect (and eventually standardize) those differences. Once implementations pass all the ~accessiblenode tree walker tests consistently, then we could work on standardizing the tree dump functionality, right? Until then, substring tree dump tests (even if not identical) may provide short term benefit.

For accessible nodes, the only comment we have is that we recommend specifying it as a bidi extension, rather than in webdriver classic.

Since all engines support webdriver classic, isn't that the better near term approach? Nothing in the AOM #203 proposal prevents longer-term adoption into webdriver bidi. It even mentions certain features (live region testing, for example) should be deferred until bidi support is universal.

[jcsteh]

That it will allow some limited amount of WPT testing potentially more quickly the other path. For example, though the output trees may be slightly different, substring searches on leaf nodes could allow us to test things like the hidden state of certain nodes faster than the other path.

Why would this be faster with tree dump vs ~AccessibleNode? Either way, we will (for now) need to potentially fuzzy match things to make it work across browsers. As I understand it, either approach exposes a bunch of properties for a given node. It's just that tree dump exposes them for all nodes, whereas ~AccessibleNode exposes them for a single node (and you then walk to other nodes to query those).

[cookiecrook]

~AccessibleNode would be known list of standardized, normalized properties that all implementations approve, not the entirety of each implementation's internal representation.

In contrast, the tree dump path would just be a standardized accessor to a serialized, non-standard, engine internal representation... In Chrome and WebKit, those internal tree accessors already exist as Accessibility.getFullAXTree and dumpAccessibilityTree. I assume Gecko has something similar, or something that could be converted without too much effort.

It'd be more work to normalized the tests to accept substring variants of the same info from different engines (hidden: true vs isHidden: YES perhaps), but lower implementation complexity, so hence it being referred to above as a "faster" path to making things testable.

[cookiecrook]

I wrote this:

I assume Gecko has something similar, or something that could be converted without too much effort [into a serialized tree].

backtracking a bit since @zcorpan said above:

From Gecko's point of view, since we don't already have a tree dump API, we're reluctant to invest time into implementing two things instead of one.

Completely reasonable. In any case, the point is moot until someone makes progress on the Chrome prototype. If that ends up being abandoned (or if Gecko stakeholders feel strongly enough to object), we can pull it from the scoring criteria, too.

For what it's worth, my priority and focus is also on the ~AccessibleNode path.

[zcorpan]

@boazsender

@zcorpan would Gecko consider implementing tree dumping in 2025 if there were specs?

I discussed with @jcsteh. We think the AccessibleNode idea is more flexible and prefer to go in that direction.

[cookiecrook]

Based on the group consensus in today's meeting, I've adjusted the scoring criteria significantly above... The most significant change (abandonment of the serialized tree dump) is called out specifically. Please comment, change, or otherwise let me know if you have any questions or concerns.

[cookiecrook]

score updated: 20*(6/17) + 5 = 12%

• update 2024 a11y investigation score wpt.fyi#3878

[cookiecrook]

Current score: 20*(6/13) + 15 = 24.231% 🎉
Mostly due to @rahimabdi's documentation progress.

• Update score of accessibility investigation. wpt.fyi#3949

[cookiecrook]

Should we bring in @spectranaut and @alice's work on the Acacia AAM testing in the scoring criteria for the 2024 Accessibility Investigation? They made significant progress recently and including it would bring the RFC more visibility.

[zcorpan]

• [Update: Aug 13, 2024] Recommend dropping the following from the Miscellaneous section unless a DRI steps up

  • Add comp_labelledby_recursion tests once AccName ambiguity is resolved #68
  • Organize Test Writing Priority by Feature Usage… #87
  • Resolve accessibility issues in datalist #95
  • Write test for accname edge case: datalist sub options #23

We resolved to drop these in #140 (comment)

[zcorpan]

Should we bring in @spectranaut and @alice's work on the Acacia AAM testing in the scoring criteria for the 2024 Accessibility Investigation? They made significant progress recently and including it would bring the RFC more visibility.

Yes per #140 (comment)

[cookiecrook]

Scoring Criteria updated per meeting resolution: Add Acacia, and drop the proposed DRI-less miscellaneous issues.

• Meeting: September 3, 2024, @ 9 AM PST #140 (comment)

[cookiecrook]

5 + 30 + 20*(7/13) = 45.769%

• update accessibility investigation score for Oct 1, 2024 wpt.fyi#4027