Gamepad API input events

[nondebug]

Ya ya yawm TAG!

I'm requesting a TAG review of Gamepad API input events.

The Gamepad API requires applications to poll to detect new inputs. Many applications follow the spec's recommendation to coordinate polling with requestAnimationFrame. However, polling in this manner is likely to lose inputs since gamepads typically update more quickly than the animation frame rate. Polling will also increase average input latency by half the polling interval. This proposal addresses both the missed inputs and latency by adding input events that fire when the gamepad state has changed.

• Explainer: Gamepad API Button and Axis Events Explainer
• Specification URL: Pull request | Preview | Diff
• Tests: None yet, web platform tests will be added with the prototype implementation
• Security and Privacy self-review: No security or privacy risks are expected since the events only expose data that is already available through the polling interface.
• Primary contacts:
  • Matt Reynolds (@nondebug), Google, Spec editor, Implementer
  • James Hollyer (@jameshollyergoogle), Google, Spec editor, Implementer
• Organization(s)/project(s) driving the specification: Web Apps WG
• Chrome Status: https://chromestatus.com/features/5989275208253440

Further details:

• I have reviewed the TAG's Web Platform Design Principles
• Relevant time constraints or deadlines: none
• The group where the work on this specification is currently being done: Gamepad API is part of the Web Apps WG
• Major unresolved issues with or opposition to this specification: Event spamminess, see below
• This work is being funded by: Google

You should also know that...

Gamepads can potentially produce a lot of input events and we are interested in feedback on how to mitigate the spamminess of these events. Consider a DualShock 4 that has two 2-axis thumbsticks and updates at 1000 Hz. When both thumbsticks are moving it will generate 4 axis change events per input frame, or 4000 events per second. If the application allows events to queue then it will be handling stale inputs, adding input latency.

This is mostly an issue for analog axis inputs but also affects pressure-sensitive buttons and analog triggers.

We have some ideas for mitigating this:

• Combine all the button and axis changes that occur within an input frame into a single event so that at most one event is fired per input frame. This is analogous to how the PointerEvents API combines the X and Y axes of a pointing device into a single event object. The downside is that applications only interested in one type of input (for instance, button presses) will still be notified for every other input.
• Don't allow events to queue. Always deliver the most recent data, but provide a method for retrieving events that were not delivered. This is analogous to getCoalescedEvents in the PointerEvents API.

See here for more discussion on this proposal, the above mitigations, and comparison with the earlier Firefox implementation of these events.

We'd prefer the TAG provide feedback as:

💬 leave review feedback as a comment in this issue and @-notify @nondebug and @jameshollyergoogle

[torgo]

Hi - we're just having a look at this today and on first pass it looks good. 👍 One question: what is the current status of any additional implementer support? Right now the ChromeStatus page says "unknown" for Safari and Firefox.

[kenchris]

Do the coalesced events contain a timestamp? I think that would be useful.

Also many modern controllers and event gaming keyboards can check the amount a button is pressed, and even the PS5 has adaptive triggers. Have you considered that use-case and how would you augment this API to support that? Maybe in that case buttonrelease and buttonpress makes more sense as names

[nondebug]

@torgo Chrome Status should have "Positive" for Firefox, referencing Ted's comment on the public-webapps mailing list. That's pretty old so I've submitted a standards position request and will update the status once that's resolved.

I haven't heard from Safari yet.

@kenchris, please take a look at this doc for our updated thoughts on how pressure-sensitive buttons and axis changes should be handled. The proposal in the explainer (called Proposal 1 in the doc) is probably not the most useful for developers since it makes it hard to get an accurate picture of the current gamepad state without adding unnecessary latency.

A typical gamepad is implemented using the HID protocol (or something like it) where all button and axis inputs are delivered in a single packet. If we fire separate events for each button and axis then it makes it harder to understand which change events are from the same update. For example, if you're handling joystick axes then you always want to handle the X and Y updates from the same update at the same time. If you try to handle X and Y updates sequentially, you'll trace out a stairstep pattern which is a poor representation of the actual joystick movement. It's possible to work around this, but it means you have to delay handling some inputs until you're sure you've received all the relevant events.

Pointer events avoid this issue by delivering X and Y axis updates in the same pointermove event. To follow the same pattern, we should combine all axis changes into a single axesmove event. And since some buttons can behave like axes, we should include them as well. Proposal 2 combines everything into a single event (we called it "change") that's fired on every input frame if there was any change to any button or axis.

Currently I'm leaning toward Proposal 3 which keeps buttondown/buttonup events along with the change event. It's rare that you would ever want to subscribe to all of them since the change event would include all the same information included in buttondown/buttonup but I think it does the best job of satisfying everyone's use cases.

buttonrelease and buttonpress makes more sense as names

Those seem reasonable and I'm open to changing the names. I prefer buttondown/buttonup because it keeps the API more consistent with mousedown/mouseup and pointerdown/pointerup.

adaptive triggers

DualSense adaptive trigger haptics, Xbox Impulse Triggers, Switch HD Rumble and other advanced haptics aren't supported through the Gamepad API. There's a Microsoft Edge proposal for a Haptic Device API that we're hoping we can adopt in order to support more advanced haptics.

[kenchris]

Adaptive triggers is more than just controlling haptic feedback such as vibration. It is also about having non-binary values for buttons. Like a machine gun (in a game) might start shooting faster and faster the longer down you press the button.

Adaptive triggers can also allow developers to configure the tension the user will experience as they press the button:

https://www.theverge.com/2020/11/3/21547303/sony-ps5-dualsense-adaptive-triggers-haptics-teardown-features

[kenchris]

Other comments:

• ongamepadchange sounds like the gamepad itself is being changed. I think "onchange" or "oninputchange" would be better names.
• If you have oninputchange, I don't think we should add onbuttondown and onbuttonup - It is better to teach developers to actually just use the other event. I don't think onbutton* adds any convenience - but they might add confusion

[kenchris]

rawgamepadchange is dispatched if any button or axis state has changed. When multiple changes occur at the same time, one event is dispatched that encapsulates all the changes.

A fourth gamepadchange event is proposed that is created whenever a rawgamepadchange event would be dispatched and placed into an internal queue. The event may be dispatched immediately or delayed for performance reasons. Queued events must be dispatched before animation callbacks are run and before a buttondown or buttonup event is dispatched for that gamepad.

That is very complex and I would never have guesses that was the behavior from the event names.

Also, events cannot have side-effects so you would have to have the implementation be prepared to dispatch both events at all times. Listeners to one event type cannot change how events are being dispatched internally.

Are there strong cases to be able to configure how events are being dispatched? If not, please just choose one. If you really need it, then couldn't this be configured with a method instead, with a sane default.

[nondebug]

Thanks for the comments, @kenchris

Are there strong cases to be able to configure how events are being dispatched? If not, please just choose one.

There's a strong case to dispatch input events immediately to support streaming game clients. For cloud gaming, inputs need to be uploaded with minimal latency. The rawgamepadchange event is intended to satisfy that use case and any other case where low latency is needed. Currently, the only "good" way to do this is to poll getGamepads() as quickly as possible, which still adds a small amount of latency. Events tied to the animation frame loop aren't useful for this use case because they add latency, on average half the interval between animation frames which will always be worse than rapid polling.

The case for having a coalesced gamepadchange event is that there can potentially be a huge number of rawgamepadchange events. The input rate is determined by the gamepad hardware, not the user agent. If script can't keep up with the events then they'll queue and add latency. By flushing all events any time one event is fired, script is always processing fresh input data but still has access to all the historical input data since the last event. Synchronizing on the animation frame callback is convenient because that's where most applications currently process gamepad inputs.

If you really need it, then couldn't this be configured with a method instead, with a sane default.

I think configuring the dispatch behavior with a method on the Gamepad object could work. Probably gamepadchange's coalescing behavior should be default so the user agent can control the event rate if needed. Changing from a coalescing mode to a non-coalescing mode should flush pending events to ensure no inputs are lost.

Is there any precedent for configuring the event dispatch behavior after an event listener is registered?

Also, events cannot have side-effects so you would have to have the implementation be prepared to dispatch both events at all times. Listeners to one event type cannot change how events are being dispatched internally.

I may have worded this confusingly, I want to specify the same behavior as in Pointer Events where "The user agent MUST fire a pointer event named pointermove when a pointer changes button state." The intent is to flush the gamepadchange queue whenever a button's pressed attribute changes regardless of whether any event listeners are registered.

If you have oninputchange, I don't think we should add onbuttondown and onbuttonup - It is better to teach developers to actually just use the other event. I don't think onbutton* adds any convenience - but they might add confusion

I agree, if the gamepadchange overhead is tolerable then buttondown and buttonup don't add much. I think it's reasonable to drop them.

Adaptive triggers is more than just controlling haptic feedback such as vibration. It is also about having non-binary values for buttons. Like a machine gun (in a game) might start shooting faster and faster the longer down you press the button.

I may have misunderstood your original suggestion. Non-binary button values have always been supported by Gamepad API and in this proposal we'd fire a change event whenever a trigger value changes. I've only heard the term "adaptive triggers" used in the context of DualSense's adjustable tension.

I think you're suggesting buttondown and buttonup could be generalized in a way that better supports non-binary buttons. Instead of firing a single buttondown when the trigger value crosses the button press threshold, we would fire multiple buttonpress events as the value increases.

If we're dropping buttondown and buttonup then this is moot.

[kenchris]

Is there any precedent for configuring the event dispatch behavior after an event listener is registered?

@domenic @annevk do you know of such a case, or do you have any other comments regarding this?

[torgo]

@nondebug one process issue - can you please move the explainer over to w3 space (webapps wg repo) along with the spec?

[annevk]

@kenchris it's not entirely clear to me what is being asked, but event listener registration is supposed to be side effect free, apart from performance optimizations. See https://dom.spec.whatwg.org/#observing-event-listeners in particular:

In general, developers do not expect the presence of an event listener to be observable. The impact of an event listener is determined by its callback. That is, a developer adding a no-op event listener would not expect it to have any side effects.

[kenchris]

Maybe we could do something like addEventListener("gamepadchange", callback, { coalesce: false })

Event listeners already have options like once: true

[kenchris]

I am not a fan of having two different event handlers when you are only supposed to use one of them and choose. Also, as they deliver data differently you can argue that they are not side effect free (as they configure this change) and that is thus against the web design principles (as Anne pointed out).

The Generic Sensor API works around this by having start() and stop() methods and start() can take arguments, so you could do something like start( { coalesce: false })

[torgo]

Hi @nondebug any feedback on the above?

[nondebug]

Maybe we could do something like addEventListener("gamepadchange", callback, { coalesce: false })

I like this.

The Generic Sensor API works around this by having start() and stop() methods

With Generic Sensor you can create a new sensor object for each consumer but for Gamepad all consumers always see the same Gamepad object. We'd need to add a Gamepad.getGamepadSensor() method to return something that can be unique for each consumer so we can call its start() and stop() without interfering with other consumers. This could work but it seems overly complex to me.

[cynthia]

We'd need to add a Gamepad.getGamepadSensor() method to return something that can be unique for each consumer so we can call its start() and stop() without interfering with other consumers. This could work but it seems overly complex to me.

Understood. Thanks for clarifying - I think this is something that we may want to revisit if there are user needs in the future. For now, based on our last breakout discussion we think it's fine to move forward with this proposal as is.

Thank you for bringing this to our attention!