Should fire events instead of using passive model

[AshleyScirra]

(This is based on an email sent to public-webapps)

The Gamepad API is the only web platform input API that relies on passively polling the input state. Gamepads should fire events like "gamepadbuttondown", "gamepadbuttonup" and "gamepadaxismove". This would have the following benefits:

• consistent with other input APIs (e.g. "mousedown", "mouseup", "mousemove" / "touchstart", "touchmove", "touchend" / "pointerdown", "pointermove", "pointerup")
• easier to program for: if you want a "gamepadbuttondown" event, you have to implement it yourself by polling the state, keeping the previous polled state, and comparing the differences looking for a previously up but currently down state and then run your handler
• easily integrates with browser's existing "user gesture" model for sensitive features like opening popup windows, copying to the clipboard, and playing audio or video on mobile. Since gamepads do not fire events, currently it is not possible to use any "user gesture" features with gamepad input alone in any existing browsers.

[pyalot]

Note that using events exlusively without control over the event loop has severe data handling drawbacks.

If events are introduced, they should be issued at a controlled point in time (when polling) which clears the entire event queue up to that point, and they should not substitute "passive" (it's not really passive, it's active polling) querying of values, particularly not axis values.

[luser]

/cc @sgraham @toji

This has been discussed on public-webapps in the past:
https://lists.w3.org/Archives/Public/public-webapps/2012AprJun/0469.html

and it's on my list of features for a next version of the spec:
https://www.w3.org/wiki/Webapps/GamepadFeatures

Firefox includes non-standard events that you can enable by flipping a pref (dom.gamepad.non_standard_events.enabled), you'll get GamepadButton{Down,Up} and GamepadAxisMove events.

I think spec'ing the button events would be fairly simple and uncontroversial and solves the most important use case--missing button presses that happen between polls.

Events for axis values (or analog button values) are harder to get right. They're likely to be spammy and not incredibly useful--most consumers just want to know the current value of an axis. It would be nice to have some sort of event for this so that consumers aren't forced to constantly poll. Maybe we could write some spec text such that we'd fire a "datachanged" event whenever anything changes, but the browser can roll up a series of them into a single event, like if you had:

<button 1 pressed>
<axis 1 moved>
<axis 2 moved>
<axis 1 moved>
<axis 2 moved>
<button 1 released>
<axis 1 moved>

the browser could compress this to:

ButtonPress {button: 1}
DataChanged {axes: [1,2]}
ButtonRelease {button: 1}
DataChanged {axes: [1]}

[tkodw]

When thinking about what sort of event api would be useful, I think it is helpful to divide the use cases into categories based on how they expect input.

Reactive: input is a cue for something to start happening. Examples: video player, slideshow viewer

In this category we mostly get web apps that want to be able to interact or navigate occasionally with a gamepad. These apps want events because the current api forces them to poll continuously even though they rarely process input.

For the majority of cases, reactive apps only want to know about:

1. Button fully pressed
2. Button no longer fully pressed
3. Axis reaches +1.0 or -1.0
4. Axis no longer at +1.0 or -1.0

Tracking slight changes in joystick inputs does not make sense in events, because it produces many more events than a reactive app can reasonably respond to. An additional catch-all event GamepadChanged could be used if necessary but it might need both rate limiting and noise tolerance to be useful.

Animated: input is processed continuously and at regular intervals. Examples: games, simulations

An event driven api could help these types of applications catch missed button presses and improve timing, but the reality is that events are a poor fit for these types of apps. @pyalot pointed this problem out earlier.

Note that using events exclusively without control over the event loop has severe data handling drawbacks.

These classes of applications aren't idle, and therefore don't need an event to fire to wake them up. The need to listen for and store events to be used for the next game/simulation loop is a nuisance.

What these apps ideally want is a way to get all available information about the gamepad between the current game/simulation loop and the previous one. Specifically that means providing a function that can retrieve for snapshots of gamepad state or gamepad state changes over a specified time interval.

An example api change that can accomplish this is to add an optional parameter of type DOMHighResTimeStamp on getGamepads, which will cause each gamepad to instead be returned as a sequence of all gamepad states after the provided time instead of a single gamepad.

Providing all gamepad state updates has important benefits:

1. Timing accuracy no longer limited by polling rate, and by extension, frame rate.
2. Inputs will not be missed or have inaccurate timing if the browser freezes briefly.
3. Can consistently determine the order and combinations buttons are pressed in.

[pyalot]

A footnote on events for axes. A common failure mode of this kind of thing is when you have say an X and Y axis and you're plotting say a position on screen from them. So the X axis event arrives first, you draw a line there, then the Y axis event arrives, and you draw a line there. Now you've got a staircase instead of a continous line.

Although getting the whole state (as is presently the case) avoids that kind of issue (and it is among the reasons that shouldn't be dismantled), there is also sometimes a case to be made for having the ability to collect the events for axes and then aggregate them yourself (for instance instead of just using the last known, you can do a weighted average). For very twitchy games, this can provide players with a more satisfactory response.

[tkodw]

A common failure mode of this kind of thing is when you have say an X and Y axis and you're plotting say a position on screen from them. So the X axis event arrives first, you draw a line there, then the Y axis event arrives, and you draw a line there. Now you've got a staircase instead of a continous line.

This problem is easily avoidable if you can get multiple events at once. In your example you get an X axis but have no way of checking if the Y axis event has already fired and is waiting to be processed.

• If you implement an event that provides an array of changes, you always see the events together.
• If you implement an event that provides only a timestamp of when changes occured and allow getGamepads to request states over a time range, you will see both axis changes within the states.
• As long as the api provides a mechanism to deal with this problem, it may be okay to leave in some events that suffer from this issue if it makes the implementation of gamepad support easier for simple use cases.

Although getting the whole state (as is presently the case) avoids that kind of issue

Take the simple case that a controller has a directional pad, and you want to perform different actions for up, diagonal (up + left) , and right. When polling the complete state, just left or just right can (and frequently will) show up in a poll before both show up together unless the are pressed together at exactly the same time. You can't avoid doing the left or right action before diagonal unless you buffer inputs for some arbitrary interval before performing the left or right actions. There are a lot of similar issues like this that need to be tackled by the application and are unlikely to be solved directly by the api.

there is also sometimes a case to be made for having the ability to collect the events for axes and then aggregate them yourself (for instance instead of just using the last known, you can do a weighted average). For very twitchy games, this can provide players with a more satisfactory response.

I am very much in favor of games aggregating events or state provided from the API, and believe this is the ideal way to handle input. The question is how to collect the aggregated events.

1. Poll Spamming (the current way)
Call getGamepads on a timer with a short interval and collect the gamepad states.

• It already works and is a simple and understandable solution.
• There is no way to know what polling rate the hardware is using or sync to it, so we are frequently undersampling (increased latency, missed inputs) or oversampling (wasted timer calls).
• If the main javascript context stalls for any reason inputs are missed and timing loses accuracy.
• The polling loop is constantly active even if the controller is idle.

note: Timing is much worse than accessing hardware directly in Chrome due to a counter being provided instead of timestamps and internally buffering with a long polling interval. (In my test case 125hz hardware is polled and buffered at 60hz)

2. Event Listeners (same as mouse and keyboard)
Api would provide events to subscribe to that fire for every possible state change on connected gamepads. The application subscribes to all event types with functions that collect the events.

• Fits in better with existing input handling of mouse and keyboard events.
• When the controller is idle, no events need to be handled.
• High update rate hardware can result in quite a lot of event handling spam.
• No missed inputs or inaccurate timing.

3. Query State/Events (my suggested method)
Have the browser internally poll or accept input from all active controllers with as much accuracy as possible and buffer this information over a short period. Expose an API that takes a time range and returns an collection of states or events from gamepads over that time range.

• No input events or timers in javascript. (better efficiency?)
• No missed inputs or inaccurate timing.

A minimal API to satisfy all possible use cases:

• Add one event for when any gamepads have change state after a specified time.
• Add an optional start time range to getGamepads so it can return multiple states.

[jakearchibald]

Could a model like https://developer.mozilla.org/en-US/docs/Web/API/PointerEvent/getCoalescedEvents help here?

[AshleyScirra]

I think there still need to be events to be able to work with user gesture requirements. For example, you still cannot start any media playback with a gamepad on Android, because you never get an input event (crbug.com/454849). Something like getCoalescedEvents doesn't seem to change that.

[jakearchibald]

There'd still be a root event to get the coalesced events from, and that would be a gesture event.

[AshleyScirra]

Oh, right. Yeah, it makes sense to use a similar mechanism to pointer events for better-than-rAF resolution.

[jakearchibald]

A similar option is a method where you declare which buttons & axis you're interested in, and it returns an EventTarget which fires a "change" event in the render steps of the event loop, and getCoalescedEvents gives you the full details.

[jakearchibald]

Thinking a little more about this, it could just be:

gamepad.addEventListener('buttondown', listener);
gamepad.addEventListener('buttonup', listener);

// Event interface:
event.index; // index of button changed

gamepad.addEventListener('axischange', listener);

// Event interface:
event.index; // index of axis changed
const events = event.getCoalescedEvents();
events[0].index; // index of axis changed
events[0].value; // value

gamepad.addEventListener('buttonvaluechange', listener);

// Event interface:
event.index; // index of button changed
const events = event.getCoalescedEvents();
events[0].index; // index of button changed
events[0].value; // value

The timing of these events should follow mouse events in Firefox & Chrome, as in:

• A task is queued to fire an event for each buttonup & buttondown.
• If an axis or button changes value, an axischange/buttonvaluechange event will fire in the render steps of the event loop (so they're limited to once per frame).
• In the task to fire a buttonup or buttondown, if an axischange/buttonvaluechange event is pending with an earlier timestamp, fire that event within the same task, before the buttonup/buttondown.

[jakearchibald]

I guess event.button already exists for mouse events, so maybe it should be event.button and event.axis instead of event.index.

[pyalot]

On Tue, Jan 2, 2018 at 12:36 PM, Jake Archibald ***@***.***> wrote: - If an axis or button changes value, an axischange/buttonvaluechange event will fire in the render steps of the event loop.

Please be aware that if you opportunistically fire events for changed axes, and you as a programmer react to those events immediately, you will experience unintended behavior, such as drawing a staircase instead of a diagonal line (because the X event fires first, at which time the Y value isn't available yet, and so forth).

[AshleyScirra]

If you wait until rAF to draw any changes, and you received both axischange events by then, it should correctly handle simultaneous X/Y changes. I think a more interesting question is how to handle that with getCoalescedEvents(). Will the coalesced events for different axes be guaranteed to have the same number of updates at the same time? Perhaps it could combine axes and fire events for them simultaneously to avoid this? I don't know if the underlying APIs tell you about axis associations or not.

[jakearchibald]

@pyalot maybe I'm not explaining my proposal properly.

My proposal:

• If an axis or button changes value, an axischange/buttonvaluechange event will fire in the render steps of the event loop (so they're limited to once per frame).

Whereas your comment suggests I'm proposing firing these events outside of the render steps, and perhaps even firing one event per axis – that isn't what I'm proposing.

Take mousemove in Firefox & Chrome as an example. If the OS receives 20 mouse position updates in a single frame, the browser will not dispatch 20 mousemove events, nor will it fire separate events for x/y positions. Instead, it dispatches a single mousemove event during the render steps, reflecting the final mouse position. If you want access to all 20 event objects, you can use event.getCoalescedEvents() to get them.

You only need the full 20 event objects in very particular cases, such as a painting application, where you want the full data for all mouse points despite any in-page jank.

https://event-timing.glitch.me/ might be useful – it shows how this works with mouse events.

I'm proposing axischange behaves the same. One event, queued once an axis changes, dispatched in the render steps.

[jakearchibald]

@AshleyScirra

If you wait until rAF to draw any changes, and you received both axischange events by then, it should correctly handle simultaneous X/Y changes.

I'm proposing this event should be fired in the render steps, so it's already in the same phase of the event loop as rAF, and debounced in the same way.

I think a more interesting question is how to handle that with getCoalescedEvents(). Will the coalesced events for different axes be guaranteed to have the same number of updates at the same time?

In my proposal the objects look like this:

gamepad.addEventListener('axischange', listener);

// Event interface:
event.index; // index of axis changed
const events = event.getCoalescedEvents();
events[0].index; // index of axis changed
events[0].value; // value

Each event of events may have a different axis. They'll be in order of event.timeStamp. If multiple axes were updated at the same time, you'd get multiple event objects with the same timestamp.

I think needing getCoalescedEvents is an edge case (as it is with mouse events). In most cases you'll just do something like:

gamepad.addEventListener('axischange', () => {
  doSomethingWithThese(gamepad.axes[0], gamepad.axes[1]);
});

[pyalot]

On Wed, Jan 3, 2018 at 2:48 PM, Jake Archibald ***@***.***> wrote: gamepad.addEventListener('axischange', listener); // Event interface:event.index; // index of axis changedconst events = event.getCoalescedEvents(); events[0].index; // index of axis changed events[0].value; // value

I'm not sure why you'd call it Coalesced events. It's just fetching the event queue, the same way "getEvents" is implemented pretty much anywhere else (USB, SDL, pyglet, xinput, directinput, etc.). You get a list of events that occured since last time you called that function. It gives you sufficient information to know what kind of event it was, and what values it transports.

[pyalot]

It's also usually the case that these "getEvents" functions support some rudimentary filters for event types, so you don't have to dig trough all events to get to the ones that're of interest to you. It's a convenience function.

…

On Thu, Jan 4, 2018 at 11:18 AM, Florian Bösch ***@***.***> wrote: On Wed, Jan 3, 2018 at 2:48 PM, Jake Archibald ***@***.***> wrote: > gamepad.addEventListener('axischange', listener); > > // Event interface:event.index; // index of axis changedconst events = event.getCoalescedEvents(); > events[0].index; // index of axis changed > events[0].value; // value > > I'm not sure why you'd call it Coalesced events. It's just fetching the event queue, the same way "getEvents" is implemented pretty much anywhere else (USB, SDL, pyglet, xinput, directinput, etc.). You get a list of events that occured since last time you called that function. It gives you sufficient information to know what kind of event it was, and what values it transports.

[jakearchibald]

@pyalot

I'm not sure why you'd call it Coalesced events.

I'm not calling it that. It already exists for pointer events https://developer.mozilla.org/en-US/docs/Web/API/PointerEvent/getCoalescedEvents.

You get a list of events that occured since last time you called that function.

Not really, you get a list of events that were coalesced into that render step event. If you don't call getCoalescedEvents, those additional events are gone. If it behaved as you describe, it'd be a massive memory leak – you'd quickly have millions of events buffered that couldn't be GC'd.

[pyalot]

On Thu, Jan 4, 2018 at 11:33 AM, Jake Archibald ***@***.***> wrote: Not really, you get a list of events that were coalesced into that render step event. If you don't call getCoalescedEvents, those additional events are gone. If it behaved as you describe, it'd be a massive memory leak – you'd quickly have millions of events buffered in memory that couldn't be collected.

That's why these other eventing functions usually let you specify (or have an implicit) event buffer size.

[jakearchibald]

Right. Or there's the pointer events model which doesn't need that, and fits in with the event model already present on the web.

[luser]

Could a model like https://developer.mozilla.org/en-US/docs/Web/API/PointerEvent/getCoalescedEvents help here?

This sounds promising! Gamepad axis events are pretty similar to pointer move events: they're an analog input that can be highly variable (even if the user isn't touching an analog stick, it's likely to be changing just due to jitter).

I suspect that for most gamepad usecases the API that users want is "what state on the gamepad has changed since the last time I looked?" For button presses it's important to have a record of every pressed/released event so that you don't miss quick button presses, but for axes you probably just want the current value, but you probably want to know if it has changed.

[jakearchibald]

@luser that's exactly what I was aiming for! getCoalescedEvents is useful for completeness, but (like mouse events) the majority of use-cases will just read the current axis values in the axischange listener.

[luser]

One thing that might make this slightly awkward for the common use case in games is that AIUI most game loops have a process input / update state phase and a render phase. Currently most JS code I've seen just mashes those both into an rAF callback, handling input and updating state first, then rendering. Presumably the slightly better option would be "run a setInterval callback at a fixed rate to do input handling/state updates, do rendering in rAF", but I don't think that matters much for the purposes of this discussion. The Gamepad API currently provides getGamepads() to poll for immediate state, which matches the expectations of game programmers given the APIs available on other platforms. If we add events, then users will have to manually track state between event handler callbacks and the input handling phase. Maybe this isn't so bad, someone like @Antumbral can chime in.

Some notes on native platform APIs:
XInput's main API is XInputGetState , which polls the state of a gamepad, and apparently this is similar to the API provided by other consoles. SDL is a little odd in that it does provide events like SDL_ControllerButtonEvent, but it does also provide direct APIs like SDL_GameControllerGetButton, and the event loop needs to be manually run by the application anyway.

[jakearchibald]

@luser

One thing that might make this slightly awkward for the common use case in games is that AIUI most game loops have a process input / update state phase and a render phase. Currently most JS code I've seen just mashes those both into an rAF callback, handling input and updating state first, then rendering.

Doesn't this proposal make this a bit more formal? You'd use the events as a place to update state, then queue a rAF for the render phase.

[mrmcpowned]

I'd like to quote what I wrote in #15 since it sums up my thoughts on an event based model:

Has there been any more movement on this? I also think it'd be welcoming to have an event-driven model, but only as an inclusion alongside the polling model as well.

The following question was asked in #22:

With that in mind, I guess my question to you is: do you know any practical examples where lost gamepad events are actually a real problem?

Lost gamepad events can be a real problem when the intended use is for accurate input visualization. A lost frame of data can be reasonable in terms of displaying said data, but an inconsistency between axis values or when two or more buttons were pressed simultaneously is not.

The suggested approach to #4 to remedy such a scenario (the API providing an event subscription strategy for multiple axes/buttons/etc) doesn't account for this kind of scenario because it could create a disconnect between axes and button represented at a fixed point in time due to the asynchronicity.

We could then propose something like onGamepadChange that would subscribe to the entire gamepad, but something like that would feel like a bastardization of the polling strategy, and would that really be better than polling from the start?

It's why I'm not against the inclusion of events, but the removal of polling wouldn't allow for the kind of scenarios I'm depicting. For something like adding gamepads as a form of navigational input for a webpage, events would be the perfect approach since it'd be idiomatic in comparison to other event-driven practices and wouldn't necessarily encounter the kind of issues I mentioned above.

This also isn't a theoretical application either. I run and manage gamepadviewer.com, which does exactly what it says, and the amount of usage is not insignificant, either. According to the getGamepads() feature data, the site ranks at least 34th in the list, so everything I've posited isn't hyperbole.

Since there hasn't been any discussion on this in a while, it'd be nice to get some talk going again as there doesn't seem to be a consensus so far.

[marcoscaceres]

thanks @mrmcpowned - our goal right now is to do some cleanup of the underlying model (see #136) of the spec and improve the privacy and security (#120 + #119). In particular, #136 should give us a better foundation on which to specify a better eventing model. However, I don't expect us to focus on the events until we land the PRs I mentioned.

[juanitogan]

I have a use case that hasn't been precisely addressed yet: network gaming. When my game detects a change in input state, I want to send that off as soon as possible. I don't want to wait up to another 20ms (or whatever) for the game loop to pick it up and send it off. I currently handle keyboard input outside the game loop and am wondering what to do about gamepads. Latency is already bad enough across the web. Why add to that? Even more important than total latency is variation in latency. Any polling solution (buffered or otherwise) adds to that variation.

I believe a proper gamepad solution would have the buttons performing just as well as keyboard keys (and/or mouse buttons) in regards to responsiveness, and can be implemented with the same coding patterns (events and/or polling). Do that and you can stop second-guessing use cases.

Thus, while buffering events could solve the button-press-duration-sensitivity problem, it does not solve the responsiveness problem overall. True, most use cases will not need responsiveness tighter than a typical game loop interval, but that is no reason to ignore it either.

---

Personally, I don't see a good use case for analog-change events* and I've built a variety of games. Yet, to be fair, I haven't looked at mouse-versus-gesture cases like others have, so I can't comment much on those details.

If a mouse fires positional events, then it makes sense a joystick should too... except for a possibly-important detail: potentiometers (joysticks) versus spinners (mice and trackballs). Sure, the output may look similar in some floating-point-vector ways, but the source and meaning of the data is not similar. Pots have extents, spinners do not. (Ignoring touch pads/screens for now.) Pots are notoriously much more twitchy producing spammy data (that should probably be smoothed as well). Pots also often have centering springs attached which, well, just makes them more weird and noisy. Thus, perhaps, if looking for an excuse to not tie joystick positioning to events, the origin of data from pots would be a good one.

* This does not include the case of using analog controls as digital controls, such as faking a button press when an analog value hits 0.5. Such an action is fuzzy to begin with and the use case can tolerate additional slop from the poll timing.

Again, personally, if needing an event system for pot changes, I wouldn't be surprised if I needed to do that myself in a loop with a 1ms interval or whatever. Then, I could also cache it, smooth it, or whatever else it needed.

[jakearchibald]

Pointer events now has a pointerrawupdate event for immediate updates. Seems reasonable that gamepad should have the same.

[marcoscaceres]

Related to #56

[tomayac]

The new explainer sounds really interesting and was well received when I shared it. Are there plans regarding getCoalescedEvents(), as discussed earlier on this thread?

[nondebug]

Thanks for sharing the explainer!

Yes, I think event coalescing will be necessary and we are currently figuring out how it should work. We want to be able to support use cases where every input frame must be received with minimal latency as well as use cases that can benefit from events but don't want the overhead of handling every input frame.

Please take a look at the proposals in this doc. We've also opened a TAG design review and comments are welcome on the pull request.

Right now I'm leaning toward Proposal 3 which would keep buttondown/buttonup (with no coalescing) but remove axischange/buttonchange in favor of a unified change event that encapsulates all the changes within an input frame and provides a getCoalescedEvents method to retrieve events for skipped input frames. I think these events, along with the current polling interface, do a good job of covering all the use cases called out in this thread.