Exposing mouse events from captured surfaces

[eladalon1983]

Web applications can observe mouse events occurring over themselves. But what about events over a captured display surface? Mouse coordinates can be read by inspecting frames and searching for the cursor, but that is neither simple, nor efficient, nor robust.

During the first Screen Capture Community Group meeting (slides, minutes), @ldenoue and I described the use-cases this would unlock and presented a proposed API. The proposal was met with enthusiasm, with unanimous agreement (via poll) for the presenters to produce a draft and present it to the group during the next meeting. This draft is now ready: https://screen-share.github.io/captured-mouse-events/)

I'd like to encourage everyone interested to join the discussion. Comments on this thread, issues on the proposal's repo, participation in the next SCCG meeting, courier pigeon notes - you name it.

(CC @happylinks)

[happylinks]

Thanks for kicking this off @eladalon1983! Happy to get this proposal off the ground :) Added some questions to the github repo to get the discussion started.

[jozefchutka]

It seems I have just proposed very same thing w3c/mediacapture-screen-share#267 ... closing my issue now as it is being in progress here

[youennf]

Looking quickly at the proposal, there might be some potential sync issues between video frames and the mouse events.
Hopefully, this should not be a very big issue. Was it discussed though?

An alternative is to attach the mouse information as metadata to VideoFrames directly.
It would be exposed to JS either via VideoFrameCallbackMetadata or via VideoFrameMetadata.

Has this option been investigated?

[eladalon1983]

Has this option been investigated?

There's an issue for roughly that question. There were a few discussions on the topic, but they've not all been captured on GitHub. Attaching mouse metadata is indeed one option that's come up. Another option is to ensure there are good timestamps from the same clock source on both VideoFrames and the new events, so that the relative order can be accurately determined. And François also mentioned RVFC. You're very welcome to contribute to that discussion.