Vote: Adopt spec by Screen Capture Community Group

[eladalon1983]

This issue collects votes for the Screen Capture Community Group to adopt the Screen-Capture Mouse Events spec.

The recommended format for casting your vote is to use one of the following:

1. I support adopting the Element Capture specification.
2. I support adopting the Element Capture specification; we at {company} intend to use it soon for {purpose}.
3. I object to the adoption of the Element Capture specification. My list of blocking-issues is {list-blocking-issues}.

Mentioning which company you work with, and whether you have a specific use-case in mind for this API, helps browser vendors set the prioritization of implementing this API. For some browser vendors, such "Web developer signals" are taken into account when deciding whether an API is ready to be shipped.

[happylinks]

I support adopting the Screen Capture Mouse Events specification; we at Tella.tv intend to use it soon for automating video zooming based on mouse movement.

[AdamJaggard]

I support adopting the Element Capture specification

edit: this should say Screen Capture Mouse Events Specification, OP needs updating.

[ldenoue]

I support adopting Screen-Capture Mouse Events specification; ScreenRun web app intends to use it for enhancing screen recordings by highlighting parts where the mouse moves and generate zooms when users click.

[fideltian]

There should be lack of returning Cursor Type or Cursor Image if the cursor type changes. for example, from cross to array, etc.
Otherwise, it is hard if the viewing side would like to replay the cursor move.

[eladalon1983]

There should be lack of returning Cursor Type or Cursor Image if the cursor type changes. for example, from cross to array, etc. Otherwise, it is hard if the viewing side would like to replay the cursor move.

Thanks for the feedback. There is an active discussion of this possible extension in issue #4. I'd love to continue engaging there.

Btw, could you please clarify whether you see this as a (a) blocking issue, (b) non-blocking issue or (c) proposal for future extension? (I see it as c, fwiw. The rationale is that some apps don't require the cursor shape; for example, those apps that only wish to auto-zoom on the cursor, not enhance it.)

[fideltian]

i think it is a blocking issue for on-line meeting case. we would like to let the viewer's view be the same as presenter's view.

With the proposal, we could handle the cursor event in App iteself and gain performance improvement.

[eladalon1983]

i think it is a blocking issue for on-line meeting case. we would like to let the viewer's view be the same as presenter's view.

With the proposal, we could handle the cursor event in App iteself and gain performance improvement.

I expect that the approach taken by the current spec will be easily extended to also tackle issue #4, which is why I don't see issue #4 as blocking adoption. I believe in iterative progress. Could you please explain why this is not the case for you, @fideltian?

[fideltian]

Hi Elad, There should be no issue if Issue #4 could be counted.

[arnaudbud]

I support adopting the Screen-Capture Mouse Events specification.

[jozefchutka]

I have just proposed a very similar API on w3c/mediacapture-screen-share#267 .

Please have a look if there are any bits in my proposal that could be helpful for this draft:

• Have an API that would provide cursor events live during capturing.
• These events should provide x, y coordinates relative to the currently captured area
• ...type of system cursor currently being in use (i.e. these https://developer.mozilla.org/en-US/docs/Web/CSS/cursor ).
• ...a pressed button (i.e. similar to https://developer.mozilla.org/en-US/docs/Web/API/MouseEvent/button )
• ...a timestamp somehow related to capture start so events can be eventually synced with recording video in postproduction
• Events should only get triggered on change (x, y, type, button)

[eladalon1983]

Hi Jozef! Thanks for the feedback. If you could also mention which company you work for and how you expect to be using this API, it would help get it shipped eventually. (Totally optional, of course.)

• Have an API that would provide cursor events live during capturing.
• These events should provide x, y coordinates relative to the currently captured area

This sounds like what the spec currently provides, right?

• ...type of system cursor currently being in use (i.e. these https://developer.mozilla.org/en-US/docs/Web/CSS/cursor ).

That sounds like issue #4.

• ...a pressed button (i.e. similar to https://developer.mozilla.org/en-US/docs/Web/API/MouseEvent/button )

That sounds like issue #1.

• ...a timestamp somehow related to capture start so events can be eventually synced with recording video in postproduction

That sounds like issue #3.

Events should only get triggered on change (x, y, type, button)

I think we're aligned here.

[jozefchutka]

Thanks for confirmation @eladalon1983 . I have shared my ideas into some of these threads.

I am working on a web based video editor wide.video which has a feature to record screen. With this API in we will be able to generate awesome videos like these https://www.screen.studio/videos/spotify.mp4 - using just web browser

[eladalon1983]

It looks like this got closed by a PR that referenced #6 but intended to reference #16, so I am reopening this.

[fred-wang]

Oops, sorry about that. I guess I'll stop doing "closes XXX" on PRs and close manually instead...

[eladalon1983]

Oops, sorry about that.

No worries.

I guess I'll stop doing "closes XXX" on PRs and close manually instead...

My experience shows that this is not immune to error either. :-)
Technically speaking, I guess having the "Fixes XXX" is still better because it gave a second person a chance to catch the error.

[k0nserv]

I object to the adoption of the Screen-Capture Mouse Events specification in its current form. At Lookback we'd want to use this to highlight when the participant interacts with the application under test in recorded user research. To do this, at minimum, we need click events (as suggested in #5), in addition to the proposed positional events.

[eladalon1983]

Hello @k0nserv. I am not sure that it's productive to object to A on the grounds that you want A+B. I think the former lays the groundwork for the latter. Or do you think that the current spec somehow closes the door on future expansions to tackle your extended use-case?

[k0nserv]

Hey, no I was just attempting to balance signalling support, with the caveat that this will only be useful to us if click events are exposed, sorry about the confusion.

Ideally we, of course, think the first version should support click events, but if the spec were to evolve to this in the future that would also make us happy.

[eladalon1983]

Thanks for clarifying your position.

Exposing coordinates is possible from Chrome's POV as per the review by Chrome Security and Privacy. When it comes to exposing clicks, the story might become more complicated. It's hard to say what the outcome would be from Chrome's perspective - whether it'd be seen as fine, or as requiring a permissions prompt, or outright rejected. I think it's best to proceed incrementally, producing now the functionality that's in consensus, thereby unlocking other developers (and possibly partially unblocking you), then explore expansions of the spec at a later time.

[atticoos]

I support adopting the Screen-Capture Mouse Events specification; we at Rally Space intend to use it soon to automatically perform effects like zoom & pan, based on certain mouse event criteria, on the recorded video, without having to direct users to a native desktop application or browser extension.