-
Notifications
You must be signed in to change notification settings - Fork 136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Announcement: Human face metadata entry to WebCodecs VideoFrame Metadata Registry #607
Comments
I'd prefer a more general approach, providing segmentation metadata rather than just metadata specific to face detection. The reason is that encoders can potentially use segmentation metadata to focus their efforts. So if we created face detection metadata, we might also need background blur metadata, whereas if we specify segmentation metadata, we could potentially use that for both face detection and background blur, as well as generic segmentation. Related: w3c/mediacapture-extensions#79 |
FYI, the metadata proposal has been modified to make it more generic so as to make it more useful for segmentation. This in turn raises the question as to how a WebCodecs Encoder could take advantage of the segmentation information (e.g. per-macroblock QP). |
One concern I have is that this adds face detection specific attributes under a top level So I'd prefer to either rename Aside from that, I'd be happy to review a PR that adds to the registry. |
Discussed at TPAC:
|
I've just noticed that this is already a part of Media Capture and Streams Extensions editor draft |
@Djuffin If a registry entry would not affect the behavior of WebCodecs or any other MEDIA WG specification, I am wondering why defining the metadata entry in Media Capture and Streams Extensions isn't sufficient. Are we concerned about naming conflicts? That could be addressed without a registry. Or is the concern about poorly designed extensions? If the entries don't affect the behavior of WebCodecs or other MEDIA WG specs, why burden the MEDIA WG with the need to review other WG documents? The lack of convincing motivation could explain why no registry entries have been approved since the registry was created. |
Monkey patching of other specs is bad practice and frowned upon, they should open a PR here and we should merge their stuff. Let's revert it from their spec. |
@padenot Do you think we should continue with the registry approach here or merge into WebCodecs spec? |
The registry has all the right characteristics I think. |
AIUI the plan we agreed between WebRTC and Media WGs was to add the face detection metadata as a registry entry, so I'm not clear why that didn't happen and it was added directly to Media Capture and Streams Extensions. |
The experience with use of registries is that when review is required, the effective use of registries requires speedy review to acceptance or (justified) rejection. If registries don't have effective reviewers attached, people will search for other solutions to get their extensions published while they wait for the review to complete. |
AIUI, all we seem to be missing is the PR on the metadata registry to add this entry, which would point to the Media Capture and Streams Extensions section. We could consider moving the metadata definition in a separate document but I do not think this is a blocker. |
Review times aren't a problem in Web Codecs, reviewers are well known and the issues are triaged every week or even day. Worst case it's fairly trivial to ping one of us to get something merged. Please do not monkey-patch specifications, and instead do the correct thing, which is to define the interface in a centralized location, which is one of the purpose of a registry. You can then link to it from other specifications, and all is good, and it's easy and clear. |
@alvestrand One reason that the review was not done is that it is not clear who was to do it or what they were supposed to have done. There is no "Designated Expert" assigned to review VideoFrame registry entries, nor are there guidelines for the review that such an Expert would do. If the registry policy was "WG Consensus" then a WG Consensus call should have been done in the MEDIA WG, but that didn't happen either, and in any case it's not clear what the MEDIA WG would have been asked to come to consensus about. |
There is no monkey patching AFAICS.
Ah, I do not think we have the same understanding of what the registry is then. The actual definition of the entry, including its interface, can be in another document, which does not necessarily have to be managed by the Media WG. AISI, what the Media WG adds in terms of coordination is related to the top level member names. |
It's perfectly conceivable that a https://www.w3.org/2023/Process-20231103/#registries has some info on what registries are and are for. Of the list in the introduction, that outlines the purpose of registries:
I'm fine with your PR, at least now it's possible for someone that implements the Web Codecs API to know of the existence of this feature, which is mostly what I care about here, and it's also not possible for someone else to use the same name. I'd say someone that want to extend this interface would click on all members to understand what they are about, so non-duplication is also guaranteed. It would be nicer to have all the interfaces in a centralized location, since this is what is done for all the other registries in Web Codecs, it really isn't hard. This was discussed by the right people, including a Web Codecs editor in the other PR, everybody agreed. |
Notes from 14 May 2024 Media WG meeting here. |
The key part of the requirments we came up with at the time were:
Item 3 enables WebRTC WG to define its own spec as a registry entry, rather than requiring them all to be Media WG specs, or live in this repository even. But we can change that if it makes sense to, but I worry about the complexity of having multiple WGs managing different specs in this repo. Item 5 means the Media WG has to agree to add the registry entry, which I hope wouldn't delay things coming from WebRTC WG by too much, but it would give the opportunity for (at least partially) independent review of proposed additions. We discussed in the meeting yesterday about evolving the requirements as we learn, so if we need to write guidance on what's expected from a metadata spec we can. On monkey-patching, do we want all possible registry entries to be in a single spec? I thought the partial dictionary approach was a good one, allowing different metadata use cases/applications to have their own spec. |
OK, let's go with this PR in the short term then, this seems inline with the current requirements quoted by @chrisn.
The codecs registry is just a list of pointers to other specs, the current model is following that approach.
Sure, but it would probably not change the WebCodecs spec algorithms, so no need for monkey patching. To be noted also that the segment metadata is also related to other getUserMedia APIs hence why its current location is not a bad choice either. |
What level of interest or implementer commitment is needed to add a feature to Media Capture and Streams Extensions? And similar for how the face metadata spec might "graduate" to Media Capture and Streams (or possibly it's own standalone spec?) I'm wondering if we want the registry requirement to require a /TR spec. |
It's the same as for mediacapture-main. New features are merely triaged away from mediacapture-main which is preparing for REC (which unfortunately is taking longer than anticipated). This is modeled on webrtc-extensions which existed ahead of its present post-REC role, as a way to get webrtc-pc to REC. If face metadata were to gain two implementations (pre- or post-REC), we'd move it to mediacapture-main, similar to w3c/webrtc-pc#2953. How does the Media WG manage updates to EME and the two implementation requirement? |
As per requirement of the VideoFrame Metadata Registry, this issue has been filed to allow discussion about the face metadata on being added into VideoFrameMetadata dictionary. The proposal is in the Media capture extensions PR #78 and the face detection explainer shall be updated.
To prevent discussions in multiple places, I do suggest that the discussion about the metadata is actually done in Media Capture Extensions Media capture extensions PR #78 where the explainer and spec will be updated, instead of this issue.
The text was updated successfully, but these errors were encountered: