-
Notifications
You must be signed in to change notification settings - Fork 136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Long term reference support for AVC/HEVC #743
Comments
We (Chrome implementers) discussed this recently. It's possible, but there are some design considerations that need to be solved:
It's also not clear to me what the exact behavior is when a large number of IDs are outstanding. There is only so much room to store references, eventually the encoder has to drop them. How much latency can there be before these schemes fall apart? How does this vary between encoder implementations? |
Are you suggesting to add something to be checked by MediaCapabilities API?
I may create prototype CL in Chromium to examine the behavior on Windows at least. |
[BA] The viability of LTR in conferencing scenarios relies on assumptions about how participants manage long-term references. They're called long-term references because the participants need to keep them long-term or else an encoder can't be assured that it can rely on an LTR to create a delta frame that all the conference participants will be able to decode. And of course, regardless of how long LTRs are kept, you can have a recent joiner who won't have received the LTR. So it is quite possible for an encoder to create a delta frame based on an LTR and then receive a PLI in response. Ooops! |
More likely a flag in |
sandersdan@ max number of outstanding long term ref picture indices(as well as max value of long_term_ref_frame_idx + 1 ) must not exceed maximum possible reference pictures (for AVC/HEVC, it is 16); |
@Djuffin / @aboba To simplify agent implementation, we could support 1 active long term reference picture and refresh it time by time. To make the LTR usable, a few settings needs to be added to spec:
|
@taste1981 There has been some work on low level encoding APIs that could support reference control. We will discuss the status at the W3C TPAC 2024 joint meeting of the MEDIA and WEBRTC WG. A link to the 2023 joint meeting: |
Happy to have low level API for that. However, LTR is a special case, that specifying updateBuffers/referenceBuffers is not sufficient. For AVC/HEVC it's not only about updating reference slots, it also about marking of reference picture type. Of course by using short term reference it's possible for you to mimic behavior of long term reference, but you lose the interop with applications that really relies on true "LTRP" for error resilience. |
@Djuffin @sandersdan @alvestrand A question about the new encoder API which was recently checked into Chrome (for AV1). On the receive side, is there a way to ensure that an LTR is available to the decoder? For example, in a conferencing use case, it is not useful for the SFM to forward an LTR recovery request to a sender unless the participants to whom the new frame will be forwarded can decode it (e.g. the participants need to have received the LTR, decoded it and retained it as an potential reference). It turns that that the RTCP RPSI message does not provide sufficient information for the SFM to make the decision (it's only useful for the 1-1 case), so implementations have created proprietary RTCP messages (e.g. LTN for libwebrtc, a PLI extension for Teams, etc.). |
Long term reference picture marking and control is very useful to cope with frame lost in video conferencing, to avoid asking key frames that typically generates a large frame.
On Windows LTR marking is through three properties if HMFT is used:
https://learn.microsoft.com/en-us/windows/win32/medfound/codecapi-avencvideouseltrframe
https://learn.microsoft.com/en-us/windows/win32/medfound/codecapi-avencvideomarkltrframe
https://learn.microsoft.com/en-us/windows/win32/medfound/codecapi-avencvideoltrbuffercontrol
On Mac/iOS:
https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_enableltr
https://developer.apple.com/documentation/videotoolbox/kvtencodeframeoptionkey_forceltrrefresh
https://developer.apple.com/documentation/videotoolbox/kvtsampleattachmentkey_requireltracknowledgementtoken
https://developer.apple.com/documentation/videotoolbox/kvtencodeframeoptionkey_acknowledgedltrtokens
It is expected AVC/HEVC codec registration defines VideoEncoderEncodeOptionsForAvc and VideoEncoderEncodeOptionsForHevc with the LTR marking/refresh control added.
Can we add this?
The text was updated successfully, but these errors were encountered: