Text Track Renderer slot and <text-track-renderer> media-chrome extra element

[gkatsev]

Using native captions is great, since it means less code in the player and there are some features that only native captions can take advantage off, such as captions in PiP. However, there are some limitations, namely making native captions not be obscured by overlays such as custom controls and doing so in a cross browser way.

This proposal adds a new slot, that's positioned in front of the gestures layer.

<slot name="text-track-renderer"></slot>

When an element is slotted into it, it would signal to media-chrome that when captions are toggled, they should toggle between disabled and hidden, compared to disabled and showing as it is normally. A currently enabled text track would have its mode change to hidden.

This would make the expectation of media slotted elements is that captions with mode hidden work more concrete. It should also be possible to switch tracks by changing the modes to hidden. Hls.js based players would work, but would require some tweaks or configurations to properly support switching of tracks while keeping them hidden.

The element slotted into the new slot is expected to handle all the rendering of the tracks based on the textTracks of the media element and the chosen track that's exposed in the media-controller.

If the slotted rendered is removed, rendering should transition back to native rendering by changing the mode back to showing. Knowing when an element is added or removed from a slot should be possible with the slotchange event.

A slotted element should use media-controller's media-captions-showing attribute to get the selected textTrack and render the cues appropriately when the active cues change.

By default, no element should be slotted, but there should be a media-chrome extras element <text-track-renderer> which would use vtt.js to render the cues appropriately. Keeping a separate element that's in extras would open it up for media-chrome users to create their own element should they want to over using the media-chrome extras element.
Having this element would allow us to style it smaller when the top or bottom chrome is showing so that captions don't overlap (keeping things away from center chrome is a lot more complicated).

In addition, since we're relying on the native text tracks to still be there, we can swap the track's mode from hidden to showing when pip is requested so that for browsers that support captions in PiP would receive them. Fullscreen on iPhones is another place we'd want to do this.

[gkatsev]

I created a POC here #236. You can play around with it here: https://media-chrome-git-fork-gkatsev-text-track-renderer-mux.vercel.app/examples/advanced.html.
It has a sample text track render element. Ideally, it would use vtt.js for proper captioning rendering. This POC also simulates setting the track to hidden and back to showing when the renderer element is present and when it's removed.

The key point is that the text track renderer doesn't have much state itself. It mostly grabs the correct track via the media-controller and then using the media it has access to the text tracks.
When the element is slotted, in the example, the media-controller gets the media-custom-text attribute set so that relevant components could check to know whether that's happening or not. In addition, we'd want the media-controller to emit some kind of event that would notify users that if they were setting the captions to showing previously, they should change it to hidden (though, I believe that's handled by the media-controller as is).

[heff]

Thanks for writing this up! At a high level, I see three different places we could solve this.

1. Require this be a feature of the media element itself, potentially polyfilling vtt regions to support moving captions out of the way of controls.
2. Making it a media element wrapper
3. Building it into media chrome in some way as described here

Number 1 honestly feels ideal to mean. I don't know if regions are meant for this, but I want the video element to solve for this.

With Number 3, the thing I'm most worried about is the layout complexity it introduces. How exactly do captions get out of the way of controls? I don't see an easy and clean way of doing that. Maybe if we could put captions in "middle-chrome" and have middle chrome expand when there's nothing in top/bottom-chrome.
I wouldn't be excited about a solution that puts more work on component authors, to make their components "captions aware" somehow.
I didn't see this solved in your demo at least, so let me know if you have thoughts on that.

What are the other reasons today, besides moving out of the way of controls, for solving this in a custom way?

[gkatsev]

Having the media element be in charge of this would sort of be ideal; however, we need to know the height of the control bars and potentially the size and locations of other overlapping elements and such an API doesn't exist right now, and I don't have any ideas for a good API for this right now. However, this could feed into w3c/webvtt#503 if we find a good API for this. I also don't think this should be a burden on media-element maintainers, so, I would say we'd want to do #​2 or something similar to the cast API, so, any media element could take advantage.

Because of the lack of this API, I figured we should handle it outside the media element. With the Text Track Render element having a display element that is aware of the media-chrome and can be resized appropriately when the controls are visible. For example, for bottom-chrome-only, the height of the display area can be made smaller by the size of the control bar. Won't help with arbitrary overlays, but that's a lot trickier to handle anyway, and I think we can ignore that for now.

I don't think that many components would need to be aware that non-native captions are set up, since the media-controller is the one in charge of setting the mode is can handle switching the mode to showing when there are cases that benefit from it, like starting a casting session or entering pip or fullscreen on iPhones.

This proposal actually wouldn't solve VTT Regions because that would require enabling custom parsing of text tracks, since most browsers/players don't parse it currently; so, this proposal is only for rendering. VTT Regions were mainly for being able to offer 608-like features (with animated scrolling when new lines are added).

Making sure that the controls don't overlay the captions is a huge accessibility issue. A lot of players do nothing, or I've seen some that just hide the captions when the controls are showing. For people who rely on captions, being able to see the captions while interacting with the controls is a big thing.
In Mux Player, I currently have a mechanism for moving the captions, which is pretty platform independent. However, it's coming up with lots of edge cases due to sizing differences between how browsers render the captions, and recently we also ran into a pretty major bug in Safari where we'll get duplicated captions after seeking. Custom rendering would make the rendering consistent across all supported platforms. Since we'll still be relying on the native text tracks and being able to switch those from hidden to showing at a moment's notice would mean that we can still take advantage of native platform features like captions in PiP but also have more consistent cross-browser rendering other times.

[heff]

due to sizing differences between how browsers render the captions

Is there any way we can use CSS or other settings to create more of a consistent default across browsers?

I would say we'd want to do #​2 or something similar to the cast API, so, any media element could take advantage.

That makes sense.

This proposal actually wouldn't solve VTT Regions because that would require enabling custom parsing of text tracks

Ah right, makes sense.

I could see a solution that starts with media-container understanding the layout of its children and maybe giving x,y,w,h box coordinates of "uncovered" video display space. Then those coordinates could be passed to the custom captions component, or any other listener, or used to inform the media element some day. Maybe not that exactly, but if the Container could be smart in that way so nothing else needs to be smarter, that feels like a nice result.

[gkatsev]

due to sizing differences between how browsers render the captions

Is there any way we can use CSS or other settings to create more of a consistent default across browsers?

Unfortunately, not in a browser-consistent manner 😅. Particularly, not with accounting for user preferences, which is important for accessibility.
With the custom rendering, we'll eventually need to build a captions user settings dialog.

if the Container could be smart in that way so nothing else needs to be smarter, that feels like a nice result.

Yeah, I think the bulk of the logic can be contained inside the media-container or the media-controller. The main thing that I'm uncertain is whether we can easily know from within the media-container/controller what the sizes of the control bar are. Though, could still likely have sensible defaults. The main thing I'm thinking about is having the captions move up a bit when the controls are up, and then move up even more when the thumbnails are then shown, which I think would be easier for having media-chrome users (or themes!) apply styles to the text track display since they would definitely know the size of the control bar and other things that might be in the way.
Originally, I was thinking that having the slotted element would be simpler, but I think we can expose a part for the text track display from the media element wrapper and that would work fine too.

I think the main benefit of the separate slotted text track renderer over wrapped media element is that we can turn on and off the custom rendering without requiring swapping out media elements.

[heff]

and then move up even more when the thumbnails are then shown

Is this a real need? It starts to feel like different use cases, but mostly that adds some real complication for knowing total height if the thumbnails aren't impacting their parent's size.

the sizes of the control bar are

We do need to be careful about making the control bar "known", as opposed to maybe "whatever's in bottom chrome".

apply styles to the text track display since they would definitely know the size of the control bar and other things that might be in the way

Yeah, that would work. If it's a separate component you have to choose to include, that feels like it sets up the right expectations.

Originally, I was thinking that having the slotted element would be simpler, but I think we can expose a part for the text track display from the media element wrapper and that would work fine too.

Do you think we'll ultimately want this included by default? If no I could go either way, but if yes, then I'm starting to lean towards the default slot.

we can turn on and off the custom rendering without requiring swapping out media elements

Is that to support the PIP situation? Swapping out media elements isn't really an option here.

I'm seeing maybe a few potential options that might solve the 95% use case at least. Assuming a slotted text track renderer.

1. Utilize middle-chrome, set the the track renderer to 100% of middle chrome, and make sure top and bottom chrome collapse to zero height when everything in them is hidden.
2. When controls are showing (user active) report the bounding box of middle-chrome, so wherever the renderer is it can absolutely position itself.

I'm assuming we don't have to move captions around the centered chrome/mobile UI when it shows, right?

[gkatsev]

and then move up even more when the thumbnails are then shown

Is this a real need? It starts to feel like different use cases, but mostly that adds some real complication for knowing total height if the thumbnails aren't impacting their parent's size.

I do think it's useful to have. We should try hard so that users that rely on captions will always have the captions visible. Though, one alternative is to make the captions move to the top of the display rather than just push up (then when the controls are active it's always at the top and won't move). But that also gets complicated unless we know for sure there aren't captions at the top.

Originally, I was thinking that having the slotted element would be simpler, but I think we can expose a part for the text track display from the media element wrapper and that would work fine too.

Do you think we'll ultimately want this included by default? If no I could go either way, but if yes, then I'm starting to lean towards the default slot.

I think we might want to have it by default, since Firefox doesn't currently have the ability for the viewer to set caption styling.

we can turn on and off the custom rendering without requiring swapping out media elements

Is that to support the PIP situation? Swapping out media elements isn't really an option here.

Not necessarily for PiP, more of if someone wanted finer controls over whether the captions are rendered natively or not. However, thinking about it some more, I think we can have the attribute on the media-container/controller act as a signal instead, if necessary. Like, add it to get custom render and remove it to have native rendering.

I'm seeing maybe a few potential options that might solve the 95% use case at least. Assuming a slotted text track renderer.

1. Utilize middle-chrome, set the the track renderer to 100% of middle chrome, and make sure top and bottom chrome collapse to zero height when everything in them is hidden.

2. When controls are showing (user active) report the bounding box of middle-chrome, so wherever the renderer is it can absolutely position itself.

Reporting the bounding box might work. The idea is that this is the default behavior, but users media-chrome should make sure that they style the Renderer's display appropriately depending on their theme?
And we'll want to make sure that controls are still accessible to Screen Readers when hidden.

I'm assuming we don't have to move captions around the centered chrome/mobile UI when it shows, right?

Ideally, we'd want to make sure that nothing overlaps the captions, but the bottom controls are the big issue because most captions are at the bottom.

An alternative to this whole thing is to do what Crunchyroll did and have a control bar design that is never in the way of the default caption location. I'm not sure that we want to go that far for the default theme, but we should definitely have a theme like that.

[heff]

An alternative to this whole thing is to do what Crunchyroll did and have a control bar design that is never in the way

If we were just building a player that could be an option but we can't make that decision for everyone using media chrome.

Ideally, we'd want to make sure that nothing overlaps the captions

Yeah, seeing as media chrome users can put controls anywhere in the display area, making sure nothing ever overlaps captions feels impossible, so there is no perfect here and we've gotta consider the most common use cases.

I'm thinking you're right that the cleanest way to solve for this is just make theme builders or media chrome users manually set the size of the text renderer as they're designing the controls. But with that being the case, I don't think this can be the default. It has to be the advanced version of captions, so people are opting into the complexity, not being forced into it, and it's clear how it works because you're intentionally adding the custom element in the clear, not working with a semi secret part.

Here's what I'm understanding could be the option then:

1. A (empty) media container slot for a custom track renderer to sit higher (z-index-wise) than the media but lower than controls
2. A custom text track renderer that media chrome users can optionally add to the custom slot if they want more control over captions layout and more consistency between browsers
3. A media-controller attribute that allows you to turn of media caption rendering, so they're not duplicating the custom renderer.

How does that sound?

[gkatsev]

An alternative to this whole thing is to do what Crunchyroll did and have a control bar design that is never in the way

If we were just building a player that could be an option but we can't make that decision for everyone using media chrome.

Yes, though, a crunchyroll-like theme would be a great custom theme.

1. A (empty) media container slot for a custom track renderer to sit higher (z-index-wise) than the media but lower than controls
2. A custom text track renderer that media chrome users can optionally add to the custom slot if they want more control over captions layout and more consistency between browsers
3. A media-controller attribute that allows you to turn of media caption rendering, so they're not duplicating the custom renderer.

How does that sound?

Sounds good, though I'm not sure that 3 is necessary. media-controller can assume custom rendering when the slot is filled, unless you're thinking that the attribute allows toggling between native and custom rendering without removing the slotted item?

[heff]

media-controller can assume custom rendering when the slot is filled

I'm good with that.