Allow selection of the active stream

[Ceylo]

A media can have several audio, video or subtitle streams.

CA:

• be able to know what are the available streams
• be able to select the active stream for audio, video or subtitle before playback did start
• be able to select the active stream for audio, video or subtitle at any time

[Ceylo]

After some study I found out it's not possible to correctly implement stream selection without seeking support. See this page if interested :)

[bluekirby0]

It is still helpful to support stream selection before playback begins, even if it is not possible (without seeking) to support changing streams on-the-fly. This can be helpful for videos that contain audio streams in different languages, and the general configuration of the calling program can determine which language audio stream is desired before playback occurs.

Subtitles could be rendered on a separate surface and simply shown or hidden by a toggle if desired. This has the serious disadvantage of needing a separate ffmpeg instance to render it to a surface with alpha.

Of course if you plan to add in proper seeking support then this is all rather moot.

[Ceylo]

Ah yes good point! I didn't thought of this case. And selecting the good stream before starting indeed doesn't require seeking support, which makes it much easier to support :)

It'll be studied once work on this task resumes.

As for subtitle, it's not supported yet so… but do you mean rendering all found subtitles from all streams then choose which one to display eventually? Why do you say a separate ffmpeg instance would be needed?

[bluekirby0]

Oh I mean if you wanted to support subtitles without supporting seeking, you could render them alongside the video on a separate surface and overlay that surface on top of the video when subtitles are desired or move the surface offscreen (or otherwise hide it) when they are not. Then you would just need to tie all streams into the same "playback" controls.

With how ffmpeg handles subtitles, though, there is no way to render subtitles separate from the video without treating it like an entirely separate video (which means instantiating much of ffmpeg AGAIN, even though you are probably working from the same source file).

So what you would have (overly-simplified) is:

FFMpeg Source Filter->Splitter->Video Decoder->Renderer1
    ->Audio Decoder->Audio Sink
    ->Subtitle Parser->NULL
FFMpeg Fake Source Filter (to generate an alpha render surface)->Renderer2
FFMpeg Source Filter->Splitter->Video Decoder->NULL
    ->Audio Decoder->NULL
    ->Subtitle Parser->Renderer2

Again this last part is moot if you plan to support proper seeking eventually, as the work put into this setup would be wasted.

[feliwir]

We already have a semi-working subtitle implementation which you can look in the "subtitle" branch. It's still WIP and i need to fix some issues

[Ceylo]

@bluekirby0 I'm really having a hard time understanding your solution :(

What is currently being done and what @feliwir uses on the subtitle branch is the following:

• sfeMovie's demuxer reads the container and finds all streams
• all streams are notified when playback start and it's their responsability to gather the required data from sfeMovie's demuxer, process it and tell the Movie class that they have data ready to render
• for subtitle it's more or less the same except that the stream will probably be passive (ie. won't request data from the demuxer) because subtitles packets can be much later in the movie, and requesting them requires reading and buffering all packets found before the subtitle packet. We don't want that. Instead, we plan to let other streams (audio & video) request their packet, and when a subtitle packet is found by the demuxer it's fed to the subtitle stream which will use it.

So no separate FFmpeg instance or source filters here, I don't exactly know what you mean with them.

[bluekirby0]

Yeah, don't worry about what I was saying then. I was looking at it from a different approach (and not a very suitable one). It would be a work-around at best.

[Ceylo]

Haha ok fine then :)

[Ceylo]

Support for stream selection when the movie is stopped has been merged into master. Support for selection during playback will be done when seeking is ready.

I've tested with multiple audio streams only, it's supposed to work with multiple video streams too, but I've no such media file to give it a try. If you've one I would be interested!

[bluekirby0]

I don't have a short clip handy but I know of a perfect source I can use to make one. I think Matroska and VOB are the only containers that actually support multiple video tracks though (and VOB does it in a really roundabout way) so I'll prepare something in mkv.

[Ceylo]

What I also wonder is what is the interest of such medias? For network streams it allows using the one adapted to your network speed. But for files… what's the point?

[bluekirby0]

It was actually commonly used on DVDs (by abusing the "multi-angle" and "playlist" features of the format) to change the title screen (or other on-screen text that subtitles were not suitable for) for multi-language releases, particularly in western Europe.

You could probably accomplish something similar by using the matroska "ordered chapters" feature, or by using splitters that support selecting tracks based on language codes.

[Ceylo]

Hmm I don't think we want to go that far with sfeMovie (especially for DVDs that are not meant to carry on). So if it's the only usage, this support won't be provided.

[bluekirby0]

It is of considerably less use than proper soft-sub support, considering libass provides enough functionality to generate nice, language-specific titles/signboards/etc... without the compression artifacts involved with hard-subbed video with sharp text.

Granted, some hard-core Adobe AFX users will disagree with how useful libass is for rendering fancy effects in real-time and that only hard-coded typesetting can produce the results they want without grinding playback speed to a slow crawl.

Overall the usefulness of this greatly hinges on whether this library will be used for multi-lingual releases that will take advantage of advanced typesetting techniques to support several languages without massively inflating the release size by creating video-per-language files for playback.

Again, you can probably still manipulate the matroska "ordered chapters" feature for this to some extent without modifying your library by creating "dummy" videos for each language that just define the proper chain of videos to play for that language. I am assuming that ffmpeg's internal splitter handles ordered chapters correctly, though, and that it does not require special handling on the playback side.

[Ceylo]

Do you know where I can find such matroska file?
If it requires non-negligible fixes it'll probably not be supported. sfeMovie's purpose isn't about going that far at the moment.

[bluekirby0]

I can probably generate something as a sample for you to test...the tools are readily available if you are familiar enough with them to use them.

[Ceylo]

Do you mean the tools to create such files?

[bluekirby0]

Yes. MKVToolnix provides both command-line and gui tools to produce and edit matroska containers and valid chapter files (which are XML). From some initial testing it looks like ffmpeg does not support ordered chapters that load segments from external files, but it may still be possible to use segments based on keyframes within the same file, for example:

File layout:
Scene1->Scene2English->Scene2French->Scene2German->Scene3

Playback branching via chapters (which in mkv can also be language coded):
Scene1->Scene2English->Scene3
Scene1->Scene2French->Scene3
or
Scene1->Scene2German->Scene3

So basically you can create a "super" clip that contains every scene for every language and then use the chapter file to determine which scenes will be played and in what order.

[Ceylo]

Ok I got it :)
So if I sum it up:

• it does not look to be supported by FFmpeg
• it's quite an advanced feature compared to sfeMovie's purpose
• it would require stream selection during playback
• it's mostly used by DVD-like movies, and even among these, it only concerne the movies for which libass isn't enough

These are several reasons which make me think it won't be supported in the short term. Which brings us with the fact that we don't care if stream selection for video streams actually does work.

[bluekirby0]

Fair enough. If this was a general purpose video player it would be slightly more important, but as simple library where video can be tailored to meet the requirements of the program it is far less important.