-
Notifications
You must be signed in to change notification settings - Fork 24
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'feature/media-pipeline' into feature/media-info
- Loading branch information
Showing
23 changed files
with
1,051 additions
and
32 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file added
BIN
+74.4 KB
requirements/images/specifications/media/media-pipeline/media/image1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+156 KB
requirements/images/specifications/media/media-pipeline/media/image2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+74.2 KB
requirements/images/specifications/media/media-pipeline/media/image3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+178 KB
requirements/images/specifications/media/media-pipeline/media/image4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+155 KB
requirements/images/specifications/media/media-pipeline/media/image5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,267 @@ | ||
| # Media Pipeline Requirements | ||
|
|
||
| Document Status: Working Draft | ||
|
|
||
| See [Firebolt Requirements Governance](./governance.md) for more info. | ||
|
|
||
| **NOTE**: Update this link based on your directory depth ^^ | ||
|
|
||
| | Contributor | Organization | | ||
| | -------------- | -------------- | | ||
| | Lucien Kennedy-Lamb | Sky | | ||
| | Bijal Shah | Sky | | ||
| | Yuri Pasquali | Sky | | ||
| | Stuart Pett | Sky | | ||
| | Wouter Meek | Comcast | | ||
| | Jeremy LaCivita | Comcast | | ||
| | Kevin Pearson | Comcast | | ||
| | Phillip Stroffolino | Comcast | | ||
|
|
||
| ## 1. Overview | ||
|
|
||
| Playback of audio/video media is a first-class concern for Firebolt. As | ||
| such, Firebolt must provide a consistent, secure Media Pipeline API for | ||
| sending audio and video streams from an app container out to a decoder | ||
| and CDM for content decryption, decoding, and rasterization. | ||
|
|
||
| Additionally, Firebolt browsers must leverage the same Media Pipeline | ||
| implementation as native apps, so that content plays back the same in | ||
| both environments and can have resources managed in the same way. | ||
|
|
||
| This is achieved with the open source [Rialto Media | ||
| Pipeline](https://github.com/rdkcentral/rialto) API and implementation: | ||
|
|
||
|  | ||
|
|
||
| ## 2. Table of Contents | ||
| - [1. Overview](#1-overview) | ||
| - [2. Table of Contents](#2-table-of-contents) | ||
| - [3. Media Pipeline](#3-media-pipeline) | ||
| - [3.1. Media Pipeline Commands](#31-media-pipeline-commands) | ||
| - [3.2. MediaPipeline Notifications](#32-mediapipeline-notifications) | ||
| - [3.3. Media Pipeline Management](#33-media-pipeline-management) | ||
| - [4. W3C Media APIs](#4-w3c-media-apis) | ||
| - [4.1. MediaSource](#41-mediasource) | ||
| - [4.2. MediaElement](#42-mediaelement) | ||
| - [5. Supported Media](#5-supported-media) | ||
| - [5.1. Container formats](#51-container-formats) | ||
| - [5.2. Video codecs](#52-video-codecs) | ||
| - [5.3. Audio codecs](#53-audio-codecs) | ||
| - [6. Supported Decryption](#6-supported-decryption) | ||
| - [6.1. DRM Key Systems](#61-drm-key-systems) | ||
| - [6.2. Decryption Schemes](#62-decryption-schemes) | ||
|
|
||
| ## 3. Media Pipeline | ||
|
|
||
| The Firebolt Media Pipeline consists of an API for passing audio & video | ||
| streams *out* of individual app containers *into* managed media sessions | ||
| that support decryption, decoding, and rasterization. Additionally there | ||
| is an API for sending notifications out of the decoder and into the app | ||
| container. | ||
|
|
||
| The goal of this abstraction is to decouple individual Firebolt Apps and | ||
| containers from the underlying implementations of these features. | ||
| Additionally, this abstraction allows for the same component to manage | ||
| media sessions in a consistent manner regardless of implementation. | ||
|
|
||
| To ensure consistency, all native containers, including browsers, that | ||
| run on Firebolt platforms **MUST** use Rialto as the only pipeline to | ||
| Media decoders and CDMs. | ||
|
|
||
| The Media Pipeline API uses a client-server model to allow Firebolt | ||
| devices to manage media pipeline resources and visibility. This enables | ||
| multiple media pipeline sessions to be running at once, even though they | ||
| will not all have access to the underlying decoder. Command messages are | ||
| passed using an efficient binary RPC protocol. Media data is passed | ||
| using memory shared by only the app container and its respective Rialto | ||
| session server. | ||
|
|
||
|  | ||
|
|
||
| ### 3.1. Media Pipeline Commands | ||
|
|
||
| Media Pipeline commands go from the app to the Rialto Media Pipeline | ||
| server. | ||
|
|
||
| This section describes the use cases that the Rialto API solves. For | ||
| actual API semantics, please refer to the Rialto Github repository. | ||
|
|
||
| The MediaPipeline API **MUST** support pushing data into the Pipeline, | ||
| in response to a notification that more data is needed. | ||
|
|
||
| The MediaPipeline API **MUST** support informing the Pipeline that it | ||
| has reached the end of a stream and no more data is available. | ||
|
|
||
| The MediaPipeline API **MUST** support initializing (aka loading) a new | ||
| Media session, typically done before starting a new stream. | ||
|
|
||
| The MediaPipeline API **MUST** support attaching one video and one audio | ||
| source buffer. | ||
|
|
||
| The MediaPipeline API **MUST** support removing source buffers. | ||
|
|
||
| The MediaPipeline API **MUST** support pausing and playing. | ||
|
|
||
| The MediaPipeline API **MUST** support stopping playback, such that it | ||
| cannot be resumed without reinitializing. | ||
|
|
||
| The MediaPipeline API **MUST** support setting the playback rate. | ||
|
|
||
| The MediaPipeline API **MUST** support setting the current playback | ||
| position. | ||
|
|
||
| The MediaPipeline API **MUST** support setting the video location and | ||
| dimensions. | ||
|
|
||
| The MediaPipeline API **MUST** support dynamic audio decode switching to | ||
| support Netflix use cases. | ||
|
|
||
| ### 3.2. MediaPipeline Notifications | ||
|
|
||
| Media Pipeline notifications come from the Rialto Media Pipeline server | ||
| to the app. | ||
|
|
||
| This section describes the use cases that the Rialto API solves. For | ||
| actual API semantics, please refer to the Rialto Github repository. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app when more audio | ||
| or video data is needed in order for playback to continue without error. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app when a need for | ||
| more audio or video data is no longer relevant, e.g. due to switching | ||
| buffers or stopping. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app of the current | ||
| Media duration. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app of the current | ||
| Media position. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app of the current | ||
| native dimensions of the current Media. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app of the current | ||
| buffer state, e.g. buffered, idle, error. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app of the current | ||
| playback state, e.g. playing, paused, seeking, stopped. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app whether the | ||
| current media contains video and/or audio data. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app when frames of | ||
| video or samples of audio are dropped due to performance. | ||
|
|
||
| The MediaPipeline API **MUST** support notifying the app when EIA/CEA | ||
| 608/708 closed caption data is extracted and ready for presentation. | ||
|
|
||
| ### 3.3. Media Pipeline Management | ||
|
|
||
| Rialto also exposes a management API for deciding which Rialto session | ||
| is active and visible. The Firebolt Execution Environment, typically | ||
| Ripple interacts with this API as apps move in and out of focus. | ||
|
|
||
|  | ||
|
|
||
| **TODO**: add decode pool requirements, e.g. minimum decoders for a | ||
| Firebolt platform, etc. \-- This needs to be written out. | ||
|
|
||
| **TODO**: add colloquial description of API surface in Rialto Server | ||
| Manager. We need this written out for the management api. | ||
|
|
||
| ## 4. W3C Media APIs | ||
|
|
||
| The Media Source API enables the entire functional scope of the W3C | ||
| MediaSource and MediaElement APIs, (e.g. MediaSource.addSourceBuffer, | ||
| video.src, video.play(), etc.) | ||
|
|
||
| Firebolt browsers will use Rialto to implement `MediaElement` and | ||
| `MediaSource` APIs. | ||
|
|
||
| ### 4.1. MediaSource | ||
|
|
||
| Browser MediaSource APIs **MUST** be powered by Rialto. Higher level | ||
| concerns such as source buffer management, etc., are outside the scope | ||
| of this document. | ||
|
|
||
| When using MediaSource APIs, demuxing is the concern of the app, not the | ||
| browser. | ||
|
|
||
|  | ||
|
|
||
| ### 4.2. MediaElement | ||
|
|
||
| Browser Media Element implementations **MUST** support progressive | ||
| download and demuxing of the following media container formats: | ||
|
|
||
| - Progressive MPEG4 video | ||
|
|
||
| - Progressive MP3 audio | ||
|
|
||
| Demuxed segments **MUST** be sent to Rialto for decoding. | ||
|
|
||
| MediaElement implementations **MUST** not support additional formats, | ||
| e.g. HLS or DASH. These formats should be parsed and fetched by a player | ||
| implemented on top of MSE. | ||
|
|
||
| Legacy support for direct playback of HLS is outside the scope of this | ||
| document. | ||
|
|
||
|  | ||
|
|
||
| ## 5. Supported Media | ||
|
|
||
| The Rialto Media Pipeline supports the following media types. | ||
|
|
||
| ### 5.1. Container formats | ||
|
|
||
| The Media Pipeline **MUST** support the following container formats: | ||
|
|
||
| - Demuxed MP4 audio and video streams, via MSE | ||
|
|
||
| - MPEG 1 for audio-only playback (progressively fetched upstream) | ||
|
|
||
| ### 5.2. Video codecs | ||
|
|
||
| The Media Pipeline **MUST** support the following video codecs: | ||
|
|
||
| - AVC | ||
|
|
||
| - HEVC | ||
|
|
||
| - VP9 | ||
|
|
||
| - H.264 | ||
|
|
||
| ### 5.3. Audio codecs | ||
|
|
||
| The Media Pipeline **MUST** support the following audio codecs: | ||
|
|
||
| - HE-AAC | ||
|
|
||
| - AAC-LC | ||
|
|
||
| - HE-AACv2 | ||
|
|
||
| - MPEG-1 Audio Layer III (for audio-only media) | ||
|
|
||
| ## 6. Supported Decryption | ||
|
|
||
| The Rialto Media Pipeline supports the following media decryption types. | ||
|
|
||
| ### 6.1. DRM Key Systems | ||
|
|
||
| - Widevine | ||
|
|
||
| - PlayReady | ||
|
|
||
| - *I know there\'s **legacy use cases**\... do we call them out?* | ||
|
|
||
| ### 6.2. Decryption Schemes | ||
|
|
||
| - CMAF CBCS decryption |
Oops, something went wrong.