Media file

[Joery-M]

This document explains how a file is represented as a media item.

Importing

When a file has been imported by the user, 2 things will be stored.

1. Metadata

First is the metadata that will be used to get information about the file.
For video, it are things like codec, colorspace, duration, framerate and anything required for VideoDecoder.configure
For audio, it are things like sample rate, bit depth, channel count and anything required for AudioDecoder.configure
For images, it are things like width, height, animated or not, frame count and anything required for ImageDecoder

Plugins are be able to add their own data alongside the default data. This data needs to be segmented per plugin to avoid overwriting data.

2. File contents

The original file is not stored, instead, only the most important parts of the file are stored. If a plugin requires more information about a file, the file needs to be re-imported. (Open for discussion)

File contents stored:

• EncodedVideoChunk
• EncodedAudioChunk
• Image data

Reference

A media file item should inherit from the base class of Media Item (#286).

Example media item implementation:

interface MediaFileItem extends MediaItem {
    /**
     * The mime type of this media item.
     */
    mimeType: string;
    /**
     * Whether this media item is currently able to be used.
     */
    ready: boolean;

    getMetadata: <Data = any>(plugin?: string) => Data | null;
    addMetadata: (plugin?: string) => boolean;
}

interface ChunkedMediaItem extends MediaItem {
    /**
     * Read a specific encoded chunk.
     *
     * Chunk index should be derived from metadata.
     */
    readChunk: (index: number) => Promise<EncodedVideoChunk | EncodedAudioChunk>;
}

Example video metadata:

interface VideoMetadata {
    /**
     * Width of the video frame in pixels
     */
    width: number;
    /**
     * Height of the video frame in pixels
     */
    height: number;
    /**
     * Duration of the video in milliseconds
     */
    duration: number;
    /**
     * Duration of a single frame in milliseconds
     *
     * If the video has variable frame-rate, this value is the maximum frame duration.
     */
    frameDuration: number;
    /**
     * Only present if the video has variable frame-rate
     */
    variableFrameDuration?: {
        /**
         * Shortest frame duration in milliseconds
         */
        min: number;
        /**
         * Longest frame duration in milliseconds
         */
        max: number;
        /**
         * Average frame duration in milliseconds
         */
        mean: number;
    };
    displayAspectWidth?: number;
    displayAspectHeight?: number;
    colorSpace: VideoColorSpace;
    /**
     * Valid codec string to be used in VideoDecoder.configure
     *
     * @see https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder/configure#codec
     */
    codec: string;
    /**
     * Codec-specific bytes used for decoding
     *
     * @see https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder/configure#description
     */
    codecDescription?: ArrayBuffer;
}

Limitations

• No support for dynamic frame size.

Storage

The file contents are written to an Origin private file system to allow high performance and partial reads.
Metadata is written to the metadata field of the IndexedDB entry, Including a description of where the file is stored in OPFS.

For chunked items, like audio and video, chunk indexes are written to IndexedDB in a way that chunks start with an i-frame.