Add HLS Support to SiteMedia

[charleswaddell]

Figure out what the data structure is going to look like. We have a meeting to do this on March 10th at 10 AM.

[isagrant]

Summary of things we'll need to do to setup HLS:

• Sort out data structure changes to accommodate HLS and non-HLS encodings (for download and Android) - @charleswaddell
• Review current ingest workflow and plan out future workflow with a new encoding backend - @isaacgrant
• Figure out how much effort it is to split old encodings into HLS streams. - @charleswaddell
• Plan transition

[isagrant]

Also discussed - moving SiteMedia stuff into it's own package.

Things to remember:

• SiteMedia is not just used for Video. It also supports audio on the Rap sites.
• MediaSets are currently awkward to switch between, both because of physical moving of files on disk, and also because of encoding profiles not being the same between sets.

[isagrant]

Met today to discuss the data-structure of this.

@charleswaddell looked into data structure, and it looks like all we need for a set of HLS encodings is a single playlist file. We're currently thinking we can just add a hls_encoded flag to the Media table to handle this.

I still owe the group a ingest pipeline report, which will include where and how we store the files.

@charleswaddell is going to see how fast/expensive it would be to split all our existing encodings locally then upload to s3, and on ec2 with ffmpeg directly into the s3 bucket, and then compare those costs to just re-encoding all the video from the biggest encoding already on s3, so we can decide how to proceed with existing video.

[charleswaddell]

Next steps:

• Add HLS structure
• EC2 Instance to split existing (don’t use reduced redundancy).
• Setup sites to use HLS
• Copy current into new structure
• Keep old structure around until better tested
• Download split files to pomelo for on-site backup

[nburka]

During the conversion process, let's also temporarily use the has_hls boolean to say whether the original has been copied into the new /media/id/full/id-encoding.mp4 file structure. If has_hls is true, getUri will use the new file structure will be used, and if false, the old file structure will be used.

[nburka]

Let's not delete the original from /media/media-set/encoding/id.mp4 right away. This will prevent people currently streaming videos from having their videos cut off. Let's wait 24 or 48 hours before removing the originals.

[isagrant]

I think this could be closed. The new encoding pipeline isn't set up, but is really a separate task.