Video StreamingDataset setup

[kshitijkg]

No description provided.

[t46]

Option 1:
We store entire youtube video and captions using mds writer, we can store bytes. Then in the data loader, in the collate fn, we split the youtube data into multiple clips and return both text and vision in batched format where the batched vision data has dimensions: (B, C, T, H, W, Ch)

Option 2:
We store splitted youtube videos as arrays and dont do the splitting online

[t46]

Uploaded a sample script for Option 2.
https://github.com/t46/video-dataset

[t46]

Uploaded the script for option 1 as well, and stored scripts both for option 1 and 2 under the videorl repo.

https://github.com/TheDuckAI/videorl/tree/feature/streaming-dataset/code/data/streamingdataset/preprocess

[t46]

Updated for 1fps
https://github.com/TheDuckAI/videorl/tree/feature/streaming-dataset/code/data/streamingdataset/preprocess

[t46]

Updated

• save all frames including those with no subtitles
• split frames and subtitles equally
• update speed comparison b/w option 1 and 2
  https://github.com/TheDuckAI/videorl/tree/feature/streaming-dataset/code/data/streamingdataset/preprocess