📝 Update Docs: Merge multiple optimized datasets into one

[bhimrazy]

What does this PR do?

• Update docs to include merge multiple optimized datasets into one

Usage

import numpy as np
from PIL import Image

from litdata import StreamingDataset, merge_datasets, optimize


def random_images(index):
    return {
        "index": index,
        "image": Image.fromarray(np.random.randint(0, 256, (32, 32, 3), dtype=np.uint8)),
        "class": np.random.randint(10),
    }


if __name__ == "__main__":
    out_dirs = ["fast_data_1", "fast_data_2", "fast_data_3", "fast_data_4"]  # or ["s3://my-bucket/fast_data_1", etc.]"
    for out_dir in out_dirs:
        optimize(fn=random_images, inputs=list(range(250)), output_dir=out_dir, num_workers=4, chunk_bytes="64MB")

    merged_out_dir = "merged_fast_data" # or "s3://my-bucket/merged_fast_data"
    merge_datasets(input_dirs=out_dirs, output_dir=merged_out_dir)

    dataset = StreamingDataset(merged_out_dir)
    print(len(dataset))
    # out: 1000

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78%. Comparing base (e253c6c) to head (aebe65a).
Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #385   +/-   ##
===================================
  Coverage    78%    78%           
===================================
  Files        34     34           
  Lines      5022   5022           
===================================
  Hits       3931   3931           
  Misses     1091   1091           

[tchaton]

Love it ! I left it un-done to see if you would do it @bhimrazy ;) Test passed !