Potential Mirage projects

[jiazhihao]

List of potential projects on top of Mirage:

• CUDA code generator: currently Mirage's cutlass-based interpreter involves a non-trivial overhead since it visits an block graph every iteration to call the corresponding cutlass/cuda functions. This task implements a CUDA code generator (similar to our Triton code generator) for producing CUDA compilable code for a given ugraph.
• Python package. We will make Mirage a python package that can be directly installed by users using pip or conda.
• AMD support. The current implementation of Mirage targets NVIDIA GPUs. We need to work on adding AMD support.
• Mix precision support. Mirage currently assumes all intermediate tensors are stored in fp16. We will be adding fp8, fp32, and integer type support.

[jeromeku]

@jiazhihao Great paper!

Any plans on extending Mirage optimizations to account for newer NVIDIA architectures (Hopper+) that introduced additional hardware features and levels to the traditional GPU hierarchy -- e.g., TMA / distributed shared memory / GMMA ?

[nullhook]

re: AMD support, is there a way to add custom backends?

[structure-charger]

reuse tinygrad? https://github.com/tinygrad/tinygrad

[structure-charger]

or reuse opencl backend implementation ? https://github.com/artyom-beilis/dlprimitives or taichi backend implementation https://github.com/taichi-dev/taichi ?