Full dynamic shapes support design

[CharlieL7]

Discussion of the current design for dynamic batch support and how it might be extended for full dynamic shape support.

[qianqing13579]

I think the reasons why migraphx does not support dynamic shape can list as follows:

1. MIGraphX IR does not support dynamic models
the following ops in MIGraphX IR can not support dynamic models

• shape
• reshape
• constantofshape
• slice
• expand
• range
• resize
• pad
• tile

the output shapes of some ops are depend on the value of input tensor which are called data dependent. These ops should be extended to support dynamic shape.

2. MIGraphX compilation needs shape,but dynamic models can not provide shape information in compilation.

3. gpu kernels can not run with different shapes
MIGraphX JIT kernel can not run with different shapes, such as pointwise jit kernels. what's more, some ops are replaced with other ops,such as resize and lstm,which can cause ops do not support dynamic shape.

4. MIGraphX can not compute shape in runtime
dynamic models need compute the output shapes of each op according to input shapes in runtime.

[CharlieL7]

@qianqing13579 I'm writing up a document that I will put in this repository's Wiki page about the current design of dynamic shapes in MIGX. We can then discuss the plan fowards.

[CharlieL7]

Here is the Wiki page https://github.com/ROCmSoftwarePlatform/AMDMIGraphX/wiki/Dynamic-Shapes-Design.

[CharlieL7]

The dynamic batch design that splits each size into submodules will only work for one dynamic dimension. This is intentional. For operators with a data-dependent shape function we will either have to use dynamic GPU kernels or change the instructions such that it works with zero padding. There is also the option to use the ref target for these operators and subsequent ones to get something "working".

[qianqing13579]

@CharlieL7 ，How to run resnet50 ONNX model with dynamic batch? can you give me a sample?

[CharlieL7]

Yes, here's a example using the C++ API that should be easy to changed to run resnet50: https://github.com/ROCmSoftwarePlatform/AMDMIGraphX/tree/develop/examples/migraphx/cpp_dynamic_batch. Let me know if you have any questions.

[qianqing13579]

@CharlieL7 ,resnet18 can run successfully. but yolov4 model can not run successfully. can the current design for dynamic batch support the data-dependent op?
the yolov4 model link : https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/yolov4/model/yolov4.onnx

[CharlieL7]

Data-dependent operators will have to be handled separately from dynamic batch. It's possible to handle with the same idea, but would be limiting. We're working on updates to get yolov4 and transformer based models working with dynamic batch. For yolov4 we're updating how we're handling Resize from ONNX.

[yiakwy-xpu-ml-framework-team]

The current dynamic batch solution means multiple copies in GPU : that is impossible for LLM models in LLM era.

Let's forget about this solution (this is designed for ASIC which compile model into chip in advance, in which case batch size must be fixed number).

GPU is indead naturally support dyanmic batches : activations can be allocated when produced and removed when data dependecy goes to zero!

Unlike General purpose ASIC such as Graphcore IPU chip, you have to reserve memory in compile stage (stack memory), while GPU ASIC can request memory just in need (HBM memory pool dynamically).

Hence the solution should turn to memory dynamically allocation for actiations .

Note you cannot perfectly infer shape from an arbitrary model with Reshape and Resize ops (this has been proved) when you don't know the batch size.

@CharlieL7

[CharlieL7]

Yes, this is possible with GPU kernels that support dynamic shapes. Many of the current graph optimizations and fusions will not be doable with dynamic kernels however. We have not yet planned to implement that feature. We're open to code contributions for this though!