[Feature Request] More realistic benchmark and throughput optimization

[tp-nan]

TensorRT-LLM has better kernel optimization, better model ecosystem support, and more quantization algorithm support compared to other large model inference libraries. However, during use, it is not easy to demonstrate its advantages in throughput scenarios(#73 #1097 #819 #965 #1255). The current benchmark is limited to designated parameters( for instances, batch sizes) that is not very close to reality.

Is it possible to replace the benchmark method with vLLM's type of benchmark_throughput, and improve throughput in this scenario? #632

[Missmiaom]

Any updates?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

[github-actions]

This issue was closed because it has been stalled for 15 days with no activity.