Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we have pushed an update to the development branch (and the Triton backend) this Oct 29, 2024.

This update includes:

• Features
  • Supported Python plugin, see examples/python_plugin/README.md.
  • Supported different rank dimensions for LoRA modules when using the Hugging Face format, thanks for the contribution from @AlessioNetti in Allow for LoRA modules with different rank dimensions when using HF format #2366.
  • Supported automatic embedding sharing when possible, no need to explicitly enable.
  • [BREAKING CHANGE] NVIDIA Volta GPU support is removed in this and future releases.
• Bug fixes
  • Fixed a slice error in forward function. (Encountered an error in forward function: slice 712 exceeds buffer size 471 #1480)
• Infra
  • The base Docker image for TensorRT-LLM is updated to nvcr.io/nvidia/pytorch:24.09-py3.
  • The base Docker image for TensorRT-LLM Backend is updated to nvcr.io/nvidia/tritonserver:24.09-py3.
  • The dependent TensorRT version is updated to 10.5.0.
  • The dependent CUDA version is updated to 12.6.1.
  • The dependent PyTorch version is updated to 2.4.1.
• Known Issues
  • There is a known issue in TensorRT 10.5 when building the MLLaMa model. Downgrade tensorrt package version to 10.4.0 as a workaround. This issue will be fixed in a future release.

Thanks,
The TensorRT-LLM Engineering Team