Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding cuda kernel (optimized for sm80) for block-wise 4b quantized float 16 GEMM. #18619

Merged
merged 13 commits into from
Mar 5, 2024
Merged

Adding cuda kernel (optimized for sm80) for block-wise 4b quantized float 16 GEMM. #18619

merged 13 commits into from
Mar 5, 2024

Conversation

chenfucn
Copy link
Contributor

Description

Adding CUDA kernel for block-wise 4b quantized float 16 GEMM, this is specially optimized for Nvidia Ampere GPUs.

Motivation and Context

Trying to improve quantized LLM inference performance on Nvidia Ampere GPUs

Note:

This is implemented by extending CUTLASS, so it has a hard dependency on CUTLASS. However, in current build system, loading of CUTLASS dependency is guarded with:

(onnxruntime_USE_FLASH_ATTENTION OR onnxruntime_USE_MEMORY_EFFICIENT_ATTENTION)

If both of these options are turned off, then compilation will fail.

Why CUTLASS dependency is guarded at all? It's a header file only library that does not introduce any binary if not instantiated. What's the downside of removing all the guards and just include CUTLASS unconditionally?

snnn
snnn reviewed Nov 30, 2023
View reviewed changes
cmake/CMakeLists.txt Outdated Show resolved Hide resolved
yufenglee
yufenglee approved these changes Mar 5, 2024
View reviewed changes
Copy link
Member

@yufenglee yufenglee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@chenfucn chenfucn merged commit 06e684c into microsoft:main Mar 5, 2024
91 of 94 checks passed
zz002 pushed a commit to zz002/onnxruntime that referenced this pull request Mar 7, 2024
@chenfucn
Adding cuda kernel (optimized for sm80) for block-wise 4b quantized f…
7b7179a 
…loat 16 GEMM. (microsoft#18619)

### Description
Adding CUDA kernel for block-wise 4b quantized float 16 GEMM, this is
specially optimized for Nvidia Ampere GPUs.


### Motivation and Context
Trying to improve quantized LLM inference performance on Nvidia Ampere
GPUs

### Note:
This is implemented by extending CUTLASS, so it has a hard dependency on
CUTLASS. However, in current build system, loading of CUTLASS dependency
is guarded with:

(onnxruntime_USE_FLASH_ATTENTION OR
onnxruntime_USE_MEMORY_EFFICIENT_ATTENTION)

If both of these options are turned off, then compilation will fail.

Why CUTLASS dependency is guarded at all? It's a header file only
library that does not introduce any binary if not instantiated. What's
the downside of removing all the guards and just include CUTLASS
unconditionally?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@snnn snnn snnn left review comments

@yufenglee yufenglee yufenglee approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@chenfucn @snnn @yufenglee