Code

Implement FlashAttention for CPU#20805

Code

Merged

yufenglee merged 36 commits intomicrosoft:mainmicrosoft/onnxruntime:mainfrom duanqn:qiduan/flashduanqn/onnxruntime:qiduan/flashCopy head branch name to clipboardJul 11, 2024

+363-5

Commits

Commits on Jul 11, 2024

• Register new contrib op FlashAttention

  Show description for 37a3175
  duanqn

  committed
  37a3175

• Move getenv to constructor

  duanqn

  committed
  4ebe454

• Get Env

  duanqn

  committed
  0d65ce2

• Renaming

  duanqn

  committed
  42b2acb

• Check for T==float

  duanqn

  committed
  88a2600

• Lintrunner

  duanqn

  committed
  53e2e85

• Remove mlas function

  duanqn

  committed
  945f656

• Handle scale; Require present_k and present_v to be empty

  duanqn

  committed
  ee323fb

• Check is_unidirectional_

  duanqn

  committed
  63e76ad

• fix build

  

  tianleiwu

  authored and

  duanqn

  committed
  3d6368b

• Merge with mlas.h

  duanqn

  committed
  1fba73a

• Add comment and MLASCALL

  duanqn

  committed
  9479623

• Remove unnecessary change

  duanqn

  committed
  1e63e82

• Fix onnxruntime_mlas.cmake

  duanqn

  committed
  1fd0813

• Pick onnxruntime/test/python/transformers/benchmark_mha.py from latest master

  duanqn

  committed
  afb7466

• Disable FlashAttention by default

  duanqn

  committed
  327b4c2

• Fix value choice of row_size_q and row_size_kv; Add comments

  duanqn

  committed
  ab0da5b

• Fix order

  duanqn

  committed
  8b19094

• causal=False

  duanqn

  committed
  8b2270a

• Add MLASCALL on implementation

  duanqn

  committed
  b449524

• Improve comment

  duanqn

  committed
  06251b1

• Enable FlashAttention by default

  duanqn

  committed
  27b18d4

• lintrunner -a

  duanqn

  committed
  3059b44

• Remove memset

  duanqn

  committed
  412f219

• Fix l2_cache_size_

  duanqn

  committed
  44ff8f0

• Fix PREfast

  duanqn

  committed
  5421335

• #include <algorithm>

  duanqn

  committed
  03d8f36

• Fix bug

  duanqn

  committed
  d63e528

• lintrunner

  duanqn

  committed
  7a3d4a6

• Renaming

  duanqn

  committed
  bf014d0

• Renaming

  duanqn

  committed
  baff456

• Use MlasSgemmOperation

  duanqn

  committed
  72f3c67

• Move threading inside MLAS kernel

  duanqn

  committed
  e8a4373

• Remove MLASCALL

  

  Qingnan Duan

  authored and

  duanqn

  committed
  46a8ce9

• Remove 1 TODO

  duanqn

  committed
  e1cf289

• Renaming

  duanqn

  committed
  852fd98