Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Implementing disaggregated prefilling, and caching KV cache in CPU/disk/database. #8498

Open
wants to merge 323 commits into
base: main
Choose a base branch
from
Open

[Core] Implementing disaggregated prefilling, and caching KV cache in CPU/disk/database. #8498

wants to merge 323 commits into from
+2,539 −16
This pull request is big! We’re only showing the most recent 250 commits.

Commits on Jul 19, 2024

  1. check

    @KuntaiDu
    KuntaiDu committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    bb8c08a View commit details
    Browse the repository at this point in the history

  2. locate the hanging line

    @KuntaiDu
    KuntaiDu committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    25a7cf3 View commit details
    Browse the repository at this point in the history

  3. add rank to CPU group

    @KuntaiDu
    KuntaiDu committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    999bd72 View commit details
    Browse the repository at this point in the history

  4. narrow case

    @KuntaiDu
    KuntaiDu committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    3428ea6 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2024

  1. bug fix: need to align the distributed groups between prefill and dec… 

    …ode instances
    @KuntaiDu
    KuntaiDu committed Jul 20, 2024
    Configuration menu
    Copy the full SHA
    91e3ed2 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. add disaggregated prefilling for flashinfer

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    3dd2275 View commit details
    Browse the repository at this point in the history

  2. adjust comments

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    2b13f3c View commit details
    Browse the repository at this point in the history

  3. add logging for send and recv

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    8c3f209 View commit details
    Browse the repository at this point in the history

  4. turn off chunked prefill to use flashinfer kernel

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    c6a5e57 View commit details
    Browse the repository at this point in the history

  5. confirm which backend is being used

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    b3c47f3 View commit details
    Browse the repository at this point in the history

  6. remove debugging from parallel_state, its too much...

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    f05540c View commit details
    Browse the repository at this point in the history

  7. add disagg prefill for flash attn backend

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    eb96fe7 View commit details
    Browse the repository at this point in the history

  8. edit flash attn to assign prefill_meta first

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    09d5588 View commit details
    Browse the repository at this point in the history

  9. use print instead of attn

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    43077e7 View commit details
    Browse the repository at this point in the history

  10. make data contiguous

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    f716737 View commit details
    Browse the repository at this point in the history

  11. add more debug message

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    0d07251 View commit details
    Browse the repository at this point in the history

  12. turn on logging

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    2177737 View commit details
    Browse the repository at this point in the history

  13. more debug prints in flash_attn

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    a293bd0 View commit details
    Browse the repository at this point in the history

  14. remove enforce eager

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    cc7f646 View commit details
    Browse the repository at this point in the history

  15. adjust printing order in flash attn

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    68f3d16 View commit details
    Browse the repository at this point in the history

  16. avoid sending & receiving output tensor during profile run

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    21a61b9 View commit details
    Browse the repository at this point in the history

  17. also log the device

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    691cad7 View commit details
    Browse the repository at this point in the history

  18. adjust implementation

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    c057f19 View commit details
    Browse the repository at this point in the history

  19. finish adjustment

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    82b73bb View commit details
    Browse the repository at this point in the history

  20. fall back to original flashinfer

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    6db1d48 View commit details
    Browse the repository at this point in the history

  21. Merge branch 'vllm-project:main' into kuntai-disagg

    @KuntaiDu
    KuntaiDu authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    9e53071 View commit details
    Browse the repository at this point in the history

  22. add space

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    dbaade7 View commit details
    Browse the repository at this point in the history

  23. clean config.py

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    f572db8 View commit details
    Browse the repository at this point in the history

  24. keep flashattn implementation

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    9ebf3ad View commit details
    Browse the repository at this point in the history

  25. commit changes that will be merged

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    67b1c2e View commit details
    Browse the repository at this point in the history

  26. Merge branch 'kuntai-disagg' of https://github.com/KuntaiDu/vllm into… 

    … kuntai-disagg

merge vLLM remote codebase
    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    4acad6a View commit details
    Browse the repository at this point in the history

  27. revert custom allreduce changes

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    3abca47 View commit details
    Browse the repository at this point in the history

  28. remove debug logs from the file

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    0ce251b View commit details
    Browse the repository at this point in the history

  29. revert changes to prefix_caching_block --- unnecessary

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    1f3ac2b View commit details
    Browse the repository at this point in the history

  30. revert changes

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    c93bf33 View commit details
    Browse the repository at this point in the history

  31. fix typos

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    8dcaf43 View commit details
    Browse the repository at this point in the history

  32. add example usage to disaggregated prefill

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    4d83813 View commit details
    Browse the repository at this point in the history

  33. can only use print instead of log.debug...

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    11c3ace View commit details
    Browse the repository at this point in the history

  34. kill vllm instance after run

    @KuntaiDu
    KuntaiDu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    0bd0cc9 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. add proxy server for disaggregated prefilling

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    39973bb View commit details
    Browse the repository at this point in the history

  2. update disagg proxy server

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    13a6d12 View commit details
    Browse the repository at this point in the history

  3. add debug message for proxy server

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    81cad25 View commit details
    Browse the repository at this point in the history

  4. fix bug

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    198931b View commit details
    Browse the repository at this point in the history

  5. increase nccl buff size

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    7412767 View commit details
    Browse the repository at this point in the history

  6. increase nccl buffer size

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    bd6f41b View commit details
    Browse the repository at this point in the history

  7. add debug flag

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    20f9de1 View commit details
    Browse the repository at this point in the history

  8. reduce gpu memory usage

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    11850d5 View commit details
    Browse the repository at this point in the history

  9. fix syntax bug

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    d6ad9bd View commit details
    Browse the repository at this point in the history

  10. temporarily lift up nccl buffer size for send and recv

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    57dd656 View commit details
    Browse the repository at this point in the history

  11. reduce nccl buffer size and see if bug fixed

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    9379fbb View commit details
    Browse the repository at this point in the history

  12. fix

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    c23d841 View commit details
    Browse the repository at this point in the history

  13. add debug info -- see which layer the prefill instance got stuck

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    7fc62b4 View commit details
    Browse the repository at this point in the history

  14. remove nccl debug -- it is too loud

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    e542366 View commit details
    Browse the repository at this point in the history

  15. change buffer size only for disagg communicator

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    e9f7dc2 View commit details
    Browse the repository at this point in the history

  16. disable nccl debug

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    18ded4c View commit details
    Browse the repository at this point in the history

  17. use isend and irecv

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    e814f82 View commit details
    Browse the repository at this point in the history

  18. try to increase the buffer size

    @KuntaiDu
    KuntaiDu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    a3399b3 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Merge branch 'main' into kuntai-disagg

    @KuntaiDu
    KuntaiDu authored Jul 30, 2024
    Configuration menu
    Copy the full SHA
    5e18bd7 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. bug fix, now disaggregated prefill should work as expected

    @KuntaiDu
    KuntaiDu committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    e4e60d9 View commit details
    Browse the repository at this point in the history

  2. add proxy server

    @KuntaiDu
    KuntaiDu committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    87fbfae View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. startr slow -- using pp=1 and tp=1

    @KuntaiDu
    KuntaiDu committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    fa664c0 View commit details
    Browse the repository at this point in the history

  2. adjust the API

    @KuntaiDu
    KuntaiDu committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    6bf7583 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. support batch size >1

    @KuntaiDu
    KuntaiDu committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    6aad5cc View commit details
    Browse the repository at this point in the history

  2. update model runner

    @KuntaiDu
    KuntaiDu committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    e934286 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2024

  1. move group coordinator to a separate file, move disagg implementation… 

    …s to a separate file
    @KuntaiDu
    KuntaiDu committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    b68435a View commit details
    Browse the repository at this point in the history

  2. no need to send during attention

    @KuntaiDu
    KuntaiDu committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    e54f7a3 View commit details
    Browse the repository at this point in the history

  3. debug tp

    @KuntaiDu
    KuntaiDu committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    23c9949 View commit details
    Browse the repository at this point in the history

  4. resolve conflicts

    @KuntaiDu
    KuntaiDu committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    87cb78b View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Fix several bugs: tensor device placement, misc performance optimizat… 

    …ions, handle the case where 2 request has identical input, refactor the code
    @KuntaiDu
    KuntaiDu committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    06a526a View commit details
    Browse the repository at this point in the history

  2. remove useless comments

    @KuntaiDu
    KuntaiDu committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    34e6bb3 View commit details
    Browse the repository at this point in the history

  3. update disaggregated prefill example

    @KuntaiDu
    KuntaiDu committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    55bf3bf View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. add disaggregated prefill overhead benchmark

    @KuntaiDu
    KuntaiDu committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    b525510 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. change disagg prefill proxy server to support non-streaming case

    @KuntaiDu
    KuntaiDu committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    ee6a6ec View commit details
    Browse the repository at this point in the history

  2. avoid detokenizing the first token in prefill instance -- for shorter… 

    … latency
    @KuntaiDu
    KuntaiDu committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    f3cc91d View commit details
    Browse the repository at this point in the history

  3. add failure test cases --- try switching to another machine

    @KuntaiDu
    KuntaiDu committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    0582265 View commit details
    Browse the repository at this point in the history

  4. update

    @KuntaiDu
    KuntaiDu committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    89d4ca4 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. remove debugging information

    @KuntaiDu
    KuntaiDu committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    9f4dba2 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. avoid broadcast by finding seqlen inside the attn metadata

    @KuntaiDu
    KuntaiDu committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    aa55883 View commit details
    Browse the repository at this point in the history

  2. update examples

    @KuntaiDu
    KuntaiDu committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    95df023 View commit details
    Browse the repository at this point in the history

  3. support pipeline parallel

    @KuntaiDu
    KuntaiDu committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    d92223a View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2024

  1. update benchmark --- compare chunked prefill w.r.t. disagg prefill

    @KuntaiDu
    KuntaiDu committed Aug 10, 2024
    Configuration menu
    Copy the full SHA
    a8c202c View commit details
    Browse the repository at this point in the history

  2. mute round_robin_proxy -- too loud

    @KuntaiDu
    KuntaiDu committed Aug 10, 2024
    Configuration menu
    Copy the full SHA
    310f3a3 View commit details
    Browse the repository at this point in the history

  3. bug fix: racing conditions, and rare cases where input hash is not ca… 

    …ched
    @KuntaiDu
    KuntaiDu committed Aug 10, 2024
    Configuration menu
    Copy the full SHA
    118aab1 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2024

  1. add visualization script

    @KuntaiDu
    KuntaiDu committed Aug 11, 2024
    Configuration menu
    Copy the full SHA
    96d38b4 View commit details
    Browse the repository at this point in the history

  2. fix bug: when KV transfer fails, do not return hidden state

    @KuntaiDu
    KuntaiDu committed Aug 11, 2024
    Configuration menu
    Copy the full SHA
    3fc0c5c View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. add new abstractions

    @KuntaiDu
    KuntaiDu committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f9aadd8 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. major revision: add 3-layer abstractions. Transport, lookup buffer, a… 

    …nd adapter.
    @KuntaiDu
    KuntaiDu committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    db66a1e View commit details
    Browse the repository at this point in the history

  2. add kv transfer test

    @KuntaiDu
    KuntaiDu committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    e04430c View commit details
    Browse the repository at this point in the history

  3. add test cases for pipe

    @KuntaiDu
    KuntaiDu committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    30f9bb6 View commit details
    Browse the repository at this point in the history

  4. bug fix

    @KuntaiDu
    KuntaiDu committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    bbce62e View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. finalize send-recv test

    @KuntaiDu
    KuntaiDu committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    927800d View commit details
    Browse the repository at this point in the history

  2. update test case so that there are both send and recv

    @KuntaiDu
    KuntaiDu committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    6680ea7 View commit details
    Browse the repository at this point in the history

  3. update kv lookup buffer --- I am TOOOOOOO sleepy

    @KuntaiDu
    KuntaiDu committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    dfbfe80 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. add lookup buffer test

    @KuntaiDu
    KuntaiDu committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    b566b18 View commit details
    Browse the repository at this point in the history

  2. update lookup buffer

    @KuntaiDu
    KuntaiDu committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    fc2c972 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. finish lookup buffer test

    @KuntaiDu
    KuntaiDu committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    b2c765c View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2024

  1. update parallel state to use the new class method

    @KuntaiDu
    KuntaiDu committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    8aef9dc View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2024

  1. move the implementatio to worker_base.py

    @KuntaiDu
    KuntaiDu committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    1b6125d View commit details
    Browse the repository at this point in the history

  2. update test

    @KuntaiDu
    KuntaiDu committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    c4102ef View commit details
    Browse the repository at this point in the history

  3. update a new implementation for distributed pipe. Much less CPU commu… 

    …nication
    @KuntaiDu
    KuntaiDu committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    a576532 View commit details
    Browse the repository at this point in the history

  4. update tensor sending and receiving. Use CPU to transfer metadata ins… 

    …tead.
    @KuntaiDu
    KuntaiDu committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    24a231e View commit details
    Browse the repository at this point in the history

  5. update benchmark: use small model for quick iteration

    @KuntaiDu
    KuntaiDu committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    dca877a View commit details
    Browse the repository at this point in the history

  6. update implementation

    @KuntaiDu
    KuntaiDu committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    9f81f41 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. [Add] optimized implementation for KV transfer pipe

    @ApostaC
    ApostaC committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    bb86588 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Merge pull request #3 from KuntaiDu/yihua-kv-pipe 

    Optimize the KV transfer pipe implementation
    @KuntaiDu
    KuntaiDu authored Sep 11, 2024
    Configuration menu
    Copy the full SHA
    1377912 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. [Fix] the implementation of KV lookup buffer

    @ApostaC
    ApostaC committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    ffb792b View commit details
    Browse the repository at this point in the history

  2. remove unused file

    @ApostaC
    ApostaC committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    d7d32c1 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #4 from KuntaiDu/yihua-lookup-buffer 

    Lookup buffer implementation
    @KuntaiDu
    KuntaiDu authored Sep 13, 2024
    Configuration menu
    Copy the full SHA
    c5b7232 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #5 from KuntaiDu/kuntai-disagg-refactor 

    Kuntai disagg refactor
    @YaoJiayi
    YaoJiayi authored Sep 13, 2024
    Configuration menu
    Copy the full SHA
    4db6446 View commit details
    Browse the repository at this point in the history

  5. update vllm adapter

    @YaoJiayi
    YaoJiayi committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    417ccb3 View commit details
    Browse the repository at this point in the history

  6. update worker_base

    @YaoJiayi
    YaoJiayi committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    0176ebb View commit details
    Browse the repository at this point in the history

  7. update comm initialization

    @YaoJiayi
    YaoJiayi committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    84fd0b8 View commit details
    Browse the repository at this point in the history

  8. update

    @ApostaC
    ApostaC committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    826ca70 View commit details
    Browse the repository at this point in the history

  9. update documentation

    @ApostaC
    ApostaC committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    3425ab6 View commit details
    Browse the repository at this point in the history

  10. adjust vllm adapter: now we separate CPU and device into different pipes

    @ApostaC
    ApostaC committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    9f3a3a5 View commit details
    Browse the repository at this point in the history

  11. build 2 pipes in vLLM adapter

    @ApostaC
    ApostaC committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    ce79d59 View commit details
    Browse the repository at this point in the history

  12. documentation chagne

    @ApostaC
    ApostaC committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    34dfdde View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2024

  1. Merge branch 'jiayi-dev-v2' into kuntai-disagg-refactor

    @YaoJiayi
    YaoJiayi authored Sep 14, 2024
    Configuration menu
    Copy the full SHA
    80b4200 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #6 from KuntaiDu/kuntai-disagg-refactor 

    Kuntai disagg refactor
    @YaoJiayi
    YaoJiayi authored Sep 14, 2024
    Configuration menu
    Copy the full SHA
    9eefec2 View commit details
    Browse the repository at this point in the history

  3. update vllm_adapter

    @YaoJiayi
    YaoJiayi committed Sep 14, 2024
    Configuration menu
    Copy the full SHA
    9355be3 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2024

  1. minor fix

    @YaoJiayi
    YaoJiayi committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    54b68c9 View commit details
    Browse the repository at this point in the history

  2. fix type hint

    @YaoJiayi
    YaoJiayi committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    2dff658 View commit details
    Browse the repository at this point in the history

  3. fix comm init

    @YaoJiayi
    YaoJiayi committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    c6a6714 View commit details
    Browse the repository at this point in the history

  4. bug fix: remove self from bypass_model_exec

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    fef35b2 View commit details
    Browse the repository at this point in the history

  5. bug fix: should init SimpleKVLookupBuffer with signal pipe first and … 

    …then data pipe
    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    4d0b5cd View commit details
    Browse the repository at this point in the history

  6. adjust torch distributed logging

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    31b891d View commit details
    Browse the repository at this point in the history

  7. remove unnecessaqry comments

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    7e68d08 View commit details
    Browse the repository at this point in the history

  8. remove unnecessary comments

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    85c7a64 View commit details
    Browse the repository at this point in the history

  9. update documentation

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    01fe335 View commit details
    Browse the repository at this point in the history

  10. Merge pull request #7 from KuntaiDu/jiayi-dev-v2 

    Add support for LMCache
    @KuntaiDu
    KuntaiDu authored Sep 15, 2024
    Configuration menu
    Copy the full SHA
    1f47731 View commit details
    Browse the repository at this point in the history

  11. update overhead benchmark

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    caaaeb8 View commit details
    Browse the repository at this point in the history

  12. Merge pull request #8 from KuntaiDu/jiayi-dev-v2 

    update overhead benchmark
    @KuntaiDu
    KuntaiDu authored Sep 15, 2024
    Configuration menu
    Copy the full SHA
    0dd3571 View commit details
    Browse the repository at this point in the history

  13. resolve merge conflict

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    9c98d5f View commit details
    Browse the repository at this point in the history

  14. remove group coordinator import

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    515c47b View commit details
    Browse the repository at this point in the history

  15. remove syntax bug

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    f166cf8 View commit details
    Browse the repository at this point in the history

  16. update round robin proxy. Prior bash-based impl is buggy

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    f320518 View commit details
    Browse the repository at this point in the history

  17. update docs for disagg overhead benchmark

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    5b4a3e3 View commit details
    Browse the repository at this point in the history

  18. use new round robin proxy in performance benchmark

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    01b2fd3 View commit details
    Browse the repository at this point in the history

  19. update

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    54bd11f View commit details
    Browse the repository at this point in the history

  20. update benchmarking script

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    b19f346 View commit details
    Browse the repository at this point in the history

  21. revert changes in model_runner.py --- no change needed for disagg pre… 

    …fill
    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    cb7ff06 View commit details
    Browse the repository at this point in the history

  22. no I was wrong

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    dd8c86d View commit details
    Browse the repository at this point in the history

  23. update benchmark

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    4e8043c View commit details
    Browse the repository at this point in the history

  24. remove sonnet 4x --- it can be automatically generated via benchmarki… 

    …ng script
    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    b51f891 View commit details
    Browse the repository at this point in the history

  25. revert change in flash attn and flash infer to clean up the diff

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    168452f View commit details
    Browse the repository at this point in the history

  26. update the example

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    784d905 View commit details
    Browse the repository at this point in the history

  27. make format checker happy

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    17d2505 View commit details
    Browse the repository at this point in the history

  28. resolve circular import

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    36a382c View commit details
    Browse the repository at this point in the history

  29. fix redundant import

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    a0867dd View commit details
    Browse the repository at this point in the history

  30. rename to a shorter name

    @ApostaC
    ApostaC committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    7f90903 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. remove unnecessary file

    @ApostaC
    ApostaC committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    5ca22fb View commit details
    Browse the repository at this point in the history

  2. update kv transfer test

    @ApostaC
    ApostaC committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    073642b View commit details
    Browse the repository at this point in the history

  3. update tests

    @ApostaC
    ApostaC committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    70d6571 View commit details
    Browse the repository at this point in the history

  4. make fmt checker happy

    @ApostaC
    ApostaC committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    4d6b00a View commit details
    Browse the repository at this point in the history

  5. constraint the model length

    @ApostaC
    ApostaC committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    7c13e03 View commit details
    Browse the repository at this point in the history

  6. adjust path

    @ApostaC
    ApostaC committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    cf5b84c View commit details
    Browse the repository at this point in the history

  7. add disagg prefill test to test pipeline

    @ApostaC
    ApostaC committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    eb751d6 View commit details
    Browse the repository at this point in the history

  8. Merge pull request #9 from KuntaiDu/kuntai-disagg-refactor 

    Kuntai disagg refactor
    @KuntaiDu
    KuntaiDu authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    f101b40 View commit details
    Browse the repository at this point in the history

  9. use new round robin proxy in performance benchmark

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    1e23e99 View commit details
    Browse the repository at this point in the history

  10. update

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    b4225f8 View commit details
    Browse the repository at this point in the history

  11. update benchmarking script

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    fa47857 View commit details
    Browse the repository at this point in the history

  12. revert changes in model_runner.py --- no change needed for disagg pre… 

    …fill
    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    46f82a4 View commit details
    Browse the repository at this point in the history

  13. no I was wrong

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    8d7bb78 View commit details
    Browse the repository at this point in the history

  14. update benchmark

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    b5f9db5 View commit details
    Browse the repository at this point in the history

  15. remove sonnet 4x --- it can be automatically generated via benchmarki… 

    …ng script
    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    0fc0091 View commit details
    Browse the repository at this point in the history

  16. revert change in flash attn and flash infer to clean up the diff

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    afd7a29 View commit details
    Browse the repository at this point in the history

  17. update the example

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    cbf24b3 View commit details
    Browse the repository at this point in the history

  18. make format checker happy

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    4f4ea50 View commit details
    Browse the repository at this point in the history

  19. resolve circular import

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    f78a2eb View commit details
    Browse the repository at this point in the history

  20. fix redundant import

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    44dfa3f View commit details
    Browse the repository at this point in the history

  21. rename to a shorter name

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    822f3dc View commit details
    Browse the repository at this point in the history

  22. remove unnecessary file

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    7682269 View commit details
    Browse the repository at this point in the history

  23. update kv transfer test

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    b6e5eb3 View commit details
    Browse the repository at this point in the history

  24. update tests

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    58f5080 View commit details
    Browse the repository at this point in the history

  25. make fmt checker happy

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    8f0538c View commit details
    Browse the repository at this point in the history

  26. constraint the model length

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    dda1f31 View commit details
    Browse the repository at this point in the history

  27. adjust path

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    85d72fa View commit details
    Browse the repository at this point in the history

  28. add disagg prefill test to test pipeline

    @KuntaiDu
    KuntaiDu committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    60ede08 View commit details
    Browse the repository at this point in the history

  29. Merge pull request #10 from KuntaiDu/kuntai-disagg-refactor 

    Kuntai disagg refactor
    @KuntaiDu
    KuntaiDu authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    0d81aaf View commit details
    Browse the repository at this point in the history

  30. bugfix

    @YaoJiayi
    YaoJiayi committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    0df7566 View commit details
    Browse the repository at this point in the history

  31. bugfix

    @YaoJiayi
    YaoJiayi committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    73c1683 View commit details
    Browse the repository at this point in the history

  32. Merge pull request #11 from KuntaiDu/jiayi-dev-v2 

    Jiayi dev v2
    @KuntaiDu
    KuntaiDu authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    2297c19 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Merge branch 'main' into kuntai-disagg-refactor

    @KuntaiDu
    KuntaiDu authored Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b2e0254 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. rename the environment variable to KV producer and KV consumer, for m… 

    …ore clarity
    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    70bec94 View commit details
    Browse the repository at this point in the history

  2. revert worker to vllm main

    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    e787e42 View commit details
    Browse the repository at this point in the history

  3. bug fix

    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    9874b42 View commit details
    Browse the repository at this point in the history

  4. fix typo: Distributerd -> Distributed

    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    5950ad5 View commit details
    Browse the repository at this point in the history

  5. remove the debug flag in example -- user don't need it

    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    c116684 View commit details
    Browse the repository at this point in the history

  6. fix typo

    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    44e8875 View commit details
    Browse the repository at this point in the history

  7. fixing benchmark_serving.py

    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    181928f View commit details
    Browse the repository at this point in the history

  8. fix the example

    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    c17d18d View commit details
    Browse the repository at this point in the history

  9. update build partial prefill input

    @KuntaiDu
    KuntaiDu committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    0b00876 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. bug fix for LMCache -- adjust vLLM's rebuild input, and merge the log… 

    …ic to reduce random exit branch
    @KuntaiDu
    KuntaiDu committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    94a5086 View commit details
    Browse the repository at this point in the history

  2. make format checker happy

    @KuntaiDu
    KuntaiDu committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    8099fb3 View commit details
    Browse the repository at this point in the history

  3. make ruff and yapf happy, also fix test bug

    @KuntaiDu
    KuntaiDu committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    603864e View commit details
    Browse the repository at this point in the history

  4. remove empty file

    @KuntaiDu
    KuntaiDu committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1d7a1c9 View commit details
    Browse the repository at this point in the history

  5. fix bug when world_size == -1

    @KuntaiDu
    KuntaiDu committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    10ad09c View commit details
    Browse the repository at this point in the history

  6. adjust comments

    @KuntaiDu
    KuntaiDu committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    38e3a57 View commit details
    Browse the repository at this point in the history

  7. make yapf and ruff happy

    @KuntaiDu
    KuntaiDu committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    e2bd481 View commit details
    Browse the repository at this point in the history

  8. relaunch CI

    @KuntaiDu
    KuntaiDu committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    4979337 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. change get_open_port so that it is easier to understand

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    a2007dc View commit details
    Browse the repository at this point in the history

  2. adjust comment

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    ce434f5 View commit details
    Browse the repository at this point in the history

  3. make format checker happy

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    f224c71 View commit details
    Browse the repository at this point in the history

  4. adjust model runner docstring

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    5d9b007 View commit details
    Browse the repository at this point in the history

  5. make format checker happy

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    6255dca View commit details
    Browse the repository at this point in the history

  6. change data == [] to not data (thanks Cody)

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    71ae275 View commit details
    Browse the repository at this point in the history

  7. fix misleading to available

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    80164ea View commit details
    Browse the repository at this point in the history

  8. add new line and run format checker

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    52c2d10 View commit details
    Browse the repository at this point in the history

  9. add docstring for lookup buffer

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    09478ef View commit details
    Browse the repository at this point in the history

  10. align docstring syntax

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    06cb15c View commit details
    Browse the repository at this point in the history

  11. add docstring for abstract classes

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    7c11a39 View commit details
    Browse the repository at this point in the history

  12. put assertion at the end of the function

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    37bac34 View commit details
    Browse the repository at this point in the history

  13. add fp8 support to pipe

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    111abb4 View commit details
    Browse the repository at this point in the history

  14. adjust docstrings

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    394afaa View commit details
    Browse the repository at this point in the history

  15. bug fix: check isinstance(torch.Tensor) before checking NOne

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    76019f1 View commit details
    Browse the repository at this point in the history

  16. make format check happy

    @KuntaiDu
    KuntaiDu committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    93ec62b View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. Merge branch 'main' into kuntai-disagg-refactor

    @KuntaiDu
    KuntaiDu authored Oct 8, 2024
    Configuration menu
    Copy the full SHA
    87b82cc View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Adjust to latest changes of kv_caches: it is now always a tensor.

    @KuntaiDu
    KuntaiDu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    c5bdf64 View commit details
    Browse the repository at this point in the history

  2. debug

    @KuntaiDu
    KuntaiDu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    596eb64 View commit details
    Browse the repository at this point in the history

  3. bug fix: kv_caches will be list of torch.tensor([]) in profile run.

    @KuntaiDu
    KuntaiDu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    683bd9c View commit details
    Browse the repository at this point in the history

  4. Merge branch 'vllm-project:main' into kuntai-disagg-refactor

    @KuntaiDu
    KuntaiDu authored Oct 10, 2024
    Configuration menu
    Copy the full SHA
    81aa825 View commit details
    Browse the repository at this point in the history

  5. Relax server start timeout limit

    @KuntaiDu
    KuntaiDu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    521daba View commit details
    Browse the repository at this point in the history

  6. Merge branch 'kuntai-disagg-refactor' of https://github.com/KuntaiDu/… 

    …vllm into kuntai-disagg-refactor
    @KuntaiDu
    KuntaiDu committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    516f9ca View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Merge branch 'main' into kuntai-disagg-refactor

    @KuntaiDu
    KuntaiDu authored Nov 7, 2024
    Configuration menu
    Copy the full SHA
    6edb723 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2024

  1. Adjust folder format

    @KuntaiDu
    KuntaiDu committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    7efdf60 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2024

  1. config fix

    @KuntaiDu
    KuntaiDu committed Nov 9, 2024
    Configuration menu
    Copy the full SHA
    1c608e6 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2024

  1. misc fixes

    @KuntaiDu
    KuntaiDu committed Nov 10, 2024
    Configuration menu
    Copy the full SHA
    303ff85 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. stage changes

    @KuntaiDu
    KuntaiDu committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    0f172d5 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into kuntai-disagg-refactor

    @KuntaiDu
    KuntaiDu committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    228f78d View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. debugging pynccl pipe

    @KuntaiDu
    KuntaiDu committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    6f3d1b3 View commit details
    Browse the repository at this point in the history

  2. bug found: NONE Tensor did not return none

    @KuntaiDu
    KuntaiDu committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    20e0450 View commit details
    Browse the repository at this point in the history

  3. save code for Kaichao to debug

    @KuntaiDu
    KuntaiDu committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    cc9e8f4 View commit details
    Browse the repository at this point in the history

  4. adjust

    @KuntaiDu
    KuntaiDu committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    3e7e341 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. NCCL pipe bug fix: only transmit metadata when the tensor is None

    @KuntaiDu
    KuntaiDu committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    49e89a2 View commit details
    Browse the repository at this point in the history

  2. Update docstring using GPT, and clean up unnecessary variables

    @KuntaiDu
    KuntaiDu committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    b6e83a2 View commit details
    Browse the repository at this point in the history

  3. Bug fix on PyNcclPipe: the device of sending tensor should be inferre… 

    …d by class variable instead of the config
    @KuntaiDu
    KuntaiDu committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    8d20116 View commit details
    Browse the repository at this point in the history

  4. Fix lookup buffer

    @KuntaiDu
    KuntaiDu committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    fdc4aad View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'upstream/main' into kuntai-disagg-refactor

    @KuntaiDu
    KuntaiDu committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    c0b9574 View commit details
    Browse the repository at this point in the history

  6. Adjust init parameters for connector.

    @KuntaiDu
    KuntaiDu committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    e7432e9 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2024

  1. Move KVTransferConfig outside ParallelConfig

    @KuntaiDu
    KuntaiDu committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    d1ce09d View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into kuntai-disagg-refactor

    @KuntaiDu
    KuntaiDu committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    a478522 View commit details
    Browse the repository at this point in the history

  3. A series of bug fix: previous merge is buggy and need to manually rev… 

    …ert things.
    @KuntaiDu
    KuntaiDu committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    9c4cbc5 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2024

  1. Fix typo (input_token wrongfully typed as input) and make default kv_… 

    …buffer_device to cuda.
    @KuntaiDu
    KuntaiDu committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    9e8affc View commit details
    Browse the repository at this point in the history

  2. Make sure the output is shown at the end of the run by sleeping longe… 

    …r -- killing engine really takes time
    @KuntaiDu
    KuntaiDu committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    d8e79fa View commit details
    Browse the repository at this point in the history

  3. A series of changes to clean up the diff

    @KuntaiDu
    KuntaiDu committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    62f3966 View commit details
    Browse the repository at this point in the history

  4. Clean up the diff

    @KuntaiDu
    KuntaiDu committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    6478fb2 View commit details
    Browse the repository at this point in the history

  5. Clean up the diff in several executor files

    @KuntaiDu
    KuntaiDu committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    c2ebcb6 View commit details
    Browse the repository at this point in the history

  6. clean up wierd spaces in tokenizer groups

    @KuntaiDu
    KuntaiDu committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    9158549 View commit details
    Browse the repository at this point in the history

  7. Remove previous environment variable -- now we initialize distributed… 

    … communication via CLI args
    @KuntaiDu
    KuntaiDu committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    98a44df View commit details
    Browse the repository at this point in the history

  8. add a new line at the end of parallel_state.py to clean up the diff

    @KuntaiDu
    KuntaiDu committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    744a40f View commit details
    Browse the repository at this point in the history