Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llama 70b model fusion and shardding #18175

Merged
merged 15 commits into from
Nov 2, 2023
Merged

llama 70b model fusion and shardding #18175

merged 15 commits into from
Nov 2, 2023

Conversation

frank-dong-ms
Copy link
Contributor

Description

Support llama-70b model fusion and shardding

Motivation and Context

This change enables shard and export llama-70b model into Onnx as this model is too large for single GPU.
This change also fuses llama-70b model with repeat_kv pattern different with llama-7b and llama-13b.

github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 30, 2023
View reviewed changes
Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lintrunner found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

@tianleiwu
Copy link
Contributor

Lint/python format pipeline failed. Please run lintrunner -a and fix all warnings.
see https://github.com/microsoft/onnxruntime/blob/main/docs/Coding_Conventions_and_Standards.md#linting

@kunal-vaishnavi
Copy link
Contributor

Since device-id is being removed in favor of adding CUDA_VISIBLE_DEVICES=<comma-separated list of device ids to use> to support multi-GPU use, can you update its usage in benchmark_all.py?

onnxruntime/onnxruntime/python/tools/transformers/models/llama/benchmark_all.py

Lines 103 to 108 in 9e8ad39

parser.add_argument(
"--device-id",
type=int,
default=0,
help="GPU device ID",
)

It would also be useful in the README to show an example command with CUDA_VISIBLE_DEVICES for both benchmark.py and benchmark_all.py (just as there's an example command with CUDA_VISIBLE_DEVICES for export).

tianleiwu
tianleiwu previously approved these changes Nov 2, 2023
View reviewed changes
@frank-dong-ms
Copy link
Contributor Author

/azp run Windows GPU CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@frank-dong-ms
Copy link
Contributor Author

/azp run Windows GPU CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@frank-dong-ms
Copy link
Contributor Author

/azp run Windows GPU TensorRT CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 2, 2023
View reviewed changes
onnxruntime/python/tools/transformers/models/llama/dist_settings.py
elif "OMPI_COMM_WORLD_LOCAL_RANK" in os.environ:
from mpi4py import MPI

comm = MPI.COMM_WORLD # noqa: F841

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable comm is not used.
@kunal-vaishnavi kunal-vaishnavi mentioned this pull request Nov 2, 2023
Closed
@frank-dong-ms frank-dong-ms merged commit dabd395 into microsoft:main Nov 2, 2023
84 of 86 checks passed
tianleiwu pushed a commit that referenced this pull request Nov 2, 2023
@frank-dong-ms @tianleiwu
llama 70b model fusion and shardding (#18175)
dae740d 
### Description
Support llama-70b model fusion and shardding



### Motivation and Context
This change enables shard and export llama-70b model into Onnx as this
model is too large for single GPU.
This change also fuses llama-70b model with repeat_kv pattern different
with llama-7b and llama-13b.
kleiti pushed a commit to kleiti/onnxruntime that referenced this pull request Mar 22, 2024
@frank-dong-ms @kleiti
llama 70b model fusion and shardding (microsoft#18175)
656aee4 
### Description
Support llama-70b model fusion and shardding



### Motivation and Context
This change enables shard and export llama-70b model into Onnx as this
model is too large for single GPU.
This change also fuses llama-70b model with repeat_kv pattern different
with llama-7b and llama-13b.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hanbitmyths hanbitmyths hanbitmyths left review comments

@mindest mindest mindest left review comments

@kunal-vaishnavi kunal-vaishnavi kunal-vaishnavi approved these changes

@tianleiwu tianleiwu tianleiwu left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@frank-dong-ms @tianleiwu @hanbitmyths @kunal-vaishnavi @mindest