vLLM model deployer #3032

dudeperf3ct · 2024-09-20T14:47:29Z

Describe changes

This PR adds support for vLLM as a model deployer.

TODO

~~Clarity on Endpoints here for vllm. For now, I have implemented a static version as part of get_prediction_url and get_healthcheck_url of VLLMDeploymentService~~.
~~Ideal approach to add additional arguments specified under EngineArgs dataclass here. For now, only model and tokenizer are added.~~
~~Add documentation for new model deployer.~~

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
If my change requires a change to docs, I have updated the documentation accordingly.
I have added tests to cover my changes.
I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

coderabbitai · 2024-09-20T14:49:09Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

safoinme

Thank you for the contribution @dudeperf3ct and Great work! We've been looking forward to this and can't wait to test it out.

safoinme · 2024-09-24T17:37:29Z

src/zenml/integrations/vllm/__init__.py

+
+    NAME = VLLM
+
+    REQUIREMENTS = ["vllm", "openai"]


can we please pin some version here to avoid conflicts and also make sure the integration will keep working

For now, I have pinned this to the following

REQUIREMENTS = ["vllm >= 0.6.0", "openai >= 1.0.0"]

@safoinme should we add a upper bound of <0.7.0 for vllm? The library is changing rapidly and we might not be sure if the implementation will work for the next major release.

safoinme · 2024-09-25T21:08:38Z

src/zenml/integrations/vllm/services/vllm_deployment.py

+
+    blocking: bool = True
+    model: Optional[str] = None
+    tokenizer: Optional[str] = None


We need to add all other config we want here, but maybe let's start only with essential one as start
tokenizer_mode, trust_remote_code, dtype, revision, served_model_name,

Added it in.

safoinme · 2024-09-25T21:13:10Z

src/zenml/integrations/vllm/steps/vllm_deployer.py

It would be better if we took this file out and alternatively documented to users how they would deploy models, can you please also provide a full example of a pipeline that deploys using vllm and create a PR for it in https://github.com/zenml-io/zenml-projects

I have removed it. I will create a separate PR on https://github.com/zenml-io/zenml-projects that creates the step and pipeline for using vllm.

@safoinme Tracked here : zenml-io/zenml-projects#131

hyperlint-ai

1 files reviewed, 4 total issue(s) found.

docs/book/component-guide/model-deployers/vllm.md

hyperlint-ai

1 files reviewed, 4 total issue(s) found.

Note:
This, and prior reviews, were resolved because we updated the style guide.
We'll leave a new review below.

docs/book/component-guide/model-deployers/vllm.md

hyperlint-ai

1 files reviewed, 5 total issue(s) found.

We noticed a change to the style guide files, We resolved the existing comments to account for any changes to your style guide.

docs/book/component-guide/model-deployers/vllm.md

safoinme · 2024-10-25T07:07:43Z

src/zenml/integrations/vllm/flavors/vllm_model_deployer_flavor.py

+        Returns:
+            The flavor logo.
+        """
+        return "https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-dark.png"


@schustmi can we please upload this logo to our s3 buckets in logo's path and share the URL here so it can be changed

https://public-flavor-logos.s3.eu-central-1.amazonaws.com/model_deployer/vllm.png

safoinme · 2024-10-25T07:20:24Z

src/zenml/integrations/vllm/services/vllm_deployment.py

+        description="vLLM Inference prediction service",
+    )
+    config: VLLMServiceConfig
+


Is there any specific reasons for not having endpoint implementation here? while having a static implementation for healthcheck_url and predection_url would work in same cases but it becomes invalid in case of port 8000 is not used

Updated the code to reflect the same.

safoinme · 2024-10-25T07:30:33Z

src/zenml/integrations/vllm/services/vllm_deployment.py

+            args = parser.parse_args()
+            # Update the arguments in place
+            args.__dict__.update(self.config.model_dump())
+            uvloop.run(run_server(args=args))


One of the main added things in the LocalDaemonServiceEndpoint is looking for a free port in the local environment which then can be used when starting the server to avoid problems when the main given port is locked to some other process. check this.

we can then add port to the args to make sure that our server is starting on it, example here

Thanks @safoinme for the pointers.

I have updated the code to use endpoint configuration.

Also updated the example on zenml-projects repo and tested the example.

…o feature/vllm-model-deployer

dudeperf3ct force-pushed the feature/vllm-model-deployer branch from a674cb7 to 3d5a694 Compare September 20, 2024 14:51

dudeperf3ct changed the title ~~Feature/vllm model deployer~~ vLLM model deployer Sep 23, 2024

htahir1 requested a review from safoinme September 23, 2024 07:08

safoinme requested changes Sep 25, 2024

View reviewed changes

dudeperf3ct force-pushed the feature/vllm-model-deployer branch from 3d5a694 to 80f867a Compare September 26, 2024 07:56

safoinme marked this pull request as ready for review September 27, 2024 16:40

dudeperf3ct mentioned this pull request Sep 30, 2024

vLLM deployment example zenml-io/zenml-projects#131

Open

dudeperf3ct force-pushed the feature/vllm-model-deployer branch from 9a973c5 to c4e6526 Compare October 1, 2024 07:43

hyperlint-ai bot reviewed Oct 1, 2024

View reviewed changes

dudeperf3ct added 9 commits October 3, 2024 13:00

First pass vllm model deployer

f8e6820

Fix the first pass implementation

ae7546b

Add first working version

bbd385c

Remove debugging log statements

9ffee17

Pin vllm integration packages

32e92cc

Support for additional arguments from vllm engine

f0a7285

Remove vllm deployer step code

ec23cdd

Add vllm documentation

104f94e

Update vllm documentation

18f423f

dudeperf3ct force-pushed the feature/vllm-model-deployer branch from 5355006 to 18f423f Compare October 3, 2024 07:30

hyperlint-ai bot reviewed Oct 3, 2024

View reviewed changes

dudeperf3ct requested a review from safoinme October 3, 2024 07:32

strickvl and others added 3 commits October 23, 2024 13:22

Merge branch 'develop' into feature/vllm-model-deployer

ef5db8a

Update readme

8097355

Merge branch 'develop' into feature/vllm-model-deployer

4121b43

safoinme approved these changes Oct 25, 2024

View reviewed changes

safoinme self-requested a review October 25, 2024 07:06

safoinme requested changes Oct 25, 2024

View reviewed changes

Update service implementation and logo url

6d2c6c1

dudeperf3ct added 3 commits October 25, 2024 13:31

Merge remote-tracking branch 'origin/feature/vllm-model-deployer' int…

013159b

…o feature/vllm-model-deployer

Use correct endpoint url

b56523f

Update vllm endpoint config

70949a3

dudeperf3ct requested a review from safoinme October 25, 2024 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vLLM model deployer #3032

vLLM model deployer #3032

dudeperf3ct commented Sep 20, 2024 •

edited

Loading

coderabbitai bot commented Sep 20, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

safoinme left a comment

safoinme Sep 24, 2024

dudeperf3ct Sep 26, 2024

dudeperf3ct Oct 25, 2024

safoinme Sep 25, 2024

dudeperf3ct Sep 26, 2024

safoinme Sep 25, 2024

dudeperf3ct Sep 26, 2024

dudeperf3ct Sep 30, 2024 •

edited

Loading

hyperlint-ai bot left a comment

hyperlint-ai bot left a comment •

edited

Loading

hyperlint-ai bot left a comment

safoinme Oct 25, 2024

schustmi Oct 25, 2024

safoinme Oct 25, 2024

dudeperf3ct Oct 25, 2024

safoinme Oct 25, 2024

dudeperf3ct Oct 25, 2024

dudeperf3ct Oct 25, 2024

vLLM model deployer #3032

Are you sure you want to change the base?

vLLM model deployer #3032

Conversation

dudeperf3ct commented Sep 20, 2024 • edited Loading

Describe changes

Pre-requisites

Types of changes

coderabbitai bot commented Sep 20, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

safoinme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dudeperf3ct Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

hyperlint-ai bot left a comment

Choose a reason for hiding this comment

hyperlint-ai bot left a comment • edited Loading

Choose a reason for hiding this comment

hyperlint-ai bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dudeperf3ct commented Sep 20, 2024 •

edited

Loading

coderabbitai bot commented Sep 20, 2024 •

edited

Loading

dudeperf3ct Sep 30, 2024 •

edited

Loading

hyperlint-ai bot left a comment •

edited

Loading