Multinode batch inference #2
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR introduces initial support for multinode batch inference using Ray in the vLLM batch inference process. The changes focus on setting up the infrastructure for distributed computing while maintaining compatibility with the existing system.
- Added
init_ray.pyto initialize a Ray cluster for multinode batch inference - Modified
Dockerfile_vllmandbuild_and_upload_image.shto includeinit_ray.pyand improve script flexibility - Updated
vllm_batch.pywith TODOs for multinode setup, indicating ongoing implementation - Potential conflict between
JOB_COMPLETION_INDEXusage invllm_batch.pyand multinode setup noted in TODO comment
4 file(s) reviewed, 4 comment(s)
Edit PR Review Bot Settings
| if result.returncode == 0: | ||
| print(f"Worker: Ray runtime started with head address {ray_address}:{ray_port}") | ||
| sys.exit(0) | ||
| print(result.returncode) |
There was a problem hiding this comment.
style: Print the error message from the result object for better debugging
| is_leader = os.getenv("JOB_COMPLETION_INDEX") == "0" | ||
| ray_address = os.getenv("MASTER_ADDR") | ||
| ray_port = os.getenv("MASTER_PORT") | ||
| ray_cluster_size = os.getenv("NUM_INSTANCES") |
There was a problem hiding this comment.
logic: Ensure NUM_INSTANCES is converted to an integer
| @@ -312,12 +312,14 @@ def tool_func(text: str, past_context: Optional[str]): | |||
|
|
|||
|
|
|||
| async def batch_inference(): | |||
| job_index = int(os.getenv("JOB_COMPLETION_INDEX", 0)) | |||
| job_index = int(os.getenv("JOB_COMPLETION_INDEX", 0)) # TODO this conflicts with multinode | |||
There was a problem hiding this comment.
logic: This TODO suggests a conflict with multinode setup. Consider resolving this before merging.
| download_model( | ||
| request.model_cfg.checkpoint_path, MODEL_WEIGHTS_FOLDER | ||
| ) # TODO move this out |
There was a problem hiding this comment.
style: Moving model download out of this function could improve performance for multinode setups.
Pull Request Summary
What is this PR changing? Why is this change being made? Any caveats you'd like to highlight? Link any relevant documents, links, or screenshots here if applicable.
Test Plan and Usage Guide
How did you validate that your PR works correctly? How do you run or demo the code? Provide enough detail so a reviewer can reasonably reproduce the testing procedure. Paste example command line invocations if applicable.