[pull] master from ray-project:master #2395
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
Data arrow 6 test jobs are failing on master and blocking folks from merging. There are a lot of tests so I soft fail the entire jobs. There are still the arrow 12 job that protect bad PRs that break data tests. Signed-off-by: can <[email protected]>
Signed-off-by: Cindy Zhang <[email protected]> Co-authored-by: Edward Oakes <[email protected]>
This change is for code refactoring, including several changes: Remove the logic for generating the keys, which has been handled by Packer Remove one meaningless, e.g., mock-too-much, UT case. We have e2e pipline to cover that case. Rewrite the scheduler for scheduling the next Ray node. The previous "factory mode" looks not very good. Remove the useless code for reusing the node. On vSphere, thanks to instant clone, we don't need to cache the VM. We can always launch a new Ray node quickly. For the old ones, we just delete the VM after the Ray process on it quits the Ray cluster. Remove the cache for the nodes. The cache could be outdated when the node information changes on vSphere side. We have only one place leverages the node cache so the benefit for keeping the cache doesn't worth its potential issue. Moved some functions which have no dependencies on the VsphereNodeProvider instance to the outside scope, instead of being a member function of the VsphereNodeProvider instance. Some changes caused by above changes. Some Python grammar level optimization. Signed-off-by: Chen Jing <[email protected]>
…mote_storage`, etc.) (#40207) This PR removes some legacy utilities, including `air._internal.remote_storage`, `TrainableUtil`, and more. --------- Signed-off-by: Justin Yu <[email protected]>
As a byproduct of the recent documentation rewrites, the Train docs contain several code snippets that aren't tested. This PR updates the snippets to test the ones that can be reasonably tested. --------- Signed-off-by: Balaji Veeramani <[email protected]>
This PR adds the support to run MPI based code on top of Ray.
The support is done with runtime env plugin. To enable it, the following decorator needs to be added inside ray remote options:
@ray.remote(
runtime_env={
"mpi": {
"args": ["-n", "4"],
"worker_entry": "mpi_worker.run",
}
}
)
def f():
pass
Here the mpi_worker.run is the function the process with rank > 0 will run. It'll run as import mpi_worker; mpi_worker.run(). The parameter needs to be passed with MPI comm.bcast.
Here the process with rank 0 sill will run the remote function f.
Signed-off-by: Yi Cheng <[email protected]>
Follow up with the pr to implement the dynamic logging. --------- Signed-off-by: Sihan Wang <[email protected]>
…n. (#41062) Signed-off-by: Yi Cheng <[email protected]>
When you launch a read task generated byFileBasedDatasource, Ray serializes the FileBasedDatasource instance because the read task calls FileBasedDatasource._read_stream. This wasn't an issue before, because FileBasedDatasource was stateless and therefore quick to serialize. However, #40900 moved state like the input file paths from _FileBasedDatasourceReader to FileBasedDatasource. As a result, FileBasedDatasource is now slow to serialize and read tasks can be slow to launch. To fix this issue, this PR stores paths and file sizes in the object store, as only stores references to them in FileBasedDatasource. --------- Signed-off-by: Balaji Veeramani <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )