[NO MRG] Try Rick's example in the benchmark #141
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: This is merely intended to be illustrative. No need for merging
This tries @rjzamora's example ( dask/dask#7615 (comment) ) as shown in the diff below. So is a follow up to the profile in issue ( #137 ).
To make this more comparable to existing work with
shuffle
, this makes a couple of notable tweaks to Rick's example. First it uses the same input data that is used withshuffle
. However this doesn't show much in a single (or even a few) iterations. So this does 20 iterations instead of the typical 5 iterations withshuffle
.With these modifications, this starts to look more analogous to the results that we see in a normal
shuffle
like in issue ( #140 ) (even though this is not ashuffle
). Namely time is spent mostly in communication followed by finished task transitions. IOW backing up what was found before ( #139 (comment) ).cc @quasiben