Skip to content

Commit

Permalink
update docstring
Browse files Browse the repository at this point in the history
  • Loading branch information
rjzamora committed Dec 7, 2023
1 parent 18b49ca commit 61a3eb2
Showing 1 changed file with 14 additions and 13 deletions.
27 changes: 14 additions & 13 deletions distributed/shuffle/_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -476,19 +476,20 @@ class UnloadedPartition:
Otherwise, the in-memory partition may need to be spilled
back to disk before the dependent task is executed anyway.
If ``shuffle_unpack`` returns an ``UnloadedPartition`` object,
``P2PShuffleLayer`` must be followed by an extra ``Blockwise``
call to ``_get_partition_data`` (to load and covert the data).
We want an extra ``Blockwise`` layer here so that the loading
and conversion can be fused into down-stream tasks. We do NOT
want the original ``shuffle_unpack`` tasks to be fused into
dependent tasks, because this would prevent effective load
balancing after the shuffle (long-running post-shuffle tasks
may be pinned to specific workers, while others sit idle).
Note that serialization automatically converts to
``LoadedPartition``, because the object may be moved to a
worker that doesn't have access to the same local storage.
If the output tasks of a ``P2PShuffleLayer`` return objects
of type ``UnloadedPartition``, that layer must be followed
by an extra ``Blockwise`` call to ``load_output_partition``
(to ensure the partitions are actually loaded). We want this
extra layer to be ``Blockwise`` so that the loading can be
fused into down-stream tasks. We do NOT want the original
``shuffle_unpack`` tasks to be fused into dependent tasks,
because this would prevent load balancing after the shuffle
(long-running post-shuffle tasks may be pinned to specific
workers, while others sit idle).
Note that serialization automatically loads the wrapped
data, because the object may be moved to a worker that
doesn't have access to the same local storage.
"""

def __init__(
Expand Down

0 comments on commit 61a3eb2

Please sign in to comment.