Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement simple key ranges for broadcast #230

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

devreal
Copy link
Contributor

@devreal devreal commented Apr 6, 2022

Problem

Broadcasting a range of keys currently requires application to populate the full range into a container and pass it to ttg::broadcast. These keys then have to be processed and send to peers, leading to large messages (in addition to the actual data transfer).

Proposal

Many (dense) applications send a range of keys that can be expressed as a linear function in an N-dimensional space. With key ranges, application can express the start and end of a range as well as a step size for iterating through the range (a vector in the N dimensional space). Some applications may require splitting the range, i.e., providing a list or ranges (e.g., the Floyd-Warshall example).

These ranges can be serialized and send to peers instead of the fully populated list of keys. Such compact representation of the key range is cheaper to transfer and store.

An example for using key ranges:

auto range1_low = ttg::make_keyrange(Key(I, 0, K), /* start */
                                     Key(I, J, K), /* end, i.e., last + 1 */
                                     Key(0, 1, 0)); /* step vector */

It is possible to define a different type for the step vector by defining Key::difference_type or by specializing the ttg::key_diff_type trait. In order to use a key type in a range, it must provide operator!= as well as operator+ or operator+=. A version of make_keyrange without the step size tries to figure out the step vector by applying operator- to the start element and the ++start value so both operators must be provided to use that variant. Unfortunately, due to the coupling between the input terminal and the TT, the key range must be compilable even if these operators are not provided. I've added the appropriate checks to make_keyrange and moved the LinearKeyRange into ttg::detail to prevent users from accidentally instantiate a key range even though not all requirements are met.

This PR uses floyd warshall as example but using key ranges will be most useful in #222 as well (see https://github.com/TESSEorg/ttg/pull/222/files#diff-6f4fdbb279035f63f17441c5c665d4a7f64c2a9e858463aa8859016a90c73122R177 where the dispatcher has to populate multiple vectors to feed to ttg::broadcast, all of which represent linear key ranges).

Signed-off-by: Joseph Schuchart [email protected]

@devreal devreal requested review from evaleev, therault and pnookala April 6, 2022 15:46
Problem: broadcasting a range of keys currently requires application
to populate the full range into a container and pass it to ttg::broadcast.
These keys then have to be processed and send to peers, leading to
large messages (in addition to the actual data transfer).

Many (dense) applications send a range of keys that can be expressed
as a linear function in an N-dimensional space. With key ranges,
application can express the start and end of a range as well as
a step size for iterating through the range (a vector in the N dimensional
space). Some applications may require splitting the range, i.e.,
providing a list or ranges (e.g., the Floyd-Warshall example).

These ranges can be serialized and send to peers instead of the fully
populated list of keys. Such compact representation of the key range
is cheaper to transfer and store.

An example for using key ranges:

```
auto range1_low = ttg::make_keyrange(Key(I, 0, K), /* start */
                                     Key(I, J, K), /* end, i.e., last + 1 */
                                     Key(0, 1, 0)); /* step vector */
```

It is possible to define a different type for the step vector by defining
`Key::difference_type` or by specializing the `ttg::key_diff_type` trait.

Signed-off-by: Joseph Schuchart <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant