Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace buffer tracking with serialization inspection #301

Open
wants to merge 24 commits into
base: master
Choose a base branch
from

Conversation

devreal
Copy link
Contributor

@devreal devreal commented Nov 8, 2024

We have been tracking the buffers of structures by having the buffer register itself with some metadata. This is clumsy and error-prone (no support for nesting, for example). Now we use a specialized archive to inspect all members of a value and apply some functor to each buffer (or its internal parsec data).

This is also the first step towards a structure derived from parsec_data_copy_t to better manage the life-time of the host copy. We now only allow shared_ptr for non-owning buffers so some adjustments must be made by users.

@devreal devreal marked this pull request as draft November 8, 2024 02:11
@devreal devreal marked this pull request as ready for review November 13, 2024 20:11
@devreal
Copy link
Contributor Author

devreal commented Nov 13, 2024

I think this is ready. However, we can only achieve such introspection using the madness archive. With boost, we would have to specialize traits on the inspector archive, which is not possible because the inspector archive is templated on the callable. Even if we were to type-erase the callable into std::function we'd still have to pick the value and allocator type for the Buffer that gets passed in. Not possible, because objects can hold different types of buffers at once.

I don't think this is an issue. Moving from boost to madness serialization is not hard and users have to touch their data structures anyway to integrate buffers. More reliance on madness though, at least for the time being.

@devreal devreal marked this pull request as draft November 13, 2024 20:41
@devreal
Copy link
Contributor Author

devreal commented Nov 13, 2024

Nevermind, not ready yet. This requires ICLDisco/parsec#695 to land in PaRSEC. Reverted back to draft.

@devreal devreal force-pushed the serialize-buffer-query branch 3 times, most recently from d84ae7b to 99dd848 Compare November 15, 2024 17:41
@devreal devreal force-pushed the serialize-buffer-query branch from da4545c to f23639d Compare November 25, 2024 22:19
We have been tracking the buffers of structures by having the buffer
register itself with some metadata. This is clumsy and error-prone
(no support for nesting, for example). Now we use a specialized
archive to inspect all members of a value and apply some functor
to each buffer (or its internal parsec data).

This is also the first step towards a structure derived from
parsec_data_copy_t to better manage the life-time of the host copy.
We now only allow shared_ptr for non-owning buffers so some adjustments
must be made by users.

Signed-off-by: Joseph Schuchart <[email protected]>
Signed-off-by: Joseph Schuchart <[email protected]>
We cannot use boost serialization to inspect objects for the
buffers they contain because boost archives cannot be templated properly
and need explicit instantiations for its archives, which is impossible
for the BufferInspectorArchive. We just rely on madness from now on.

Signed-off-by: Joseph Schuchart <[email protected]>
For now only allow T[] on smart pointers passed to buffers.
We may want to support both T and T[] on ttg::Buffer
so we can distinguish the two later.

Signed-off-by: Joseph Schuchart <[email protected]>
The data copy will call deallocate no matter what, so be graceful.
We will still catch cases where we try to allocate through
the empty allocator.

Signed-off-by: Joseph Schuchart <[email protected]>
We can now inspect the buffers of the tile to extract its memory regions.

Signed-off-by: Joseph Schuchart <[email protected]>
This allows us to mark buffers as allocate-only. SyncIn is the default.
Ignored in the MANDESS backend.

Signed-off-by: Joseph Schuchart <[email protected]>
@devreal devreal force-pushed the serialize-buffer-query branch from f23639d to 04b67fd Compare December 20, 2024 20:57
@devreal devreal marked this pull request as ready for review December 20, 2024 21:00
So we can delay evaluation until the derived struct has been populated.

Signed-off-by: Joseph Schuchart <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant