Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Serialize/deserialize with libcudf pack/unpack #5025

Closed
wants to merge 114 commits into from
Closed

[WIP] Serialize/deserialize with libcudf pack/unpack #5025

wants to merge 114 commits into from

Conversation

shwina
Copy link
Contributor

@shwina shwina commented Apr 27, 2020 •
edited
Loading

Still WIP. Depends on #4941 and #5309

devavret and others added 27 commits April 18, 2020 01:02
@devavret
table_view pack: first commit.
7a9c48f
@devavret
Add cudf::unpack
3a884e7
@devavret
Pack-unpack tests fixed width nullable columns
6e3f23a 
And fix for pack: was using moved-from data.
Forgot to add first metadata element which stands in for the table
@devavret
Fix bug of pack still using old table view after contiguous split
d43f287
@devavret
Fixed the case where columns could have nulls.
1862f0b
@devavret
Tests and fix for string columns
a19b5ef
@devavret
changelog
055fada
@devavret
Use stream and mr in pack. Remove both from unpack
0ac61f2
@devavret
Change table_data ref to base_ptr as per review
b3cd355
@devavret
Change packed_table to hide serialized_column type
d0d20f5
@devavret
Change API to take ownership of packed_table in unpack
f211a85
@devavret
Merge branch 'branch-0.14' into pack-unpack
e80b8d9
@devavret
clang format of code in this PR
9fd6985
@devavret
Change packed_table.table_metadata to unique_ptr
4428907 
Because that is also expensive to copy
so we need to enforce taking ownership
@devavret
Initial pack support in Cython
aac9b14
@devavret
Cython bindings for unpack
38a6dff
@devavret
Move BufferArrayFromVector from parquet to utils
7e3fb84
@devavret
Documentation for pack/unpack
2adc6b1
@devavret
Fixing style CI
a52d18b
@devavret
Refactor unpack to avoid passing counter around
7f0bef8
@shwina
Merge branch 'branch-0.14' of https://github.com/rapidsai/cudf into s…
110df91 
…erialize-with-pack-unpack
@devavret
Modified pack to work with vector of columns
44acd2e
@shwina
Merge branch 'pack-unpack' of https://github.com/devavret/cudf into s…
b3ca181 
…erialize-with-pack-unpack
@shwina
Add pack/unpack based serialization
66eaeee
@shwina
Handle any kind of device object in dask_serialize_cudf_object
124a509
@shwina
Add level_names setter
fa8c207
@shwina
Better handling of metadata in serialize/deserialize
7963f5f
@shwina shwina requested review from a team as code owners April 27, 2020 19:29
revans2 and others added 11 commits May 27, 2020 10:56
@revans2
spell check
1bc72f5
@rgsl888prabhu
moving the nvstring condition below
45858a6
@karthikeyann
Merge pull request rapidsai#5216 from karthikeyann/doc-uniform-doc
4520d0f 
Make documentation uniform for stream and memory resource params in cpp
@shwina
Merge branch 'pack-unpack' of https://github.com/devavret/cudf into s…
2bc016c 
…erialize-with-pack-unpack
Merge pull request rapidsai#5254 from harrism/fea-deprecate-nvstrings
9dca5ab 
[REVIEW] Deprecate nvstrings
@codereport
Merge pull request rapidsai#5285 from codereport/bug-fp-debug-fail
5a607ff 
[REVIEW] Fix DEBUG compilation failure due to `fixed_point.hpp`
@revans2
Merge branch 'branch-0.14' into fix-deps
7d7904c
@revans2
Merge pull request rapidsai#5298 from revans2/fix-deps
f7470a8 
Remove unused native deps from java library
@shwina
Hack around current Serializable requirements
afc2b5e
@shwina
Merge branch 'branch-0.14' of https://github.com/rapidsai/cudf into s…
eb2975b 
…erialize-with-pack-unpack
@jakirkham
Handle host frames in serialization
ff406d8 
As some serialization strategies may rely on some or all frames on host,
add logic to `Serializable` to track which frames are on host or device
and ensure they are handled appropriately.
@jakirkham jakirkham mentioned this pull request May 28, 2020
Merged
@jakirkham
Copy link
Member

FWIW I think PR ( #5309 ) should make the needed changes to Serializable so that a mixture of host and device frames is handled correctly if you want to give that a try ;)

@shwina shwina requested a review from a team as a code owner May 28, 2020 20:32
@shwina shwina changed the base branch from branch-0.14 to branch-0.15 May 29, 2020 11:33
@jakirkham
Copy link
Member

PR ( #5309 ) was merged into branch-0.15. So merging with the latest changes in branch-0.15 should include that change.

@harrism harrism added 2 - In Progress Currently a work in progress Python Affects Python cuDF API. labels Jun 16, 2020
@shwina
Copy link
Contributor Author

shwina commented Jun 16, 2020

@VibhuJawa tested this PR and reported that it actually slowed things down for some of his work. Are we interested in pursuing this further? cc: @kkraus14 @devavret @jakirkham @jrhemstad

@devavret
Copy link
Contributor

This could be handy if/when we have cudf configuration. Use/don't use packing could be a switchable option.

@jakirkham
Copy link
Member

Yeah that is what we are discussing in issue ( #5311 ).

@jakirkham jakirkham mentioned this pull request Jul 1, 2020
Closed
@kkraus14
Copy link
Collaborator

@shwina going to close this for now since it's stalled. We can reopen if desired.

@kkraus14 kkraus14 closed this Aug 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@codereport codereport codereport requested changes

@harrism harrism Awaiting requested review from harrism

Assignees

@shwina shwina

Labels
2 - In Progress Currently a work in progress Python Affects Python cuDF API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.