[WIP] Allocate validity masks on output arrays of groupbys #246
Conversation
|
This looks to break the current groupby sort based implementation, I guess it doesn't set the valid bits as it fills the output columns? |
|
Yeah, it looks like it is reducing the array to length-0 on output. |
kkraus14
changed the title
|
Depends on #261 |
|
@kkraus14 what is the status now that #261 is merged? I know this is blocking rapidsai/libgdf#140 |
|
rerun tests |
1 similar comment
|
rerun tests |
|
@kkraus14 now that the join syntax error is gone these are failing with |
pygdf/cudautils.py
Outdated
| @@ -300,8 +301,9 @@ def gpu_mask_from_devary(ary, bits): | |||
|
|
|||
| def mask_from_devary(ary): | |||
| bits = make_mask(len(ary)) | |||
| gpu_fill_value.forall(bits.size)(bits, 0) | |||
| gpu_mask_from_devary.forall(bits.size)(ary, bits) | |||
| if bits.size > 0: | |||
There was a problem hiding this comment.
is this function ensuring that the bitmasks are 64 byte padded as in the arrow spec? Or if not that at least 64 bit aligned since there are places in the code that assume this so that we can read 64 values at a time from memory?
There was a problem hiding this comment.
No, currently it seems like it's not padded and uses 1 byte as underlying type. @jrhemstad these are functions we'd want to move to the bit utils anyway, correct?
There was a problem hiding this comment.
@felipeblazing we're working on updating the gdf_valid_type and associated bit utility functions. I believe @BradReesWork can comment on that effort.
harrism
added
Python
kkraus14
added
2 - In Progress
|
Looks like a lot of the test failures were related to the concat bug fixed in #427, hopefully there's a more manageable number now. |
|
@kkraus14 we will not be getting null support for groupby into v0.5. I believe this PR is solely needed for null support on groupby, so can we also remove this from v0.5? |
|
Closing this as it's outdated. |
No description provided.