Make into_arrow truly zero-copy, rewrite DataFusion operators #451
Conversation
Before we would go through the ArrowBuffer::from(&[u8]) constructor, which would actually do a full data copy. Now we're just doing pointer tricks. Shaved about 30ms (~7%) off of the tpc-h benchmark.
| data.into(), | ||
| nulls, | ||
| )), | ||
| PType::I32 => Arc::new(unsafe { |
There was a problem hiding this comment.
I'm not sure how much this actually saves us, once I get to ~parity with arrow numbers I'll turn the validations back on and see what difference it makes
|
I'm using this as a more general improve-the-benchmarks PR, found a few more allocations and rewriting the vortex memory scan exec, should have something ready to ship later this morning |
|
Rewrote the datafusion operators to operate over This plus eliminating all of the extra allocations brings us down to parity with DataFusion's builtin Arrow 
|
| f.debug_struct("VortexScanExec") | ||
| .field("array_length", &self.array.len()) | ||
| .field("array_dtype", &self.array.dtype()) | ||
| .field("scan_projection", &self.scan_projection) | ||
| .field("plan_properties", &self.plan_properties) | ||
| .finish_non_exhaustive() | ||
| } |
There was a problem hiding this comment.
I went with custom impl because Array logs the entire buffer contents as part of its Debug impl
Stdlib has a gated unstable feature to log a field using the contents of a closure, so in the future you'd be able to do something like
f.debug_struct("VortexScanExec")
.field_with("array_length", |mut f| {
f.debug_struct("Array")
.field("length", &self.array.len())
.field("dtype", self.array.dtype())
.finish()
})
https://doc.rust-lang.org/std/fmt/struct.DebugStruct.html#method.field_with
a10y
changed the title
|
The spread between vortex-pushdown and vortex-nopushdown is all of the 
|
Before we would go through the ArrowBuffer::from(&[u8]) constructor, which would actually do a full data copy.
Now we're just doing pointer tricks. Shaved about 30ms (~7%) off of the tpc-h benchmark.