feat: File row splits #1709
feat: File row splits #1709
Conversation
| @@ -26,11 +26,11 @@ pub(crate) trait ReadMasked { | |||
|
|
|||
| /// Read an array with a [`RowMask`]. | |||
| pub(crate) struct ReadArray { | |||
| layout: Box<dyn LayoutReader>, | |||
| layout: Arc<dyn LayoutReader>, | |||
There was a problem hiding this comment.
So LayoutReaders are no longer stateful?
There was a problem hiding this comment.
Seems like I overlooked some layers of of state/locks in the readers, that will take some thinking and probably additional refactoring.
There was a problem hiding this comment.
ok after a more careful reading and a small test, as far as I can tell while the LayoutReader implementations are stateful (with interior mutability), they can read independently from each other as the LayoutBufferedReader is the part that actually controls the overall flow and each stream initializes a new one.
|
|
||
| // Set up a stream of RowMask that result from applying a filter expression over the file. | ||
| let mask_iterator = if let Some(fr) = &self.filter_reader { | ||
| Box::new(BufferedLayoutReader::new( |
There was a problem hiding this comment.
Ideally we pull the splits_stream out of this thing and just stream::iter(self.splits().iter()).map(|s| self.read_range(s).boxed()).buffered(64) or whatever?
Or is that not possible?
There was a problem hiding this comment.
Honestly worried about using buffered everywhere
Introduces a new
VortexReadHandlethat can be used cloned and used to either generate a full stream over the underlying file OR read a specific row range.The handle also implements
Cloneso it can be re-used over the same file.DataFusion support will be in a followup PR.