Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Separate datasource from reader parameters in cuIO #17159

Open
ttnghia opened this issue Oct 23, 2024 · 1 comment
Open

[FEA] Separate datasource from reader parameters in cuIO #17159

ttnghia opened this issue Oct 23, 2024 · 1 comment
Labels
cuIO cuIO issue feature request New feature or request

Comments

@ttnghia
Copy link
Contributor

ttnghia commented Oct 23, 2024

Currently, for reading various file formats, we do this:

auto opts = cudf::io::xxx_reader_options::builder(cudf::io::source_info{input_data_source_info})
    .some_options(...)
    ...
    .build();

auto output = read_xxx(opts);

When we have a large number of options to specify and have different datasources, it would be very burdensome to set the parameters repeatedly for every datasource.

We can avoid doing so by separating datasource from reading options. By doing so, we can just set the reading parameters once, then reuse the options instance multiple times:

auto opts = cudf::io::xxx_reader_options::builder(cudf::io::source_info{input_data_source_info})
    .some_options(...)
    ...
    .build();

auto output1 = read_xxx(source1, opts);
auto output2 = read_xxx(source2, opts);
....
@ttnghia ttnghia added cuIO cuIO issue feature request New feature or request labels Oct 23, 2024
@ttnghia
Copy link
Contributor Author

ttnghia commented Oct 23, 2024

CC @vuule @karthikeyann.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cuIO cuIO issue feature request New feature or request
Projects
Status: In Progress
Development

No branches or pull requests

1 participant