-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature request]Custom column list support for csv dump #122
Comments
I thought less columns during load and dump meant little more speed boost. I am trying to reduce the time for mft to csv conversion so xsv won't help. |
from my intuition i doubt that would be meaningful. simply transmuting fields to csv is a trivial amount of compute. you're welcome to try and modify the code and check the results if you'd like, or you can integrate the library to whatever downstream task you're doing for better performance. |
Reducing columns didn't help much. So I tried a different approach. I tried simply skipping the parse process if it's deleted in
And skipped it from serialization/writing it in
Got about 60% faster conversion for my data. |
sounds very reasonable, i'm glad you found something that works for you :) |
Should I make MR out of my changes so that it works with a flag like |
We don't always need all columns when generating the csv/json dump. Requesting a way to supply column list for csv dump.
The text was updated successfully, but these errors were encountered: