Large data processing

[lmtr0]

Hello there, good day!

I have a rather large csv (about 13 million rows) that I need to run some processes on (just a function).
I'm using this library to parse it and then process it, however it takes up to an hour to do all the processing, is there any way to do this faster? Like creating multiple readers that work on chunks of the file at the same time in parallel? Pollars is not an option to me as I run out of memory before reaching the middle of the file (I've about 16gb of memory in my computer)

[BurntSushi]

13 million rows really is not that much. There's no way this library is taking an hour to parse that. This library should be able to handle 13 million rows in ~seconds on commodity hardware. (Unless each row is truly huge.)

This feels to me more like a general Rust help question rather than something specific to csv.

I'd suggest:

1. Work a little harder to make it easier for others to help you. This means sharing a program that does the work you want to do, or models it in some way such that its performance characteristics match your real program. This also means sharing how you compiled your program (you're using --release right?) and how to run it. It also means sharing input. Maybe not the full 13 million rows, but perhaps a public data set or some way to generate something similar so that others can run your benchmark.
2. Ask this on a general help forum like https://users.rust-lang.org/.

[lmtr0]

running it in release solved the problem, thx,

Sorry I can't share any of the data unfortunally.
Thank you for your time.

yes the rows are massive

[BurntSushi]

Sorry I can't share any of the data unfortunally.

To clarify, for the future, you'll note that I specifically acknowledged this. And sharing the exact data you're using is not a blocker for providing a minimal reproduction when asking for help.

[lmtr0]

Yes I agree. Thank you