Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io.lines() equivalent for better iteration over files #235

Open
mxruben opened this issue Aug 6, 2024 · 3 comments
Open

io.lines() equivalent for better iteration over files #235

mxruben opened this issue Aug 6, 2024 · 3 comments
Labels
enhancement New feature or request pending / needs design Awaiting design and/or implementation discussion and decisions

Comments

@mxruben
Copy link
Contributor

mxruben commented Aug 6, 2024

Hello, I am currently looking to write some tools with lune that deal with very large files. The only option for opening files is to read the entire thing into a string in memory. This works fine for smaller files, but when the file is gigabytes large (10 GB in my case), it becomes a real issue. If the fs library could open a file pointer and had something like vanilla lua's io.lines for iteration, it would solve the cases where reading an entire file into a string isn't feasible.

@filiptibell filiptibell added enhancement New feature or request pending / needs design Awaiting design and/or implementation discussion and decisions labels Aug 7, 2024
@CleverSource
Copy link

I wrote an implementation for this here. I can make a PR if the design for this is improved or we can discuss.

I have a use case that makes this useful. @filiptibell opinions?

@CompeyDev
Copy link
Contributor

Hey! As a part of #211, I implemented a basic async stream system for the new process.create API. Can we abstract the stream implementation into the lune-utils crate and extend it for this sort of chunked reads of files?

That way, we can expand more built-ins with streaming functionality relatively easily.

@CleverSource
Copy link

CleverSource commented Dec 23, 2024

Sounds good to me. I'll update my changes when possible and create a PR when it's ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request pending / needs design Awaiting design and/or implementation discussion and decisions
Projects
None yet
Development

No branches or pull requests

4 participants