-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve resource management for block repository backends #77
Comments
- Added sql.Rows.Close() calls when missing - Allows the BlockRepository to be closed, and thus closing the underlying sql.DB connections. Fixes minetest-go#77
- Added sql.Rows.Close() calls when missing - Allows the BlockRepository to be closed, and thus closing the underlying sql.DB connections. Fixes #77
(reopening, looks like there are still 2 tasks open) |
yeah! thanks! I'm going to test how Iterator() performs on a 80G/150M blocks map cleanup task and if it indeed causes resource usage issues due to a single reference to sql.Rows I'll batch them. I have split the IteratorTo() as a new task. It is kind of a very similar impl. but may benefit from the perf. improvement if any. |
- Uses modernc.org/sqlite package - Properly used row filters for Iterator - Small updates for easy local testing Refs minetest-go#77
#80 should resolve this. |
* fixs: pure-go build, iterator for postgres - Uses modernc.org/sqlite package - Properly used row filters for Iterator - Small updates for easy local testing Refs #77 * doc: updated Iterator behavior.
It may be needed to allow for callers to manage resources when using the map data from the block repository. I see two opportunities for improvement/fixes:
1/ Implement defer Rows.Close() and DB.Close() on all backends: this may solve memory usage issues as reported here minetest-go/mapcleaner#179 as well as resolve final transactions/connections not being properly released (potentially what is happening here: minetest-go/mapcleaner#180.
2/ Implement a new version of Iterator(), IteratorTo() which receives a block bounding box. It could make concurrency easier to implement like for exporting all protected areas in separate go-routines. Also, perhaps a single iterator that keeps sql.Rows always open and only closes at the end could be a bad idea. If there are just too many blocks in the map (the main use case for cleaning the map) it can use just too much memory.
Open for feedback/thoughts on this!
Checklist:
The text was updated successfully, but these errors were encountered: