Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn off threading for rasterizing really large rasters #532

Closed
rafaqz opened this issue Sep 30, 2023 · 2 comments
Closed

Turn off threading for rasterizing really large rasters #532

rafaqz opened this issue Sep 30, 2023 · 2 comments

Comments

@rafaqz
Copy link
Owner

rafaqz commented Sep 30, 2023

Because we need to make separate rasters for each thread to write to, threading can use all RAM when a single thread would still work.

We should look at how to check available memory and raster size and use some heuristic.

See #529

@alex-s-gardner
Copy link
Contributor

One option would be to thread by tiles, cropping the shapes to each tile would also greatly reduce the search space... though I suspect this would be a more involved mod.

@rafaqz
Copy link
Owner Author

rafaqz commented Sep 30, 2023

Cropping the shapes may take longer than rasterizing them unfortunately.

And if there are just a few big polygons then the tiles are broken.

I suspect the key problem is actually that I switched the burning arrays from BitArray to Array{Bool} and that means 8x more memory. Maybe we can just swap that back and keep the threading in most cases... it only doubles the memory use for UInt8 arrays and 8 cores.

@rafaqz rafaqz closed this as completed Oct 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants