Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use BitArray for burning buffer #533

Merged
merged 6 commits into from
Oct 21, 2023
Merged

use BitArray for burning buffer #533

merged 6 commits into from
Oct 21, 2023

Conversation

rafaqz
Copy link
Owner

@rafaqz rafaqz commented Sep 30, 2023

@alex-s-gardner you could try this branch on your huge array without threaded=false

It should use 1/8th of the memory.

@alex-s-gardner
Copy link
Contributor

This branch speeds things up considerably but for the large use case threaded=false is still much faster

Branch = Rasters v0.8.4
threaded=true: 260.770344 seconds (1.24 M allocations: 336.461 GiB, 0.09% gc time, 6.75% compilation time)
threaded=false: 8.425360 seconds (876.21 k allocations: 10.768 GiB, 0.88% gc time)

Branch = use_BitArray
threaded=true: 30.647470 seconds (932.37 k allocations: 42.203 GiB, 0.98% gc time)
threaded=false: 2.602775 seconds (913.17 k allocations: 1.491 GiB, 2.14% gc time)

@rafaqz
Copy link
Owner Author

rafaqz commented Oct 3, 2023

Ok thats way lower allocation at least.

Maybe the threads are running into each other somehow, there is a lock in there but mostly it can be avoided. I'll have to look into that.

@rafaqz rafaqz force-pushed the use_BitArray branch 2 times, most recently from b422d28 to 73101f5 Compare October 8, 2023 18:28
@codecov-commenter
Copy link

codecov-commenter commented Oct 21, 2023

Codecov Report

Merging #533 (6441f08) into main (ff90b8f) will increase coverage by 7.92%.
Report is 2 commits behind head on main.
The diff coverage is 61.53%.

@@            Coverage Diff             @@
##             main     #533      +/-   ##
==========================================
+ Coverage   73.06%   80.98%   +7.92%     
==========================================
  Files          59       57       -2     
  Lines        4633     4108     -525     
==========================================
- Hits         3385     3327      -58     
+ Misses       1248      781     -467     
Files Coverage Δ
ext/RastersArchGDALExt/warp.jl 96.00% <100.00%> (ø)
src/Rasters.jl 100.00% <ø> (ø)
src/methods/burning/array_init.jl 92.30% <100.00%> (-0.38%) ⬇️
src/methods/burning/geometry.jl 68.57% <ø> (ø)
src/methods/burning/point.jl 51.35% <100.00%> (ø)
src/methods/coverage.jl 92.92% <100.00%> (ø)
src/methods/mask.jl 90.00% <100.00%> (+0.38%) ⬆️
src/methods/rasterize.jl 89.91% <100.00%> (-0.03%) ⬇️
src/plotrecipes.jl 83.17% <ø> (+9.91%) ⬆️
src/stack.jl 84.47% <100.00%> (+0.09%) ⬆️
... and 1 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@rafaqz rafaqz merged commit 88af714 into main Oct 21, 2023
8 of 9 checks passed
@rafaqz rafaqz deleted the use_BitArray branch October 21, 2023 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants