Use a dynamic sampler to speed-up weighted sampling #30
Conversation
|
okay, I think this can be merged, it gives a quite cool speedup: from >1400 seconds of main to <85 seconds for one epoch of all austrian economy |
|
Thanks @Tortar, this looks good to me and we can merge the PR. |
| @@ -6,8 +6,10 @@ version = "0.1.2" | |||
| [deps] | |||
| Dates = "ade2ca70-3891-5945-98fb-dc099432e06a" | |||
| Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f" | |||
| DynamicSampling = "2083aeaf-6258-5d07-89fc-32cf5060c837" | |||
There was a problem hiding this comment.
So, this PR is based on the use of this package. Would you have a paper that describes the technique implemented there?
There was a problem hiding this comment.
Yes, I think you can find some references of the technique of the package in
There was a problem hiding this comment.
Will add something to the package explaining the origin of the technique
| b = C_d_h .* b_HH_g[g] .- pos!(@view(C_d_hg[1:H]) .- b_CFH_g[g] .* I_d_h) | ||
| c = C_d_h .* b_HH_g[g] .+ b_CFH_g[g] .* I_d_h .- @view(C_d_hg[1:H]) | ||
| d = pos!(b_CFH_g[g] .* I_d_h .- @view(C_d_hg[1:H])) | ||
| b = @~ C_d_h .* b_HH_g[g] .- pos.(@view(C_d_hg[1:H]) .- b_CFH_g[g] .* I_d_h) |
There was a problem hiding this comment.
could you explain what @~ does?
There was a problem hiding this comment.
@~ comes from LazyArrays.jl and allows us to "wait" until that operations are actually required to run, we can then fuse the operations, so that we don't need more passes over data
Still wip because https://github.com/Tortar/DynamicSampling.jl needs still to be released, but this speeds up big models more almost by an order of magnitude
Superseeds #20