-
-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
map_dtc is unreasonably slow when .f returns data.table #78
Comments
@mb706 I've found the same but on a much smaller scale, but the memory allocation was higher than it should be, this is due to the map_dtc = function(.x, .f, ...) {
cols = map(.x, .f, ...)
setDT(unlist(cols, recursive = FALSE))[]
} perhaps we can do something like the following to accomodate both use-cases: map_dtc = function(.x, .f, ...) {
cols = map(.x, .f, ...)
j = map_lgl(cols, function(x) !is.null(dim(x)) && !is.null(colnames(x)))
names(cols)[j] = ""
if (inherits(cols[[1L]], "data.table")) {
cols = unlist(cols, recursive = FALSE)
}
setDT(cols)[]
} There is also PR for a C implementation for cbindlist, but seems to take quite a while till that is merged: Rdatatable/data.table#4370 |
When the function in
map_dtc
returns adata.table
with many rows,map_dtc
appears to be slower than it needs to be by a factor of about 100.profvis tells me this this is because
name_dots
is called in data.table.The text was updated successfully, but these errors were encountered: