Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding a future backend and possibly refactoring parallelization backends #152

Open
coatless opened this issue Sep 6, 2024 · 0 comments

Comments

@coatless
Copy link

coatless commented Sep 6, 2024

Hi @ecpolley! Hope all is well.

I'm reaching out today to see if you'd be interested in receiving a PR that contains a {future} backend for SuperLearner. future is part of a collection of R packages by Henrik Bengtsson that seeks to unify the underlying parallelization frameworks within R. The reason for preferring a future-oriented backend is it allows us to easily work with remote and cluster-based resources for intense computations.

For a future backend to exist, very few changes are required. For example, the following diff based on a slight code change with mcSuperLearner (based on multicore) seems to work out quite well.

A screenshot showing the diff between a possible `futureSuperLearner()` function and the already existing `mcSuperLearner()` function

Let me know if you would be interested in receiving this backend.

Moreover, glancing at the code differences between snow*(), mc*(), and future*() SuperLearner backends, there is a lot of similar code that could potentially be combined with minimal disruption.

/ cc @SG-TLR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant