Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple forward per backward #81

Open
ClashLuke opened this issue Sep 11, 2022 · 0 comments
Open

Multiple forward per backward #81

ClashLuke opened this issue Sep 11, 2022 · 0 comments
Labels
core Improves core model while keeping core idea intact engineering Software-engineering problems that don't require ML-Expertise research Creative project that might fail but could give high returns

Comments

@ClashLuke
Copy link
Member

Currently, our model does one forward pass and uses the intermediate states to do one backward pass. However, a backward pass is over 3x as expensive as a forward pass, so we could change the ratio of forward to backward passes to speed up the model.
One such approach would be MESA, which adds KL(model(x), ema_model(x)). Another method is RHO-Loss, which prioritizes some samples over others, by running (model(x) - oracle(x)).topk(). Both of these methods claim to improve sample efficiency by up to 18x.

@ClashLuke ClashLuke added research Creative project that might fail but could give high returns engineering Software-engineering problems that don't require ML-Expertise core Improves core model while keeping core idea intact labels Sep 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Improves core model while keeping core idea intact engineering Software-engineering problems that don't require ML-Expertise research Creative project that might fail but could give high returns
Projects
None yet
Development

No branches or pull requests

1 participant