New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix_untrained_tokens doesn't work with zero-3 #1944

Open

6 of 8 tasks

winglian opened this issue Oct 4, 2024 · 0 comments

Open

6 of 8 tasks

fix_untrained_tokens doesn't work with zero-3 #1944

winglian opened this issue Oct 4, 2024 · 0 comments

Labels

bug

Collaborator

winglian commented Oct 4, 2024

Please check that this issue hasn't been reported before.

I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

n/a

Current behaviour

n/a

Steps to reproduce

n/a

Config yaml

No response

Possible solution

i think this is because the embeddings are sharded

Which Operating Systems are you using?

Linux
macOS
Windows

Python Version

3.11

axolotl branch-commit

main

Acknowledgements

My issue title is concise, descriptive, and in title casing.
I have searched the existing issues to make sure this bug has not been reported yet.
I am using the latest version of axolotl.
I have provided enough information for the maintainers to reproduce and diagnose the issue.

winglian added the bug label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment