Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug Report] ActivationsStore fails for models without a tokenizer #186

Open
1 task done
chanind opened this issue Jun 16, 2024 · 0 comments
Open
1 task done

[Bug Report] ActivationsStore fails for models without a tokenizer #186

chanind opened this issue Jun 16, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@chanind
Copy link
Collaborator

chanind commented Jun 16, 2024

Describe the bug
For models without a tokenizer, e.g. Othello, an error is thrown in the ActivationsStore since it assumes a tokenizer is present and attempts to insert BOS tokens. If the model has no Tokenizer, we should gracefully handle this and not attempt to read anything form the tokenizer.

Code example
See #177

Checklist

  • I have checked that there is no similar issue in the repo (required)
@chanind chanind added the bug Something isn't working label Jun 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant