Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tokenizer class IndicBERTSentencePieceTokenizer does not exist or is not currently imported. #50

Open
alvynabranches opened this issue Sep 24, 2024 · 1 comment

Comments

@alvynabranches
Copy link

Code

import torch
from transformers import pipeline, AutoModel, AutoTokenizer

model_id = "ai4bharat/IndicBERTv2-alpha-SentimentClassification"
tokenizer = AutoTokenizer.from_pretrained(model_id, keep_accents=True)
model = AutoModel.from_pretrained(model_id).to("cuda")

Error

ValueError                                Traceback (most recent call last)
[<ipython-input-8-faa308d63e88>](https://localhost:8080/#) in <cell line: 2>()
      1 model_id = "ai4bharat/IndicBERTv2-alpha-SentimentClassification"
----> 2 tokenizer = AutoTokenizer.from_pretrained(model_id, keep_accents=True)
      3 model = AutoModel.from_pretrained(model_id).to("cuda")

[/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    892                 tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
    893             if tokenizer_class is None:
--> 894                 raise ValueError(
    895                     f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
    896                 )

ValueError: Tokenizer class IndicBERTSentencePieceTokenizer does not exist or is not currently imported.
@Jeet009
Copy link

Jeet009 commented Nov 16, 2024

Also getting the same error. Any updates on the issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants