Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add RoBERTaConfig to TransformerConverter #1510

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

bhavnicksm
Copy link

Hey!

I did a simple hack to add support for RoBERTa models that I have been using for a bit. Please give me feedback if this makes sense or if this is wrong as well.

Thanks! 😄

@michaelfeil
Copy link
Contributor

@bhavnicksm The code between Roberta and Bert is quite different - e.g. compare to https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py and https://github.com/huggingface/transformers/blob/main/src/transformers/models/roberta/modeling_roberta.py - I am sorry to say that your PR likley will need quite some more work to support Roberta.

@mhillebrand
Copy link

mhillebrand commented Feb 18, 2024

What if you did this instead?

@register_loader("RobertaConfig")
class RobertaLoader(XLMRobertaLoader):
    @property
    def architecture_name(self):
        return "RobertaForSequenceClassification"

XLMRobertaConfig and RobertaConfig look nearly identical under the hood.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants