Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What accuracy can you actually achieve? #21

Open
xiajinxiong opened this issue Sep 21, 2021 · 3 comments
Open

What accuracy can you actually achieve? #21

xiajinxiong opened this issue Sep 21, 2021 · 3 comments

Comments

@xiajinxiong
Copy link

xiajinxiong commented Sep 21, 2021

Hi! I just cloned and ran your code but after half a day's trainig the accuracy on dev set just achieved 67%, way beyond what was reported in the original paper, which was 82% on test set.
Also, when I reproduced the model with my own code and data proprocessing techniques, the best I can achieve with hierarchical GRU was around 76% on dev set and 74% on test set.
Is the accuracy reported in the paper truly reproducable? Has anyone spotted the same issue with me?
Thanks

@glicerico
Copy link
Contributor

I reached a max accuracy as you found. Please see #8 (comment)

@xiajinxiong
Copy link
Author

xiajinxiong commented Sep 21, 2021

@glicerico Thanks. Can you find a way to reach higher accuracy? My Roberta with transformer Encoder reached 79.9% on dev set, but was not a fair comparison with this model.

@glicerico
Copy link
Contributor

I'm not the author of this repository. No, I explored a few options but didn't reach more than what I report in the comment above. Good luck!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants