Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix float limit override #2325

Merged
merged 3 commits into from
Oct 7, 2024
Merged

Conversation

cjluo-omniml
Copy link
Contributor

See: #2324

The float limit will be override with the previous int limit of multiple tasks are triggered together.

This PR fix this issue

See: EleutherAI#2324

The float limit will be override with the previous int limit of multiple tasks are triggered together.

This PR fix this issue
@CLAassistant
Copy link

CLAassistant commented Sep 19, 2024

CLA assistant check
All committers have signed the CLA.

lm_eval/evaluator.py Outdated Show resolved Hide resolved
@baberabb
Copy link
Contributor

baberabb commented Sep 27, 2024

sorry @cjluo-omniml , didn't know you had to commit review comments :S. Basically limit is used a couple of times in the script to gather the outputs and results for the respective tasks, so need to make sure the same limits are used downstream as well now.

@cjluo-omniml
Copy link
Contributor Author

@baberabb could you help rebase this? Not sure why the unittests are failing

Copy link
Collaborator

@haileyschoelkopf haileyschoelkopf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@baberabb this looks ready for merge now, correct?

@haileyschoelkopf
Copy link
Collaborator

Test failure unrelated (due to a change in tokenizers) and since fixed by @baberabb !

@haileyschoelkopf haileyschoelkopf merged commit aa457ed into EleutherAI:main Oct 7, 2024
4 of 8 checks passed
mariagrandury pushed a commit to somosnlp/lm-evaluation-harness that referenced this pull request Oct 9, 2024
* Fix float limit override

See: EleutherAI#2324

The float limit will be override with the previous int limit of multiple tasks are triggered together.

This PR fix this issue

* Update evaluator.py

* Update evaluator.py
maaxap pushed a commit to maaxap/lm-evaluation-harness that referenced this pull request Nov 16, 2024
* Fix float limit override

See: EleutherAI#2324

The float limit will be override with the previous int limit of multiple tasks are triggered together.

This PR fix this issue

* Update evaluator.py

* Update evaluator.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants