Bug in the float limit handling

[cjluo-omniml]

Hi, LM eval team,

In this line: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L439

The limit is reassigned to an int if the original limit flag is a float. However, this does not count the case if I'm running two tasks together.

E.g. TASK A: Full data samples 100, TASK B: Full data samples 1000. And I use a limit of 0.1.

I'm expecting the eval to run 10 samples from TASK A and 100 samples from TASK B.

However, the current logic does not process the TASK B limit compute as the limit is set to 10 already after processing TASK A.

So it ends with we run only 10 samples for TASK B as well.

This should be an easy fix, could you help update your code? Thanks

[baberabb]

Hi! nice catch! yes, a PR will be appreciated!

[cjluo-omniml]

@baberabb could you help review the fix above?

[cjluo-omniml]

@baberabb friendly ping? This should be an easy fix

[baberabb]

Hi! @cjluo-omniml . left a comment in the PR earlier. The script uses limit again downstream as well so need to handle that as well.

[cjluo-omniml]

@baberabb fixed. Could you review again?

[cjluo-omniml]

@baberabb friendly ping?