Setting limit_mm_per_prompt for vllm_vlm fails argument parser

[mgoin]

Using the latest vllm release and current main of lm-eval, I see this error:

lm_eval --model vllm-vlm --model_args pretrained=llava-hf/llava-1.5-7b-hf,limit_mm_per_prompt="image=50" --tasks mmmu_val

Traceback (most recent call last):
  File "/home/mgoin/venvs/vllm/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/utils.py", line 398, in _wrapper
    return fn(*args, **kwargs)
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/evaluator.py", line 199, in simple_evaluate
    f"Initializing {model} model, with arguments: {simple_parse_args_string(model_args)}"
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/utils.py", line 107, in simple_parse_args_string
    args_dict = {
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/utils.py", line 108, in <dictcomp>
    k: handle_arg_string(v) for k, v in [arg.split("=") for arg in arg_list]
ValueError: too many values to unpack (expected 2)

I printed out the intermediate arguments before that line and the issue seems to be the overuse of "=" in the value of the argument

[['pretrained', 'llava-hf/llava-1.5-7b-hf'], ['limit_mm_per_prompt', 'image', '50']]

Maybe we could get past this by directly passing a dict in the parameter and using another parser, such as limit_mm_per_prompt="{'image':50}"

[haileyschoelkopf]

We've worked around this for now with #2387 ! -- limit_mm_per_prompt="{'image':50}" isn't a bad idea but looks a little irritating for a user to specify.