Batch inference

[mtsysin]

Hi!

I'm evaluating the model on a relatively large dataset (single question, single answer). I was able to fine-tune the Bunny-1.1-Llama-3-8B-V model using one of the scripts provided. What is the best strategy to implement batch inference?

[Isaachhh]

Sorry for that we don't support batch inference currently. You may split the dataset into multiple parts and launch a model on each GPU, like evaluating on VQA, GQA and SEED-Bench.

[Isaachhh]

@mtsysin

You may refer to batch_inference.py.

However, we failed to set the attention_mask of left-padding tokens to be 0. So the attention_mask of inputs are all 1 and the outputs may be a little different from single-sample inference.

[Isaachhh]

Close the issue for now if there's no further discussions. Feel free to reopen it if there's any other questions.

[Artanic30]

Hi, I observe a significant performance drop with batch inference. The accuracy for mmmu decreased from 41 to 34 when inference with bs 8 in bunny-1.1 4B. I'm wondering if you have any plans for resolving the issue.

[Artanic30]

@mtsysin

You may refer to batch_inference.py.

However, we failed to set the attention_mask of left-padding tokens to be 0. So the attention_mask of inputs are all 1 and the outputs may be a little different from single-sample inference.

I notice the issue is largely caused by the padding tokens. I would like to recommend a workaround. If the authors could provide a code to group the input questions by their token length during batch inference, the negative impact of padding tokens may be reduced.