[Bug] pytorch backend 's precision points loss 1.0-2.5 points between main code and v0.6.1 on some models.

[zhulinJulia24]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

Precision points of pytorch backend loss between main code and v0.6.1.

Reproduction

1. install opencompass
2. copy .github/scripts/eval_chat_config.py, add following code on end of config

models=[pytorch_internlm2_5_7b_chat, pytorch_llama3_8b_instruct, pytorch_qwen1_5_moe_2_7b_chat]
datasets=[*mmlu_datasets]

3. run opencompass .github/scripts/eval_chat_config.py to do evaluation

Environment

v0.6.2

Error traceback

No response