Why does the CSV extracted according to the steps show all the scores as nan?

[SsmallSong]

file name, index, score
cos, 0, nan
cos, 1, nan
cos, 2, nan
cos, 3, nan
cos, 4, nan
cos, 5, nan
cos, 6, nan
cos, 7, nan
cos, 8, nan
cos, 9, nan
cos, 10, nan
cos, 11, nan
cos, 12, nan
cos, 13, nan
cos, 14, nan
cos, 15, nan
cos, 16, nan
cos, 17, nan
cos, 18, nan
cos, 19, nan
cos, 20, nan

[xiamengzhou]

I think it's largely because you were not getting the correct gradients from the models, could you check if the raw gradients before compression are correct?

[totoers]

I also encountered this problem. I found that the execution of less/scripts/get_info/grad/get_eval_lora_grads.sh might be the earliest place that caused NaN. I used tydiqa as validation data, but some data point had a loss of NaN. What should I do in this situation?