When I finetuned the model, an error occurred during the decoding process: IndexError: Out of range: piece id is out of range.

[HypherX]

Thank you for your amazing work!
I use the generate_batch() function which you provide in another issue, when I run my decode code:

pred_sents = [
    tokenizer.decode(
        g
    )
    for g in pred_ids
]

It seems that there is a problem, I found that it might be an issue with the tokenizer, how can it be solved?