Whisper large fine-tuning on wenetspeech, mutli-hans-zh

[yuekaizhang]

This PR adds:

• SpeechIO test set recipe: decoding results on whisper, zipformer, see https://github.com/yuekaizhang/icefall/blob/whisper_zh/egs/speechio/ASR/RESULTS.md

Model	SpeechIO 001-026 Avg WER	Comment
zrjin/icefall-asr-multi-zh-hans-zipformer-ctc-2023-10-24	7.48%	Zipformer trained on 14k hours ZH data
whisper-large-v2-wenetspeech	8.01%	large-v2 fine-tuned with wenetspeech 10k hours, suffer deletion errors: see wenet-e2e/WenetSpeech#54
whisper-large-v2-wenetspeech + zipformer	6.93%	Utilize zipformer to reduce deletion errors: see here

• Whisper fine-tuning recipe on wenetspeech, multi-hans-zh

Following PR TODOs:

• wenetspeech/whisper RESULTS.md update
• multi-hans-zh/whisper RESULTS.md update

[yuekaizhang]

Not sure about deletion errors, but the key aspect of whisper is that it is multiobjective both transcription and translation. Translation task is not just for fun, it helps model to factorize out grammar and sense aspect and improve ASR accuracy.

If you want to get a good numbers on high-resource language, you also need multi-objective, at least translation task. Ideally speakerid objective too.

Good point. It would be great if you know some experiment results or papers which use multi-objective to fine-tune whisper.

[marcoyang1998]

Have other people reported similar observations when fine-tuning Whisper on WenetSpeech?

Similarly, we experienced severe deletion errors while training Zipformer on WenetSpeech (see #1130), could this be the problem of the dataset?

[JinZr]

thanks! i'll look into it.

[yuekaizhang]

Have other people reported similar observations when fine-tuning Whisper on WenetSpeech?

Similarly, we experienced severe deletion errors while training Zipformer on WenetSpeech (see #1130), could this be the problem of the dataset?

@marcoyang1998 You're correct, see wenet-e2e/WenetSpeech#54.

One solution is to retrain with the new labels provided by wenet-e2e/WenetSpeech#54, but I'm thinking that for such colloquial scenarios, there might be a better way to evaluate, such as reducing the weight of errors related to modal particles.

When people want to use ASR to add subtitles to their videos, it's clear that it would be more helpful if the model could automatically omit these colloquial words.

[JinZr]

hi, thank you for your work!

i went through the pr and left a comment and few modifications, if you feel like those are proper then i think this pr is ready to merge.

[JinZr]

LGTM, waiting for CI tests to be done

thanks!