Question about Hyperparameters

[yongzx]

Thank you for this wonderful repo!

I have a quick question: are the hyperparameters listed in the README optimized for both finetuning on a specific task and finetuning to support a new language (i.e., language adaptation)?

[youran-qi]

Hi @yongzx, thank you and hope this repo is helpful in your use case!

The hyperparameters in the README are not optimized for a specific task or for language adaptation. We decided these hyperparameters according to experiments in some very generic use cases. Therefore, they are more like some hyperparameters you can start with, but you probably want to do some hyperparameter search to find the best hyperparameters for your specific task, e.g., language adaptation, etc.

Also, there are some validation codes to make sure the hyperparameters are in the "valid ranges" (see here). If you believe the best hyperparameters for your specific task are out of these "valid ranges", you can also use your out-of-range hyperparameters, but don't forget to remove / comment these validation codes.