Lora rank vs Lora Alpha

[AdrianAcala]

Hi,

I just got into training lora models recently and so I'm a noob here so please forgive me if I'm mistaken.

I thought it was generally a best practice to have the lora alpha be double what the lora rank is, but in both your Mistral and Phi examples, the lora alpha is half of what the lora rank is.

[NanoCode012]

I think there's conflicting info on what the best hyper parameter selection is for that. The examples are good as a start.