[Feature Request] Auto-detect LoRA dimensions and precision when merging

[TomLucidor]

The SVD multi-LoRA merging procedure requires "New Rank" and "New Conv Rank", as well as "Merge Precision" and "Save Precision". Those can be automated through reading the metadata of the source LoRA.
Ideally, the ranks would use the maximum of the source LoRAs for maximum information preservation. There can be a secondary option for the median/mode of the source, assuming there are weird outliers.
As for the "Merge Precision" and "Save Precision" (for both regular Merge LoRA and SVD versions), maximizing the precision to be "float" and then collapsing back to the majority precision (most are either BF16 or FP16) would work.

Q1: is it true that sometimes people want to "fix" a bad 128-dimensional LoRA with this?
Q2: What are the benefits in changing "merge precision" to 16 bits when it reduces accuracy? (maybe lower computation cost?)

[rafstahelin]

I'd love a follow up on this