Merge Two New Checkpoints without Training Data

[wenyi-li]

Hi, Thanks for your great work! @gstoica27
Now I have two checkpoints of the same semantic segmentation model trained on differente datasets. The backbone is ResNet. I don't have the raw data to train, can I use the ZipIt Tool to only merge these two ckpts into one without training data? If it is possible, could you please give me a tutorial?

[gstoica27]

Hi,

I apologize for such a late response.

You absolutely can - one way you can do this is by creating a dummy dataset containing only the standard basis vectors and passing this through your models to compute the mergings. This would simulate weight-matching approaches found in prior work such as Git-Rebasin.

Does this make sense? Would you still be interested in this?

[f-amerehi]

Hi @gstoica27, hope all is well. I have a question similar to the one mentioned above and would appreciate any input.

I have two checkpoints from resnet50--- trained on ImageNet but with a different labels. I now want to merge them using ZipIt without retraining.

What does 'standard basis' mean here? Does that mean having vectors where one element is 1 and all other elements are 0? So, for example, to resemble ImageNet, I should create 1000 vectors, each corresponding to one of the 1000 classes in ImageNet. And each vector is with a single 1 in the position representing its class and 0s elsewhere. is that right?

One more question; if one Resnet network contains a batch norm layer and the other does not, would merging them create branches and next layers will form heads?

Thank you very much.