Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect

[ZhanYang-nwpu]

Loading checkpoint shards: 50%|█████ | 1/2 [00:13<00:13, 13.30s/it]
Traceback (most recent call last):
File "/data/zhanyang/GeoChat/test.py", line 55, in
tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name,
File "/data/zhanyang/GeoChat/geochat/model/builder.py", line 105, in load_pretrained_model
model = GeoChatLlamaForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs)
File "/home/zhanyang/.conda/envs/env_zy39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2881, in from_pretrained
) = cls._load_pretrained_model(
File "/home/zhanyang/.conda/envs/env_zy39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3228, in _load_pretrained_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
File "/home/zhanyang/.conda/envs/env_zy39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 720, in _load_state_dict_into_meta_model
set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
File "/home/zhanyang/.conda/envs/env_zy39/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 285, in set_module_tensor_to_device
raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

[bubblebrow]

Loading checkpoint shards: 50%|█████ | 1/2 [00:13<00:13, 13.30s/it] Traceback (most recent call last): File "/data/zhanyang/GeoChat/test.py", line 55, in tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, File "/data/zhanyang/GeoChat/geochat/model/builder.py", line 105, in load_pretrained_model model = GeoChatLlamaForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs) File "/home/zhanyang/.conda/envs/env_zy39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2881, in from_pretrained ) = cls._load_pretrained_model( File "/home/zhanyang/.conda/envs/env_zy39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3228, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/home/zhanyang/.conda/envs/env_zy39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 720, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/home/zhanyang/.conda/envs/env_zy39/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 285, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

请问你知道该如何解决这个问题了吗

[LiPang]

修改geochat/model/multimodal_encoder.py的第91和97行为self.clip_interpolate_embeddings(image_size=336, patch_size=14)即可。报错原因为作者根据图像大小的不同对embedding做了插值，然而作者先做了插值（图像为504时插值长度为1297），再加载的模型参数，模型参数里embedding的长度却仍然是577（和原始的clip保持一致），导致了报错。修改该bug则应该先加载参数，再对embedding插值
To fix the error, simply modify lines 91 and 97 in geochat/model/multimodal_encoder.py to: self.clip_interpolate_embeddings(image_size=336, patch_size=14).

The original error occurred because the author interpolated the embeddings based on different image sizes. However, the author performed the interpolation first (with an interpolation length of 1297 for a 504-pixel image) and then loaded the model parameters. The loaded model parameters still had an embedding length of 577 (consistent with the original CLIP), causing the error.

To fix this bug, the parameters should be loaded first, and then the embeddings should be interpolated.