Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning

Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, [Yingya Zhang], Changxin Gao, [Deli Zhao], Nong Sang
In ICCV, 2023. [Paper].

Latest

[2023-09] Codes and models are available!

This repo is a modification on the TAdaConv repo.

Installation

Requirements:

• Python>=3.6
• torch>=1.5
• torchvision (version corresponding with torch)
• simplejson==3.11.1
• decord>=0.6.0
• pyyaml
• einops
• oss2
• psutil
• tqdm
• pandas

optional requirements

• fvcore (for flops calculation)

Model Zoo

Dataset	architecture	pre-training	#frames	acc@1	acc@5	checkpoint	config
SSV2	ViT-B/16	CLIP	8	68.7	91.1	[google drive]	vit-b16-8+16f
SSV2	ViT-B/16	CLIP	16	70.2	92.0	[google drive]	vit-b16-16+32f
SSV2	ViT-B/16	CLIP	32	70.9	92.1	[google drive]	vit-b16-32+64f
SSV2	ViT-L/14	CLIP	32	73.1	93.2	[google drive]	vit-l14-32+64f
K400	ViT-B/16	CLIP	8	83.6	96.3	[google drive]	vit-b16-8+16f
K400	ViT-B/16	CLIP	16	84.4	96.7	[google drive]	vit-b16-16+32f
K400	ViT-B/16	CLIP	32	85.0	97.0	[google drive]	vit-b16-32+64f
K400	ViT-L/14	CLIP	32	88.0	97.9	[google drive]	vit-l14-32+64f
K400	ViT-L/14	CLIP + K710	32	89.6	98.4	[google drive]	vit-l14-32+64f

Running instructions

You can find some pre-trained models in the Model Zoo.

For detailed explanations on the approach itself, please refer to the paper.

For an example run, set the DATA_ROOT_DIR and ANNO_DIR in configs/projects/dist/vit_base_16_ssv2.yaml, and OUTPUT_DIR in configs/projects/dist/ssv2-cn/vit-b16-8+16f_e001.yaml, and run the command for fine-tuning:

python runs/run.py --cfg configs/projects/dist/ssv2-cn/vit-b16-8+16f_e001.yaml

We use 8 Nvidia V100 GPUs for fine-tuning, and each GPU contains 32 video clips.

Citing DiST

If you find DiST useful for your research, please consider citing the paper as follows:

@inproceedings{qing2023dist,
  title={Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning},
  author={Qing, Zhiwu and Zhang, Shiwei and Huang, Ziyuan and Yingya Zhang and Gao, Changxin and Deli Zhao and Sang, Nong},
  booktitle={ICCV},
  year={2023}
}