Skip to content

Latest commit

 

History

History
723 lines (547 loc) · 56.7 KB

VideoRelated.md

File metadata and controls

723 lines (547 loc) · 56.7 KB
Raw

Table of Contents

Video Super Resolution

Frame Interpolation

  • AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation
    Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, Ming-Ming Cheng
    [CVPR 2023] [Project] [Pytorch-Code]

  • BiFormer: Learning Bilateral Motion Estimation via Bilateral Transformer for 4K Video Frame Interpolation
    Junheum Park, Jintae Kim, Chang-Su Kim
    [CVPR 2023] [Pytorch-Code]

  • A Unified Pyramid Recurrent Network for Video Frame Interpolation
    Xin Jin, Longhai Wu, Jie Chen, Youxin Chen, Jayoon Koo, Cheul-hee Hahm
    [CVPR 2023] [Pytorch-Code]

  • Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation
    Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, Limin Wang
    [CVPR 2023] [Pytorch-Code]
    [EMA-VFI]

  • Beyond a Video Frame Interpolator: A Space Decoupled Learning Approach to Continuous Image Transition
    Tao Yang, Peiran Ren, Xuansong Xie, Xiansheng Hua, Lei Zhang
    [arXiv 2203] [Pytorch-Code]
    [SDL]

  • FILM: Frame Interpolation for Large Motion
    Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, Brian Curless
    [ECCV 2022] [Project] [TF2-Code]
    [★★] end-to-end的插帧网络, 使用类似PWCNet的多尺度flow预测结构, 特征提取部分使用共享权值对图像金字塔提特征, 首次使用gram matrix loss, 使生成帧更清晰. 网络结构很简单明了, 但效果很好, 训练和模型设计的细节应该是功不可没.

  • Enhancing Deformable Convolution based Video Frame Interpolation with Coarse-to-fine 3D CNN
    Duolikun Danier, Fan Zhang, David Bull
    [arXiv 2202] [Pytorch-Code]
    [EDC]

  • A Subjective Quality Study for Video Frame Interpolation
    Duolikun Danier, Fan Zhang, David Bull
    [arXiv 2202] [Project]
    [BVI-VFI]

  • Splatting-based Synthesis for Video Frame Interpolation
    Simon Niklaus, Ping Hu, Jiawen Chen
    [arXiv 2201]

  • Deep Bayesian Video Frame Interpolation
    Zhiyang Yu, Yu Zhang, Xujie Xiang, Dongqing Zou, Xijun Chen, Jimmy S. Ren
    [ECCV 2022] [Pytorch-Code]
    [DBVI]

  • IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation
    Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, Jie Yang
    [CVPR 2022] [Pytorch-Code]<Br

  • ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation
    Duolikun Danier, Fan Zhang, David Bull
    [CVPR 2022] [Project] [Pytorch-Code]

  • Real-time Spatial Temporal Transformer
    Zhicheng Geng, Luming Liang, Tianyu Ding, Ilya Zharkov
    [CVPR 2022] [Pytorch-Code]
    [RSTT]

  • Video Frame Interpolation Transformer
    Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, Ming-Hsuan Yang
    [CVPR 2022] [Pytorch-Code]

  • Many-to-many Splatting for Efficient Video Frame Interpolation
    Ping Hu, Simon Niklaus, Stan Sclaroff, Kate Saenko
    [CVPR 2022] [Code]
    [M2M] [★★] 首先用off-the-shell model预测0->1/1->0的光流, 再通过motion refinement network预测N个光流图及置信度map, 最后利用N个光流进行forward warp并fuse结果. 使用Low-rank Feature Modulation加强光流的低秩约束.

  • DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting
    Jihyong Oh, Munchurl Kim
    [arXiv 2111] [Pytorch-Code]

  • EA-Net: Edge-Aware Network for Flow-based Video Frame Interpolation
    Bin Zhao, Xuelong Li
    [arXiv 2105]
    [★] 基于光流的插帧, 加入了edge信息

  • Asymmetric Bilateral Motion Estimation for Video Frame Interpolation
    Junheum Park, Chul Lee, Chang-Su Kim
    [ICCV 2021] [Pytorch-Code]
    [ABME] [★] 先大致估计t->0, t->1的光流, 生成初始It, 再refine光流和生成帧

  • Revisiting Adaptive Convolutions for Video Frame Interpolation
    Simon Niklaus, Long Mai, Oliver Wang
    [WACV 2021] [Pytorch-Code]
    [SepConv++] [★] 使用adaptive conv做插帧, 在sepconv的基础上提出了一些trick. 要点为: 1.预测x和y方向的conv kernel; 2.在预测kernel时不padding; 3.输入两帧一起逐channel归一化到0均值单位方差; 4. 对kernel做norm; 5.使用VGG loss

  • XVFI: eXtreme Video Frame Interpolation
    Hyeonjun Sim, Jihyong Oh, Munchurl Kim
    [ICCV 2021 Oral] [Pytorch-Code]
    [★★] 1. 提出了一个4K, 1000fps的插帧数据集; 2. 提出一个共享参数的多尺度插帧网络, 通过调整预测的scale级数, 处理不同分辨率和偏移.

  • FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
    Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, Du Tran
    [CVPR 2021] [Project] [Pytorch-Code]
    [★★] 首次提出用3D卷积做视频插帧, 结构为UNet, 输入为前后四帧, 输出为需要插的k-1帧

  • CDFI: Compression-Driven Network Design for Frame Interpolation
    Tianyu Ding, Luming Liang, Zhihui Zhu, Ilya Zharkov
    [CVPR 2021] [Pytorch-Code]
    [★] 通过加入L1正则引入稀疏性, 然后将模型输入层数逐次减小, 得到压缩后的模型. 论文中表示压缩后的结构更合理, from scratch训练该网络就能得到与大模型相近的性能. 之后在小模型上加入了一些改进模块, 进一步提高了精度

  • Deep Animation Video Interpolation in the Wild
    Li Siyao, Shiyu Zhao, Weijiang Yu, Wenxiu Sun, Dimitris N. Metaxas, Chen Change Loy, Ziwei Liu
    [CVPR 2021] [Pytorch-Code]
    [AnimeInterp] [★] 动画的插帧, 针对动画纹理平滑和位移大的特点, 设计了segment匹配模块和coarse-to-fine的光流匹配模块.

  • RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation
    Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, Shuchang Zhou
    [arXiv 2011] [Pytorch-Code] [Software]
    [★★] 使用一个coarse-to-fine的网络IFNet预测f1,f2到t时刻的光流, 融合部分使用网络预测fusion map和residual. 为更好训练光流, 使用leakage distillation的方法, 先用一训练好的大网络预测中间光流的值.

  • Channel Attention Is All You Need for Video Frame Interpolation
    Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, Kyoung Mu Lee
    [AAAI 2020] [Project] [Pytorch-Code]
    [CAIN] [★★] 设计了一个pixelshuffle + attention residual block的网络, 无需光流估计和warp操作.

  • Video Frame Interpolation Via Residue Refinement
    Haopeng Li, Yuan Yuan, Qi Wang
    [ICASSP 2020] [Pytorch-Code]
    [RRIN] [★] 残差和UNet结构预测光流, warp, refine

  • BMBC: Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation
    Junheum Park, Keunsoo Ko, Chul Lee, Chang-Su Kim
    [ECCV 2020] [Pytorch-Code]
    [BMBC] [★] 基于光流的插帧, 提出了用中间光流值取得前后帧的feature组成bilateral cost volume

  • All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling
    Zhixiang Chi, Rasoul Mohammadi Nasiri, Zheng Liu, Juwei Lu, Jin Tang, Konstantinos N Plataniotis
    [ECCV 2020] [Project] [Code]
    [★] 1.使用三次多项式对光流建模; 2.用temporal pyramidal形式, 有易到难逐次估计t1/t7->t2/t6->t3/t5->t4; 3. 估计光流时, 使用relaxed loss, 即允许估计的光流有小范围误差

  • Enhanced Quadratic Video Interpolation
    Yihao Liu, Liangbin Xie, Li Siyao, Wenxiu Sun, Yu Qiao, Chao Dong
    [ECCVW 2020] [Pytorch-Code]
    [EQVI] [★] 1.用最小二乘计算quadratic光流; 2.用resnet18提取的contextual特征和原图一起预测残差; 3.对两个尺度的输入用相同网络处理, 并用一个fusion net预测map进行融合

  • Video Frame Interpolation without Temporal Priors
    Youjian Zhang, Chaoyue Wang, Dacheng Tao
    [NeurIPS 2020] [Pytorch-Code]
    [UTI-VFI] [★] 用残差网络先从模糊帧中预测清晰的起始和结束关键帧, 再用二次光流refine

  • Softmax Splatting for Video Frame Interpolation
    Simon Niklaus, Feng Liu
    [CVPR 2020] [Pytorch-Code]
    [SoftSplat] [★☆] 使用前向光流warp的插帧, 大致浏览, 效果不错. 主要创新点为使用I0和I1_warp的亮度一致性作为权重Z, 并用一网络refine Z, 最后在融合时使用exp保证了尺度不变性(此处是从深度图作为Z来论述的).

  • Scene-Adaptive Video Frame Interpolation via Meta-Learning
    Myungsub Choi, Janghoon Choi, Sungyong Baik, Tae Hyun Kim, Kyoung Mu Lee
    [CVPR 2020] [Project] [Pytorch-Code]
    [Meta Interpolation]

  • FeatureFlow: Robust Video Interpolation via Structure-to-Texture Generation
    Shurui Gui, Chaoyue Wang, Qihua Chen, Dacheng Tao
    [CVPR 2020] [Pytorch-Code]
    [★] 首先, 根据两帧输入图像和edge, 预测中间帧的structure图; 然后设计了一个texture compensator生成纹理细节. 网络用deformable conv做插帧和纹理补偿.

  • Blurry Video Frame Interpolation
    Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, Zhiyong Gao
    [CVPR 2020 Oral] [Pytorch-Code]
    [BIN] [★] 空间采用金字塔型结构, 逐层利用多帧信息, 为保证帧间一致性, 使用了ConvLSTM挖掘temporal关系

  • AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation
    Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, Sangyoun Lee
    [CVPR 2020] [Pytorch-Code]

  • Deep Slow Motion Video Reconstruction with Hybrid Imaging System
    Avinash Paliwal, Nima Kalantari
    [TPAMI 2020] [Project] [Pytorch-Code]
    [Deep-SloMo]

  • Robust Video Frame Interpolation With Exceptional Motion Map
    Minho Park, Hak Gu Kim, Sangmin Lee, Yong Man Ro
    [TCSVT 2020]

  • ALANET: Adaptive Latent Attention Network for Joint Video Deblurring and Interpolation
    Akash Gupta, Abhishek Aich, Amit K. Roy-Chowdhury
    [MM 2020] [Project] [Code]
    [★] 用帧内和帧间attention做去模糊和插帧

  • Depth-Aware Video Frame Interpolation
    Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang
    [CVPR 2019] [Project] [Pytorch-Code]
    [DAIN] [★★☆]

  • Unsupervised Video Interpolation using Cycle Consistency
    Fitsum A. Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J. Shih, Andrew Tao, Jan Kautz, Bryan Catanzaro
    [ICCV 2019] [Project] [Pytorch-Code]

  • IM-Net for High Resolution Video Frame Interpolationn
    Tomer Peleg, Pablo Szekely, Doron Sabo, Omry Sendik
    [CVPR 2019] [Project] [Pytorch-Code]
    [★] 预测motion field vector(光流)和occlusion map做插帧

  • Quadratic video interpolation
    Xiangyu Xu, Siyao Li, Wenxiu Sun, Qian Yin, Ming-Hsuan Yang
    [NeurIPS 2019] [Project] [Pytorch-Code]
    [QVI] [★] 利用-1,0,1三帧的信息对光流进行二次插值

  • Deep Video Frame Interpolation using Cyclic Frame Generation
    Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, Yung-Yu Chuang
    [Project] [TF-Code]
    [CyclicGen] [★☆] 用2个预测帧预测输入帧, 作为consistency loss, 提高生成帧的质量. 另外提出了光流线性约束loss, 并将边缘信息加入到输入中

  • MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement
    Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang
    [TPAMI 2019] [Project] [Pytorch-Code]
    [★★] 提出一个自适应warping layer, 将warping中简单的bilinear操作改为学习的插值kernel与双线性核相结合的操作

  • Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
    Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz
    [CVPR 2018] [Project] [Pytorch-Code]
    [★☆] 视频插帧. 首先预测双向光流, 接下来在每个要插值的时刻t, 用一个网络refine光流并预测visibility map, 最后根据光流和visibility map插值生成t时刻图像.

  • Context-aware Synthesis for Video Frame Interpolation
    Simon Niklaus, Feng Liu
    [CVPR 2018] [Project]

  • Video Frame Interpolation via Adaptive Separable Convolution
    Simon Niklaus, Long Mai, Feng Liu
    [ICCV 2017] [Project] [Pytorch-Code]
    [sepconv-slomo]

  • Video Frame Interpolation via Adaptive Convolution
    Simon Niklaus, Long Mai, Feng Liu
    [CVPR 2017] [Project]
    [adaconv-slomo]

Video Enhancement and Restoration

Video Stabilization

Video Debluring

  • Deep Discriminative Spatial and Temporal Network for Efficient Video Deblurring
    Jinshan Pan], Boming Xu, Jiangxin Dong, Jianjun Ge, Jinhui Tang
    [CVPR 2023] [Pytorch-Code]
    [DSTNet]

  • Multi-Scale Memory-Based Video Deblurring
    Bo Ji, Angela Yao
    [CVPR 2022] [Pytorch-Code]

  • DeFMO: Deblurring and Shape Recovery of Fast Moving Objects
    Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Jiri Matas, Marc Pollefeys
    [CVPR 2021] [Pytorch-Code]
    [★] 从单张模糊图像和背景图中恢复t张清晰的图像, 网络结构为一个encoder+t个renderers, 采用了几个loss分别用于处理目标外观, sharpness及空间一致性等.

  • ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring
    Dongxu Li, Chenchen Xu, Kaihao Zhang, Xin Yu, Yiran Zhong, Wenqi Ren, Hanna Suominen, Hongdong Li
    [CVPR 2021] [Project] [Pytorch-Code]

  • Learning Event-Driven Video Deblurring and Interpolation
    Songnan Lin, Jiawei Zhang, Jinshan Pan, Zhe Jiang, Dongqing Zou, Yongtian Wang, Jing Chen, Jimmy Ren
    [ECCV 2020]

  • Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring
    Zhihang Zhong, Ye Gao, Yinqiang Zheng, Bo Zheng
    [ECCV 2020 Spotlight] [Pytorch-Code]
    [ESTRNN]

  • Cascaded Deep Video Deblurring Using Temporal Sharpness Prior
    Jinshan Pan, Haoran Bai, Jinhui Tang
    [CVPR 2020] [Project] [Pytorch-Code]
    [CDVD-TSP] [★☆] 使用光流将先后帧warp到当前帧, 和一个新提出的sharpness prior进行concat, 送入网络进行处理. 通过级联(2阶段)的方式提高精度.

  • Spatio-Temporal Filter Adaptive Network for Video Deblurring
    Shangchen Zhou, Jiawei Zhang, Jinshan Pan, Haozhe Xie, Wangmeng Zuo, Jimmy Ren
    [ICCV 2019] [Project] [Pytorch-Code]
    [STFAN]

Video Deraining

Video Dehazing

  • Learning to Restore Hazy Video: A New Real-World Dataset and A New Method
    Xinyi Zhang, Hang Dong, Jinshan Pan, Chao Zhu, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Fei Wang
    [CVPR 2021]

Video Deflicker

Video Inpainting

Video Matting

Video Demoireing

Video Synthesis

Video Editing

Misc