Synthetic Data for Vision: Insights, Progress and Applications

A personal survey of Synthetic Data for Computer Vision

Overview

This repository collects the studies on currunt progress of Synthetic Data for computer vision research.

Overall, the paper collection is organized as follows. If you find some work/repo/blog/talk is missing, feel free to raise an issue or create a pull request. We appreciate contributions in any form.

• Insightsn
• Datasets
• Application

1. Insights

• [Talk] Toward an ImageNet Moment for Synthetic Data by Prof.Jia Deng | Recording

• [Talk] N=0: Learning Vision Without Visual Data by Prof.Phillip Isola | Slides

• [Tutorial] Blender pipeline to generate images for deep learning | Youtube

• [Tutorial] Generating synthetic data for deep learning | Youtube

• How Far is Video Generation from World Model? — A Physical Law Perspective
  arxiv / Project Page / Code

• Scaling Laws of Synthetic Images for Model Training ... for Now
  CVPR 2024 / Project Page / Code / Youtube

2. Datasets

• FlyingThings3D: A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
  CVPR 2016 / Project Page

• Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
  ICCV 2021 / Project Page / Code

• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
  CVPR2017 / Project Page / Code

• BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
  CVPR2024 / Project Page / Code

• Kubric: A scalable dataset generator
  CVPR2022 / Project Page / Code

• PointOdyssey A Large-Scale Synthetic Dataset for Long-Term Point Tracking
  ICCV 2023 / Project Page / Code

• EgoGen: An Egocentric Synthetic Data Generator
  CVPR 2024 / Project Page / Code

• MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond
  ICCV 2023 / Project Page / Code

• TartanAir: A Dataset to Push the Limits of Visual SLAM
  IROS 2020 / Project Page

• MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments
  arxiv 2022 / Project Page / Code

3. Applications

Visual Representation Learning

• Learning to See by Looking at Noise
  NeurIPS 2021 / Project Page / Code

• Procedural Image Programs for Representation Learning
  NeurIPS 2022 / Project Page / Code

• StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners
  arxiv 2024 / Code

• Learning Vision from Models Rivals Learning Vision from Data
  CVPR 2024 / Code

• Visual Representation Learning from Synthetic Data
  Dr.Lijie Fan's Phd Thesis

• Learning Video Representations without Natural Videos
  arxiv 2024 / Project Page

• How Transferable are Video Representations Based on Synthetic Data?
  NeurIPS 2022 / Code

• Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data
  CVPR 2022

Depth Estimation

• Marigold: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
  CVPR 2024 / Project Page / Code

• Depth Anything V2
  NeurIPS 2024 / Project Page / Code

• GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image
  ECCV 2024 / Project Page / Code

• ChronoDepth: Learning Temporally Consistent Video Depth from Video Diffusion Priors
  arxiv 2024 / Project Page / Code

• Depth Any Video with Scalable Synthetic Data
  arxiv 2024 / Project Page / Code

• DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos
  arxiv 2024 / Project Page / Code

Motion Estimation

Optical&Scene Flow

• FlowNet: Learning Optical Flow with Convolutional Networks
  FlowNet ECCV 2020 / FlowNet2 CVPR 2017 / FlowNet2 Official Code / FlowNet2 Pytorch Code

• PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
  CVPR 2018 / Code

• RAFT: Recurrent All Pairs Field Transforms for Optical Flow
  ECCV 2020 / Code

• RAFT-3D: Scene Flow using Rigid-Motion Embeddings
  CVPR 2021 / Code

• GMFlow: Learning Optical Flow via Global Matching
  CVPR 2022 / Code

• SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow
  ECCV 2024 / Code

Tracking Any Points

• Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories
  ECCV 2022 / Project Page / Code

• TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
  ICCV 2023 / Project Page / Code

• CoTracker: It is Better to Track Together
  CoTracker ECCV 2024 / CoTrackerV3 arxiv 2024 / Project Page / Code

• SpatialTracker: Tracking Any 2D Pixels in 3D Space
  CVPR 2024 / Project Page / Code

• LocoTrack: Local All-Pair Correspondence for Point Tracking
  ECCV 2024 /Project Page / Code

3D Reconstruction

• DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras
  arxiv 2024 / Project Page / Code

• LRM-Zero: Training Large Reconstruction Models with Synthesized Data
  NeurIPS 2024 / Project Page / Code

• MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data
  arxiv 2024 / Project Page / Code

Visual Content Genereation

• SimGen: Simulator-conditioned Driving Scene Generation
  arxiv 2024 / Project Page / Code

• Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis
  arxiv 2024 / Project Page / Code

• 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation
  arxiv 2024 / Project Page / Code

• SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints
  arxiv 2024 / Project Page / Code

Autonomous Driving & Robotic

TODO