Diffusion Models: A Comprehensive Survey of Methods and Applications

This repo is constructed for collecting and categorizing papers about diffusion models according to our survey paper——Diffusion Models: A Comprehensive Survey of Methods and Applications

Overview

Catalogue

Algorithm Taxonomy

Sampling-Acceleration Enhancement

• Learning-Free Sampling
• Learning-Based Sampling

Likelihood-Maximization Enhancement

• Noise Schedule Optimization
• Learnable Reverse Variance
• Continuous-Time VLB
• Exact Log likelihood

Data-Generalization Enhancement

• Manifold Structures
• Data with Invariant Structures
• Discrete Data

Application Taxonomy

• Computer Vision
  • Image Super Resolution, Inpainting and Translation
  • Semantic Segementation
  • Video Generation
  • Point Cloud Completion and Generation
  • Anomaly Detection
• Natural Language Processing
• Temporal Data Modeling
  • Time-Series Imputation
  • Time-Seires Forecasting
  • Waveform Signal Processing
• Multi-Modal Learning
  • Text-to-Image Generation
  • Text-to-Aaudio Generation
• Robust Learning
• Molecular Graph Modeling
• Material Design
• Inverse Problem Solving (Medical Imaging)

Connections with Other Generative Models

• Variational Autoencoder
• Generative Adversarial Network
• Normalizing Flow
• Autoregressive Models
• Energy-Based Models

Algorithm Taxonomy

1. Sampling-Acceleration Enhancement

1.1.1 Learning-Free Sampling

1.1.1.1 SDE Solver

Score-Based Generative Modeling through Stochastic Differential Equations

Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Gotta Go Fast When Generating Data with Score-Based Models

Elucidating the Design Space of Diffusion-Based Generative Models

1.1.2 ODE Solver

Denoising Diffusion Implicit Models

gDDIM: Generalized denoising diffusion implicit models

Elucidating the Design Space of Diffusion-Based Generative Models

DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Step

Fast Sampling of Diffusion Models with Exponential Integrator

1.2 Learning-Based Sampling

1.2.1 Dynamic Programming

Learning to Efficiently Sample from Diffusion Probabilistic Models

1.2.2 Knowledge Distillation

Progressive Distillation for Fast Sampling of Diffusion Models

Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed

1.2.3 Early Stopping

Accelerating Diffusion Models via Early Stop of the Diffusion Process

Truncated Diffusion Probabilistic Models

2. Likelihood-Maximization Enhancement

2.1. Noise Schedule Optimization

2.1.1 Deterministic Schedule

Improved denoising diffusion probabilistic models

2.1.2 Learnable Schedule

Variational diffusion models

2.2. Learnable Reverse Variance

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

Improved denoising diffusion probabilistic models

2.3. Continuous-Time VLB

Maximum likelihood training of score-based diffusion models

A variational perspective on diffusion-based generative models and score matching

2.4 Exact Log likelihood

Score-Based Generative Modeling through Stochastic Differential Equations

Maximum Likelihood Training for Score-based Diffusion ODEs by High Order Denoising Score Matching

3. Data-Generalization Enhancement

3.1. Manifold Structures

3.1.1 Mapping to Manifolds

Score-based generative modeling in latent space

Diffusion priors in variational autoencoders

3.1.2 Diffusion on Manifolds

Pseudo Numerical Methods for Diffusion Models on Manifolds

Riemannian Score-Based Generative Modeling

3.2. Data with Invariant Structures

GeoDiff: A Geometric Diffusion Model for Molecular Conformation Generation

Permutation invariant graph generation via score-based generative modeling

Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations

3.3 Discrete Data

Vector quantized diffusion model for text-to-image synthesis

Structured Denoising Diffusion Models in Discrete State-Spaces

Vector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation

Deep Unsupervised Learning using Nonequilibrium Thermodynamics.

Application Taxonomy

1. Computer Vision

• Image Super Resolution, Inpainting and Translation

  • SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models

  • Image Super-Resolution via Iterative Refinement

  • High-Resolution Image Synthesis with Latent Diffusion Models

  • Repaint: Inpainting using denoising diffusion probabilistic models.

  • Palette: Image-to-image diffusion models.

  • Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models

  • Cascaded Diffusion Models for High Fidelity Image Generation.

  • Conditional image generation with score-based diffusion models

  • Unsupervised Medical Image Translation with Adversarial Diffusion Models

  • Score-based diffusion models for accelerated MRI

  • Solving Inverse Problems in Medical Imaging with Score-Based Generative Models

  • MR Image Denoising and Super-Resolution Using Regularized Reverse Diffusion

  • Sdedit: Guided image synthesis and editing with stochastic differential equations

• Semantic Segmentation
  • Label-Efficient Semantic Segmentation with Diffusion Models.
  • Decoder Denoising Pretraining for Semantic Segmentation.

• Video Generation
  • Flexible Diffusion Modeling of Long Videos
  • Video diffusion models
  • Diffusion probabilistic modeling for video generation
  • MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model.

• Point Cloud Completion and Generation
  • 3d shape generation and completion through point-voxel diffusion
  • Diffusion probabilistic models for 3d point cloud generation
  • A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion
  • Let us Build Bridges: Understanding and Extending Diffusion Generative Models.

• Anomaly Detection
  • AnoDDPM: Anomaly Detection With Denoising Diffusion Probabilistic Models Using Simplex Noise
  • Remote Sensing Change Detection (Segmentation) using Denoising Diffusion Probabilistic Models.

2. Natural Language Processing

• Structured denoising diffusion models in discrete state-spaces
• Diffusion-LM Improves Controllable Text Generation.
• Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning

3. Temporal Data Modeling

• Time Series Imputation
  • CSDI: Conditional score-based diffusion models for probabilistic time series imputation
  • Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models
  • Neural Markov Controlled SDE: Stochastic Optimization for Continuous-Time Data

• Time Series Forecasting
  • Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting
  • Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models

• Waveform Signal Processing
  • WaveGrad: Estimating Gradients for Waveform Generation.
  • DiffWave: A Versatile Diffusion Model for Audio Synthesis

4. Multi-Modal Learning

• Text-to-Image Generation
  • Blended diffusion for text-driven editing of natural images
  • Hierarchical Text-Conditional Image Generation with CLIP Latents
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  • GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
  • Vector quantized diffusion model for text-to-image synthesis.
  • Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

• Text-to-Audio Generation
  • Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
  • Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data
  • Diffsound: Discrete Diffusion Model for Text-to-sound Generation
  • ItôTTS and ItôWave: Linear Stochastic Differential Equation Is All You Need For Audio Generation
  • Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models
  • EdiTTS: Score-based Editing for Controllable Text-to-Speech.
  • ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech.

5. Robust Learning

• Diffusion Models for Adversarial Purification
• Adversarial purification with score-based generative models
• Threat Model-Agnostic Adversarial Defense using Diffusion Models
• Guided Diffusion Model for Adversarial Purification
• Guided Diffusion Model for Adversarial Purification from Random Noise
• PointDP: Diffusion-driven Purification against Adversarial Attacks on 3D Point Cloud Recognition.

6. Molecular Graph Modeling

• Torsional Diffusion for Molecular Conformer Generation.
• Equivariant Diffusion for Molecule Generation in 3D
• Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models
• GeoDiff: A Geometric Diffusion Model for Molecular Conformation Generation
• Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem
• Diffusion-based Molecule Generation with Informative Prior Bridge
• Learning gradient fields for molecular conformation generation
• Predicting molecular conformation via dynamic graph score matching.

7. Material Design

• Crystal Diffusion Variational Autoencoder for Periodic Material Generation
• Antigen-specific antibody design and optimization with diffusion-based generative models

8. Inverse Problem Solving (Medical Imaging)

• Solving Inverse Problems in Medical Imaging with Score-Based Generative Models
• MR Image Denoising and Super-Resolution Using Regularized Reverse Diffusion
• Score-based diffusion models for accelerated MRI

Connections with Other Generative Models

1. Variational Autoencoder

• Understanding Diffusion Models: A Unified Perspective
• A variational perspective on diffusion-based generative models and score matching
• Score-based generative modeling in latent space

2. Generative Adversarial Network

• Diffusion-GAN: Training GANs with Diffusion.
• Tackling the generative learning trilemma with denoising diffusion gans

3. Normalizing Flow

• Diffusion Normalizing Flow
• Interpreting diffusion score matching using normalizing flow

4. Autoregressive Models

• Autoregressive Diffusion Models.
• Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting.

5. Energy-Based Models

• Learning Energy-Based Models by Diffusion Recovery Likelihood
• Latent Diffusion Energy-Based Model for Interpretable Text Modeling

Citing

If you find this work useful, please cite our paper:

@article{Yang2022DiffusionMA,
  title={Diffusion Models: A Comprehensive Survey of Methods and Applications},
  author={Yang, Ling and Zhang, Zhilong and Hong, Shenda},
  journal={arXiv preprint arXiv:2209.00796},
  year={2022}
}

or

@article{Yang2022DiffusionMA,
  title={Diffusion Models: A Comprehensive Survey of Methods and Applications},
  author={Ling Yang and Zhilong Zhang and Yang Song and Shenda Hong and Runsheng Xu and Yue Zhao and Yingxia Shao and Wentao Zhang and Bin Cui and Ming-Hsuan Yang},
  journal={arXiv preprint arXiv:2209.00796},
  year={2022}
}