Skip to content

Latest commit

 

History

History
237 lines (184 loc) · 18 KB

ISP.md

File metadata and controls

237 lines (184 loc) · 18 KB
Raw

Table of Contents

ISP

  • Rendering Nighttime Image Via Cascaded Color and Brightness Compensation
    Zhihao Li, Si Yi, Zhan Ma
    [CVPRW 2023] [Project] [Pytorch-Code]
    [CBUnet] 提供了一个夜景raw2rgb数据集

  • Transform your Smartphone into a DSLR Camera: Learning the ISP in the Wild
    Ardhendu Shekhar Tripathi, Martin Danelljan, Samarth Shukla, Radu Timofte, Luc Van Gool
    [ECCV 2022] [Code]

  • RAWtoBit: A Fully End-to-end Camera ISP Network
    Wooseok Jeong, Seung-Won Jung
    [ECCV 2022]

  • Day-to-Night Image Synthesis for Training Nighttime Neural ISPs
    Abhijith Punnappurath, Abdullah Abuolaim, Abdelrahman Abdelhamed, Alex Levinshtein, Michael S. Brown
    [CVPR 2022 Oral] [Pytorch-Code]

  • Abandoning the Bayer-Filter to See in the Dark
    Xingbo Dong, Wanyan Xu, Zhihui Miao, Lan Ma, Chao Zhang, Jiewen Yang, Zhe Jin, Andrew Beng Jin Teoh, Jiajun Shen
    [CVPR 2022] [Pytorch-Code]
    [★] 从bayer raw预测一个mono. 采集了一个bayer和mono配对的数据集.

  • Model-Based Image Signal Processors via Learnable Dictionaries
    Marcos V. Conde, Steven McDonagh, Matteo Maggioni, Aleš Leonardis, Eduardo Pérez-Pellitero
    [AAAI 2022 Oral] [Project]
    [Model ISP] [★★] 对ISP过程做了较详细的拆分, 每一步尽量做到可逆, end-to-end训练和推理

  • Mobile Computational Photography: A Tour
    Mauricio Delbracio, Damien Kelly, Michael S. Brown, Peyman Milanfar
    [arXiv 2102]
    [★★★] 很好的一篇ISP综述

  • Learning RAW-to-sRGB Mappings with Inaccurately Aligned Supervision
    Zhilu Zhang, Haolin Wang, Ming Liu, Ruohao Wang, Jiawei Zhang, Wangmeng Zuo
    [ICCV 2021] [Project]

  • ReconfigISP: Reconfigurable Camera Image Processing Pipeline
    Ke Yu, Zexian Li, Yue Peng, Chen Change Loy, Jinwei Gu
    [ICCV 2021] [Project]

  • End-to-end High Dynamic Range Camera Pipeline Optimization
    Yazhou Xing, Zian Qian, Qifeng Chen
    [CVPR 2021] [Project] [Pytorch-Code]
    [★] 将ISP参数与下游任务一同优化

  • Invertible Image Signal Processing
    Nicolas Robidoux, Luis E. Garcia Capel, Dong-eun Seo, Avinash Sharma, Federico Ariza, Felix Heide
    [CVPR 2021] [Project]

  • Neural Camera Simulators
    Hao Ouyang, Zifan Shi, Chenyang Lei, Ka Lung Law, Qifeng Chen
    [CVPR 2021] [Pytorch-Code]
    [★] 提出了一个controllable raw图像生成流程, 通过控制曝光时间, ISO, 噪声level和aperture, 控制生成图像的亮度, blur, 噪声等属性

  • Learning sRGB-to-Raw-RGB De-rendering with Content-Aware Metadata
    Seonghyeon Nam, Abhijith Punnappurath, Marcus A. Brubaker, Michael S. Brown
    [CVPR 2021]
    SRGB->RAW

  • CycleISP: Real Image Restoration via Improved Data Synthesis
    Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao
    [CVPR 2020 Oral] [Pytorch-Code]
    [★] 提出了一个从sRGB到RAW相互转换的网络, 在Raw图像上注入高斯噪声用于生成RGB噪声样本.

  • Replacing Mobile Camera ISP with a Single Deep Learning Model
    Andrey Ignatov, Luc Van Gool, Radu Timofte
    [CVPR 2020] [Code]
    [PyNet] [★] 1) 提出了一个端到端的深度学习网络, 用以代替现有的ISP处理流程. 2) 提出了一个华为P20 RAW 和Canon 5D的RAW-RGB图像对, 用以训练ISP模型. 3) 提出的算法与自带的ISP流程相比, 色彩上有一定提升, 但没有明显优势, 且存在晕影. 另外速度也是个问题. 因此对于用一个DL模型代替ISP流程的方案可行性还是有待确认.

  • Generating Training Data for Denoising Real RGB Images via Camera Pipeline Simulation
    Ronnachai Jaroensri, Camille Biscarrat, Miika Aittala, Frédo Durand
    [arXiv 1904] [Project] [Pytorch-Code]
    [★★] 提出了一个模拟isp流程, 使用多种demoasic, denoise, tone mapping算法, 从raw生成rgb, 用于去噪. 可以学习它的pipeline和算法.

  • DeepISP: Learning End-to-End Image Processing Pipeline
    Eli Schwartz, Raja Giryes, Alex M. Bronstein
    [TIP 2018]
    [☆] 大致浏览, 一个end-to-end的网络, 分为保持分辨率的low level部分和逐层下采样的high level部分. 使用了conv+relu, conv+tanh, 直连三个分支并行的设计, 比较少见

  • Learning to See in the Dark
    Chen Chen, Qifeng Chen, Jia Xu, Vladlen Koltun
    [CVPR 2018] [Project] [TF-Code]
    [★★] 1) 提出了SID数据集, 包括RGB和Raw数据. 2) 提出了一个end-to-end的isp网络, 以RAW和增益信息为输入, 输入RGB图像, 代替传统ISP流程

  • Reconfiguring the Imaging Pipeline for Computer Vision
    Mark Buckler, Suren Jayasuriya, Adrian Sampson
    [ICCV 2017] [Project] [Code1] [Code2]
    [★★] 1) 针对计算机视觉任务(而不是photography)简化ISP流程. 提出只有demosaicing和gamma校正是CV任务中必需的, 并据此设计了系统. 2) 系统包括3个步骤: reduced resolution readout, subsampling to produce RGB images, lower-precision logarithmic ADC configuration. 能将总体能耗降低约75% 3) 公布了一套用于模拟imaging pipeline及其逆过程的工具CRIP

  • Learning the image processing pipeline
    Haomiao Jiang, Qiyuan Tian, Joyce Farrell, Brian Wandell
    [TIP 2017]
    [L3] [★] 将RAW到sRGB的转换用局部相信滤波器完成, 类似于RAISR.

  • A Software Platform for Manipulating the Camera Imaging Pipeline
    Hakki Can Karaimer, Michael S. Brown
    [CVPR 2016] [Project]

AWB

Demosaicing

Deep Learning Methods

  • Make the Most Out of Your Net: Alternating Between Canonical and Hard Datasets for Improved Image Demosaicing
    Yuval Becker, Raz Z. Nossek, Tomer Peleg
    [arXiv 2303]
    [★] samsung以色列研究院的文章, 提出了一个训练demosaicing的策略: 1) 先用普通数据集训练网络, 根据指标挑出难样本; 2) 将难样本与全部数据交替训练. 在非线性域做的

  • Rethinking Learning-based Demosaicing, Denoising, and Super-Resolution Pipeline
    Guocheng Qian, Yuanhao Wang, Jinjin Gu, Chao Dong, Wolfgang Heidrich, Bernard Ghanem, Jimmy S. Ren
    [ICCP 2022] [Pytorch-Code]
    [TENet] [★★] 1. 使用具有pixel shift技术的相机收集了一可以做demoasic的数据集, 避免了用普通RGB数据做真值时内置demoasic过程带来的误差. 2. 提出了一端到端的demosaic, 去噪和超分的网络, 采用residual + dense block的形式, 没什么特别的

  • Searching for Fast Demosaicking Algorithms
    Karima Ma, Michael Gharbi, Andrew Adams, Shoaib Kamil, Tzu-Mao Li, Connelly Barnes, Jonathan Ragan-Kelley
    [TOG 2022]
    NAS搜索demosacing网络结构

  • End-to-End Learning for Joint Image Demosaicing, Denoising and Super-Resolution
    Wenzhu Xing, Karen Egiazarian
    [CVPR 2021] [Pytorch-Code]
    [★] 一个网络联合做demosaic, 去噪和超分, 退化很简单, 没在真实数据上实验.

  • A Compact High-Quality Image Demosaicking Neural Network for Edge-Computing Devices
    Shuyu Wang, Mingxin Zhao, Runjiang Dou, Shuangming Yu, Liyuan Liu, Nanjian Wu
    [Sensor 2021]
    [★]

  • HighEr-Resolution Network for Image Demosaicing and Enhancing
    Kangfu Mei, Juncheng Li, Jiajie Zhang, Haoyu Wu, Jie Li, Rui Huang
    [ICCVW 2019] [Pytorch-Code]
    [★]

  • Deep Joint Demosaicking and Denoising
    Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, Frédo Durand
    [SIGGRAPH Asia 2016] [Project] [Pytorch-Code]
    [★★] sRGB域用CNN端到端demosaicing, 重点是提出了难样本挖掘的策略, 用HDR-VDP2找到亮度上的artifact, 用频域上低频分量的增益找到moire artifact.

Traditional Methods

  • Multiscale gradients-based color filter array interpolation
    Ibrahim Pekkucuksen, Yucel Altunbasak
    [TIP 2012]
    [MSG] [★★] 使用色差梯度

  • Adaptive homogeneity-directed demosaicing algorithm
    K Hirakawa, TW Parks
    [TIP 2005]
    [AHD] [★★]

  • High-quality linear interpolation for demosaicing of Bayer-patterned color images
    HS Malvar, L He, R Cutler
    [ICASSP 2004]
    [★★] 微软. 使用其他颜色的梯度修正双线性结果, 修正系数用最小化MSE求解得到. 简单速度快, 但在边缘容易有格子artifact和伪彩

  • Adaptive color plane interpolation in single sensor color electronic camera
    James E. Adams, John F. Hamilton.
    [US Patent 1999] [★★★] Hamilton & Adams插值算法. 首先根据色差和水平垂直梯度方向先插值G, 再用插值好的G计算色差, 根据对角线方向梯度插值R和B

Misc

  • Improving Color Reproduction Accuracy on Cameras
    Thomas Eboli, Jean-Michel Morel, Gabriele Facciolo
    [ECCV 2022] [Project] [Pytorch-Code]

  • Optical aberrations Correction in Postprocessing using Imaging Simulation
    Shiqi Chen, Huajun Feng, Dexin Pan, Zhihai Xu, Qi Li, Yueting Chen
    [TOG 2021]

  • Removing chromatic aberration by digital image processing
    SW Chung, BK Kim, WJ Song
    [Optical Engineering 2010] [Unofficial-Cpp-Code]
    [★★] (传统算法) 首先归纳了正常无color fringe的边缘过渡区域的性质, 即: 过渡区域的色差值(R-G, B-G)在过渡区域边缘色差值的范围内; 接下来先根据G找到边缘p, 在p附近根据梯度相似原则找到过渡区域边界lp和rp; 最后根据前面归纳的边缘性质, 去除过渡区域的色差

Resources

Articles