GTA-HDR: High Dynamic Range Content Creation and Qualtiy Assessment using Deep Learning [WACV 2025]

**This Graphics is retrieved from Google Search**

This page contains all the Datasets and Code bases (experiments and evaluations) involved in experimenting and establishing our newly proposed HDR → LDR dataset (GTA-HDR) for Deep Learning models. In this work, we propose GTA-HDR, a large-scale synthetic dataset for HDR image reconstruction, sampled from the photo-realistic (i.e., HDR-10, link enabled game Grand Theft Auto V (GTA-V) by Rockstar Games.

The official repository of the paper with supplementary: GTA-HDR!!

About the project

This project is a collaboration between Monash University, Malaysia campus and Human-Centered AI Lab in the Faculty of Information Technology, Monash University, Melbourne (Clayton), Australia.

Project Members -

Hrishav Bakul Barua (Monash University and TCS Research, Kolkata, India),
Kalin Stefanov (Monash University, Melbourne, Australia),
KokSheik Wong (Monash University, Malaysia),
Abhinav Dhall (Indian Institute of Technology (IIT) Ropar, India and Flinders University, Adelaide, Australia) , and
Ganesh Krishnasami (Monash University, Malaysia).

Thw work is accepted in IEEE/CVF WACV 2025 !!

Funding details

This work is supported by the prestigious Global Excellence and Mobility Scholarship (GEMS), Monash University. This research is also supported, in part, by the E-Science fund under the project: Innovative High Dynamic Range Imaging - From Information Hiding to Its Applications (Grant No. 01-02-10-SF0327).

Overview

Given that the performance of any data-driven learning- based method for HDR image reconstruction largely depends on the size and diversity of the datasets used for development, there is a significant gap in the publicly available datasets required to advance this research direction. Moreover, currently, there are no available datasets that adequately address the problem of no- reference HDR image quality assessment (other link1, link2), which demands vast collections of ground truth HDR (HDR_GT) and distorted HDR (HDR_Dis) pairs. In summary, there is a substantial research gap pertaining to benchmark datasets needed to advance the research on HDR image reconstruction, motivating the creation of an appropriate large-scale dataset.

Overview of Tone-mapping and Inverse tome-mapping (using Deep Learning)

Tone mapping is the process of mapping the colors of HDR images capturing real-world scenes with a wide range of illumination levels to LDR images appropriate for standard displays with limited dynamic range. Inverse tone mapping is the reverse process accomplished with either traditional (non-learning) methods or data-driven learning-based approaches.

The proposed GTA-HDR dataset addresses the identified gaps in the current publicly available datasets for HDR image reconstruction, including:

a) GTA-HDR is a large-scale (i.e., 40K ground truth HDR images) synthetic dataset sampled from the GTA-V video game data which utilizes ray tracing technology that simulates the physics behind light and shadows.

b) GTA-HDR includes HDR images with sufficient resolution (i.e., 512 × 512 and 1024 × 1024).

c) GTA-HDR includes HDR images capturing a diverse set of scenes including, different locations (e.g., indoor, urban, rural, in-the-wild), different lighting conditions (e.g., morning, midday, evening, night), and different weather and season conditions (e.g., summer, winter, snowy, rainy, sunny, cloudy).

NOTE: The official GTA-HDR Benchmark Dataset is releasing soon!!

Data collection pipeline

Steps	Details/Link
1(a). GTA-V has built-in HDR-10 support for displaying video sequences on HDR displays	The details can be found in Link1, Link2
1(b). We used Script Hook V plugin to capture HDR frames from the GTA-V game-play sequences	The details can be found in Link1, Link2
1(c). Our pipeline is inspired by the code base adopted from RenderDoc for Game data (IEEE TIP 2021)	The details can be found in Link1, Link2
2. We removed frames that are similar to the previous or next frames in the sequence to avoid unnecessary increase in dataset size	We find the Frame Similarity using Chamfer Similarity Metric: Link
3(a). We performed transformations on the original LDR images to generate multiple exposure LDR images (i.e., exposure values EV 0, ±1, ±2, ±3, and ±4) and different contrast levels	For exposure: Link and contrast: Link
3(b). We also generated distorted HDR images by randomly utilizing state-of-the-art methods	Some of the methods: M1, M2, M3

We stored the final HDR and LDR images in ".hdr" and ".png" formats, respectively

NOTE: Kindly fillup the agreement form to get access to the novel GTA-HDR Dataset used in this work.

Please fillup the Dataset Agreement Form and mail to Hrishav Bakul Barua ([email protected]) to get access.

Dataset characteristics

Scene diversity

GTA-HDR dataset scene diversity. Samples from the GTA-HDR dataset with multiple variations in location, weather, objects and time. The scene diversity ensures a thorough coverage of pixel colors, brightness, and luminance.

Image diversity

GTA-HDR dataset image diversity. For any image-to-image translation dataset, it is important to include a sufficient amount of samples with diverse range of color hues, saturation, exposure, and contrast levels. The final set of images in the dataset amounts to a total of 40K × 25 = 1M LDR, 40K HDR, and 40K distorted HDR images.

Our work utilizes the following:

State-of-the-art learning models for HDR reconstruction

ACM TOG 2017 | HDRCNN - HDR image reconstruction from a single exposure using deep CNNs | Code

ACM TOG 2017 | DrTMO - Deep Reverse Tone Mapping | Code

GlobalSIP 2019 | FHDR - HDR Image Reconstruction from a Single LDR Image using Feedback Network | Code

CVPR 2020 | SingleHDR - Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline | Code

IEEE TIP 2021 | HDR-GAN - HDR Image Reconstruction from Multi-Exposed LDR Images with Large Motions | Code

WACV 2023 | SingleHDR - Single-Image HDR Reconstruction by Multi-Exposure Generation | Code

APSIPA 2023 | ArtHDR-Net - Perceptually Realistic and Accurate HDR Content Creation | Code

ICIP 2024 | HistoHDR-Net - Histogram Equalization for Single LDR to HDR Image Translation| Code

LDR-HDR datasets

VPQM 2015 | HDR-Eye - Visual attention in LDR and HDR images | Dataset

ICCV 2017 | City Scene, site1 - Learning high dynamic range from outdoor panoramas | Dataset

City Scene, site2- The Laval HDR sky database | Dataset

ACM TOG 2017 | Kalantari et al. - Deep high dynamic range imaging of dynamic scenes | Dataset

ACM TOG 2017 | Endo et al. - Deep reverse tone mapping | Dataset

ACM TOG 2017 | Eilertsen et al. - HDR image reconstruction from a single exposure using deep CNNs | Dataset

IEEE Access 2018 | Lee et al. - Deep chain hdri: Reconstructing a high dynamic range image from a single low dynamic range image | Dataset

IEEE TIP 2018 | Cai et al. - Learning a deep single image contrast enhancer from multi-exposure images | Dataset

IEEE ICCP 2019 | Prabhakar et al. - A fast, scalable, and reliable deghosting method for extreme exposure fusion | Dataset

IEEE Access 2020 | LDR-HDR Pair - Dynamic range expansion using cumulative histogram learning for high dynamic range image generation | Dataset

CVPR 2020 | HDR-Synth & HDR-Real - Single-image HDR reconstruction by learning to reverse the camera pipeline | Dataset

SIGGRAPH 2022 | SI-HDR - Comparison of single image hdr reconstruction methods — the caveats of quality assessment | Dataset

Common link to some commonly used datasets - RAISE (ACM MMSys'15), HDR-Synth, HDR-Real, and HDR-Eye

Another huge dataset by Deep-SR-ITM, Kim et al. (ICCV 2019) (not used in our evaluation due to reasons stated in the paper)

Dataset: Sen et al. (ACM TOG 2012) (not used in our evaluation due to reasons stated in the paper)

Dataset: Trusun et al. (Wiley CGF 2016) (not used in our evaluation due to reasons stated in the paper)

Comprehensive surveys

IEEE TPAMI 2022 | Deep Learning for HDR Imaging: State-of-the-Art and Future Trends | Link

Elsevier DSP 2023 | High Dynamic Range Image Tone Mapping: Literature review and performance benchmark | Link

Dataset summary table

Dataset	Type	#HDRGT	Resolution	In-the-wild	HDRDis	Scene diversity	Image diversity
HDR-Eye (2015)	Synthetic	46	512✗512	❌	❌	❌	❌
City Scene (2017)	Mixed	41222	128✗64	❌	❌	✅	❌
Kalantari et al. (2017)	Real	89	1500✗1000	❌	❌	❌	❌
Endo et al. (2017)	Synthetic	1043	512✗512	❌	❌	❌	❌
Eilertsen et al. (2017)	Synthetic	96	1024✗768	❌	❌	❌	❌
Lee et al. (2018)	Synthetic	96	512✗512	❌	❌	❌	❌
Cai et al. (2018)	Synthetic	4413	3072✗1620	❌	❌	❌	❌
Prabhakar et al. (2019)	Real	582	1200✗900	❌	❌	❌	❌
LDR-HDR Pair (2020)	Real	176	1024✗1024	❌	❌	❌	❌
HDR-Synth & HDR-Real (2020)	Mixed	20537	512✗512	❌	❌	❌	✅
SI-HDR (2022)	Real	181	1920✗1280	❌	❌	✅	❌
GTA-HDR (ours) (2024)	Synthetic	40000	512✗512	✅	✅	✅	✅

Experiments and Results

Impact of the GTA-HDR dataset on the performance of the state-of-the-art in HDR image reconstruction. Impact of the GTA-HDR dataset on the performance of the state-of-the-art in HDR image reconstruction. R: Real data combines the datasets proposed in Kalantari et al. (2017), Prabhakar et al. (2019), LDR-HDR Pair (2020) and the real images from the datasets proposed in City Scene (2017); R ⊕ S: This combination contains the mixed datasets (including both real and synthetic data) proposed in City Scene (2017) and the real datasets proposed in Kalantari et al. (2017), Prabhakar et al. (2019), LDR-HDR Pair (2020); GTA-HDR: Proposed synthetic dataset; E2E: End-to-end training; FT: Finetuning of the original pre-trained models. The performance of all methods is evaluated on a separate dataset proposed in HDR-Synth & HDR-Real (2020).

For more details and experimental results please check out the paper!!

HDR images reconstructed with and without GTA-HDR as part of the training dataset, along with the RGB histograms and KL-divergence values. Base: HDR images reconstructed with ArtHDR-Net trained without GTA-HDR data; Ours: HDR images reconstructed with ArtHDR-Net trained with GTA-HDR data; GT: Ground truth.

For more details and experimental results please check out the paper!!

A page for awsome Deep Learning based HDR Reconstruction models: Link1, Link2

A page for awsome Deep Learning based Underexposed Image Enhancement models: Link

Citation

If you find our work (i.e., the code, the theory/concept, or the dataset) useful for your research or development activities, please consider citing our work as follows:

@article{barua2024gta,
  title={GTA-HDR: A Large-Scale Synthetic Dataset for HDR Image Reconstruction},
  author={Barua, Hrishav Bakul and Stefanov, Kalin and Wong, KokSheik and Dhall, Abhinav and Krishnasamy, Ganesh},
  journal={arXiv preprint arXiv:2403.17837},
  year={2024}
}

Related works:

@inproceedings{barua2023arthdr,
  title={ArtHDR-Net: Perceptually Realistic and Accurate HDR Content Creation},
  author={Barua, Hrishav Bakul and Krishnasamy, Ganesh and Wong, KokSheik and Stefanov, Kalin and Dhall, Abhinav},
  booktitle={2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)},
  pages={806--812},
  year={2023},
  organization={IEEE}
}

@article{barua2024histohdr,
  title={HistoHDR-Net: Histogram Equalization for Single LDR to HDR Image Translation},
  author={Barua, Hrishav Bakul and Krishnasamy, Ganesh and Wong, KokSheik and Dhall, Abhinav and Stefanov, Kalin},
  journal={arXiv preprint arXiv:2402.06692},
  year={2024}
}

License and Copyright

----------------------------------------------------------------------------------------
Copyright 2024 | All the authors and contributors of this repository as mentioned above.
----------------------------------------------------------------------------------------

Please check the License Agreement.