How to measure PSNR of screenshot images correctly?

[JasonLSC]

Hi! I want to measure the quality of images rendered by instantNGP with DLSS.

I referred to the python script that you provided in the project and issues. For enabling DLSS, I create a window via testbed.init_window and set testbed.dlss = True. Then, I use testbed.set_nerf_camera_matrix to set target pose, and use testbed.frame() and testbed.screenshot() to get screenshot.

However, it seems that when DLSS is enabled, the PSNR of screenshot degrades a lot (33dB -> 31dB). Is this a normal phenomenon or is there a problem with my metric script?

[Tom94]

Hi, thank you very much for this report! I found (and fixed) a few issues, so I'd like to ask you to pull the latest commit and try again. DLSS still produces slightly worse PSNR than native full-res rendering, but less severe than before. On the lego dataset, I see ~34dB with DLSS and ~36dB without, so a slight decrease vs. native rendering is expected.

Information about what has been fixed is below:

• testbed.screenshot() was lagging behind by one frame, which might have also negatively affected non-DLSS PSNR numbers
• DLSS was broken in combination with testbed.snap_to_pixel_centers = True. I'm not sure whether you've used this option, but in any case: it should be False when doing super-resolution and True otherwise (when just wanting to do temporal accumulation).

The code I used to render a reference image + a DLSS'd image is:

resolution = testbed.nerf.training.dataset.metadata[i].resolution
testbed.render_ground_truth = True
testbed.set_camera_to_training_view(i)
ref_image = testbed.render(resolution[0], resolution[1], 1, True)
testbed.render_ground_truth = False
testbed.dynamic_res = False
testbed.fixed_res_factor = 8 * <downsampling factor, e.g. 2 for half resolution>
testbed.snap_to_pixel_centers = <False for superresolution, True otherwise>
testbed.nerf.render_min_transmittance = 1e-4
testbed.frame()
image = testbed.view()

where i is the test frame. A detail to keep in mind is that test frames are automatically sorted (in natural order) by their filename, so should have correctly ordered filenames for temporal accumulation to work well.

[Tom94]

That is very weird -- I suspect that something on the OpenGL side is going wrong (because that's where the screenshot pixels are coming from), so I quickly implemented a workaround that lets you directly read the contents of the render buffer without involving OpenGL.

Simply pull the latest commit and use testbed.view() instead of testbed.screenshot(), which will hopefully work. I am also updating the above code snippet to reflect the change.

[JasonLSC]

Thank you very much for such a quick response! Now, I can get reasonable rendered image via testbed.view().

I conducted some experiments to test the reconstructed image quality under three different rendering configurations. The first one is using testbed.render() to get full res. rendering without DLSS. The second one is enabling DLSS but just setting downsample rate to 1. The last one is enabling DLSS and setting downsample rate to 2.

I took 100 views in training split of Lego scene to train for 35000 iteration. Then, I evaluated the model under these 3 settings. I found out the PSNR on test split is close to 36dB under the first setting. However, the PSNR under the second setting is only 33.68dB and the PSNR under the last setting drop to 31.85dB.

Still a gap from the number you reported. I guess maybe I need to check some other rendering parameters, like nerf_compatibility ?

[Tom94]

I'd say 33.68dB is close enough to be attributable to random variation. I've gotten something slightly below 34 (don't remember the exact number) with nerf_compatibility enabled (when rendering at native resolution + DLSS enabled), but my runs have varied by +-0.2dB due to randomness in training. Thank you very much for being so diligent!

[JasonLSC]

Thank you for sharing information! So if I understand this correctly, 'rendering at native resolution + DLSS enabled' means testbed.fixed_res_factor = 8？In other word, filters from DLSS will still be applied to the volume rendered image in this setting, which leads to a minor decrease in PSNR number(36dB->33.68dB) but is not noticeable by human vision system.

The avg. PSNR of 'rendering at half resolution + DLSS enabled' from my experiment is only 31.85dB(min: 25.92dB max:33.94), which means super-resolution could lead to 3dB drop in PSNR compared to full res. volume rendering. Even though the number differs a lot, visual quality maintains at a high level. Is this a reasonable phenomenon on PSNR drop? (Still worry about my metric scripts for wrong results..)

[Tom94]

Yes, your understanding is correct! If you dump the images, you can actually notice the quality drop as a human. For example, specular highlights lag a bit behind the animation as DLSS is not able to deduce them purely from previous frames and motion vectors, so takes a bit of time to blend them in over multiple frames.

What you describe about half-resolution upscaling sounds very reasonable to me, also.