Display Color Spaces and Formats

[bjornbytes]

Can be handled separately for headset and window

Window

• Enable VK_EXT_swapchain_colorspace
• Add t.graphics.colorspace (or t.graphics.hdr?) to set it, but probably also need a way to query supported colorspaces and apply one explicitly post-conf (not hard), for people that want precise control
• lovr.graphics.getColorSpace probably
• Do shaders need to apply the transfer function?

Headset

• Enable XR_FB_color_space
• Can ask for supported color spaces (lovr.headset.getSupportedColorSpaces?)
• Can get current/native color space (lovr.headset.getColorSpace)
• Can set the color space (lovr.headset.setColorSpace)
• It seems like Oculus does the conversion during compositing and LÖVR wouldn't need to change shaders?

I still don't understand color spaces well enough

[bjornbytes]

This will probably involve some behavior in shaders to map the output to the target color space.

[bjornbytes]

If HDR is active, just try to select rgb10a2 backbuffer w/ rec2020 colorspace (and optionally hdr10 pq transfer function)?

[shakesoda]

rgb10a2 with rec2020 will get you wcg (useful!) but not hdr, for hdr you need rec2100 pq (or hlg, but i don't think hlg is relevant for anything but debugging in lovr). 2100's the one that should be opted into, 2020's just gonna be exposing the expected display behavior though (assuming lovr changes to do the appropriate srgb color conversion, which i think it doesn't already do)

[bjornbytes]

Rephrasing holo's plan from chat to ensure I understand, with minor modifications:

• You can request a color space for the desktop window in conf.lua (t.graphics.colorspace(s)).
  • This color space is really a smoosh of color space + EOTF, like in Vulkan.
  • At least srgb and hdr10 (rec2020 primaries with PQ encoding, plus an rgb10a2 backbuffer) are supported.
    • Is linear rec2020 useful?
    • Ignore P3/HLG stuff for now.
• You can check the window's color space with lovr.graphics.getColorSpace. No setter right now because lovr can't recreate swapchains yet.
• Assume fragment shader working space remains linear sRGB, because everyone's used to that.
  • Similarly, background color, setColor, etc. all continue to take sRGB colors (converted to linear sRGB).
• Shaders have flags for applying color space conversion, tonemapping (ACES), and display encoding (inverse EOTF), in that order. You can return linear radiance from lovrmain and have the builtin flags do everything for you, or disable them and do it yourself.
  • Not 100% sure on the specifics -- how they're split up, which are enabled by default, etc.
  • Pass objects for the window and headset set shader globals for the 3x3 color space conversion matrix and a flag for whether PQ encoding is required, based on the current colorspace of the display.
• lovr.math has helpers for conversions, similar to existing sRGB ones.

[bjornbytes]

Another slightly more simplified/concrete stab at this:

• Add an "HDR mode". It's a simplified boolean that indicates whether the backbuffer is using HDR10 (technically PQ10). When active:
  • Backbuffer format is rgb10a2.
  • Backbuffer colorspace is rec2020.
  • Backbuffer transfer function is PQ.
• It has to be opt-in. You do t.graphics.hdr = true to request it.
• lovr.graphics.isHDR tells you if HDR mode is active (requested && supported).
• Shaders rendering to an HDR backbuffer will perform color space conversion and apply transfer function encoding.
  • Color space conversion is srgb aka rec709 -> rec2020. Display encoding is linear -> PQ.
  • There's a shader flag to disable both of those.
  • I think tonemapping has to be sandwiched between the color conversion and the PQ encoding, which is fine.
  • The functions used would be exposed as helper functions.
• If you're rendering to an intermediate canvas that you're treating as HDR, you're on the hook to do the conversions yourself for now. There's no like "mark a texture as HDR" flag yet, but maybe that would be useful in the future.
• Add lovr.math.srgbToHDR and lovr.math.hdrToSRGB except maybe renamed to avoid camel case acronym hell.
  • (background color / setColor still use srgb, but these functions could be used to convert WCG-specified colors).
• This is only done for the window right now. OpenXR hasn't figured out its color space story yet.
• No HDR metadata for displays (brightness levels etc.) because I haven't wrapped my brain around that stuff yet.

[bjornbytes]

It seems like HDR metadata is required for this stuff. It mostly consists of metadata about the color space (primaries, white point), plus some data about the range of light levels in the scene. Vulkan has an extension where you can send the metadata to the display, but apparently it's flawed because drivers/displays don't always use it.

A seemingly better approach is to have Vulkan tell you some metadata about the display (basically the brightness range it supports), and then have LÖVR incorporate that into tonemapping. However, Vulkan can't do that yet. DXGI can though.

This is so gnarly!