Float16Array + HTML Canvas

[palemieux]

FYI. Below is input from the Color on the Web CG regarding Float16Array in the context of HTML Canvas. This input was prompted by w3c/ColorWeb-CG#87

-- W3C Color on the Web CG chairs
https://www.w3.org/community/colorweb/
[email protected]

====================

CG proposal re: Float16Array + HTML Canvas

The ColorWeb CG is seeking consensus with parties interested in
discussing floating-point backing stores for canvas rendering
contexts.

The CG spec proposal is:
https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md

The primary outstanding issue at present is how to represent values
which are read back from the canvas into typed arrays which can be
manipulated from ECMAScript.

The current spec proposal supports readbacks into both
Uint8ClampedArray and Float32Array via ImageDataColorType.

It has been suggested to also add support for readbacks into
Float16Array (w3c/ColorWeb-CG#87). This
would make the specification dependent on the Float16Array type
currently being defined by the ECMAScript committee (ISO/TC 39).

The CG supports the idea of adding support for Float16Array to
ECMAScript. However, the CG feels it is most appropriate to decouple
the ongoing improvements to ECMAScript from the work on floating-point
canvases. Users of the web platform are expressing the need for
floating-point canvas backing stores now; in Chromium, implementation
is underway in crbug.com/1230619. Moreover, readback and CPU-side
manipulation paths are not high-performance, so the additional memory
bandwidth of using Float32Array for this task will not critically
impact applications.

The ColorWeb CG therefore suggests:

1. Moving ahead with
   https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md in its
   current form, with unorm8 and float32 as options for readback.

2. In parallel, working on adding Float16Array to ECMAScript, and
   prototyping it in browsers.

3. Specifying a "float16" enum in ImageDataColorType once
   implementation experience has been achieved with Float16Array, .
   Applications can feature-detect its support in browsers by catching
   exceptions raised by getImageData or createImageData.

Background

For context, introducing a Float16Array into the Typed Array hierarchy
was discussed several years ago among members of TC39, as well as the
Khronos Group where Typed Arrays originated. At the time, the costs of
supporting this type natively in ECMAScript engines outweighed the
benefits for the following reasons:

1. For practical purposes, Float16Array could be sufficiently
   polyfilled in ECMAScript:

https://github.com/petamoriken/float16

Corner cases existed regarding denormalized values, NaNs, and
infinities - but applications could generally achieve their desired
results, with good performance on all ECMAScript engines, using
existing primitives.

2. Direct CPU support for half-float numbers did not seem to exist. At
   the time, C libraries which worked with this data type universally
   emulated it, rather than using compiler intrinsics on certain
   platforms.

3. A significant amount of work would be required in every ECMAScript
   engine to make this typed array type perform well.

4. A significant amount of work would be required to specify the
   behavior of this numeric type in ECMAScript.

The landscape has changed in recent years. Per
https://en.wikipedia.org/wiki/Half-precision_floating-point_format#Hardware_support,
ARM and x86 processors either support FP16 natively, or will in the
near future. FP16 formats are increasingly heavily used in neural
network evaluation and image processing, including on the web. The
ECMAScript editors may be able to save spec work by referencing IEEE
specifications (which, to be fair, already existed when Float16Array
was originally considered).

[annevk]

I'm worried that this removes the incentive to get the work done on Float16Array.

It also ends up cementing the wrong default:

If the CanvasImageData has a colorType of "float16", then the ImageData's colorType will be "float32"

We don't want float32 there in the future.

[annevk]

cc @whatwg/canvas

[palemieux]

@annevk Can you confirm that your concern is specifically with selecting Float32Array as the default readback type? If so, we can use Uint8ClampedArray as the default readback type instead.

We are also happy to support the Float16Array proposal in TC 39. Let us know how to best coordinate the effort/proceed.

In the meantime, are there objections to supporting Float32Array as one of the (non-default) readback types and adding Float16Array once the proposal makes its way through the TC 39 process?

[annevk]

What are the use cases for Float32Array? And would it not be weird if the canvas is float16 and you get uint8 data out even if Float16Array is a thing?

[palemieux]

The use case for Float32Array that is of concern to the Color CG is the manipulation of HDR pixels, for which UInt8 is not sufficient.

The idea is for getImageData() to continue returning UInt8 data by default, even if the contents of the Canvas is higher bit-depth. An application could request higher bit depths, e.g., Float16Array , by passing a configuration parameter to getImageData().

[annevk]

Right, but Float16Array meets that need as well and shouldn't be difficult to add to the platform. And would allow for more useful defaults as well.

[palemieux]

Please provide a concrete proposal and the steps needed to get there, and the CG can consider it.

[annevk]

1. The Color CG or one of its participants follows through on Please create Stage 1 float16 repo tc39/proposals#453 to get Float16Array in ECMAScript/JavaScript done.
2. We add Float16Array to Web IDL.
3. We incorporate it in this API (and others).

[palemieux]

@annevk Are you ready to be a proponent to the work in TC 39, and are ready to implement the feature?

[bakkot]

I've made a repo to track the proposal in TC39. I'm not very familiar with the space, so while I'd be willing to champion the proposal in TC39 it would be helpful if someone from ColorWebCG could lay out why it would be helpful to have this (and, specifically, why Uint16Array and Float32Array aren't sufficient). The best thing would be if you could open an issue on that repository with your use case, but if you respond here or point me to a summary I can try to copy that into an issue myself.

[palemieux]

@bakkot Thanks for getting this started. I am happy to coordinate input from the Color CG. In the meantime:

• Uint16Array is sufficient for high-dynamic range (HDR) pixels with non-linear color values, e.g. PQ and HLG pixels.
• Uint16Array is not sufficient for high-dynamic range (HDR) pixels with linear color values. Float16 or Float32 are needed in that case.

Do you think the explainer should include background on HDR and WCG imagery?

[bakkot]

A little bit of background on why Uint16Array is not sufficient would be very helpful, yes. Most people on the committee (myself included) don't have any particular expertise here. I'm happy to take it on faith, but if there's a short explanation you can give to a layperson that would be great.

I've signed myself up to present the proposal for stages 2 and 3 (essentially skipping a stage due to the fact that the design is entirely straightforward) at the March meeting, which begins on March 21, so that's the timeline. If it gets approved at that point - which is not entirely a sure thing, but I'm hopeful - it will be ready for browsers to begin developing and shipping implementations at that point.

[ccameron-chromium]

Perhaps we should split this proposal into two proposals: (1) that adds floating-point backing for CanvasRenderingContext2D and (2) that adds whatever formats are considered useful for ImageData.

As was mentioned earlier, I now consider it a bug that I proposed Float32Array as a default format for getImageData. We should not change the default type for that function.

[annevk]

@ccameron-chromium can you explain? getImageData() using the same backing as the 2D context it is invoked upon makes sense to me. (While looking for other impacted features I found #8917.)

[ccameron-chromium]

It's a trade-off about what the "best" default behavior is. What is "best" here is a bit subjective, and I can understand people disagreeing here. I'll outline my reasoning below. Note that this is just for the default behavior -- with ImageDataSettings, the developer is in full control of the format in which data is read back.

The guiding principle that leads me to "we should use Uint8ClampedArray" is as follows: Suppose someone has written naïve code that is unaware of the existence of different CanvasRenderingContext2D colorTypes. What should happen to them? If someone writes code assuming Uint8ClampedArray, and they unexpectedly get a Float32Array or Float16Array instead, the result will be catastrophic. E.g, the developer blindly writes 255 for SDR white and instead gets a color value that is 524,947 times brighter than SDR white. E.g, the developer blindly reads the colors and gets values in 0..1, and treats them all as being black.

The guiding principle that originally lead me to "we should use Float32Array" would be that a naive getImageData/putImageData pair that has no pixel manipulation in between should not lose precision. The more I think about this, the more I think that it does not represent a realistic use case, because of the "no pixel manipulation in between" part.

I also see an analogous behavior in toDataURL. If one tries to round-trip data using that, they will have values clamped to 0..1 in the canvas' color space.

WRT #8917, I think there is a misunderstanding there with respect to the color space of the backing bitmap of a CanvasRenderingContext2D versus the parameterization of a CSS color. I put some details on that bug. Feel free to reach out if I misunderstood the question myself.

[annevk]

Presumably if you opt-in to a float16 context you do so to be able to do float16-based manipulation. Having to opt-in again for each getImageData() call seems cumbersome.

[Kaiido]

On the other hand I wonder if it's backward compatible to return something new here. There are cases where you want to grab the ImageData out of a context that you didn't initialize, e.g. to extend from a library.
Having a variable format for the default call to getImageData might break any script that were expecting an Uint8 array there when the library is updated to use HDR.

[kenrussell]

Float16Array meets that need as well and shouldn't be difficult to add to the platform

Adding Float16Array to the web platform is far from trivial. Every ECMAScript engine must be updated to support it, and it must be implemented as efficiently as all of the other Typed Array types from the start, because web authors already have expectations of high performance from the other typed array variants.

In each engine this involves revising an interpreter and one or more tiers of compilers to understand the float16 data type, register allocate it, define efficient conversions to and from other ECMAScript numeric representations, perform good per-CPU-architecture instruction selection to keep these values in the float16 representation as long as possible (again, for efficiency and performance), tightly specify and test these conversions, and finally ship them. Having participated in V8's implementation of Typed Arrays I would estimate this as a minimum of six months of dedicated work, and likely more.

In my opinion as a ColorWeb CG member, it's not acceptable for the WHATWG to require Float16Array as a hard dependency for the specification changes the ColorWeb CG is incubating and proposing. I respectfully request that the WHATWG members evaluating this issue agree with the ColorWeb CG's evaluation, and allow the specs to move forward using Float32Array as an ECMAScript-exposed representation of the canvas's float16 backing store.

If this can be agreed upon, then even if it were later changed to Float16Array (without requiring an application-level opt-in), that would likely not be a breaking change. The values read back from the canvas would be exactly the same in both representations - 0 bits are added to the mantissa and exponent when expanding float16 to float32. float32 adds more precision and range, but both would already be silently dropped / truncated when writing values to the canvas. If the application must explicitly choose the representation, then there is no risk of breakage.

[domenic]

In my opinion as a ColorWeb CG member, it's not acceptable for the WHATWG to require Float16Array

Let's be clear here that in no part of this thread is "the WHATWG" "requiring" anything. We have individuals expressing opinions.

Implicitly, those opinions often represent "what the browser engine that X individual represents would require, in order to implement the proposal", but I haven't seen anyone be explicit about speaking for any given company or browser project so even that is not clear yet.

But there's no reason to treat comments on an issue thread as some sort of organizational position. That's never really the case in the WHATWG. We specify what is implemented and shipped in browsers. If you get 2+ browsers to ship Float32Array-based or Float16Array-based or Float1337Array-based canvases, that's what will make it into the spec, per our working mode.

[othermaciej]

Setting aside differences in amount of spec work and implementation, is it a better design if we have Float16Array and can build on it? It seems to me it is. From what I see here, the main arguments against are the expediency of being able to ship somewhat faster, by avoiding the ECMAScript standards process and JS engine implementation work. If that was a very large difference, like multiple years, then maybe cutting corners in the design would be warranted. On the other hand, the future is big, and web APIs tend to last a long time, so if the delta relatively small (like less than a year), then it's probably better to do the extra work.

The above is a personal opinion, now, with more of an Apple hat on:

• The spec work for Float16Array seems relatively straightforward to us, we're looking into whether any Apple folks who have TC-39 experience can help.
• A senior engineer on JavaScriptCore told me:

Implementing fully optimized Float16Array (on all JIT tiers) takes from 2 weeks to 1 month. I think 6 months was reasonable for typed arrays because it is bringing up all typed array infrastructure from scratch. Adding one Float16Array does not require this level of thing.

That's much less than the 6 month estimate above, and tentatively seems more credible to me since it comes from an expert on the relevant VM internals.

• We like float16 color and in general improved color options in the platform, and would like to see some form of this proposal succeed.

[bakkot]

The spec work for Float16Array seems relatively straightforward to us, we're looking into whether any Apple folks who have TC-39 experience can help.

The spec work to add it to JS is entirely trivial; I have already done it as part of putting together the proposal linked above.

[ccameron-chromium]

I want to get back to the discussion of "what is the best default format for getImageData"?

I still think that changing that to match the canvas is a bad idea, for the reasons that Kaiido articulated above.

• Changing the type that getImageData returns is extremely likely to break (like, render nonfunctional) libraries. Any library that exists to day that is passed a canvas for any reason is broken if it gets passed a higher precision canvas.
• Also, if we continue on the path of adding more canvas types, it gets even messier. I can see very strong arguments for supporting 2d canvases with unorm16 and 10-10-10-2 backing. What should their defaults for getImageData be?

To be clear, I'm a huge fan of Float16Array, and I consider it to be strictly better than Float32Array for use with ImageData. But the only place that it touches this spec is the default ImageDataSettings for getImageData, and I don't think it should touch that part of the spec.

[othermaciej]

Changing the type that getImageData returns is extremely likely to break (like, render nonfunctional) libraries. Any library that exists to day that is passed a canvas for any reason is broken if it gets passed a higher precision canvas.

Some questions that might highlight how significant this risk is:

• Is it actually common for general-purpose canvas libraries to use getImageData / putImageData?
• To the extent that general-purpose canvas libraries use getImageData / putImageData, is it likely that such functionality would actually give correct behavior blindly operating on uint8 data when the underlying backing store is float16?

I’d expect the answer to both questions to be “no”; even one of them being “no” would, I think, reduce the force of the quoted argument. The first I’d expect to be somewhat uncommon because canvas readback is slow. The second because it seems this would break HDR or other intended color effects over a read-write-modify cycle. Perhaps it’s common to do pure reads and do math on them without writing back; and it’s ok if that math uses the wrong colors, but it would totally explode if given float16 values instead of uint8 values. But that doesn’t seem all that likely.

Still, if there actually is substantial compatibility risk to changing the default format at all, then perhaps the existing method should be forever uint8 and there should be a new one that gets the canvas’s most preferred representation. Leaving it to the library or the developer to figure out the best format for the backing store seems like a stumbling block, especially if the mapping is going to be non-obvious for future formats.

[Kaiido]

Is it actually common for general-purpose canvas libraries to use getImageData / putImageData?

Sure, given that as of today Safari still doesn't support ctx.filter using getImageData() / putImageData() is the only way to have some kind of cross-browser CSS filters. All "polyfills" out there are doing getImageData() / putImageData(). Many non-standard filters are also done this way. Image analysis like the ones provided by tensorflow also use getImageData(). Some very bad code are still doing collision detection by checking the pixel values. Flood fill algos are using getImageData()/ putImageData(). All color histogram scripts from more than 2 years ago are using getImageData(). All these (and probbaly many more examples) are expecting Uint8, and passing anything else will simply break them because all of the sudden they'll see just almost transparent black images when they receive a Float16 instead.

is it likely that such functionality would actually give correct behavior blindly operating on uint8 data when the underlying backing store is float16?

That sounds like a moot point. Losing HDR precision when applying a filter or doing image analysis isn't as bad as working on an almost transparent black image. In one case, at worst you lose some "cool feature", in the other, you have a broken product. Also, remember that HDR canvas is still relatively new and still not that widespread.

perhaps the existing method should be forever uint8 and there should be a new one that gets the canvas’s most preferred representation.

That's what is being proposed already: getImageData(x, y, width, height, settings)'s settings would provide a way to define the output format. All that's being discussed here is the default format.

[syg]

The spec work to add it to JS is entirely trivial; I have already done it as part of putting together the proposal linked above.

It should be very explicitly noted that the triviality of the spec changes here is not at all indicative of the implementation challenges. And each browser JS engine may have fairly different tradeoffs to consider around what architectures to support, because half-precision support on CPUs is limited and relatively new.

Implementing fully optimized Float16Array (on all JIT tiers) takes from 2 weeks to 1 month. I think 6 months was reasonable for typed arrays because it is bringing up all typed array infrastructure from scratch. Adding one Float16Array does not require this level of thing.

V8's estimate is more conservative: we think a quarter at least (exclusive of standardization time). Our supported hardware run a wider gamut and the software emulation that may be required play into the longer estimate.

[palemieux]

@bakkot Quick update. I am working on a simplified explainer re: HDR in HTML Canvas. In the meantime, some background information on why >8 bits per color channel is needed for HDR imagery.

As detailed, for example, at Ultra HD Blu-ray Format Video Characteristics , 8-bit quantization (bit depth) results in contouring and banding, even for traditional standard dynamic range (SDR) imagery, like sRGB, which covers a typical luminance range between 0 and 100 cd/m2. These quantization artifacts become unacceptable with High-Dynamic Range (HDR) imagery, supports luminance ranges between 0 and up to 10,000 cd/m2.

The table below illustrates how larger quantization step size results in visible step sizes, i.e. the gap between successive quantization levels exceeds the threshold of visibility for the human visual system (Barten threshold). The green entries are less than or equal to the Barten threshold, the yellow entries are 2.5 times the Barten threshold, and the red entries are five times the Barten threshold.

In summary, at very least 10 bits per color channels are needed for HDR imagery.

@michaeldsmith can provide more information.

[bakkot]

Update: the proposal to add Float16Array to JavaScript today reached stage 3, meaning the design is finished, the committee is in favor, and engines can start implementing and shipping it.

[litherum]

Ideally, while we're here, we should add support for 10-10-10-2 canvas backing store formats (either RGBA or BGRA) so that authors can accomplish wide color without having to use fp16 and doubling the size of their back buffer. We don't have opinions about what the data type returned by getImageData() for such backing stores.

[ccameron-chromium]

Ideally, while we're here, we should add support for 10-10-10-2 canvas backing store formats (either RGBA or BGRA) so that authors can accomplish wide color without having to use fp16 and doubling the size of their back buffer. We don't have opinions about what the data type returned by getImageData() for such backing stores.

That sounds like a great idea. In fact, it might be a better first step to land that (compared with float, which has many more things attached to it). What are your thoughts on the idea of adding:

    enum CanvasPixelFormat {
    "rgba8unorm",
    "rgb10alpha2unorm",
  };
  partial dictionary CanvasRenderingContext2DSettings {
    CanvasPixelFormat pixelFormat = "rgba8unorm";
  };

If that sounds good on its own, I can put up a PR!

I can also add a parameter to ImageDataSettings that allows the user to request a higher precision, if that's something that would be an important complement (I would default to just adding an option for Uint16Array, but that isn't a position that I hold very strongly).

[annevk]

Could we get away with returning the same uint8 view and essentially require bitwise math? I'd like our switch to bigger views to essentially be a switch to f16.

[annevk]

For the high-precision backing buffer, we shouldn't forget about #5173 (comment) from @junov:

let buffer = new Uint8ClampedArray(w*h*4);  // Any type of ArrayBuffer or ArrayBufferView should work
canvas.width = w
canvas.height = h
let ctx = canvas.getContext('2d', {storage: buffer});  // Throws if buffer is too small.

That API shape still looks quite good and addresses a number of shortcomings with the current getImageData() API. (Though not entirely sure about accepting all view types.)

[ccameron-chromium]

I've updated the proposal such that the type returned by getImageData() does not change.

Example usages are:

  // Returns an ImageData with a Uint8ClampedArray
  ctx.getImageData(1, 2, 3, 4);

  // Returns an ImageData with a Uint8ClampedArray
  ctx.getImageData(1, 2, 3, 4, {pixelFormat:'rgba8unorm'});

  // Returns an ImageData with a Float32Array
  ctx.getImageData(1, 2, 3, 4, {colorSpace:'display-p3', pixelFormat:'rgba32float'}); 

I support the idea of subsequently adding ImageData support for rgba16unorm, rgba16float, and (probably) rgb10a2-packed (which would return use a Uint32Array).

Regarding alignment with #5173, it should be emphasized that getImageData() does not ever return the actual backing buffer data from the canvas (ImageData is always tightly packed and always un-premultiplied, whereas almost all canvas backings don't have these properties).

[annevk]

When we had the in-person meeting, my impression was that we only needed one of these APIs going forward, not both.

It certainly seems like a lot of complexity to overload ImageData in this way when it's not quite clear that will be the path most web developers will use going forward.

[ccameron-chromium]

When we had the in-person meeting, my impression was that we only needed one of these APIs going forward, not both.

Sorry for being confused, which two APIs are under consideration?

[annevk]

I thought that instead of getImageData() we were considering just exposing the backing buffer (and making the backing type of that Float16Array so it can handle the new pixel formats).

[ccameron-chromium]

I thought that instead of getImageData() we were considering just exposing the backing buffer (and making the backing type of that Float16Array so it can handle the new pixel formats).

Sorry, I must have misunderstood things during the meeting.

I don't think that it is feasible to directly expose the true backing buffer of a canvas.

The main reason for this is that the true format of a canvas's backing buffer doesn't line up with ImageData in several ways:

• ImageData is always in CPU memory, back buffers can be in GPU memory or CPU memory (and often not linearly addressable even when on a unified memory architecture)
• ImageData is tightly packed, but back buffers are often padded
• ImageData is necessarily in RGBA format, but back buffers often in different formats (BGRA, ARGB, etc), depending on the CPU or GPU architecture and backing library
• ImageData is unpremultiplied, but back buffers are almost-always premultiplied

The consequence is that if something is going to access the data from a canvas as an ImageData, that data will have gone through some sort of conversion and/or copy.

The getImageData and putImageData functions already support several conversions. The only big deficiency with getImageData is that it is synchronous -- I'd love for there to be a getImageDataAsync function.

[annevk]

Even if it's not a true backing buffer, it would be the new forced-asynchronous way to access pixel data. (Potentially into an existing buffer.) Which would also allow us to switch to Float16Array. I don't think we should be doing both.

[ccameron-chromium]

That hypothetical new function would still incur a copy-and-convert of the data, for the aforementioned reasons.

Are we proposing that we restrict the types of conversions that will be supported for that new function?

E.g getImageData can be used to

• retrieve just a subset of the image (via sx,sy,sw,sh). Is that still supported?
• convert the data to the caller's requested color space (via settings). Is that still supported?
• convert the data to the caller's requested format (via settings), in this proposal. Should not introduce that?

If we are to introduce any of these restrictions, what would be the motivation?

[annevk]

I think we only really need to restrict things that we were unhappy with, such as synchronous access. But it might be wise to start out small and only add parity for features when there is clear demand and need.

[ccameron-chromium]

Back in another turn of the wheel of reincarnation, desktop OpenGL supported type conversions. When the "lower level" APIs (starting with OpenGL ES) removed those abilities, it was a big pain point for many developers. Lots of people just wanted to always specify or read data in the format that the rest of their code wanted (often floats), and lamented that TexImage2D and ReadPixels suddenly didn't support conversion anymore.

Having been through that, I wouldn't want introduce restrictions of that sort.

Also, the penalty for removing conversions is higher now, as GPUs are now much faster relative to CPUs than was the case during that previous episode, and removing conversions forces the user to do the conversions themselves, on the CPU.

But I think we're getting a bit far afield. With respect to this proposal, I hope I've made a convincing case that we shouldn't break getImageData.

[annevk]

I don't think anyone is suggesting breaking getImageData though?

[ccameron-chromium]

In the most strict sense:

• don't just throw an exception for float-backed canvases (this was suggested at the F2F)

In a wider sense:

• don't introduce changes that will cause existing code to break (e.g, changing the default return type)
• don't restrict its ability to perform format conversions (e.g, reading an 8-bit buffer as float or vice versa)

[mdrejhon]

Example usages are:

  // Returns an ImageData with a Uint8ClampedArray
  ctx.getImageData(1, 2, 3, 4);

  // Returns an ImageData with a Uint8ClampedArray
  ctx.getImageData(1, 2, 3, 4, {pixelFormat:'rgba8unorm'});

  // Returns an ImageData with a Float32Array
  ctx.getImageData(1, 2, 3, 4, {colorSpace:'display-p3', pixelFormat:'rgba32float'}); 

I think that's a great compromise without breaking getImageData;

Long term, as creator of TestUFO I need direct access to WCG/HDR data at its full precision, and eventually access to tonemapping data, so I can at least determine native target lumens (before displays' clipping decisions), to scientifically quantify display measurements. Some tests are just demos, teaching animations, while others are used for display measurements. However, just getting access to WCG/HDR in a canvas, is a big first step.

As of September 1st, 2024, beta.testufo.com now does HDR Canvas 2D in Chrome via experimental flag. Test whether WCG/HDR is working at beta.testufo.com/hdr. I've added HDR color picker support where applicable among the other selectable tests at top.

So I'm already butting against limitations of getImageData() because I would love to render a HDR pallete in a canvas (e.g. beta.testufo.com/palette and simply getImageData() from it, for an improved WCG+HDR colorpicker for certain display tests (like HDR pixel response tests), some tests like /flicker are used with an oscilloscope.

[BlobTheKat]

#10624

[mdrejhon]

What are the use cases for Float32Array?

Here's My Industry Answer As Display Scientific Expert

While float16 is good enough, there are some use cases (10 year window) for float32 such as linearized HDR

Or doing large number of canvas operations (matching two images together repeatedly).

We necessarily have to go non-linear to avoid banding artifacts with float16, which is why PQ and Hybrid Log Gamma exists.

16-bit is not enough precision for linearized HDR over the entire dynamic range when doing adjacent shades in a cave versus adjacent shades in direct sun.

This was not important in 15" VGA CRT era

But important for gigantic curved 49" super-ultra wide (32:9) displays where some pixels are now outside your peripheral vision if you sit close enough.

Or different content in different head-facing directions in VR allowing your eyes to adapt to the new dynamic range subset.

Screens that exceed FOV now show banding at 16bit linearized in these extreme cases!

A screen optimized to one dynamic range shows different banding at a different dynamic range once your eyes adjust to the new brightness of the new direction you're facing on the giant-FOV screen.

The increase in FOV means now you have to account for human effect of exiting/entering cave vs outdoors, simply by a human facing different parts of the same giant screen. This amplifies banding.

Screens are getting bigger, you see? And now Apple Vision Pro and similar use cases of virtual screens. But it also applies to fiant-FOV screens like current Corsair Xeneon Flex 45" 240Hz OLED that I use to make some of the replies here.

2030s GPUs with float32 frame buffers may want native faster access to linearized HDR if math ops are more efficient in the linear space, especially for high-processing cases with fewer workarounds than using nonlinear color spaces. Linear color space means pixel values more linearly corresponds to photons/sec count, allowing easier maths.

Obviously, this isn't important to "I just wanna browse" use cases, but if we want native accsss to future GPU color spaces, permanent float16 assumption for 2030s = gun pointed to foot for architecture-self-into-corner.

Fine, (as long as we're not doing any image math/peocessing) we don't perceptually need more than float16 -- but only as long as we're not doing LINEAR float16 on a Vegas Sphere quality VR headset with little light back scatter (e.g. improved lens like post-pancake lens). Where you are in a cave entranceway and turn your headset back and forth between the dark and bright, giving enough time for eye adjustment to see different in cave shades of colors versus later shades of daylight. And that's without doing lots of image math without banding rounding errors. We need float32 for Holodeck displays where FOV amplifies quantization due to eye adjustment simply by facing different parts of FOV.

P.S. The Apple employees in the WebKit department needs to talk to the Vision Pro people, to self-educate on these sciences. By the way, I am cited in over 30 research papers, and I also teach display-sciences training classrooms [image]

Two scientific rasion d'être's exceeds more than enough.

We do not ever need to make float32 default, or even standardize today, but there should be a decade path to supporting it natively. Support float16 today, but keep paths to other GPU-native color channel data types in future.

Note: I work with display manufacturers. Its amazing how weak links shows in extreme tests.

Either way, float16 standardization should be expedited, as long as "does our float16 support path allow future support for GPU native color channel formats such as float32?" tests pass.

The @ccameron-chromium path passes that test.

[ccameron-chromium]

Capturing some offline discussion about this.

We've decided that adding getImageDataAsync is the better solution to the "what should getImageData do by default" question. We've separated that out into #10855. That API takes an already-created ImageData, and so there are no defaults to decide upon. As for getImageData, it's a bad API. We'll leave it functional (no throwing exceptions), but the bar for adding the ability to do things like specify the desired pixel format will be extremely high (and almost certainly never reached).

As for ImageData, it looks like Float16Array has made quite a bit of progress. I've filed #10856 against adding support for creating ImageData with different pixel formats.

Having carved off those two issues (which are independent, and can be developed in parallel), I think this issue can be focused on just "add float16 backing support to CanvasRenderingContext2D". This can probably be done by updating CanvasRenderingContext2DSettings to have a pixel format parameter.

We may want to include the (currently very limited) tone mapping parameters in WebGPU in this issue (since that, too, would live in CanvasRenderingContext2DSettings), or potentially carve that off into a separate issue.

Does that approach feel reasonable to all involved?

[ccameron-chromium]

I've simplified the proposal for 2D canvas floating-point support to this explainer PR.

I've also requested a TAG review on the limited spec (this is required to ship in Chromium).

I have a spec draft here, but I'll want to send that out only once TAG and ColorWeb-CG don't have any change requests.