[css-color-4] Premultiplication in cylindrical spaces and mixing

[raphlinus]

The current spec describes premultiplication as not multiplying the hue component in cylindrical spaces. I think I understand the motivation of that for doing interpolation and gradients, but it does not seem to be the correct logic for doing mixing and compositing. I'm wondering what implementations should do, and specifically whether there need to be two forms of premultiplication, one for lerp, one for mixing.

The basic rule for x over y in premultiplied spaces is x + (1 - x.alpha) * y. But this breaks down for cylindrical spaces premultiplied as per the spec, as the sum of the weights on the hue components exceeds 1. (It's not a problem for lerping, as the sum of weights is always 1)

Another way of phrasing this is that the premultiplication method in the Color Level 4 draft mismatches the definition of premultiplication for color-mix in Color Level 5, which does not make an exception for hue.

I can think of several ways of dealing with this:

1. Simply forbid compositing in cylindrical spaces.
2. When compositing in a cylindrical space is requested, actually do it in the associated rectangular space, for example oklab when oklch is requested.
3. Define the over operation as (x.hue * x.alpha + y.hue * y.alpha * (1 - x.alpha)) / (x.alpha + y.alpha * (1 - x.alpha)) for the hue component.
4. Have PremulColor and LerpPremulColor as separate types, with the former premultiplying all components, and the latter holding out hue.

One reason that choices (1) and (2) are on this list is that I'm not sure how useful it is to do compositing in a cylindrical space. I'm happy to be pointed to evidence on this.

The difference in behavior seems subtle, and it's not obvious to me that the CSS specified behavior is clearly more better or more correct than the simpler, compositing-friendly behavior. I searched for discussion where this was decided and couldn't find it. I can generate color ramps to illustrate the difference if that would be useful.

For context, this came up when we were starting to contemplate a color_mix method in our new Rust color crate. My original hope is that the PremulColor type we defined for CSS Color Level 4 interpolation would also efficiently support mixing/compositing, but that is not looking hopeful at the moment.

[facelessuser]

It should be noted if you were to treat the cylindrical space in the rectangular space and apply the premultiplication, you'd still have the same hue, that's why it doesn't make sense to premultiply the hues. The hue doesn't change, just the colorfulness when it is mixed.

>>> color = Color('oklch', [0.5, 0.2, 85], 0.5)
>>> a, b = color.convert('oklab').get(['a', 'b'])
>>> a *= color.alpha()
>>> b *= color.alpha()
>>> Color('oklab', [color['lightness'] * color.alpha(), a, b]).convert('oklch')
color(--oklch 0.25 0.1 85 / 1)

I don't think compositing should really be done in a cylindrical space, but interpolating between two cylindrical colors, which is what color-mix is doing seems fine.

Another way of phrasing this is that the premultiplication method in the Color Level 4 draft mismatches the definition of premultiplication for color-mix in Color Level 5, which does not make an exception for hue.

If I had to guess, I think omitting the statement about not premultiplying hue is just an accident, not an explicit intention.

[facelessuser]

I do realize that, in this scenario, we are referring to this alpha blending as compositing. When I say compositing probably shouldn't be done in a cylindrical space, I mean the browser itself should not apply compositing in this way when rendering colors or overlaying images, etc. due to how hue interpolations work, but there is nothing wrong with interpolating in a cylindrical space if that is what you want to do.

[raphlinus]

Oops, I realize I made a mistake in formulating this question, as I believed that color-mix was capable of representing the Porter-Duff over operator, and, now, looking at it more closely, it seems that it is only capable of representing lerp but with an additional alpha scaling step.

So I think there are two separate concerns. One is the mismatch as pointed out, which I agree with @facelessuser is most likely a spec drafting issue. The second is whether the color representation is suitable for compositing, which is the main question I'm trying to raise right now. We do already have an interpolate method which does the color-mix functionality except for the scaling by 1 / (p1 + p2), and we have a separate mul_alpha which can perform that additional step.

To me, interpolation and compositing are closely related. In particular, I believe compositing color-a / alpha over opaque color-b should match color-mix(color-a alpha, color-b). In rectangular spaces, that is not controversial, but if we allow compositing in cylindrical spaces, it is.

So strike "another way of phrasing this," and the last paragraph should read: For context, this came up when we were starting to contemplate an over method in our new Rust color crate. My original hope is that the PremulColor type we defined for CSS Color Level 4 interpolation would also efficiently support compositing, but that is not looking hopeful at the moment.

Apologies for the confusion.

[facelessuser]

To me, interpolation and compositing are closely related

Sure, interpolation is used in compositing.

I think when defining a general-purpose interpolation function, like what is done in CSS, you need to define clear, sane rules, and the rules for cylindrical spaces seem perfectly reasonable.

Due to how hue interpolation behaves, it doesn't lend itself well to compositing images and layers, but there are plenty of reasons to desire a hue interpolation, such as when doing gradients. I'm not sure there is a reason to specifically forbid alpha blending/compositing in cylindrical spaces as the current rules are quite reasonable. Would I choose to specifically use a cylindrical space to do compositing in an image? Probably not.

[facelessuser]

It probably should be noted that there is a separate discussion talking about exposing blending (possibly) and alpha compositing of colors: #8431. I don't think that discussion goes into specific steps in compositing colors and is more discussing an interface to do so. I think if such an interface is approved, it probably shouldn't allow compositing to be applied in a cylindrical color space.

color-mix is not really a compositing interface, but more an interface for getting a color anywhere between two colors, and yes, cylindrical spaces behave very differently when interpolation is done within them simply due to how they separate the color characteristics. Even if using a rectangular color space, mixing two transparent colors at 50% will not yield the same results as applying Porter Duff compositing on those two colors. The logic is different, even if interpolation is used in Porter Duff compositing. This is why I don't think color-mix should be viewed as a compositing interface, not directly.

I think this is a pretty good read on alpha compositing: https://ciechanow.ski/alpha-compositing/.

[svgeesus]

One reason that choices (1) and (2) are on this list is that I'm not sure how useful it is to do compositing in a cylindrical space. I'm happy to be pointed to evidence on this.

There are three useful spaces for compositing:

1. CIE XYZ (or equivalently, any unbounded linear-light RGB space) to get the right result
2. gamma-encoded sRGB, to get the wrong but web-compatible result
3. gamma-encoded device RGB (because you already have your pixels in that space and want to use the hardware), which also gives a similar wrong result to 2.

[svgeesus]

If I had to guess, I think omitting the statement about not premultiplying hue is just an accident, not an explicit intention.

You guessed correctly

[raphlinus]

There seem to be a few possible issues here. One is whether the over operator is defined over arbitrary color spaces or a select few, and if the former, how it should be defined for cylindrical color spaces. From what @svgeesus has said, the answer is the latter, so that may be the end of the story for this issue. (We may choose to define over in the color crate for more spaces anyway, but that needn't concern us here)

I'm still not convinced that holding out hue from premultiplication is not needless complexity that makes things worse, though it's also possible I'm missing something. In particular, I don't follow the argument above. To me, the interpretation of premultiplied color components has an implicit division by alpha. The premultiplied component vector for oklch(0.5 0.2 85 / 50%) is [0.25, 0.1, 42.5, 0.5], which thus has a hue angle of 85 degrees, not 42.5.

Holding out premultiplication of hue, the result of a 50% lerp of oklch(0.5 0.1 120 / 50%) and oklch(0.5 0.1 240) is oklch(0.5 0.1 180 / 75%). If hue were premultiplied, the result would be oklch(0.5 0.1 200 / 75%), essentially giving more weight to the more opaque color. That actually seems better to me, in that I believe it's a more perceptually uniform ramp. As I say, I'm happy to prepare visual samples to argue for this position.

gamma-encoded sRGB, to get the wrong but web-compatible result

I don't agree with the word "wrong" here, as I believe there is a case to be made that compositing in more perceptually uniform spaces is a useful design tool. In addition, compositing in device RGB is a much closer approximation to layering paint in the Kubelka-Munk model than doing it in a linear space. That may be an additional factor in the staying power of compositing in device color spaces, not just laziness of implementors. But perhaps that's a discussion for another day.

[facelessuser]

One is whether the over operator is defined over arbitrary color spaces or a select few, and if the former, how it should be defined for cylindrical color spaces.

CSS does not currently define composition except in https://www.w3.org/TR/compositing-1/ and only in the context of RGB spaces. It has not been defined or applied outside of that. So over is currently only defined in this context. This does not apply to general interpolation.

I'm still not convinced that holding out hue from premultiplication is not needless complexity that makes things worse, though it's also possible I'm missing something. In particular, I don't follow the argument above.

Premultiplication is used essentially to weight how much each color influences the interpolation. This works well in rectangular coordinate systems. The actual hue of a color does not change in this process (speaking relative to the color space). That's why when you convert said cylindrical color to rectangular coordinates and apply premultiplication, then convert back, hue is unchanged. You should not premultiply hue because if you do, you are now interpolating something very different.

If you cherry pick certain cases, it could look okay if we apply premultiplication to hues. For instance, you gave a specific case in OkLCh which does seem to look okay:

But if we use some other colors, it quickly falls apart:

[tabatkins]

Premultiplication is used essentially to weight how much each color influences the interpolation. This works well in rectangular coordinate systems.

Right, premultiplication is a weighting method designed to model compositing. It is only a meaningful operation when applied to vector coordianates, that have a meaningful notion of "scaling". Hue, in a cylindrical space, is not such a coordinate; there is no privileged zero point. I think the technical term is that the hue is an affine coordinate, which doesn't allow addition or scaling. You can only add/scale differences between hues; the set of hue differences forms a vector space. This is why you can interpolate hue (you're scaling a hue difference, and adding it to a hue, which is also allowed).

So the concept of premultiplying a cylindrical space simply isn't coherent. You have to do premult in a rectangular space, and as Chris says, if you're premultiplying in order to composite, there's only a handful of color spaces intended for that.

[raphlinus]

I'm still needing to be convinced, both the example and the argument from math.

The huge discrepancy in the second example is caused by hue fixup. That is most definitely a complication (and I hadn't thought about that in my initial analysis), but I don't think it's a showstopper. Hue fixup can be defined in terms of un-premultiplying, and then in turn can be evaluated efficiently without division with some algebraic manipulation (basically it's h1 * a2 - h2 * a1 > 180 * a1 * a2 being equivalent to h1 / a1 - h2 / a2 > 180).

And I don't quite follow the argument about affine spaces, and don't think compositing is restricted to coordinates that have a meaningful concept of scaling. Ultimately, the final output is a weighted sum of values, and if the weights sum to 1 then I don't see a fundamental mathematical problem.

Of course, the hue fixup is a real source of trouble, and the fact it can create discontinuities is a good reason to not consider it form of compositing. But there are ways to get discontinuities from blend modes also.

[facelessuser]

I'm still needing to be convinced, both the example and the argument from math.

Fundementally, I don't think polar interpolation models the kind of compositing that premultiplication was designed to help mimic in rectangular spaces. Premultiplication helps model how more or less light makes it to the eye in transparent cases. The entire concept doesn't really translate to hue shifts in polar spaces. More or less transparency doesn't control the bending of light.

It seems you really want an analogy to how compositing works in the polar space, which is fine, but you'll likely have to make a strong argument on why this is useful and is needed. I'm not convinced it is needed or is worth the added complexity, but I'm also not who you need to convince 🙂.

I think I'll bow out of the conversation as I think I've probably said enough.

[svgeesus]

I agree with @facelessuser and @tabatkins that there are no articulated use cases for doing compositing in a polar space, beyond "it makes our code path simpler". The option in the first post

When compositing in a cylindrical space is requested, actually do it in the associated rectangular space, for example oklab when oklch is requested.

seems like an ideal way forward, if you are coding up a generalized multi-color-space compositing codebase. I might be tempted to throw in a warning when that happens, too.