GTAO occlusion is not used when colorBounce is enabled

[vorg]

pex-shaders/shaders/chunks/ambient-occlusion.glsl.js

Line 25 in 59621b2

color.rgb = vec3(rgb + aoData.rgb * color.rgb * 2.0);

should be

color.rgb = vec3(color.rgb * aoData.a + aoData.rgb * color.rgb); //not sure about *2.0

[dmnsgn]

aoData.a is used in the line above with gtao multibounce approximation:

pex-shaders/shaders/chunks/ambient-occlusion.glsl.js

Line 24 in 59621b2

vec3 rgb = mix(color.rgb, color.rgb * gtaoMultiBounce(aoData.a, color.rgb), intensity);

[vorg]

So aoData.a is actually used in gtaoMultiBounce. The problem was more that colorBounce was not scaled by ssao.mix (named intensity here, not to confuse with colorBounceIntensity or ssao.intensity (to be renamed to aoIntensity))

vec4 ssao(vec4 color, vec4 aoData, float intensity) {
  #ifdef USE_SSAO_COLORS
      vec3 albedoColor = color.rgb; //unlit color of the surface that we don't have ATM
      vec3 colorWithAO = color.rgb * gtaoMultiBounce(aoData.a, albedoColor);
      vec3 colorBounce = albedoColor * aoData.rgb;
      color.rgb = mix(color.rgb, colorWithAO + colorBounce, intensity);
      // color.rgb = vec3(aoData.aaa);
    #else
      color.rgb *= mix(vec3(1.0), vec3(aoData.r), intensity);
    #endif

  return color;
}
``

[vorg]

One way to avoid having to have both ssao.mix and ssao.aoIntensity is to reuse ssao.intensity to fade out GTAO's AO when below 1

visibility = mix(visibility, 1.0, 1.0 - clamp(uIntensity, 0.0, 1.0));

[vorg]

AO wish list

• use albedo color (data.baseColor) so we get color bounce in shadowed areas (e.g. without light probe)
• separate diffuse and indirect lighting so we can apply AO only to indirect lighting vs re-project previous depth buffer so we can compute SSAO from previous frame

[vorg]

The bounce from baseColor vs directColor should look like that this Source

We do get that reflection a bit if indirectDiffuse is present as they all (direct + indirect) add up to output color.

[vorg]

@dmnsgn let's keep current code but with better variable names like mentioned in my comment above

vec3 albedoColor = color.rgb; //TODO: unlit color of the surface that we don't have ATM
vec3 colorWithAO = color.rgb * gtaoMultiBounce(aoData.a, albedoColor);
vec3 colorBounce = albedoColor * aoData.rgb;
color.rgb = mix(color.rgb, colorWithAO + colorBounce, intensity);