8-Bit Surface Conversion Distorting and Quantizing

[geometrian]

There are some issues with converting to an 8-bit surface.

The first issue is that the result's palette appears to be a fixed default, not optimized from the surface. This can have very unexpected effects (for example, converting a 24-bit image containing grayscale data to 8-bit can suddenly add spurious color). The standard practice would be to generate an optimized palette for the surface's data, for which there are well-known approaches. Two mitigating factors of the current approach are backward compatibility and simplicity/performance, but there appears to be no standard way to choose different behavior (e.g. by passing a flag; the current flags I guess are passed to surface construction).

The second and more serious issue, also affecting the first, is that blits don't even use colors that are actually in the palette. For example, we might try to work around the previous issue by creating an 8-bit surface with a palette pre-initialized to all 256 gray levels, and then blitting to that. However, this produces an image with only 8 gray levels, instead of 2⁸.

The following simple test example shows both issues:

import os, sys
with open( os.devnull, "w" ) as f:
    oldstdout=sys.stdout; sys.stdout=f; import pygame; sys.stdout=oldstdout
pygame.init()
print(pygame.ver)

window = pygame.display.set_mode(( 512+30, 512+30 ))

#Top Left: 24-bit gradient (reference)
gradient = pygame.Surface( (256,1), depth=24 )
for i in range(256): gradient.set_at( (i,0), (i,i,i) )
gradient = pygame.transform.scale( gradient, (256,256) )

#Top Right: 8-bit conversion by `.convert(8)`
gradient8a = gradient.copy().convert(8)

#Bottom Left: 8-bit conversion by blitting
gradient8b = pygame.Surface( (256,256), depth=8 )
gradient8b.blit( gradient, (0,0) )

#Bottom Right: 8-bit conversion by blitting (with pre-inited palette)
gradient8c = pygame.Surface( (256,256), depth=8 )
gradient8c.set_palette([ (g,g,g) for g in range(256) ])
gradient8c.blit( gradient, (0,0) )

window.fill(( 64, 64, 0 ))
window.blit( gradient  , (10,       10       ) )
window.blit( gradient8a, (10+256+10,10       ) )
window.blit( gradient8b, (10,       10+256+10) )
window.blit( gradient8c, (10+256+10,10+256+10) )

looping = True
while looping:
    for event in pygame.event.get():
        if   event.type == pygame.QUIT: looping = False
        elif event.type == pygame.KEYDOWN:
            if   event.key == pygame.K_ESCAPE: looping = False
            elif event.key==pygame.K_s and pygame.key.get_mods()&pygame.K_LCTRL:
                pygame.image.save( window, "screenshot.png" );
    pygame.display.flip()

pygame.quit()

At the top left, we see a 24-bit procedural grayscale gradient. On the top right, we see it converted to 8-bit with .convert(8), and the spurious colors introduced. On the bottom left, we see it converted to 8-bit by blitting, with the same colors. On the bottom right, we see it converted to an 8-bit surface by blitting, where the palette is pre-initialized.

All of the conversion methods have the weird quantization issue, even in the case where the palette matches the input data exactly (bottom right). The direct conversion (top right) should use an optimized palette but doesn't (with no facility to request doing that). The color shifts in the conversion by blitting where the destination already exists (bottom left) are perhaps acceptable to avoid changing the palette for the destination.

The top right and bottom right should look exactly the same as the top left, and the bottom left should at least be far less quantized (some color shifting there is acceptable).

FWIW the problem of conversion, at least for pure grayscale, can be worked around by setting the palette indices directly, e.g. with surfarray. Something like the following:

gray_surf8 = pygame.Surface( gray_surf.get_size(), depth=8 )
gray_surf8.set_palette([ (g,g,g) for g in range(256) ])
pygame.surfarray.pixels2d(gray_surf8)[:,:] = pygame.surfarray.pixels_red(gray_surf)[:,:]

^{(Note: this bug first reported on the pygame Discord, now being formalized and expanded here.)}