Font handling - ASCII, non-ASCII and CJK

[alvinhochun]

Just throwing out ideas. After having the initial CJK support and Chinese translation merged, I was thinking how text handling may be improved to allow for more chars in the CJK range and possibly make multi-language firmware releases possible, so I typed out some rough notes. I haven't proofread it so please excuse me if I've wrote something really stupid.

Looking for comments.

---

Fonts

Latin/Cyrillic

• Small: 6x8 px (16x2 chars on screen)
• Large: 12x16 px (8x1 chars on screen)

CJK

• Small:
  • Full-width char: n/a
  • Half-width char (i.e. alphanumeric): 6x8 px (16x2 chars on screen), same
    as Latin/Cyrillic
• Mixed:
  • 12x12 px padded to 12x16 px (8x1 chars on screen), in conjunction with
    the Large font for Latin/Cyrillic
• Medium (optional, but would be nice to have):
  • Full-width char: 12x12 px padded to 12x16 px (8x1 chars on screen)
  • Half-width char: 6x12 px padded to 6x16 px (16x1 chars on screen)
    • Or alternatively, 6x8 px padded to 6x16 px, i.e. reusing small font
• Large (probably not worth the trouble):
  • Full-width char: 16x16 px (6x1 chars on screen)
  • Half-width char: 8x16 px (12x1 chars on screen)

Japanese using only half-width Katakana (like JIS X 0201)

• Small: 6x8 px (16x2 chars on screen)
• Large: 12x16 px (8x1 chars on screen)

Needs comment from native Japanese user. Perhaps not really useful.

Text layout

Full screen static text message

• Latin/Cyrillic:
  • 16x2 using small font
• CJK:
  • 8x1 (16x1 half-width) using medium font
  • 8x1 using mixed font
  • 6x1 using large font

Full screen scrolley text (marquee) message

• Latin/Cyrillic:
  • Single row using large font
• CJK:
  • Single row using medium font
  • Single row using large font
  • Single row using mixed font

Menu group

16x16 px graphical icon on the right.

2 px on the right for scroll bar...

Technically printable in the same way as "full screen static text message".

• Latin/Cyrillic:
  • 13x2 (13.33x2) using large font
• CJK:
  • 6x1 (6.67x1) (13.33x1 half-width) using medium font
  • 6x1 (6.67x1) using mixed font
  • 5x1 using large font (but this does not account for the scroll bar)

Menu option short description

Option value is printed on the right.

2 px on the right for scroll bar...

Technically printed in the same way as "full screen static text message".

Exact available space depends on the option, but in general it is about the
same as a menu group.

Menu option value

Option values are usually printed programmatically, so the font size is usually
defined statically.

Normal screens / detailed screens / debug menu

Fixed layouts, practically require statically defined font sizes.

Font setup

Issues with the current system

• Not possible to directly print ASCII strings onto the OLED without going
  through font table mapping transformation
• CJK characters have an entry in the small font table even if they can't be
  printed using this size
• There is a limit of a total of 254 printable characters which is inadequate
  for CJK
• Alphanumeric chars are always full-width instead of half-width in relation
  to CJK chars

Proposal 1

For \x00 to \x7F, keep the same chars as ASCII for convenient alphanumeric
display (e.g. debugging info).

For extended Latin/Cyrillic chars outside of ASCII, they should be able to fit
within \x80 to \xFE (none of the current non-CJK firmwares uses more than
128 chars and that does not exclude ASCII chars). We can use the same dynamic
font table generation method as currently in use for these extended chars.

For CJK versions, we can employ lead bytes, effectly implementing a custom
double-byte charset (DBCS), which theoretically gives us 32768 CJK chars and
should be more than enough. Or alternatively, we can designate \xF0 to \xFF
as lead bytes and leave the rest to be single-byte chars, so that the more
frequently used CJK chars only takes one byte. The dynamic font table
generation can do the work here again.

Font tables

• Small font table for ASCII + extended Latin/Cyrillic
  • 6x8 px
  • Storage format same as current
  • Index 0 starts with \x20 (space), as anything < \x20 are control chars
  • Extended Latin/Cyrillic not available for CJK languages
• Large font table for ASCII + extended Latin/Cyrillic
  • 12x16 px
  • Storage format same as current
  • Index 0 starts with \x20 (space), as anything < \x20 are control chars
  • Extended Latin/Cyrillic not available for CJK languages
• Medium font table for CJK full-width
  • 12x12 px
  • Storage format:
    • Option 1: Same as 12x16 px but with glyph pre-centred vertically
      (same as current as in large font table)
    • Option 2: Use groups of nibbles (4 bits), which saves 1/4 of space
      but with increased handling complexity
  • Index 0 starts with \x80\x00
  • Only available for CJK languages

CJK text printing

• Mixed:
  • Use medium CJK font together with large ASCII font
  • Same as current
• Medium:
  • Use medium CJK font together with small ASCII font
  • Small ASCII font needs to be offset downward to match text baseline
  • Char width is determined per char
• Small:
  • Use small ASCII font
  • All CJK chars print with a "replacement character" instead

Font selection in translation string

• All menu groups, menu options and full screen static text to use a single
  null-terminated string
  • String prefixed with \x01 to indicate usage of large (Latin/Cyrillic)
    or medium (CJK) font
  • String prefixed with \x02 to indicate usage of large (Latin/Cyrillic)
    or mixed (CJK) font
  • Unprefixed to indicate usage of small font, where \n moves to the
    second line
• Detailed screen strings always use small font
• Single char option values always use large (Latin/Cyrillic) or mixed (CJK)
  font

Proposal 2

Use UTF-8 encoding for strings.

A font lookup table can be used to map Unicode codepoints to the font table.

ASCII chars from the range of \x20 (space) to \x7F may be fixed so as to
not take space in the lookup table.

It is also possible to include different set of available chars for the small
and medium/large fonts to reduce size of font tables.

Downsides:

• Typical CJK chars needs three bytes in UTF-8
• Font lookup table takes extra space

Font lookup table may be:

struct font_entry_bmp_t {
    uint16_t codepoint;
    uint16_t font_idx;
};
font_entry_bmp_t FONT_BMP_LUT[] = {/* ... */};
constexpr uint16_t FONT_BMP_LUT_LENGTH = sizeof(FONT_BMP_LUT) / sizeof(FONT_BMP_LUT[0]);

struct font_entry_sup_t {
    uint32_t codepoint;
    uint16_t font_idx;
};
font_entry_sup_t FONT_SUP_LUT[] = {/* ... */};
constexpr uint16_t FONT_SUP_LUT_LENGTH = sizeof(FONT_SUP_LUT) / sizeof(FONT_SUP_LUT[0]);

constexpr uint16_t CJK_FONT_START_IDX = 96;

contexpr uint8_t FONT_TABLE_6[] = {/* ... */};
contexpr uint8_t FONT_TABLE_12[] = {/* ... */};

uint16_t codepoint_to_font_idx(uint32_t codepoint) {
    // Control chars has no font.
    if (codepoint < 0x20) {
        return 0;
    }
    // ASCII chars have fixed font indices.
    if (codepoint < 0x80) {
        return static_cast<uint16_t>(codepoint) - 32;
    }
    if (codepoint < 0x10000) {
        uint16_t bmp_codepoint = static_cast<uint16_t>(codepoint);
        // FIXME: Maybe use binary search.
        for (uint16_t i = 0; i < FONT_BMP_LUT_LENGTH; i++) {
            if (FONT_BMP_LUT[i].codepoint == bmp_codepoint) {
                return ONT_BMP_LUT[i].font_idx;
            }
        }
        return 0;
    }
    // FIXME: Maybe use binary search.
    for (uint16_t i = 0; i < FONT_SUP_LUT_LENGTH; i++) {
        if (FONT_SUP_LUT[i].codepoint == codepoint) {
            return FONT_SUP_LUT[i].font_idx;
        }
    }
    return 0;
}

bool is_font_idx_cjk(uint16_t font_idx) {
    return font_idx >= CJK_FONT_START_IDX;
}

[Ralim]

Hi,
Sorry to not respond earlier, I did read this but didn't really own down thoughts.

One of the main reason I went to the current system was to save flash as having full font tables was getting huge. Would prefer to not go back to burning in an entire ASCII range as a table; especially as for debugging you rarely need text often only need numbers 🤣

My initial plan here was to go to utf style leading bytes (which is why symbols are ranked by use count; to reduce the number of the leaders). But not use actual utf8 due to the overhead.

I'm open to any ideas really; but with a preference for ones that reduce flash and ram usage where sane.

One thought would be to move to variable width characters with seperate lookup + data arrays.

Akin to how adafruit do their fonts iirc (been a long time). No idea what the size difference would be to the current method though 😅

Also have wondered if some simplification can occur with all the settings screens or not as they are all trending towards a similar style 😅

hmmmm

[alvinhochun]

Actually, I was thinking of just simply pushing the large-font-only CJK chars to the back of the list so we can skip the filler in the small font table. Separating the encoding of the two font tables would obviously save more, but is the saved space worth the complexity increase?

So I tried this (alvinhochun@73d22bc) and it saved 1160 bytes.

[Ralim]

Actually, I was thinking of just simply pushing the large-font-only CJK chars to the back of the list so we can skip the filler in the small font table. Separating the encoding of the two font tables would obviously save more, but is the saved space worth the complexity increase?

So I tried this (alvinhochun@73d22bc) and it saved 1160 bytes.

We dont even need to do that ^
We can just in python track small vs large font strings seperately and then use their own index's independantly without issue.

Complexity here should be fairly small, I would be inclined to just add to the defs fine if it uses the small font or not. :)

[Ralim]

For height padding, the code wont be too complex, but i feel the easier space saver is to do small / large font index's seperately first, then leave that until we get tight on room again.

[alvinhochun]

I think the complexity will be from keeping track of which strings are printed in which font, but maybe it's not really that bad.

Regardless of whether you separate the font encoding or not, I would still like to see the CJK symbols pushed to the back of the font table. This makes it easier to try some stuff that I'm trying.

[Ralim]

Happy for a pr to push them to the back - no issues there at all :)

I might take a stab at it if i have some time to sit down and do some good testing