.NET nanoFramework graphics improvement proposition

[Ellerbach]

.NET nanoFramework graphics improvement proposition

Today the graphics library from .NET nanoFramework is available and mainly implemented in native. Over the last year, we've added multiple graphic drivers. We've been adding the ability to specify the screen size and memory buffers independently of the graphic driver.

Here is a summary of the situation:

• Need to build a specific image per graphic driver.
  • While this is ok for boards like M5Stack where we offer a full board package, it's more challenging for others using a more generic approach with an external screen.
  • The number of drivers has increased and continues to increase. All the supported drivers are using SPI and use a similar approach.
• Low memory devices can't really benefit from the advance graphic features like buttons, frames, images as those require high memory usage.
• Performance has always been the key driver for the graphics implementation.
  • All SPI drivers are using native approach.
  • Advance graphics manipulation is native based.
• Additional small screen I2C based have been added in the IoT repository.
  • All implementations are managed C# based only.
  • No easy interoperability with any of the native graphic libraries.

Some of the challenges discussed on Discord:

• Need to have a simple compatible layer (colors, font interface, primitives interface) to facilitate usage between the small I2C screen and the native ones.
• Need to simplify the life of developers who want to use a native graphic driver without having the rebuild a full image for a specific board.

Proposed approach for compatible layer

A discussion to extract common elements is present on Discord.

To summarize the discussion:

For the graphics lib, ok for 3 different ones:

• The "Core" should have very little things needed by everything, no native part at all
  • Colors, IFont, Points and maybe few others.
  • Pure managed.
• The "Primitive", only primitives with a native implementation. That should fit low memory devices and still allow to use native graphic drivers.
  • Primitive requires the Core nuget.
• The "Full" which contains everything and of course uses the native part.
  • Full requires the Primitive nuget (and Core).

This design has some short term and most impactful implications:

• Requirement to move from the current nanoFramework.Graphic code the Core part to start with.
• Adjustment of the managed codes to take this change in consideration.

As a second step, splitting the rest of the graphic libraries in 2 parts with:

• Maybe a native part with just primitives split from the current implementation. The footprint for the non primitive part on the native side is far to be negligeable.
• Maybe a common interface between the native drivers and the purely managed ones.
• All this needs to be discussed and investigated.

This second step can be improved after the first step. IT will give us a better understanding of what will have to be done. This second step can also have an impact on the unified native graphic driver implementation.

Unified native graphic drivers

Over time, with the addition of multiple drivers based on files, 1 image can only contains 1 driver. See the graphics directory on the nf-interpreter.

All graphics follows the same interface and in the Preset, one file is selected for the build. It's using the existing SPI abstraction layer for this purpose. Today only SPI drivers are natively implemented. While this discussion applies to SPI native based drivers, if some I2C native ones will be implemented, this discussion will apply the same way.

Challenges that need to be overcome:

• Performance, performance, performance
• Keep it simple for the developer while using the graphic library
• Simplify the image creation process
• Make it easy to add new native drivers

Here are few propositions to overcome those challenges and advantages/disadvantages

Move all drivers initialization and management into managed code

In this proposition, the idea is to move all the drivers logic in managed code, so the initialization sequence, the rotation, brightness adjustment, etc. The lower native part would only contains the basic logic.

Advantages

• Flexibility to add new drivers.
• Easier for C# developers to add their own drivers.
• One native driver only.

Disadvantages

• Impact on performances which needs to be measured but anything moving from native to manage has an impact.
• Moving the complexity from native to manage.
• One size won't fit all. While drivers works on a similar way, they still have differences, some using brightness natively, some using GPIO, will add quite a lot of complexity on the managed side.
• Increase code footprint as moving code from native to manage shows a non negligeable increase on the code base. This can be adjusted by building a "Core" driver library and then additional nuget per driver.
• Any adjustment to add new driver will translate in rebuilding as well the native part as the library will evolve and native signature will change. This can be adjusted by building a "Core" driver library and then additional nuget per driver.

Allow to select the driver from the managed side

In this approach, rather than having one unified driver, we'll keep all the drivers natively and just select the driver during the initialization of the graphic library.

Advantages

• Only 1 build for all drivers. So any image with graphic build can be used with any available driver.
• Specific builds with specific drivers still can be done to improve efficiency for boards like M5 Stack.
• Easy usage with minimum managed modification.
• Flexibility to add more native drivers in a transparent way.

Disadvantages

• Any adjustment to add new driver will translate in rebuilding as well the native part as the library will evolve and native signature will change.
• Small increase on the native side while adding drivers. Those can be adjusted by putting in common quite a lot of code on the native side. Only increase will mainly be the initialization sequence and register names. Those are constants and about 100 bytes max per driver.

A mix solution with native drivers and possibility to pass managed code

See this proposition as a mix of the previous ones. So you'll get native drivers available and the possibility to pass specific initialization components from the managed side.

While this can seems a good approach it will complexify even more the code and won't help bringing more native drivers. It's still possible to have this one done with the all available native driver approach.

Proposed approach

The proposed approach is to allow to select the driver from the managed side. It offers more advantages and for sure will keep the performance which in this case is the most important. The mix solution with ability to pass more commands or initialization elements should be investigated during this development. It can offer some more flexibility for developers who want to unlock some specific elements of a driver.

[TerryFogg]

This is a great review.

I had already started experimenting with some ideas a few weeks ago.

Having written the original ILI9341 and OTM8009a(MIPI) as embedded firmware drivers, this worked fine as a start.
Since then a number of drivers have been added and each requires a different firmware build.

From my experience building the OTM8009A(Mipi interface) and the RK043FN48H driver using the LTDC interface with DMA, I have found that these drivers require a greater understanding and integration of the hardware/memory involved.
For these, I suggest, the graphics support is built into the image since a typical development board usually has the enough memory already installed that the addition of graphics code doesn't cause it a memory shortage problem.

My only other experience is with the SPI displays.
Each of these has had some issues, but having completed the following

• ILI9341
• STV7789 (240x240)
• ST7789VW ( 240x135)
• ILI9488 (320x480)
• GC9A01A - 240x240 ( Round Display)

Finding the issues can be frustrating, for example the 240x135 display required offsets of 40 pixels in x and 53 pixels in y.
In addition a board I have, required one of the control signals to be inverted.
But, I have found all of these can be parameterized, and those parameters can be passed through to the initialization code.

One of the disadvantages suggested was that the C# programmer would require more knowledge.
I hope we can offset some of the complexity by creating a Class for each Display.
By using a base class for the DisplayController and inheriting this into specific displays, then a number of displays can be packaged into a Nuget. I estimate that this will consume about 170-200 bytes per display controller.
If we have 10 different SPI controllers, then 2000 bytes in Flash may be acceptable.

I see that the MDP can exclude classes for deploying to the MCU, but I'm not sure how that could work with an already built library.

A modified Graphics approach

%%{init: {'theme':'forest'}}%%

graph TD


A[C# graphics application] == Initialization codes/settings === G[Display Controller] 


Z[C# Display Initialization codes</br>for Native Flexible drivers</br>Multiple display drivers available]-->A
G === H

H{Bus type?} -- SPI Configuration --- B[SPI]
H -- I2C Configuration --- C[I2C]

A === D[Video Driver]
A === E[MIPI Driver]
A === F[Parallel Driver]

  subgraph "nanoFramework</br>_______________________</br>"
  A
 Z
 subgraph  Native[Native code]
   subgraph Generalised[Initialized at CLR startup from C# ]
    B
     C
     G
     H
   end

subgraph Specific["Specific to MCU"]
   D
   E
   F
   end

  end
    end

  classDef green fill:#9f6,stroke:#333,stroke-width:4px;
  class Specific green
  class Generalised green

Loading

[Ellerbach]

Thanks a lot for the picture, it does help much in the discussion.
We should measure the performance impact but for sure having driver initialization and elements in a nuget package will be slower than on the native side. We can think of a mixed approach where the most common drivers can be build in while we can offer possibility to also pass them from the higher level.
What does worry me as well with this approach is that there are other elements that are needed for the orientation, the brightness, the color scheme and passing those from the managed side is not the best we can do. It would somehow requires to change the logic and pass a class of commands, store that natively so it can be run.
More investigations needed for this of course but good thinking with all this.

[josesimoes]

We've keep revisiting this UI matter which proves that's a complicated one.
In several of the past iterations it was suggested to move to an existing UI library for embedded systems.
I keep thinking that this is the way to go.

To be clear: I'm not diminishing the huge efforts and work hours poured into this by Laurent and specially Terry who keeps working on this. Relentlessly! 😃

Unlike pretty much all the other APIs (and native drivers) that we've modernized (and even replaced) inherited from .NETMF, we keep stuck on the old UI API and namespace. JPEG code dates from 20 years ago! We keep using bitmaps for MCUs with limited RAM. Bitmaps are inefficient, complicated and a terrible waste of RAM. JPEG and PNG are the go to in our MCU world.

Displays and UI are hard! I get that. That's why we should rely on a robust and well maintained library.
We don't have the resources or manpower to address this in an efficient way.
Just like we do with other key building blocks (RTOS for example) we rely on 3rd party ones. And that's OK.
Why can't (shouldn't?) we do this with UI too?

LVGL comes to mind. It's probably the most popular, well maintained and supported library for this. Includes desktop tool to design interfaces. It supports hundreds of different displays. I2C. SPI. Has support for hardware acceleration when that's available on the MCU running it.
There's also Azure RTOS GUIX although that's probably not the best choice because of it's limitations on licensed hardware.

We have the mechanism in place to build a C# wrapper over the primitive functions offered by the library so this can easily be used by developers coding for nanoFramework.
Will it require effort? Yes! Absolutely!
But I truly think it's a much efficient and forward looking approach and better use of the time invested on this matter. As opposed to keep investing on an outdated approach, hard to maintain and makes a poor developer pull it's hair going thorough a cryptic datasheet with every new display that is needed to support, just to have it initialized...

Relying on a modern and well maintained UI library will make a total difference and will bring serious UI capabilities to .NET nanoFramework!

[Ellerbach]

I keep thinking that this is the way to go.

First problem with other existing implementation is the same as with our own implementation: drivers are NOT generic, you have to build it with the device.
Second problem with any other existing implementation: writing a proper wrapper in C# for nanoFramework. And this is taking a lot of time as well.

So the key problem to solve here and for any further implementation is to make the driver generic to avoid building every single device with a specific firmware but rather use a flexible way.
If later we'll change the library, we will be able to reuse that pattern. And in all cases, it will require to change that part of any library we may use.

So let's focus first on solving the problem of having more generic drivers as whatever we'll decide for the graphic library, we'll have this solved.

[alberk8]

I like the LVGL library.

[Ellerbach]

I like the LVGL library.

I4ve been looking at it and it's a lot of work to get it ported. And it won't solve the driver problem. Every driver is built in and every image needs then to be built for the proper driver. Which will make things more complicated. So reason why, I'm trying to find out to get generic drivers and pass the needed elements from the managed code.

[josesimoes]

I get the goal of being able to have a generic init for the driver, controlled by the C# application.
And how convenient it would be not having a build specific for a particular driver.
Because of the specificity of each driver, I'm finding it a bit hard to come up with a generic enough approach to make this happen. But I haven't looked at this with enough detail to have a definitive answer.

Something that strikes me against the initialization of the display from the C# application is that we'll miss display support until a C# application is properly loaded and executing. This prevents the display from being used for boot information, progress report on IFU and such. It may not be necessary or a determent in all situations but it's something to consider.

[Ellerbach]

Because of the specificity of each driver, I'm finding it a bit hard to come up with a generic enough approach to make this happen. But I haven't looked at this with enough detail to have a definitive answer.

Yes and no :-) Yes, it's not that trivial but we have already a native interface. Each method requires for each execution a specific command or set of commands. So passing a class with arrays containing those specific commands or sets of commands for each function should work. And that's what we need to investigate. The commands are just constants. The execution commands are also constants. And setting the window, writing, is standardized over the drivers. What's not are the commands themselves.
And all the purpose of the investigation is to see how we can manage this properly between native and managed code. If too complex and not working, we will know it :-)

[alberk8]

Most likely the approach is to have a generic driver for a group of graphic chipset like the STxxx, ILIxxx, etc, on on the managed side pass in the appropriate initialization command array.

[Ellerbach]

It's not just initialization, it's orientation, brightness and few others. So far that's what we need for the SPI drivers:

public class GraphicDriver
    {
        private uint _width;
        private uint _height;
        private byte _bitsPerPixel;
        private byte[][] _initializationSequence;
        private byte _memoryWrite;
        private byte _setColumnAddress;
        private byte _setRowAddress;
        private byte[][] _powerModeNormal;
        private byte[][] _powerModeSleep;
        private byte[][] _orientationPortrait;
        private byte[][] _orientationPortrait180;
        private byte[][] _orientationLandscape;
        private byte[][] _orientationLandscape180;
        private byte[][] _clear;
        private byte _brightness;

        public uint Width { get => _width; set => _width = value; }
        public uint Height { get => _height; set => _height = value; }
        public byte BitsPerPixel { get => _bitsPerPixel; set => _bitsPerPixel = value; }
        public byte[][] InitializationSequence { get => _initializationSequence; set => _initializationSequence = value; }
        public byte MemoryWrite { get => _memoryWrite; set => _memoryWrite = value; }
        public byte SetColumnAddress { get => _setColumnAddress; set => _setColumnAddress = value; }
        public byte SetRowAddress { get => _setRowAddress; set => _setRowAddress = value; }
        public byte[][] PowerModeNormal { get => _powerModeNormal; set => _powerModeNormal = value; }
        public byte[][] PowerModeSleep { get => _powerModeSleep; set => _powerModeSleep = value; }
        public byte[][] OrientationPortrait { get => _orientationPortrait; set => _orientationPortrait = value; }
        public byte[][] OrientationPortrait180 { get => _orientationPortrait180; set => _orientationPortrait180 = value; }
        public byte[][] OrientationLandscape { get => _orientationLandscape; set => _orientationLandscape = value; }
        public byte[][] OrientationLandscape180 { get => _orientationLandscape180; set => _orientationLandscape180 = value; }
        public byte[][] Clear { get => _clear; set => _clear = value; }
        public byte Brightness { get => _brightness; set => _brightness = value; }
    }

Some are just register addresses like WriteMemory or the SetColumAddress but some can be a series of commands like for the powermodesand the rest of the commands.
And I think I found as well a way to manage "sleep" times. You need to be able between commands to have a sleep. It can be from 20 to 120 ms on what I found. So I'll use an empty command for that, meaning, I'll wait for 20 ms. 6 emtpty commands will wait for 120 ms. Or a mechanism like this.

[TerryFogg]

Have you done any programming towards any of this, I will be doing some experimentation of my own soon.

[Ellerbach]

More than that :-) Already 2 PR and implemented everything. The class is a bit different to improve memory usage.
See nanoframework/nf-interpreter#2588 and nanoframework/nanoFramework.Graphics#118.
Tested on real device with real screens.
Just need now to create some nugets with existing drivers for all the native screens we have.
The build system and current implementation is not changing. Meaning the Generic driver is an additional driver that can be used with a generic image. For the M5Stack image, they'll embed their normal native drivers.
The advantage of the generic driver is as well to test orientation and other things like this, adjusting the intitialization if needed for custom scenarios.

[Ellerbach]

Closing this discussion as now implementation is done!