mcuboot: add support for new "copy with revert" algorithm

[michallenc]

Sumary

This commit introduces new algorithm discussed few weeks ago on dev mailing list. The goal of this algorithm is to speed up update process for devices with large images and a lot of free space on external flash memory. The biggest setback of swapping based algorithms is the amount of erases and writes performed on ota1 and otascratch. This may not be an issue if all partitions are in embedded flash, however having them on external flash prolongs the update by an extensive amount of time.

Presented algorithm solves this issue by removing scratch area and adding tertiary slot ota2. This algorithm does not swap images, but rather only copies the update image to primary slot. Recovery option is kept thanks to third slot as secondary or tertiary always keeps recovery image (and the other one is used for new update).

Overall, this algorithm allows to achieve the speed of overwrite only algorithm while retaining the revert/recovery option. It is especially useful for devices with larger images and a lot of free space on external flash.

Target support is currently added just for NuttX RTOS.

Testing

Tested with NuttX on SAMv7 MCU with 2 MB embedded flash. 1 Gb W25Q NOR was used as external flash. Almost 2 MB image takes about 3 minutes to update with swap algorithm with scratch area while it takes about 20 seconds with this new algorithm (once recovery image is created).

[nordicjm]

We should not be adding a third image slot without a very good justification, original comment mentions swap using scratch, so just use swap using move instead?

[michallenc]

We should not be adding a third image slot without a very good justification, original comment mentions swap using scratch, so just use swap using move instead?

But swap using move is limited to support sectors of the same sector layout (see https://github.com/mcu-tools/mcuboot/blob/main/boot/bootutil/src/swap_move.c#L296) thus often not usable for the combination of embedded and external flash. Moreover swap using move is still slower than the presented algorithm as you have to copy image to external flash during every update.

[nordicjm]

We should not be adding a third image slot without a very good justification, original comment mentions swap using scratch, so just use swap using move instead?

But swap using move is limited to support sectors of the same sector layout (see https://github.com/mcu-tools/mcuboot/blob/main/boot/bootutil/src/swap_move.c#L296) thus often not usable for the combination of embedded and external flash. Moreover swap using move is still slower than the presented algorithm as you have to copy image to external flash during every update.

That is something I believe @d3zd3z mentioned about recently and should be fixed by using the largest sector size of the two

[michallenc]

We should not be adding a third image slot without a very good justification, original comment mentions swap using scratch, so just use swap using move instead?

But swap using move is limited to support sectors of the same sector layout (see https://github.com/mcu-tools/mcuboot/blob/main/boot/bootutil/src/swap_move.c#L296) thus often not usable for the combination of embedded and external flash. Moreover swap using move is still slower than the presented algorithm as you have to copy image to external flash during every update.

That is something I believe @d3zd3z mentioned about recently and should be fixed by using the largest sector size of the two

That would be great and would definitely help, but still the algorithm is much slower. I can get to ~20 seconds for 2 MB image now (if recovery is present), but swap using move would take about ~90 seconds (just theory calculation as swap using scratch takes more than 3 minutes). IMHO this algorithm is better for devices with large external flash memory.

[taltenbach]

I'm working on a project with MCUboot on an STM32F4 device and I'm very interested by this new algorithm, which I think brings significant benefits compared to the existing strategies, in particular for MCUs having large internal flash memory sectors. For example, the STM32F4 I'm using has 1.5 MiB of flash divided in sectors of 128 KiB. This means:

• When using swap-move, one sector of internal flash is reserved for the swap-move algorithm and another sector for the image trailer.
• When using swap-scratch, at least one sector of internal flash memory must be allocated to the scratch area (which must be located in internal flash, assuming the firmware is encrypted).

Therefore, at least 256 KiB (swap-move) or 128 KiB (swap-scratch) of internal flash cannot be used for storing the app, which represents in both cases a significant loss of space (resp. ~17% and ~8%). For projects having large applications like the one I'm working on, this is an issue.

Considering the app's secondary slot is located in external flash memory, using the "copy with revert" algorithm would eliminate the need to allocate one or multiple sectors in internal flash memory for enabling the swap, while still having the possibility to revert after the update. This comes of course at the cost of higher external flash memory usage.

So, in addition to improving the update speed, this new algorithm also helps reducing the internal flash memory overhead, which would definitely be valuable for projects similar to one I've described.

[ppisa]

I support the proposed method and wait for it for more of our projects. I see another reason to use this algorithm, swapping by smaller parts requires to expose internal FLASH to more erase cycle and even if they target different areas there can be negative influence on wearing of the FLASH. If you consider some safety microcontrollers (as TMS570) then they have very strict rules for guaranteed parameters and they cannot be used in critical application after 1000 erase cycles to the main program FLASH. If at least some of the cycles are eliminated even at cost of double or even triple and larger external FLASH size use then it is real win situation.

[michallenc]

Hi @d3zd3z, @utzig, @nvlsianpu. Could you take a look at this? Thanks.

[utzig]

Hi @d3zd3z, @utzig, @nvlsianpu. Could you take a look at this? Thanks.

Sorry for the delay. I've not been actively using MCUboot, or doing any embedded stuff as of late. This PR is quite big and I would rather not approve or merge, and defer the decision for the active maintainers.

Regarding the PR, the main issue is not having any patches for the simulator to test the new method, so it would require someone to manually test, and any change to any of the "shared" code could break the functionality without it being noticed until the day someone tries to use it. I don't remember if it's a prerequisite to have sim support, but I would not merge it otherwise.

I am also curious if you tested with a large scratch partition, with a size similar to the new partition, and if it also requires all the erases that slow the process down.

Also as mentioned above, adding support for swap move with different sector sizes should work here, and it's probably very straight-forward to do, at least for the simple case where there's a fixed size on each partition!

[michallenc]

Regarding the PR, the main issue is not having any patches for the simulator to test the new method, so it would require someone to manually test, and any change to any of the "shared" code could break the functionality without it being noticed until the day someone tries to use it. I don't remember if it's a prerequisite to have sim support, but I would not merge it otherwise.

Hi, thanks for the reply. Yes, method implementation for a simulator is a reasonable requirement. I will take into it, but first I would like to have some confirmation the maintainers will be willing to consider this algorithm so I do not invest more time into simulator implementation without possibility of merging. Or at least some initial review. I got initial "go for it" from David Brown on mailing list few months ago so I hope there will be a consensus to add this to the mainline.

I am also curious if you tested with a large scratch partition, with a size similar to the new partition, and if it also requires all the erases that slow the process down.

I have not tried that large scratch partition, just 128 KB i think (with entire flash partition being about 2 MB). I can try larger scratch area by Friday, I hope, but I do not expect significant speed up as it still has to erase entire partition twice (firstly for primary -> scratch and secondly for scratch -> secondary). The presented algorithm does not erase secondary or tertiary slot at all during boot, it just erases primary slot (which is fast if located in embedded flash). Also wear leveling is lower here.

[ppisa]

Please, what is the state for the case? At least some response from MCUboot core developers, if the discussion about alternative is possible and if they hear for preserving Flash from abundant wear and tear and what are requirements to discuss and possibly merge more decent alternative. We have multiple project at university and even my company (PiKRON) where we wait if MCUboot would be acceptable or we have to continue to use our own previous solution with update over special fixed boot block including USB, CAN and RS-485 communication. I have seen MCUboot as interesting alternative which allows to join forces together and keep boot block size minimal. But I consider copying back and fort over scratch area as really bad solution for cases where larger external memory is available and two complete versions of application can be stored there. I though that MCUboot offers this alternative initially which was my reason to consider it. But when I learned that is is not supported and it seems that it could be forced so indefinitely, I am quite reluctant to look how/if to integrate it our projects.

[nordicjm]

It's a no from me, I assume it's a no from @de-nordic too but will let him comment

[ppisa]

It's a no from me, I assume it's a no from @de-nordic too but will let him comment

Thanks for response.

Please, can you elaborate alternative to keep Flash wear acceptable?

It would be even worse for MCUs like Ti TMS570/Hercules family (we have experience at PiKRON, designed HW of rapid prototyping platform for Porsche based on it) which has only one thousand guaranteed rewrites of internal program Flash. When it is used with MCUboot without external memory it will be dead immediately even that it has relatively large internal Flash. So if you do not consider keeping wear at reasonable level as valid argument for alternative algorithm, then there should be placed bold warning on the MCUboot pages that it is not suitable for such devices and will cause (unnecessarily) breakage even of other targets in time shorter by factor of two or more. There should be provided formula counting that factor from firmware size and scratch area size and again bold warning of limited usability of MCUboot.

[nordicjm]

If you want to have MCUboot in a "firmware loader" mode, this means you have MCUboot, your main image and a specific firmware update application (which would update the main image) then you can do this, it meets the goal of not doing additional erases because you would erase the main slot and replace it.

If you want to still have 2 slots, then you have upgrade only mode which will replace the main application, this has the downside that your image must be perfect, if you load a bad update then the device is bricked. The alternative is to have direct XIP with revert, which means you load an update for a specific slot and boot it directly, there is no copying of data to the alternative slot, if the image boots successfully and passes any checks, it can mark itself as confirmed and will boot that image continually, if not then upon reboot (or watchdog timeout) the image will be reverted to the other.

I do not accept this because you are adding a third slot, this just breaks everything we have in zephyr, likely other OS's too, whilst it might work for your usecase it is not needed for the vast majority of everyone else's usecases and vastly complicates things (and as said, completely breaks zephyr support)

[michallenc]

I do not accept this because you are adding a third slot, this just breaks everything we have in zephyr, likely other OS's too, whilst it might work for your usecase it is not needed for the vast majority of everyone else's usecases and vastly complicates things (and as said, completely breaks zephyr support)

It should not brake anything. Other algorithms still work as previously (tests on simulator are passing and I have done tests for swap with scratch manually on real hardware before submitting this PR). Type of used algorithm is selected by #ifdef, similarly to how you select between scratch algorithms, overwrite only etc. and definitions for third slot are required only if this algorithm is enabled. So in theory, this new code should not be even compiled for Zephyr and other systems unless correct defines are configured there. There might be some mistakes I missed so if you tried to compile my branch against Zephyr and found some issues I would appreciate if you send them.

In your initial comment, you also mentioned:

We should not be adding a third image slot without a very good justification

I personally think both me and other people (I work on some stuff with @ppisa, but @taltenbach is completely independent, so there is obviously wider usacase than just for me) provided good justification (faster boot, much lower wear leveling) for adding this algorithm.

[ppisa]

Thanks for elaborate. But I expect that if the update application needs to be updated (i.e update protocol security problem or some incremental changes, reliability increase) then there is significant trouble how to update it safely. I liked idea of MCUboot that update is done by main previously up to date application and it is guaranteed that if new image does not behave correctly (i.e. does not connects to some external controller by communication protocol which allows to run some checks and then send command to confirm the firmware as valid) then watchdog or next reset ensures return to previous up to date functional code. I really liked that idea and took it as argument for MCUboot over our previous solutions.

As the third slot protocol has been designed by @michallenc as the configurable option which is off by default, I do not see any downside for MCUboot use in the scenarios you describe. I agree that it represnets some maintenance burden but as I understood @michallenc offered to prepare test cases and infrastructure for that and I do not expect that it would cost much in build time and testing on the host system. But yes, there is valid argument that it is additional code and its maintenance cost should be compared to added value now. But I see really substantial added value and it seems that even some independent MCUboot user @taltenbach sees the same value as the CTU univesity, Elektroline and PiKRON companies. We can ask in https://github.com/robertobucher/pysimCoder (https://www.youtube.com/@robots5/videos) community which is looking for alternative to update over JTAG, but there could be approach with fxed updater and single application image functional because these devices are more for the local experiments than for production systems where they are not available for manual correction when running on the other side of the world (i.e. tram yards in Melbourne). May be more MCUboot users would see the value if the they are informed about this discussion.

[nordicjm]

Have you tested any of this with the smp_svr sample in zephyr? Have you tried erasing the third slot?

[michallenc]

Have you tested any of this with the smp_svr sample in zephyr? Have you tried erasing the third slot?

The third slot is currently implemented only for NuttX, it does not have any affect on Zephyr or other systems. Number of image slots is set to default 2 if MCUBOOT_COPY_WITH_REVERT option is not used, see:

#ifdef MCUBOOT_COPY_WITH_REVERT
#  define BOOT_NUM_SLOTS                  3
#else
#  define BOOT_NUM_SLOTS                  2
#endif

The only Zephyr related change is boot_write_image_ok rename to boot_write_image_flag, see commit 1acec48, and this should not break anything.

Regarding smp_svr sample in Zephyr. No, I did not tested this on Zephyr as I do not use the system at all. Tests were done on NuttX only (and checked if simulator tests pass). I have now tested once again NuttX build if MCUBOOT_COPY_WITH_REVERT is not set and third partition is erased from NuttX (basically revert of 8110ce5) and it did not cause any errors.

Did you have some troubles compiling this with Zephyr?

[utzig]

I am also curious if you tested with a large scratch partition, with a size similar to the new partition, and if it also requires all the erases that slow the process down.

Very straight-forward question which remains unanswered (or did I miss the answer?!)

When it is used with MCUboot without external memory it will be dead immediately even that it has relatively large internal Flash. So if you do not consider keeping wear at reasonable level

Ditto, same subject as above, there is no wear with a large scratch, maybe...

Tests were done on NuttX only

I don't think the maintainers usually use NuttX, and our CI does not have a NuttX job, maybe you'd like to PR one?

[michallenc]

I am also curious if you tested with a large scratch partition, with a size similar to the new partition, and if it also requires all the erases that slow the process down.

Very straight-forward question which remains unanswered (or did I miss the answer?!)

As I have already said, there is no reason this would significantly improve anything. The algorithm remains the same, it still has (based on the docs) to erase scratch area, copy secondary to scratch, erase secondary, copy primary to secondary and finally copy scratch to primary. Yes, number of erases is lower with larger scratch, but you have to erase larger area. As a result, number of erased pages remains the same.

I have done the comparison on real hardware now and there is no significant improvement.

When it is used with MCUboot without external memory it will be dead immediately even that it has relatively large internal Flash. So if you do not consider keeping wear at reasonable level

Ditto, same subject as above, there is no wear with a large scratch, maybe...

Yes, wear leveling will be lower, but still higher compared to the proposed algorithm in this MR as there are more erases.

Tests were done on NuttX only

I don't think the maintainers usually use NuttX, and our CI does not have a NuttX job, maybe you'd like to PR one?

I do not mind adding NuttX CI job, it would benefit both NuttX and mcuboot in my opinion. But as I have already said, I would like at least some reassurement this has a chance of being merged. Because so far it seems there is an opposition against adding the possibility of third slot in general, regardless CI and tests (which is of course your right to reject it).

[d3zd3z]

The issue with swap-move needing a separate sector for the image trailer is actually a bug, and needs to be investigated and fixed. It should be possible to do swap-move with two sectors in the first slot, and one sector in the upgrade slot. swap-move should not require more space than swap scratch.

[michallenc]

Hello all. I have just tried Zephyr for SAMv71 board and my MCUboot changes and both build and subsequent boot (tested with simple hello_world program) works fine and without errors. So the presented changes should not brake current Zephyr (or other OS) support in any way.

[michallenc]

Hello. I think it is time to resolve this. I know there was a dismissive feedback towards the presented changes, but we as a company are in the point where we need some proper bootloader. So we have to decide whether we keep investing our time and resources into getting this to mainline (which would be ideal in our opinion) or whether we look into other solutions (our fork of MCUboot or something else).

Was there any change in your opinion regarding potential merging this to the mainline if tests and simulator are implemented? Or is there still an issue with the principle of third partition slot and the algorithm itself? Thanks.

@de-nordic @nordicjm @d3zd3z

[nordicjm]

No change from me

[ppisa]

No change from me

Please, can you clarify what you refer to?

You have stated that @michallenc work breaks Zephyr

I do not accept this because you are adding a third slot, this just breaks everything we have in zephyr, likely other OS's too,

and asked the tests on Zephyr. Even that Elektroline, CTU and PiKRON have no actual project related to Zephyr (I value that OS and may be we try it in future, but NuttX, RTEMS and Linux are our actual workhorses and we contribute to core of them all), @michallenc has invested time to run the code on Zephyr and checked that there is no breakage. The "copy with revert" is fully optional so when not used it has really zero impact in addition.

At least actual MCUboot is unusable slow in the default mode and has wore at least one device Flash to dead at Elekroline (as I have heard) in the relatively short time compared to devices lifetime expectancy.

Or when the your Zephyr breakage point is resolved as proved above, then your answer should be interpreted the option would be accepted when implemented correctly and it is blocked on missing testes now?

What is stance of @d3zd3z and @utzig?

We (me, at least five my students at CTU and more people at Elekroline) have many contribution in NuttX including significant work for Espressif based projects so I can discus this option and save of Flash wear with their people as well to get additional input.

[utzig]

What is stance of @d3zd3z and @utzig?

My suggestion was to use swap with scratch with a very large scratch area. It might not be ideal as pointed out above, but adding an algorithm that solves a problem for one person, and has to be maintained by all of us is also far from great. I am just a hobbyist contributor, so no chance I want to maintain more code, this needs to be dealt by someone financially motivated.

We (me, at least five my students at CTU and more people at Elekroline) have many contribution in NuttX including significant work for Espressif based projects so I can discus this option and save of Flash wear with their people as well to get additional input.

It should have been discussed before implementation, but I suggest you join the weekly (or whatever scheduling) meeting where stuff is discussed in an online meeting, where you can push your use case, etc. There should be a link for joining somewhere in the docs.

[michallenc]

It should have been discussed before implementation, but I suggest you join the weekly (or whatever scheduling) meeting where stuff is discussed in an online meeting, where you can push your use case, etc. There should be a link for joining somewhere in the docs.

It actually was discussed before the implementation on the mailing list which (according to the documentation) is the main communication channel for MCUboot.

[d3zd3z]

So a few comments. As far as swap move requiring the same sector sizes. This needs to be fixed. I believe the correct fix is to be able to configure, somewhere, to tell mcuboot to use a sector size that is a multiple of the device sector size. Related to this, we really should get rid of all of the silliness around returning the detailed sector map. If mcuboot is running on a partition with different sector sizes within it, it should treat the whole thing as just having sectors the size of the largest sector.

So, the simplification would be: 1. mcuboot internally works with a single sector size across any partition. 2. It can adjust that upward to use the larger size when the devices differ. 3. Eventually, the flash api should be simplified to just pass a single sector size instead of a large table that doesn't help us in any way.

As far as this code goes. I agree with @utzig on this. If we do want to make use of external code, configure a large scratch area (the size of the image). The swap scratch algorithm should then just perform the 3 way copy like this algorithm does, at least as I understand it. It also uses code that gets tested.

In addition: this commit mentions being tested on NuttX, which suggests to me that NuttX is still active. I was planning on proposing removing targets that don't have active CI tests to at least build them, since other devs have no way of knowing if their change would break them. If there is still interest in NuttX, can someone please make a workflow that actually tries building it?

Related, we can't have new code, especially algorithmic code like this, that doesn't exercise the functionality in the simulator. A simple "did it work" test on a target is completely inadequate to know if the upgrades are robust, even if they are fairly simple, there are still plenty of edge cases that can be missed.