Slow start even though validate primary slot is disabled?

[Olstyle]

I am using mcuboot bare metal on an SM32U5 (from internal flash, running at 48MHz) and am a bit surprised how long a boot_go() takes even if I have validation disabled.
In pictures (the application was started when the signal is pulled down for a short time):
No validation

Including validation

Bypassing boot_go() by jumping directly to the application in Slot0

What the heck is mcuboot doing for 600ms when its is "not validating"? I expected it to read some headers and trailers for maybe single digit milliseconds, but what I am seeing is excessively more.