YUV420 on main slows output on Pi3/Zero2

[mgineer85]

Hey @davidplowman , to save memory on the Pi Zero platform I would like to use YUV420 for lores and main stream.
When doing so, I noticed some slowdown in frames received.

For testing I modified the frame_server example:

cam_config=picam2.create_still_configuration(lores={"format": "YUV420"})
picam2.configure(cam_config)

before starting camera:

pi@wigglecam-dev3:~ $ python frame_server_yuv.py
[136:51:34.768875526] [10455]  INFO Camera camera_manager.cpp:325 libcamera v0.3.2+27-7330f29b
[136:51:34.825228438] [10458]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[136:51:34.830409697] [10458]  INFO RPI vc4.cpp:447 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media3 and ISP device /dev/media0
[136:51:34.830546989] [10458]  INFO RPI pipeline_base.cpp:1126 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[136:51:34.848321811] [10455]  INFO Camera camera.cpp:1197 configuring streams: (0) 4056x3040-BGR888 (1) 4056x3040-YUV420 (2) 4056x3040-SBGGR12_CSI2P
[136:51:34.849086969] [10458]  INFO RPI vc4.cpp:622 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-pBCC
Thread1 received 11 frames
Thread2 received 11 frames
Server received 12 frames

And to compare with main also set to YUV420:

cam_config=picam2.create_still_configuration(main={"format": "YUV420"},lores={"format": "YUV420"})
picam2.configure(cam_config)

before starting camera:

pi@wigglecam-dev3:~ $ python frame_server_yuv.py
[136:52:46.031476199] [10470]  INFO Camera camera_manager.cpp:325 libcamera v0.3.2+27-7330f29b
[136:52:46.090679012] [10473]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[136:52:46.095824125] [10473]  INFO RPI vc4.cpp:447 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media3 and ISP device /dev/media0
[136:52:46.095997771] [10473]  INFO RPI pipeline_base.cpp:1126 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[136:52:46.114277282] [10470]  INFO Camera camera.cpp:1197 configuring streams: (0) 4056x3040-YUV420 (1) 4056x3040-YUV420 (2) 4056x3040-SBGGR12_CSI2P
[136:52:46.115052231] [10473]  INFO RPI vc4.cpp:622 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-pBCC
Thread1 received 6 frames
Thread2 received 6 frames
Server received 7 frames

I would not expect it do slow down, I think I did not notice this on a Pi5. Is it something about the RAW stream that seems slightly different also and involves any additional conversions on Pi3 which takes time?

[davidplowman]

On vc4 (i.e. non-Pi 5) platforms there is some colour denoise that runs in software, and I wonder if that's slowing down the framerate. On Pi 5, colour denoise is implemented in hardware and runs at the maximum pipeline rate. I think that software denoise is only support for YUV, so it probably doesn't run on RGB outputs.

To stop the colour denoise try this:

import libcamera
main = ={"format": "YUV420"},lores={"format": "YUV420"}
controls = {"NoiseReductionMode": libcamera.controls.draft.NoiseReductionModeEnum.Minimal}
cam_config=picam2.create_still_configuration(main, controls=controls)
picam2.configure(cam_config)

Minimal should give you the full pipeline rate. You could also try Fast (faster but lower quality software denoise).

[mgineer85]

Oh, I was not aware about that. I’ll dig into that and post here again

[mgineer85]

So

controls = {"NoiseReductionMode": libcamera.controls.draft.NoiseReductionModeEnum.NoiseReductionModeOff}
results in 16 frames from server

controls = {"NoiseReductionMode": libcamera.controls.draft.NoiseReductionModeEnum.Fast}
results in 15 frames from server

controls = {"NoiseReductionMode": libcamera.controls.draft.NoiseReductionModeEnum.NoiseReductionModeHighQuality }
results in 7 frames from server

controls = {"NoiseReductionMode": libcamera.controls.draft.NoiseReductionModeEnum.NoiseReductionModeMinimal}
results in 16 frames from server

controls = {"NoiseReductionMode": libcamera.controls.draft.NoiseReductionModeEnum.NoiseReductionModeZSL}
results in 7 frames from server

So, thank you, that confirmes it is about the denoise algorithm :)

But I wonder why Fast is giving just one frame less then Minimal or Off. Fast uses one additional frame internally for denoising over two frames? Just wonder because I want to sync several devices...

[davidplowman]

Off and Minimal should perform the same in terms of speed, because Minimal only enables the hardware spatial denoise block, which runs at the full rate (so no different to Off).

Fast uses a "fast" colour denoise software algorithm. It's designed to keep up at 1080p30, but it's possible that you could see occasional slippage if you're "unlucky". HighQuality is obviously much slower.

If you want to avoid dropping frames, you might want to increase the configuration's buffer_count. I think it defaults to just 1 for "still" configurations (in order to save memory), but it might be worth adding buffer_count=2 or even 3 (memory permitting - this is obviously where a Pi Zero might have trouble).

[mgineer85]

I already use 2 buffer to ensure constant flow when taking one image off the queue.
I think more is not possible on pi zero and also if it queues more frames the sync is in faulty state already …