Error with encoded frames when using setIsp3aFps

[jackforrester03]

I am currently trying to save encoded frames from all 3 cameras of an Oak D Lite, at 30 FPS@1080p and 90 FPS@400p for the colour and mono cameras respectively. I am using a python script very similar to the Encoding Max Limit example. After recording some videos, it was clear that frames were being dropped and not being received on the host (found by plotting sequence against timestamps). After a bit of investigation it was due to the CPU usage being >95%.

Using the guidance given in the docs I have looked to reduce the FPS of the 3A algorithms, using the 'setIsp3aFps' function. I have tried this on all 3 cameras individually and in various combinations and various FPS. Whilst this does reduce the CPU usage, this still results in some (however fewer) dropped frames but the camera in which the setIsp3aFps is set on, stops sending frames after a few seconds.

For example, I have set setIsp3aFps to 30FPS for both mono cameras (colour camera not set) and recorded for ~50s. The colour camera recorded as expected but the mono cameras only recorded for 3.8 and 3.9s for the left and right respectively, after which no frames were received.

Is there something I am doing wrong or is this a bug?

The full script I have used it below:

#!/usr/bin/env python3

import depthai as dai
import matplotlib.pyplot as plt
import numpy as np
import time

# Create pipeline
pipeline = dai.Pipeline()

# Define sources and outputs
camRgb = pipeline.create(dai.node.ColorCamera)
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
controlIn = pipeline.create(dai.node.XLinkIn)
controlIn.setStreamName('control')

ve1 = pipeline.create(dai.node.VideoEncoder)
ve2 = pipeline.create(dai.node.VideoEncoder)
ve3 = pipeline.create(dai.node.VideoEncoder)

ve1Out = pipeline.create(dai.node.XLinkOut)
ve2Out = pipeline.create(dai.node.XLinkOut)
ve3Out = pipeline.create(dai.node.XLinkOut)

ve1Out.setStreamName('ve1Out')
ve2Out.setStreamName('ve2Out')
ve3Out.setStreamName('ve3Out')

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setFps(30)
# camRgb.setIsp3aFps(5)

monoLeft.setCamera("left")
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setIsp3aFps(30)
monoLeft.setFps(90)

monoRight.setCamera("right")
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setIsp3aFps(30)
monoRight.setFps(90)

# Link to control
controlIn.out.link(monoRight.inputControl)
controlIn.out.link(monoLeft.inputControl)

# Setting to 26fps will trigger error
ve1.setDefaultProfilePreset(90, dai.VideoEncoderProperties.Profile.H264_MAIN)
ve2.setDefaultProfilePreset(30, dai.VideoEncoderProperties.Profile.H265_MAIN)
ve3.setDefaultProfilePreset(90, dai.VideoEncoderProperties.Profile.H264_MAIN)

# Linking
monoLeft.out.link(ve1.input)
camRgb.video.link(ve2.input)
monoRight.out.link(ve3.input)

ve1.bitstream.link(ve1Out.input)
ve2.bitstream.link(ve2Out.input)
ve3.bitstream.link(ve3Out.input)

# exposure time:     I   O      1..33000 [us]
# sensitivity iso:   K   L    100..1600
ctrl = dai.CameraControl()
ctrl.setManualExposure(5000, 1200)
m1 = []
m2 = []
c = []
M1 = []
M2 = []
C = []

# Connect to device and start pipeline
with dai.Device(pipeline) as dev:
    dev.setLogLevel(dai.LogLevel.DEBUG)
    dev.setLogOutputLevel(dai.LogLevel.DEBUG)
    controlQueue = dev.getInputQueue(controlIn.getStreamName())
    controlQueue.send(ctrl)

    # Output queues will be used to get the encoded data from the output defined above
    outQ1 = dev.getOutputQueue('ve1Out', maxSize=30, blocking=False)
    outQ2 = dev.getOutputQueue('ve2Out', maxSize=30, blocking=False)
    outQ3 = dev.getOutputQueue('ve3Out', maxSize=30, blocking=False)

    initM1Time = 0
    initM2Time = 0
    initCTime = 0
    
    # Processing loop
    with open('mono1.h264', 'wb') as fileMono1H264, open('color.h264', 'wb') as fileColorH265, open('mono2.h264', 'wb') as fileMono2H264:
        print("Press Ctrl+C to stop encoding...")
        start = time.time()
        while True:
            try:
                # Empty each queue
                while outQ1.has():
                    OUT = outQ1.get()
                    ts = OUT.getTimestamp()
                    if initM1Time == 0:
                        initM1Time = ts.seconds + ts.microseconds/1e6
                        break
                    m1.append((ts.seconds + ts.microseconds/1e6) - initM1Time)
                    M1.append(OUT.getSequenceNum())
                    OUT.getData().tofile(fileMono1H264)

                while outQ2.has():
                    OUT = outQ2.get()
                    ts = OUT.getTimestamp()
                    if initCTime == 0:
                        initCTime = ts.seconds + ts.microseconds/1e6
                    c.append((ts.seconds + ts.microseconds/1e6) - initCTime)
                    C.append(OUT.getSequenceNum())
                    OUT.getData().tofile(fileColorH265)

                while outQ3.has():
                    OUT = outQ3.get()
                    ts = OUT.getTimestamp()
                    if initM2Time == 0:
                        initM2Time = ts.seconds + ts.microseconds/1e6
                    m2.append((ts.seconds + ts.microseconds/1e6) - initM2Time)
                    M2.append(OUT.getSequenceNum())
                    OUT.getData().tofile(fileMono2H264)

            except KeyboardInterrupt:
                break

    # output_container.close()
    print("To view the encoded data, convert the stream file (.h264/.h265) into a video file (.mp4), using commands below:")
    cmd = "ffmpeg -framerate 90 -i {} -c copy {}"
    print(cmd.format("mono1.h264", "mono1.mp4"))
    print(cmd.format("mono2.h264", "mono2.mp4"))
    cmd = "ffmpeg -framerate 30 -i {} -c copy {}"
    print(cmd.format("color.h264", "color.mp4"))

print(f"Cam1 SEQNumb: {max(M1)} frames: {len(M1)}")
print(f"Cam2 SEQNumb: {max(M2)} frames: {len(M2)}")
print(f"ColorCam SEQNumb: {max(C)} frames: {len(C)}")

fig, ax = plt.subplots(1) #, sharex=True,
ax.scatter(M1,m1)
ax.scatter(M2,m2)
ax.scatter(C,c)

plt.show()    

Using M1 MacBook Air and depthai-2.24.0.0.

[jakaskerl]

Hi @jackforrester03
Try timing your host side loop. I think that your MacBook is unable to write frames that fast.

[jackforrester03]

Hi just did a quick test to check the loop times and I'm not sure that is the problem.

The first images shows a plot of sequence number vs timestamp (normalised to the timestamp of the first frame). The green dots showing the colour camera frames at 30fps and the orange dots show the left and right cams frames at 90FPS with both mono cameras set with setIsp3aFps(15). This shows the issue, as the orange dots stop after ~4.3s

The second plot shows the loop timestamp (again normalised to the loop start time) vs loop duration time. The worst case being 0.0056s which is faster than 0.0111 (90fps) so this doesn't seem long loop times are the issue.

Couple of additional things to notice is that after 4.3s the average loop time is reduced, this seems consistent with no frames being received by the host. Also, the CPU usage starts at around 45% for the first few seconds and then reduces to 17% usage, this seems to show that no frames are being received by the ISP after a few seconds (probably after ~4.3s).

Finally, this issue does not happen with the mono cams set to 60FPS, this only seems to occur at higher FPS.

Thanks for the suggestion.

[pgmurphy]

We are also experiencing the same issue with setIsp3aFps().

@jackforrester03, have you found any resolution?

[Erol444]

Hi @pgmurphy @jackforrester03 ,
Unfortunately, we found out the setIsp3aFps isn't as stable as we initially thought. So we will likely deprecate/ignore it until we can make it stable. We apologize for the inconvenience.