Input queue of Decoder - size has exceeded the threshold #1434
Unanswered
rebound-software
asked this question in
Help
Replies: 4 comments 1 reply
-
|
That log means that the Nvidia hardware decoder (NVDEC) is not operating at 1x speed. Perhaps the stream is getting slower. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I start to get these messages once I have a reasonable load on OME. Once I get to 9 camera feeds, it starts to appear. Before this, all is fine. I see the queues gradually get larger and larger. If I then drop one of the camera feeds, after the stream drops due to timing out, the queues gradually start to empty until the messages disappear again. nvtop shows about 60% load on the GPU decoder and CPU core are all around 30%. I've also noticed that the WebRTC preview window start to lag behind. Presumably that's due to the queues gradually filling? It seems to me like there is something causing an in balance between the rate that InboundWorker is filling the queue and OutboundWorker is emptying the queue? I don't think this is a problem with the RTSP camera feeds as I have 3 different camera models and all work fine until the system starts to get loaded. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I've conducted quite a lot of testing around this issue. It's pretty clear I'm hitting some kind of bottleneck and it is not the stream slowing down. The CPU load remains well within the system capacity sitting around 30% on all cores (i7 with 4 cores / 8 threads). The GPU load is about 50% with 5 streams active so seemingly plenty of overhead. However, if I add an extra stream after this point, the queue overrun messages start to appear. The more streams I add, the quicker the queues increase in size. If I remove streams, the queue gradually reduces in size until it returns to within the threshold and messages disappear. Does anyone have any insights into what limitation I might be hitting? Or is there some fundamental bottleneck in OME? 
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
IIRC GPU utilization is a very crude number and there are a number of sub
blocks within the GPU that can individually be saturated. Specifically many
GPU can only transcode a certain number of streams, so that's my guess.
Hopefully at least a good starting point for more searching
…On Tue, Nov 14, 2023 at 02:00 Mark Wild ***@***.***> wrote:
I've conducted quite a lot of testing around this issue.
It's pretty clear I'm hitting some kind of bottleneck and it is not the
stream slowing down.
The CPU load remains well within the system capacity sitting around 30% on
all cores (i7 with 4 cores / 8 threads).
The GPU load is about 50% with 5 streams active so seemingly plenty of
overhead. However, if I add an extra stream after this point, the queue
overrun messages start to appear. The more streams I add, the quicker the
queues increase in size. If I remove streams, the queue gradually reduces
in size until it returns to within the threshold and messages disappear.
Does anyone have any insights into what limitation I might be hitting? Or
is there some fundamental bottleneck in OME?
[image: image]
<https://user-images.githubusercontent.com/37187745/282547287-dcbb9e78-068f-44ae-ab51-3af644b393b4.png> [image:
image]
<https://user-images.githubusercontent.com/37187745/282547091-f313888a-f86c-4cbd-a8b5-4b291143d4c1.png>
—
Reply to this email directly, view it on GitHub
<#1434 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABV4BWMVJ7UBL6RX6S24RQDYEJN27AVCNFSM6AAAAAA7AKKVG2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TKNJXGA4TM>
.
You are receiving this because you are subscribed to this thread.Message
ID: <AirenSoft/OvenMediaEngine/repo-discussions/1434/comments/7557096@
github.com>
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks for the suggestion. I don't think it's the GPU that is the limiting factor. I have 2 systems setup. One with a Ryzen 5 and 1650 super, and the other with an i7 and 1060. I've swapped the GPUs over and when either GPU is in the i7 system, the queue starts to overrun at more or less the same point and well before the point when the problem occurs in the Ryzen 5 system with exactly the same setup. This definitely points the finger at the CPU side IMO. It feels to me like a realtime problem where something can't be serviced quickly enough on the CPU side. I'm not seeing a crash but I suspect that's because I'm not leaving it long enough to fill up memory to the point where things break which could be a few hours or more. Might this explain some of the long-run crashes that have been reported? It does explain an odd problem I noticed when operating in a production system where WebRTC feeds appear to lag behind current time, gradually getting later and later. @getroot @Keukhan Is there any debug I can turn on to pinpoint the cause of my issue ? Also, is there anything in the config I can use to control the number of streams the GPU takes on? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
What is the cause of these messages I'm seeing in my log file (using hw accel with Nvidia GPU):
[2023-11-06 23:35:24.143] W [InboundWorker:433632] ov.Queue | queue.h:268 | [0x7f651a875918] Input queue of Decoder. track(0) codec(h264/27) size has exceeded the threshold: queue: 1051, threshold: 120, peak: 1051I have monitored the thread usage of OME (top -H -p ) and I have 3 Dech264NV threads showing ~19% CPU. Using the system resource monitor(20.04 Ubuntu) shows the overall CPU usage is around 30% and each core never exceeds 40% (Hex Core, so 12 "cpus").
Nothing seems to be hitting any obvious resource limit so I'm struggling to understand what the limit is that's being hit to cause these messages?
Also, what are the problems this will cause? I don't seem to be seeing visible dropouts in the stream or obvious problems.
Beta Was this translation helpful? Give feedback.
All reactions