Buffer stalls with SRT + LLHLS

[SceneCityDev]

Hi getroot,

I think I need a helping hand here.

I have a HEVC stream coming in to the origin via SRT, and then getting published as LLHLS on an Edge. My Chrome browsers supports HEVC playback. In theory all works fine.

But no matter what I try, the LLHLS stream is getting buffer stalls all the time. It's not a matter of not having enough bandwidth or CPU on the Source, Origin, Edge or Client PC - it happens on all computers I have tried. Even on a 1 GBit/s connection with a 2 MBit/s stream it happens. I tried both OBS as a source, and by having ffmpeg deliver a TS as an alternative. And even if I still have a buffer of a full minute (!) ahead of me in the client, every couple of seconds there will be a buffer stall. I looked at it in detail in hls.js'es debugger, too. I don't get it.

I tried your recommended settings for Chunk Duration, Segment Duration, Segment Count and the mysterious Part Hold Back, and tried hundreds of other combinations. But even with Chunk Duration(10.00) Segment Duration(30) Segment Count(50): Buffer stalls.

If I play the stream in VLC, it works perfectly - but of course VLC does not do LLHLS. So I know HLS fallback works. Telling Ovenplayer or the server to treat this as HLS and not LLHLS however doesn't.

I get the feeling that somehow OME is delivering segments to the client that are there, but in some why not usable. Maybe chunks are marked Independent while they aren't, or they are re-ordered.

There is a warning in the log popping up every couple of seconds, and I find it plausible that it might be related:

[2024-03-23 10:22:10.413] W [AW-LLHLS0:1144] FMP4 Packager | fmp4_storage.cpp:341 | LLHLS stream (#default#origin2/hevctest) / track (1) - a longer-than-expected (8383.3 | expected : 6000) segment has created. Long or irregular intervals between keyframes might be the cause.
Of course I have 1 second keyframes.

So I tried increasing the segment size. When I increased to 10000 OME decided to now create 12000+ sized frames. I went all the way up to...

[2024-03-23 10:13:40.425] W [AW-LLHLS0:225167] FMP4 Packager | fmp4_storage.cpp:341 | LLHLS stream (#default#origin2/hevctest) / track (4) - a longer-than-expected (39808.0 | expected : 30000) segment has created. Long or irregular intervals between keyframes might be the cause.

...when it became very clear that no matter what size I choose, the generated ones will always be bigger.

The only thing that made things somewhat (but not perfectly) stable was to have ChunkDuration = SegmentDuration, so basically killing any option for the LLHLS client to try to range requests.

I have reset the config to your recommended values:

<PartHoldBack>1.5</PartHoldBack>
<ChunkDuration>0.5</ChunkDuration>
<SegmentDuration>6</SegmentDuration>
<SegmentCount>10</SegmentCount>

A test stream showing the problem is available here:

Origin:
HEVC + AAC Passthrough: https://origin2.scenecity.net/origin2/hevctest/hevc-bypass_llhls.m3u8
HEVC transcoded (QSV) + AAC Passthrough: https://origin2.scenecity.net/origin2/hevctest/hevc-hq_llhls.m3u8
HEVC rescaled, transcoded (QSV) + AAC Re-encoded: https://origin2.scenecity.net/origin2/hevctest/hevc-lq_llhls.m3u8

And the same on the Edge:
https://fsn1-edge3.scenecity.net/origin2/hevctest/hevc-bypass_llhls.m3u8
https://fsn1-edge3.scenecity.net/origin2/hevctest/hevc-hq_llhls.m3u8
https://fsn1-edge3.scenecity.net/origin2/hevctest/hevc-lq_llhls.m3u8

Again: All of these work fine when played in VLC, but none works without those buffer stalls with Ovenplayer or the llhls.js debugger.

Any help would be highly appreciated. It would also be no problem to send the srt test stream to an origin of yours, of course.

It's weird that you prefer LLHLS over HLS. Reading through the protocol during the last night made by brain bleed. What a complex monster of a stream transport protocol...

Thank you!

br,
scamp

[bchah]

+1, I've seen this behaviour too. It helped a bit adding the secret <PartHoldBack> tag to the LLHLS config, for how many seconds of segments you want to cache - I set it to 2.0

[bchah]

A couple other ideas - make sure HLS.js is up to date and check that your keyframe interval divides evenly into your segment duration. I usually use 2s keyframe interval.

[SceneCityDev]

I've tried multiple hls.js versions.

But both the default segment duration of 6, and the other values I've tried divide evenly to 1/2 keyframe interval....

[SceneCityDev]

As you said you have seen this before - do you now have a combination that works for you?

[getroot]

I looked at https://origin2.scenecity.net/origin2/hevctest/hevc-bypass_llhls.m3u8, which shows that no new chunks are being created. The input signal may no longer be coming in, or there may be another problem.

Buffer stalls can have many causes.

1. Because LLHLS is played by the player as soon as a chunk is created, buffering may occur if network jitter is severe. This can be resolved to some extent with PartHoldBack as bchah said.

2. Chunk creation may be slower than the wall clock. If OME's system resources are sufficient, you can assume that the input signal may not be fast enough or there may be a drop on the input side. If you are using OBS, make sure there are no drop frames.

There are many other possibilities, but I think the priority is to check whether the input signal is normal. Check whether the HEVC encoder is running fast enough in OBS.

(And sorry, this is a public community, so please don't mention me specifically.)

[SceneCityDev]

Getroot: Sorry to have mentioned you.

The stream was gone to due an OME crash, and now due to my tests. The test live stream is running again now.

Thank you once more for the recommendations. I have now found TWO issues in my setup:

1.) In my ffmpeg based test, ffmpeg wasn't passing my parameters to libx265. Due to this x265's scenecut was enabled, which may generate random keyframes in wrong places. WIth -x265-params "keyint=60:min-keyint=60:scenecut=0" I now get regular key frames. Transcoding some videos with that option, then will test-stream again with it.

2.) It's a mistake I made again and again: With my OBS-based live stream test, I passed latency=1000 to libsrt. But the parameter is in MICROseconds, not miliseconds. Due to this there had been reordered/missing packets. Stupid me.

What is weird: The buffer stalls happen both on the path-through stream AND the transcodes. I would assume that for the transcodes the original keyframes don't matter, as you will set these in the transcoder. So I wonder why the buffer stalls also happen with the transcodes..

Right now I am running with ChunkDuration=SegmentDuration and that appears to be stable.

Can you give me a recommendation of LLHLS parameters that have a focus on stability instead of low-latency? I am happy with a latency of 10 seconds or so.

[getroot]

Try adding ?_HLS_legacy="yes" querystring after the LLHLS URL. It responds with m3u8 excluding partial segments, so it can be played with legacy HLS.

Alternatively, it is a good idea to keep ChunkDuration at 0.5 and set PartHoldBack as large as 5.0.

[getroot]

I don't yet know if the QSV encoder handles the H265 keyframe interval strictly. For reference, OpenH264 is very poor at accurately processing keyframe intervals (almost impossible).

[getroot]

x264, xilinx, and nvenc do it very well.

[SceneCityDev]

So, I have finally solved the problem:

The missing magic ingredient for ffmpeg had been -pes_payload_size 0. I found that in OME's documentation for MPEG TS inputs.

But it turns out that this also applies to SRT.

I would recommend to update the documentation for SRT that if you want to feed a live stream via ffmpeg that -pes_payload_size 0 must be added.

In case anyone cares, here is an example that is randomly selecting videos from a folder (the videos must be in MPEG TS format!) and streaming them. Due to a pipe being used there is no reconnect between the different videos, so you get a continuous stream.

(filename streamts.sh)

#!/bin/bash
(for j in `ls /home/YOUR_VIDEO_DIRECTORY/*.ts | shuf`
do
        echo $j
    ffmpeg -f mpegts -i $j \
        -c:v copy -c:a copy \
        -f mpegts -
done) | ffmpeg -f mpegts -re -i - -c:v copy -c:a copy -pes_payload_size 0 -f mpegts "srt://YOUR_SRT_URL"

For the stream to recover in case ffmpeg quits with an error, I have a wrapper script that is calling the above script:

#!/bin/bash
while true;
do
  ./streamts.sh
  sleep 3
  echo "Restarting stream..."
done

Once again thanks to all who have helped out here.

[bchah]

Cool workflow! Have a look at ffplayout which wraps this concept into a server app and uses Python watchdog to detect changes to the directory listing on the fly.