Image panel stuck on black screen if stream starts with a non-keyframe

[AlexKirko]

The current version of Foxglove Studio has a bug.

Bug description and reproduction

Start streaming H.264 CompressedVideo via a Foxglove Websocket server. Connect the Studio to the client. If the first video frame does not happen to be a keyframe by accident, the screen will stay black forever.

This does not happen with MCAP playback, where the video starts playing once the first keyframe gets played.

This is extremely annoying, because you need to have the studio pre-opened and pre-configured, before you start streaming. If you don't know the names of the topics in advance, then it's simply impossible to reliably get a video playing on the first try.

Expected behavior

The video appears once a key frame is reached, same as for MCAP playback.

[defunctzombie]

It would help if you could make an mcap recording of the server data where the first messages on the topic are not keyframes.

You can record to an mcap from your existing websocket server with this one-liner (if you have nodejs installed):

npx @foxglove/ws-protocol-examples@latest mcap-record ws://localhost:8765 -o <file.mcap>

[snosenzo]

@AlexKirko I was not able to reproduce using npx @foxglove/ws-protocol-examples@latest mcap-play with an mcap containing
a CompressedVideo topic.

Are you able to provide example data that reproduces this behavior? If not, could you tell us what error(s) you're getting in the image panel and/or the browser console? Does this happen with local data as well? Could you tell us a bit more about the nature of your data? How frequently are you sending a keyframe?

[AlexKirko]

@snosenzo , this makes sense, as MCAP does not have this bug, only Websocket streaming does. I was on vacation, so I was not able to produce a fail for you. I will provide an MCAP with the same data that's causing the problem when streaming by the end of the week.

This is of limited use, however, since MCAP resets correctly on a keyframe, while the streaming does not.

[jtbandes]

FWIW, the npx @foxglove/ws-protocol-examples@latest mcap-play which Sam mentioned plays back an mcap file over a websocket connection, so on the Foxglove side the app sees only a websocket. This should make it possible to reproduce the issue. You can try it out with instructions here: https://www.npmjs.com/package/@foxglove/ws-protocol-examples

[AlexKirko]

This is very strange. I have tried skipping a bunch of key frames, but I'm not getting this bug anymore when streaming from mcap with mcap-play. Maybe due to the recent video processing update that you guys had?

Unfortunately, the exact multi-camera setup that produced the bug is in another city, so give me a few days to recruit some colleagues to check if we are still experiencing the bug. If yes, then I'll have to use mcap-record to create a proper dump from that setup.

[AlexKirko]

The colleagues will be able to test tomorrow.

[snosenzo]

Any updates on this?

[AlexKirko]

My colleagues got tied up in meetings, so they couldn't confirm whether the black screen bug is still a problem. We'll be able to test and post the results next week. Please keep the ticket open.

Also, thanks for letting me know about the mcap-record utility. It should be very useful to us outside of debugging.

[AlexKirko]

That will teach me to promise things. Got pretty sick this week, so now the failure to produce a file is on my end, sorry. Will definitely get to it next week.

[AlexKirko]

Okay, I have finally been able to produce a couple files for you. One results in a black screen Image panel with 'waiting for key frame' forever for me, and the other is fine. The only difference is that, for the first one, I ran the server first and then ran the record utility, and for the non-broken one, I did the opposite.

Since you guys do not support attaching an .mcap here, I'm putting the files into .zip archives.

[jtbandes]

Since you guys do not support attaching an .mcap here, I'm putting the files into .zip archives.

Unfortunately this is a GitHub limitation 😢 (docs, discussion)

[AlexKirko]

missing_key_frame.zip
no_missing_key_frame.zip

[jtbandes]

Testing these files:

• I see that "no_missing_key_frame" indeed has a keyframe at the beginning and plays correctly.
• It appears that "missing_key_frame" does not have a keyframe until the very end of the file (around 45.8 seconds elapsed). It also appears the key frame is missing the SPS, since I see this error message:
  

Could you try changing your encoder to include SPS in every key frame?

It is expected behavior that the video will not display until a key frame is reached – this is the nature of video compression. If you would like to be able to start playback from an arbitrary point, you could consider configuring the encoder/publisher to add key frames every so often (for example every 2 seconds) and then you won't have to wait very long before the video starts playing. If your video encoder has >45 seconds between keyframes, as in the example data you shared, then the user might have to wait that long before they see video.

[AlexKirko]

The key frame rate is a bug with this particular dataset, normally we produce one for every 10 frames.

The lacking SPS, however, is probably the reason why we are observing this behavior. It must be getting broadcast at the start of the stream and never again, s oif the client joins in the middle, we run into trouble.

[snosenzo]

Are you able to add the SPS to the keyframes that are missing it? If you do end up getting this working please share and we might consider adding any steps you took to our docs for video troubleshooting if it seems like others might run into the issue as well.

Also since this seems to be an issue with the dataset, would you say that we can close this discussion?

[AlexKirko]

I have added the SPS data to key frames, and the issue is now fixed. I can join mid-stream, and the decoding starts. Thank you.