Working with BIG Videos

[jmdelahanty]

Hello Sleapyheads!

I'm in possession of some HUGE videos of several zebrafish swimming in a dish (5-12). They're recorded from some super cool Emergent Technologies Cameras and written using HEVC at runtime with an awesome computer. The dimensions are 6400x6400. My approximately 40 minute video is 40GB and this is apparently at 100x compression(!!!). It's very big!

Here's an example when I seek past the first frame (which displays normally):

I'm reading this from disk locally (nvme) and things like VLC mediaplayer seem to be working well. I'm also greeted with this in my terminal:

Software versions:
SLEAP: 1.3.3
TensorFlow: 2.7.0
Numpy: 1.19.5
Python: 3.7.12
OS: Linux-6.9.3-76060903-generic-x86_64-with-debian-bookworm-sid

Happy SLEAPing! :)
[hevc @ 0x5fff96b38740] Could not find ref with POC 3
[hevc @ 0x5fff96b38740] Could not find ref with POC 2
[hevc @ 0x5fff96b38740] Could not find ref with POC 1
[hevc @ 0x5fff96b38740] Could not find ref with POC 0
[hevc @ 0x5fff96b38740] Could not find ref with POC 10
[hevc @ 0x5fff96b38740] Could not find ref with POC 9
[hevc @ 0x5fff96b38740] Could not find ref with POC 8
[hevc @ 0x5fff96b38740] Could not find ref with POC 7
[hevc @ 0x5fff96b38740] Could not find ref with POC 25
[hevc @ 0x5fff96b38740] Could not find ref with POC 24
[hevc @ 0x5fff96b38740] Could not find ref with POC 23
[hevc @ 0x5fff96b38740] Could not find ref with POC 22

It's not an urgent request to solve this, but it would be super cool to see how people handle labeling/training on/working with these HUGE (and just incredibly cool resolution that lets you see SO MUCH) videos might be handled. Any tips or advice?

[talmo]

Hey @jmdelahanty,

The issue here is the video encoding. HEVC (aka H.265) support is provided by opencv-python versions built with the HEVC dependencies, which is only available in newer versions. From your logs it looks like it might have the HEVC plugin, but maybe it's an older/experimental version?

Can you try the v1.4.1a2 pre-release? It pulls in a newer opencv version than SLEAP v1.3.3 and might support it now.

For other folks running into this: a workaround is also to reencode to H264 instead. The problem is that it has a max resolution limit of 2048 x 2048, so it won't work for large image sizes unfortunately.

Let us know if the pre-release one is able to decode the video properly!

Cheers,

Talmo

[jmdelahanty]

Downloading now! Will update...

[jmdelahanty]

It looks like so long as you don't seek too far from your current frame or seek between frames too quickly (like hitting left arrow very very quickly) it's decoding nicely! I notice if I try to go somewhere random in the video it'll show me that gray screen with more of these referencing errors.

[jmdelahanty]

A different tool this lab uses for labeling that's kinda cool had me install the following driver/ffmpeg combo:

wget https://developer.download.nvidia.com/compute/cuda/12.5.1/local_installers/cuda_12.5.1_555.42.06_linux.run
sudo sh cuda_12.5.1_555.42.06_linux.run

(base) jmdelahanty@shadow:~/nvidia/ffmpeg$ git branch
  master
* release/4.4

Note that this ffmpeg isn't installed in my conda environment.