Custom video frame processing

[hiroshihorie]

No description provided.

[bcherry]

This looks pretty simple which is great. is there any particular reason why the interface can't be on track instead of on capturer? that would more cleanly match how we do it on the other sdks

[hiroshihorie]

@bcherry Good point, will check other SDK interface and try to align 🙂

[hiroshihorie]

@bcherry createCameraTrack(processor:) is available now and also exposed LocalVideoTrack.processor property.
I wonder if I should add auto-frame dropping if processing frame hasn't completed 🤔

[bcherry]

thank you! lgtm.

What is the behavior if you do fail to process frames fast enough? It does sound like dropping frames would be the right behavior if we're still processing older ones. we'd want to have the same behavior across every SDK - does JS do anything like this @lukasIO ?

[lukasIO]

there's no timeout for frame processing in the JS processors we provide.
The functionality is not tied to the client SDK, but rather implemented in the processors themselves, so the behaviour might differ across (custom) processor implementations.

[bcherry]

if your input video is coming from the camera at 30fps but your processor takes longer than 1/30th of a second to process each frame, what will the users experience? It seems like we should do something here at the framework level to guide users into a happy path that just works?