Introduction

This work was done in an internship at Lightricks in collaboration with YDATA. It was selected as the one of the top 3 out of 15 other industry internships.

The tasks were:

• Basic - Classify action in a video of arbitrary length.
• Strech goal 1 - Classify subsequent actions in a video of arbitrary length.
• Strech goal 2 - Classify multiple subsequent actions in a video of arbitrary length. (Another repo)

The solutions must be lightweight and run at realtime or better.

We chose to compare models by their performance on kinetics 600. We ended up using the pytorch implementation of MoViNets.

Benefits of the MoViNets Stream Buffers :

• Allow the usage of constant memory at inference time.
• Takes into account longer temporal relationships.

Demo webapp (Streamlit)

To install:

git clone https://github.com/bf2harven/Lightricks_all_video.git
cd Lightricks_all_video
conda env create -f environment.yml
conda activate lightricks_env

To run a streamlit webapp of our model:

streamlit run main.py

In the webapp:

• You can select the granularity of the classification. (How many classes)
• You can select a file to run inference on.
• You can select how often the buffer is resetted.
  If this is too short - there's not enough information to make a good classification.
  If this is too long - There will be a lag in detection of new actions. Can even miss short actions.