Automated pipeline for processing images with SAM

MRob, Laboratory of Mobile Robotics, Skoltech

Team #3 Project

Team members:

• Maksim Alymov ;
• German Devchich ;
• Denis Fatykhoph ;
• Maksim Smirnov

The key idea of this project is to create an automated pipeline for processing images from a dataset, consisting of numerous pictures extracted from videos recorded in various outdoor and indoor environments. We utilize available odometry measurements to project human steps onto the ground. These projected points are then used as input for the Segment Anything Model (SAM). For more information on SAM, you can refer to the original repository . Below are provided examples of model results.

Getting Started

Easy start

1. You need to clone this repo

git clone https://github.com/Fatyhich/sk_fse_pr.git

2. Then you need to build image with our project

cd sk_fse_pr
docker build . -t <your name of image>

If there are no errors in terminal, that means that all requirements for our project has been successfully downloaded and builded. Additionally, pretained weights for SAM has been downloaded. They stores in container in according folder.

3. Starting container

docker run -it -v <place_of_dataset>:/mnt/data <your name of image>:latest

We separated the pipeline on 3 stages. Inside container you can rub them separately or by one-command.

Each stage description:

• Preprocessing Stage We are doing sampling several images from hole dataset and store them in other folder named preprocess_data/frames

To run this stage from local machine:

docker run -it -v <place_of_dataset>:/mnt/data <your name of image>:latest make preprocess 

• Processing Stage On this step we are generate masks for each picture from folder preprocess_data/frames using SAM. If masks was generated, they stored in new folder preprocess_data/masks_area

To run this stage from local machine:

docker run -it -v <place_of_dataset>:/mnt/data <your name of image>:latest make process 

• Postprocessing Stage On the last step we applying generated masks on each image, according on to other. And as output we receive folder output, that consist of final images.

To run this stage from local machine:

docker run -it -v <place_of_dataset>:/mnt/data <your name of image>:latest make postprocess 

Also, you can run all this steps with one command from your local machine terminal:

docker run -it -v <place_of_dataset>:/mnt/data <your name of image>:latest make all

If you want to remove processed data, here is command available:

docker run -it -v <place_of_dataset>:/mnt/data <your name of image>:latest make remove

Test

Despite the fact that tests are already checked and done on image's building stage, you can run them by the following commands:

python3 -m unittest -v test/test_<target_stage>.py

Attention! You need run them from inside of started container. If you want to run test from your local machine, without connecting to container:

docker run -it -v <place_of_dataset>:/mnt/data <your name of image>:latest python3 -m unittest -v test/test_<target_stage>.py

Video with Demo

Cloning and Building process

Demonstrating all steps with make

Finalization

Dataset Storage

The dataset used in this project is also stored using Git LFS (Large File Storage) in a separate branch. This allows for efficient storage and versioning of the dataset, while keeping the main codebase lightweight.

To access the dataset in Git LFS format and operate with small-sized repo, simply checkout the lfs_dataaset branch:

git checkout lfs_dataaset

License

The model is licensed under the Apache 2.0 license.