Panorama and Image Stitching

In this project, our goal is to make panorama from first 900 frame of the original video below:

Panorama using five key frames

To produce a panorama using five key frames. we have some challenges :

• mapping from some frames to frame 450 is difficult because they share very little area. Therefore we need to perform a two step mapping , using frame (270,630) as a guide frames and then multiply two homographies to reach the final homography

• for stitching part, we use both dp(dynamic programming) optimization (minimum cut method) and after finding best boarder , use Laplacian pyramid blending two stitch these frames with the best result

if we just combine two frames without any optimization we have :

now if we apply dp optimization(minimum error cut) we have :

dp optimization

after detecting the minimum cut , if we combine two frames without any blending , differences between both sides of boarder is very clear and obvious . so we use Laplacian pyramid with 9 levels to combine frames very smoothly and naturally . below you can see the differences :

Before Laplacian pyramid	After Laplacian pyramid

we do the above procedure for each two frames that are next to each other with following condition : • we apply dp optimization for three boarders between two frame (vertical boarder , horizontal boarder that is on the top of page and horizontal boarder that is in the bottom of the page and make the mask based one the common areas that these three boarder make).below you can see the final result :

panorama of whole video

Create background panorama

in this section we remove moving objects from the video and create a background panorama .to do this, i divided the whole panorama image vertically into 230 small windows . for each small Windows, i iterate all over the frames and if the projection of frame into frame 450 includes that specific small windows, i save that part into the list . after gathering all of the proper frames that includes the small window , we get median from this list and set the small window to that median . the advantages of this method is that it’s very fast (took around 3 minutes to extract background)

background panorama

Create background movie

in this section we should map the background panorama to the movie coordinates. For each frame of the movie, we need to estimate a projection from the panorama to each frame.

Create forground movie

in this section we want to make a video from moving objects .in each frame we calculate the absolute difference (L1 norm) between original frame and background and set the threshold for these differences. if the differences in a pixel is bigger than this threshold , it can be pixel of moving object . To remove background noise we use two morphological transformation . first we use "opening"(erosion followed by dilation) to remove noises on the mask . we use the kernel with size 5 for erosion to remove noises and then we use "closing"(dilation followed by erosion) to make the mask over moving objects bigger (make it more recognizable) below you can see the result of these procedure :

Mask after threshold	Opening morphology	Closing morphology

• Result

Remove camera shake

if we assume that camera parameters change smoothly and obtain a temporally smoothed estimate for each camera parameter, we reach shake less video.

to do so, for each element of homography matrix, we use polynomial approximation. to say it more accurately, for example for element(1,1), we have 900 data from 900 homography matrix and we have to fit the function that best approximate the model. in this section, i use savgol-filter from scipy.signal library and fit the polynomial with 7 degree and kernel size of 351 for these 900data. for better intuition about smoothing noisy data see the picture below :

smoothing data

• Result