Skip to content

Latest commit

 

History

History
60 lines (50 loc) · 3.06 KB

README.md

File metadata and controls

60 lines (50 loc) · 3.06 KB

Hypersim Multi View Geometry

This repository implements frame transformation methods to interface and use the Apple Hypersim Evermotion data set with multi view geometry applications. For example this package can be used to label pixel correspondences for self-supervised key point detection and description as in KP2D

Frame-to-frame pixel projection

Pixel correspondences between two frames of the same scene can be found through the pixel world position in the source frame and the target camera pose and camera intrinsic calibration.

warp():

Frame-to-frame pixel projection, source Frame-to-frame pixel projection, target

Occlusion detection

Pixel that are visible in the source frame but occluded in the target frame can be detected automatically. Occlusion detection

Reflectance detection

Reflecting and transparent surface can be masked using the diffuse reflectance information of the Hypersim dataset. Reflectance detection

Installation

Install the hypersim_multiview toolbox with

conda develop $HOME/hypersim_multiview

or

pip3 install -e $HOME/hypersim_multiview

Minimum required dataset

The following files are required to compute the projection, the occlusion detection, and the reflectance detection:

|_detail
|--*
|images
|--scene_cam_CC_final_preview
|----frame.FFFF.color.jpg
|----frame.FFFF.diffuse_reflectance.jpg
|--scene_cam_CC_geometry.hdf5
|----frame.FFFF.position.hdf5

The minimum dataset is still 500 Gb, thus we have to download it through the official source. Use 99991's alternative downloader to download the reduced dataset:

python3 ./download.py -d /diretory/to/download/to --contains _detail --silent
python3 ./download.py -d /diretory/to/download/to --contains .color.jpg --contains final_preview --silent
python3 ./download.py -d /diretory/to/download/to --contains .diffuse_reflectance.jpg --silent
python3 ./download.py -d /diretory/to/download/to --contains .position.hdf5 --silent

The download script will first list all files that it is skipping. This may take 15 minutes. Then it starts downloading.

Acknowledgement

This work was created within the 3D Vision class student project Viewpoint Adaptation in a Synthetic Environment with Shengqu Cai (@HexagonPrime), Menelaos Kanakis (@menelaoskanakis) and Mihai Dusmanu (@mihaidusmanu) at ETH Zurich.