Skip to content

[RA-L'22] 360-DFPE: Leveraging Monocular 360-Layouts for Direct Floor Plan Estimation

Notifications You must be signed in to change notification settings

zovelsanj/direct_360_FPE

 
 

Repository files navigation

360-DFPE: Leveraging Monocular 360-Layouts for Direct Floor Plan Estimation

This project was published in IEEE Robotics and Automation Letters RA-L
[ArXiv Paper]

Introduction

For a quick overview please visit our website project or watch our video demo depicted below. For further questions, please contact us to [email protected]


News

  • 29/12/2023: Add direct_floor_plan_estimation.v2 to wrapper vo-scale recover for different dataloaders versions e.g 360-MLC, MVL-Challenge.

  • 28/12/2023: Add setup.py to the project and rename src folder to direct_floor_plan_estimation for better import-like format.

  • 18/05/2023: Added pcl.pyl files to GoogleDrive. Fix issue #2

  • 19/07/2022: Code and pilot scenes released. HERE

  • 29/01/2023: Open access to MP3D-FPE dataset HERE.


Description

This is the implementation of the 360-DFPE, sequential floor plan estimation using only monocular 360-images as input.

Using this REPO, you can:

  • Excecute 360-DFPE for single and multiple room scenes in a sequential and non-sequential manner.
  • Register multiple 360-layouts with estimated camera poses, assuming an unknown visual odometry scale and missed camera height. For convenience, this registration can be executed isolated from the floor plan estimation pipeline. see Computing metadata
  • Evaluate corner and room metrics for floor plan estimation.

Main Requirements

  • python 3.7.7
  • vispy 0.5.3
  • numpy 1.18.5
  • opencv-python 3.4.3.18
  • pandas 1.0.5

Installation

git clone [email protected]:EnriqueSolarte/direct_360_FPE.git

cd direct_360_FPE
pip install -r requirements.txt
pip install .

Dataset

The dataset used in this project is our own collected MP3D-FPE dataset, which simulates a handled-camera walking through different rooms scenes rendered by MINOS simulator with MP3D real-world data. Upon this collected data, we annotate floor plan labels, estimated camera poses uisng OpenVSLAM[4], and estimated layout for each key-frame by HorizonNet[3]. Our dataset offers 360-rgb-images, depth maps, ground truth and estimated camera poses, floor plan labels, and estimated layouts.

DISCLAIMER:

MP3D-FPE is a collected dataset for the task of floor plan estimation. This dataset is stricly an extension of MP3D. Therefore, the data policies and copyrights of this dataset are under the the terms and condition described by Matterport.

For convenience, we have prepared a light-set of scenes which can be downloaded by running download_mp3d_fpe.py -h.

All hyperameter settings are stored in a yaml file at ./config/config.yaml. Addtitionally, for practical purposes, we define a data manager class DataManager, which handles the data in the MP3D-FPE dataset, i.e., ground truth information, estimated poses, rgb images, etc. The following is a typical initialization of our system:

from config import read_config
from data_manager import DataManager
from direct_floor_plan_estimation import DirectFloorPlanEstimation

config_file = "./config/config.yaml"

cfg = read_config(config_file=config_file)
dt = DataManager(cfg)
fpe = DirectFloorPlanEstimation(dt)
...
fpe.scale_recover.esimate_vo_scale()
...

How to execute 360-DFPE

For executing 360-DFPE, we have created three main scripts, i.e., main_eval_scene.py, main_eval_list_scenes.py, main_eval_non_seq_approach.py.

Running on a single scene (Single or Multiple rooms)
python main_eval_scene.py --scene_name 1LXtFkjw3qL_0 --results ./test/
Running from a list of scenes
python main_eval_scenes_list.py --scene_list ./data/scene_list_pilot.txt --results ./test/
python main_eval_non_seq_approach.py --scene_list ./data/scene_list_pilot.txt --results ./test/

Computing metadata

Note that our formulation rely on monocular estimated camera poses, therefore the real scale of the odometry is missed. Additionally, since every layout geometry is estimated from a monocular inference of HorizonNet[3], the layout scale is also missed. For these reasons, we additionally implemented main_compute_metadata.py, which will compute these missed scales along with some additional information such as ground truth rooms, labels, and room-corners, and the missed scales need for 360-layout registration. For convenience, this script runs isolated from the floor plan estimation pipeline.

python main_compute_metadata.py --scene_list ./data/scene_list_pilot.txt --results ./test/

Some qualitative results

Acknowledgement

  • Thanks to professor Wei-Chen Chiu for his unvaluable advises in this project.
  • The credits of this repo are shared with Yueh-Cheng Liu.

Citation

Please cite our paper for any purpose of usage.

@ARTICLE{9772341,
  author={Solarte, Bolivar and Liu, Yueh-Cheng and Wu, Chin-Hsuan and Tsai, Yi-Hsuan and Sun, Min},
  journal={IEEE Robotics and Automation Letters}, 
  title={360-DFPE: Leveraging Monocular 360-Layouts for Direct Floor Plan Estimation}, 
  year={2022},
  volume={7},
  number={3},
  pages={6503-6510},
  doi={10.1109/LRA.2022.3173730}}


References

[1]: Chen, Jiacheng, et al. "Floor-sp: Inverse cad for floorplans by sequential room-wise shortest path." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

[2]: Liu, Chen, Jiaye Wu, and Yasutaka Furukawa. "Floornet: A unified framework for floorplan reconstruction from 3d scans." Proceedings of the European conference on computer vision (ECCV). 2018.

[3]: Sun, Cheng, et al. "Horizonnet: Learning room layout with 1d representation and pano stretch data augmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[4]: Sumikura, Shinya, Mikiya Shibuya, and Ken Sakurada. "OpenVSLAM: A versatile visual SLAM framework." Proceedings of the 27th ACM International Conference on Multimedia. 2019.

About

[RA-L'22] 360-DFPE: Leveraging Monocular 360-Layouts for Direct Floor Plan Estimation

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%