3D_style_transfer

Final Project for 3D Vision course at ETH Zurich. This project was an attempt to combine established methods in image and video style transfer with 3D reconstruction techniques.

Learned Descriptors

0. Prerequisites:

   • Ubuntu 16.04
   • OpenCV 3.1.0
   • CUDA v7.5
   • Lua 5.1 with the following Luarocks:
     • nn
     • cunn
     • image
     • torch
     • lsqlite3

1. Preparing data: To get started quickly use the images proveded here. Place the entire directory named glove/mosaic inside the data folder. You can download the other directories aswell if you want to try out different styles. If you want to use your own data you will need to first run rename.sh to change the names of the files to sequential numbers.

2. Building: To build the C++ code run: cd ~/3D_style_transfer/learned_descriptors/src sudo chmod +x build.sh ./build.sh This will build all of the C++ and Cuda code.

3. Extracting features Run: ./extractor.o which will extract up to 3000 features and record them in the glove_mosaic_learned.db database. The number of features can be changed but this will take longer to match later on. Currently the code is only set to find features in the first 5 images. If you actually want to attempt 3D reconstruciton you will need to extract feature from all the images and have at least 5000 features.

4. Converting descriptors Here we use Lua to convert the image patches into 128D descriptors. Run: th descriptor.lua.

5. Matching features Here exhaustive matching is performed using the GPU. Run ./matcher.o

6. Viewing matches To view matches between the first two images Run: ./verifier.o

Voxel Carving

0. Prerequisites:

   • Ubuntu 14.04
   • OpenCV 3.1.0
   • VTK

1. Preparing data: In order to use this module the user needs to have camera.txt, images.txt both without the HEADER (commented description). Additionaly the user needs to provide "sil" folder where the silhouettes are stored. Sample assets are provided in the "sample" folder.

2. Building: To build the C++ code run: cd ~/3D_style_transfer/Voxel-Carving/ mkdir build cd build cmake .. make This will build all of the C++.

3. Carving: To run: cd ~/3D_style_transfer/Voxel-Carving/build ./main /home/user/sample/ 50

4. Viewing The output file is inside the build folder. The format is .ply so the file can be viewed in any point cloud viewer such as MeshLab.

Scripts

0. Prerequisites:

   • Ubuntu 14.04
   • COLMAP
   • CUDA v7.5

1. Running: To run the scripts modify the PROJECT_PATH and COLMAP_PATH to appropriate location. PORJECT_PATH needs to contain the images and/or the database depending on which script is ran. COLMAP_PATH is simply the path to COLMAP binaries assuming they were built but not installed.

2. Running: Simply run the chosen bash script:

./3dcnn

3. Scripts Description:

3dcnn: Runs 3D reconstruction using learned descriptor database. Assumes database already exists from the learned descriptors code.

3ddense: Runs dense reconstruction on already existing sparse reconstruction from learned descriptor code.

3dmesh: Runs mesh reconstruction. Useful for experimenting with different reconstructions from original images data but with stylized images.

3drec: Standard 3D reconstruction. Requires images folder.

3dstyle: Uses images to find and match descriptors and uses stylized images in style_image to 3D reconstruction with stylized color. Assumes folder images and style_images is present. Stylized images need to be same the same name as the original counterparts.

Point Cloud coloring

0. Prerequisites:

   • Ubuntu 14.04
   • OpenCV 3.1.0
   • PCL 1.7
   • VTK

1. Preparing data: In order to use this module the user needs to have camera.txt, images.txt both AS GENERATED BY COLMAP. Additionaly the user needs to provide a coloring folder where the images with the desired color are stored. These images should be named as found in the images.txt.

2. Building: To build the C++ code run:

cd ~/3D_style_transfer/color_projection/
mkdir build
cd build
cmake ..
make

This will build all of the code.

3. Coloring: To run:

cd ~/3D_style_transfer/Voxel-Carving/build 
./color_painting CAMERAS.TXT IMAGES.TXT IMAGES_FOLDER/

4. As used by us As we used it for this the Spring 2017 3D Vision project: You will need a set of images to run colmap with. These images may be named however you wish. Use Colmap to generate the following: images.txt, cameras.txt, and a .ply file. We used a image style transfer to generate images with interesting colors. Run the style transfer on all the images you used with colmap. Make sure the names stay the same (so if the original image is named scene0015.jpg, the stylized image should be named scene0015.jpg). Now that you have this, run

/PATH/TO/BUILD/color_painting POINT_CLOUD.ply CAMERAS.txt IMAGES.txt STYLIZED/IMAGE/FOLDER/ 

where cameras.txt and images.txt are the UNALTERED files that Colmap generated. The / at the end of the stylized image folder is also required. This will start the main program.

5. Example All of the data used in our project can be seen here

Run

cd /PATH/TO/DATA_FOLDER/data/glove

 /PATH/TO/BUILD/color_painting glove_filtered.ply cameras.txt images.txt mosaic/ 

This should run the program. You can type "all" to have the glove_few.ply cloud colored by all images in the mosaic folder. Type anything else, and you will be prompted to choose images to color the cloud with. When you are finished, type q. You will be prompted to enter a file name to save the point cloud to.

6 The Other Code There are two other executables included, which help with point cloud processing. One of them, filter_point_cloud, runs a series of filters that tries to filter out all points in the point cloud that aren't the object you wish to color. We assumed a fairly dense point cloud (upwards of 100000 points in the object of interest), where the object is sitting on a table (ie we can segment and remove a ground plane). view_point_cloud is exactly as it sounds: it just visualizes a point cloud in pcl's viewer.