Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Points not lying on the surface #1

Open
zgojcic opened this issue May 16, 2021 · 6 comments
Open

Points not lying on the surface #1

zgojcic opened this issue May 16, 2021 · 6 comments

Comments

@zgojcic
Copy link

zgojcic commented May 16, 2021

Hi,

thanks for open-sourcing this project. I have used it to render some of the shapenet models but have observed that the 3D point do not lie exactly on the surface once back-projected to 3D. The cloud2mesh distance lie in the range of 0.002.

Have you maybe observed the same? Do you know if this is an inherent precision of Blender depth rendering or if there is maybe some precision loss occurring at some point?

Zan

@yinyunie
Copy link
Owner

Hi,

which mesh was used in your cloud2mesh? If you used the fused mesh, there could be two reasons: 1. we erode the depth map with a kernel size of 2. It enlarges the shape a little bit to avoid missing small structures (like bars on the chair back), see dist_to_dep in tools/utils.py; 2. discretization error with marching cubes algorithm.

@zgojcic
Copy link
Author

zgojcic commented May 16, 2021

Hi,

thanks for the answer, maybe I was not clear in the description. I take the original meshes from the ShapnetV2 and render them using this code with Blender Cycles rendering engine. I then take the depth maps and project them to 3D based on the intrinsic/extrinsic camera parameters. If one then loads the mesh model and point cloud into meshlab or CC the points obtained from the depth image do not lie exactly on the surface of the mesh that was used to render them in the first place. If one then computes the cloud2mesh distance the distances are in the range up to 0.002.

So tldr. there are no marching cubes or TSDF fusion involved and the models are the original ones from Shapenet

@yinyunie
Copy link
Owner

Hi,

I did not come across this issue in your description. Note that, the *.exr file from blender is 'distance map', not 'depth map', where each pixel on the 2D plane stores the distance value from a point on surface to the camera center. You can check dist_to_dep in tools/utils.py to check the conversion.

@zgojcic
Copy link
Author

zgojcic commented May 17, 2021

Hi,

it actually does happen with the provided configuration (I know that cycles returns the distance map and that one has to transform it to the depth map). To reproduce what I am describing you can do the following steps:

  1. run python create_viewpoints.py
  2. run python run_render.py
  3. change python visualization/pc_painter.py such that "point_canonical" is saved to a ply or txt file.
  4. Load the ".obj" file that was used as input to the run_render.py script and the saved point clouds from step 3 to Meshlab or CloudCompare.
  5. Check that the points of the point cloud do not lie exactly on the mesh faces of the "*.obj" file.

@skymanaditya1
Copy link

@zgojcic were you able to resolve this issue? I am facing the exact same issue. This is how the pointcloud (rendered from a fixed camera viewpoint) registers with obj file -- notice the slight offset.
Screenshot from 2024-01-31 01-52-08

@skymanaditya1
Copy link

I finally got this to work. I was using custom code to read the exr files and I think that was the reason for the imprecision between the back-projected pointclouds. Here is the code for generating the pointcloud for anyone facing the same issue as me --

import imageio
import sys
sys.path.append('../')
from data_config import *
from tools.read_and_write import *
import random

def read_exr(exr_file_list):
    print(f'The exr file list is : {exr_file_list}')
    '''
    read exr files and output a matrix.
    :param exr_file_list:
    :return:
    '''
    if isinstance(exr_file_list, str):
        exr_file_list = [exr_file_list]

    im_list = []

    for exr_file in exr_file_list:
        if not exr_file.endswith('exr'):
            raise TypeError('file is not with the format of .exr.')

        im_list.append(imageio.imread(exr_file, format='exr')[:,:,0])

    return np.array(im_list)

# get the pointcloud given the rgb image and the depth map 
def get_point_cloud(depth_maps, cam_Ks, cam_RTs, rgb_imgs=None):
    '''
    get point cloud from depth maps
    :param depth_maps: depth map list
    :param cam_Ks: corresponding camera intrinsics
    :param cam_RTs: corresponding camera rotations and translations
    :param rgb_imgs: corresponding rgb images
    :return: aligned point clouds in the canonical system with color intensities.
    '''
    point_list_canonical = []
    camera_positions = []
    color_intensities = []

    if not isinstance(rgb_imgs, np.ndarray):
        rgb_imgs = 32*np.ones([depth_maps.shape[0], depth_maps.shape[1], depth_maps.shape[2], 3], dtype=np.uint8)

    for depth_map, rgb_img, cam_K, cam_RT in zip(depth_maps, rgb_imgs, cam_Ks, cam_RTs):
        u, v = np.meshgrid(range(depth_map.shape[1]), range(depth_map.shape[0]))
        u = u.reshape([1, -1])[0]
        v = v.reshape([1, -1])[0]

        z = depth_map[v, u]

        # remove infinitive pixels
        non_inf_indices = np.argwhere(z < np.inf).T[0]

        color_indices = rgb_img[v, u][non_inf_indices]
        z = z[non_inf_indices]
        u = u[non_inf_indices]
        v = v[non_inf_indices]

        # calculate coordinates
        x = (u - cam_K[0][2]) * z / cam_K[0][0]
        y = (v - cam_K[1][2]) * z / cam_K[1][1]

        point_cam = np.vstack([x, y, z]).T

        point_canonical = (point_cam - cam_RT[:, -1]).dot(cam_RT[:,:-1])
        cam_pos = - cam_RT[:, -1].dot(cam_RT[:,:-1])
        focal_point = ([0, 0, 1] - cam_RT[:, -1]).dot(cam_RT[:,:-1])
        up = np.array([0,-1,0]).dot(cam_RT[:,:-1])

        cam_pos = {'pos':cam_pos, 'fp':focal_point, 'up':up}

        # point_clouds.append(np.expand_dims(point_canonical, axis=0))

        point_list_canonical.append(point_canonical)
        camera_positions.append(cam_pos)
        color_intensities.append(color_indices)

    return point_list_canonical, camera_positions, color_intensities

depth_sample_dir = '02818832/e91c2df09de0d4b1ed4d676215f46734'
n_views = 20
assert n_views <= total_view_nums # there are total 20 views surrounding an object.
view_ids = range(1, n_views+1)

shapenet_rendering_path = '../datasets/shapenet_sample_renderings'
metadata_dir = os.path.join(shapenet_rendering_path, depth_sample_dir)
print(f'metadata_dir : {metadata_dir}')
dist_map_dir = [os.path.join(metadata_dir, 'depth_{0:03d}.exr'.format(view_id)) for view_id in view_ids]
print(f'dist map dir : {dist_map_dir}')
camera_path = '../datasets/camera_settings'
cam_RT_dir = [os.path.join(camera_path, 'cam_RT','cam_RT_{0:03d}.txt'.format(view_id)) for view_id in view_ids]

dist_maps = read_exr(dist_map_dir)
cam_K = np.loadtxt(os.path.join(camera_path, 'cam_K/cam_K.txt'))
cam_RTs = read_txt(cam_RT_dir)

depth_maps = np.float32(dist_to_dep(dist_maps, [cam_K]*len(view_ids)))
rgb_images = [os.path.join(metadata_dir, 'color_{0:03d}.png'.format(view_id)) for view_id in view_ids]
point_list_canonical, camera_positions, color_intensities = get_point_cloud(depth_maps, [cam_K]*len(view_ids), cam_RTs, rgb_imgs=read_image(rgb_images))

# visualize the pointcloud 
pcd_index = random.randint(0, len(point_list_canonical)-1)
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(np.vstack(point_list_canonical))
pcd.colors = o3d.utility.Vector3dVector(np.vstack(color_intensities)/255.0)

# pcd_combined = o3d.geometry.PointCloud()
# pcd_combined.points = o3d.utility.Vector3dVector(np.vstack(point_list_canonical))

o3d.visualization.draw_geometries([pcd])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants