Get the object pose after full pipeline

[EAST-J]

After all three stages, I tried to parse the object pose and aligned with the GT (e.g. HO3D) for visualization, but I found that the rotation part can not aligned, hera is part of the code to parse the pose from last.ckpt, I try to align the poses in the opencv coordinate. Did I do something wrong?

out, ckpt = load_data('logs/{}/checkpoints/last.ckpt'.format(exp_id))
obj_trans = out['param_dict']['model.nodes.object.params.transl.weight']
obj_rot = out['param_dict']['model.nodes.object.params.global_orient.weight']
w2c = out['w2c']
batch_size = obj_rot.shape[0]
rot_mat = axis_angle_to_matrix(obj_rot).view(batch_size, 3, 3)
# object to camera
batch_size = rot_mat.shape[0]
tf_mats = torch.eye(4).unsqueeze(0).repeat(batch_size, 1, 1)
tf_mats[:, :3, :3] = rot_mat

tf_mats[:, :3, 3] = obj_trans.view(batch_size, 3)
tf_mats = torch.inverse(tf_mats)
np.savetxt('logs/{}/saved_pose.txt'.format(exp_id), tf_mats[:, :3].reshape(-1, 12).detach().cpu().numpy())

Here are pose traj.

[zc-alexfan]

Hi, maybe you can check my evaluation code?

hold/code/src/utils/eval.py

Line 11 in 31dd612

def load_eval_space_data(sd_p):

Here, the rotation should be correct. Just that I didn't calculate the global translation as we only care about the relative translation between hand and object.

[zc-alexfan]

The object also has a scale value, which I don't see in your code.

[EAST-J]

Hi, maybe you can check my evaluation code?

hold/code/src/utils/eval.py

Line 11 in 31dd612

def load_eval_space_data(sd_p):

Here, the rotation should be correct. Just that I didn't calculate the global translation as we only care about the relative translation between hand and object.

Thx for your quick reply, I found that I forgot to multiply the w2c matrix in the out, now I can align them.

[zc-alexfan]

Great! @EAST-J , would you mind sharing some snippets of the alignment code here? I think people can really benefit from it (#17). I am currently doing an internship so I haven't had the capacity to prepare that.

[EAST-J]

Great! @EAST-J , would you mind sharing some snippets of the alignment code here? I think people can really benefit from it (#17). I am currently doing an internship so I haven't had the capacity to prepare that.

Sure, Here are part of my code to get the predicted object pose and HO3D GT poses.

out, ckpt = load_data('logs/{}/checkpoints/last.ckpt'.format(exp_id))
obj_trans = out['param_dict']['model.nodes.object.params.transl.weight']
obj_rot = out['param_dict']['model.nodes.object.params.global_orient.weight']
w2c = out['w2c']
batch_size = obj_rot.shape[0]
rot_mat = axis_angle_to_matrix(obj_rot).view(batch_size, 3, 3)
# object to camera
batch_size = rot_mat.shape[0]
tf_mats = torch.eye(4).unsqueeze(0).repeat(batch_size, 1, 1)
tf_mats[:, :3, :3] = rot_mat
tf_mats[:, :3, 3] = obj_trans.view(batch_size, 3)
tf_mats = torch.inverse(tf_mats) # bs * 4 * 4 camera2object
tf_mats[:, 1:3, :3] *= -1
tf_mats[:, :3 ,:3] = torch.inverse(tf_mats[:, :3, :3])

ho3d_pose = []
meta_dis = sorted(glob(os.path.join(ho3d_dir, "*.pkl")))[start_idx:end_idx:step]
remove_idxs = []

for idx, label_name in enumerate(meta_dis):
    with open(label_name, "rb") as f:
        label = pkl.load(f, encoding="latin1")
    # w2c (here for o2c)
    R_ = label["objRot"]
    if R_ is None:
        remove_idxs.append(idx)
        print("{}:{}".format(idx, label_name))
        continue
    R_, _ = cv2.Rodrigues(R_)
    T_ = label["objTrans"].reshape(3, -1)
    Rt = np.concatenate([R_, T_], 1)
    Rt[1:] *= -1 # ho3d pose in the pyrender coordinate
    Rt = np.concatenate([Rt, np.array([[0, 0, 0, 1]])])
    Rt = np.linalg.inv(Rt)
    ho3d_pose.append(Rt)
ho3d_pose = torch.from_numpy(np.concatenate(ho3d_pose))

For the alignment and visualization part, I employ a method from SLAM to align the trajectory using Sim(3). Detailed alignment procedures can be found here, or use the tool evo