How to train on custom synthetic dataset?

[mikkeljakobsen]

Hi,

I would like to train MoreFusion on my own synthetic dataset of boxes. I've generated my dataset using BlenderProc and I have ground truth instance masks and 6D poses for the boxes. An example train image from my generated data is shown below:

I managed to train Mask R-CNN so it generalizes to real images to some extend. But I need to do full 6D pose estimation.

So my question is: do you think it would be possible to succesfully train MoreFusion on my own custom synthetic data as described above? And what would I need to do in order to adapt the training script to custom data?

Best regards, Mikkel

[wkentaro]

To train the 6D pose estimation model, we need the below information, which probably you can extract from Blender.

morefusion/morefusion/datasets/rgbd_pose_estimation/ycb_video/dataset.py

Lines 73 to 80 in 5c1036c

	instance_ids=instance_ids,
	class_ids=class_ids,
	rgb=frame["color"],
	depth=frame["depth"],
	instance_label=frame["label"],
	intrinsic_matrix=frame["meta"]["intrinsic_matrix"],
	T_cam2world=T_cam2world,
	Ts_cad2cam=Ts_cad2cam,

For the actual training, you need to create your custom dataset class similar to the below.

morefusion/morefusion/datasets/rgbd_pose_estimation/ycb_video/dataset.py

Line 9 in 5c1036c

class YCBVideoRGBDPoseEstimationDataset(RGBDPoseEstimationDatasetBase):

And use it in the train.py script.
https://github.com/wkentaro/morefusion/blob/master/examples/ycb_video/singleview_3d/train.py

[alanxuefei]

In ./singleview_3d/train.py, YCBVideoRGBDPoseEstimationDatasetReIndexed is used.
YCBVideoRGBDPoseEstimationDatasetReIndexed uses the sample data 1 as below.

The above datasets is generated from the sample data 2 as below via MyRealRGBDPoseEstimationDataset

The sample data 2 may be collected via data_collection_server.py.

Questions:

• How does these sample data 2 generated?

[hz-ants]

@ mikkeljakobsen @alanxuefei Hi,have you completed the project task of multiple instances of similar objects?Can you share your successful experience,thank you!

[mikkeljakobsen]

@ mikkeljakobsen @alanxuefei Hi,have you completed the project task of multiple instances of similar objects?Can you share your successful experience,thank you!

Sorry, I didn't manage to train MoreFusion on my own data.