Poor quality output

[shivar2]

Hi again,
I'v given the following video to the model and have used the default configuration of the project to train model. The output after 6300 epochs is as follows. Can you help me and tell me why the output mesh isn't High quality and is damaged?Does the input have to have Special features or do I need to change the project configuration?

input video:
https://github.com/MoyGcc/vid2avatar/assets/47525205/c9ce55be-9969-401d-b4c7-53c75c40e65a

output:

[MoyGcc]

Hi, thank you for reporting the issue. I'm testing the same sequence on my end and will come back to you once I have the results.

[shivar2]

Thank you and I appreciate your quick response!

[MoyGcc]

Hi, I think now I roughly know what the problem is. The main cause is because of this purely white background. This leads to confusion about the multi-view consistency of the human body. Since the white pixels around the human in image space can be easily integrated into the canonical human SDF without violating the multi-view (cross-frame) RGB consistency. I attach an example figure after about 100 epochs to illustrate this problem. Things will be worse after more training iterations.

One solution is to incorporate mask loss since the segmentation, in this case, is no longer an issue for most off-the-shelf human matting methods for example RVM. Some helper functions are pasted here so that you may try this out faster.

def get_index_in_and_outside_of_mask(indices, mask):
    indices_floored = np.floor(indices).astype(np.int32)
    xs = indices_floored[:, 0]
    ys = indices_floored[:, 1]
    mask00 = mask[xs, ys]
    mask01 = mask[xs, ys + 1]
    mask10 = mask[xs + 1, ys]
    mask11 = mask[xs + 1, ys + 1]

    mask_all = mask00 & mask01 & mask10 & mask11
    index_inside = np.where(mask_all == 1)[0]

    mask_all = mask00 | mask01 | mask10 | mask11
    index_outside = np.where(mask_all == 0)[0]
    return index_inside, index_outside

def get_mask_loss(self, acc_map, index_inside, index_outside):
    mask_loss = self.l1_loss(acc_map[index_inside], torch.ones_like(acc_map[index_inside])) + \
                self.l1_loss(acc_map[index_outside], torch.zeros_like(acc_map[index_outside]))
    return mask_loss

[shivar2]

Thanks for your quick response! Actually, I thought a white background would help us get better results from the model, but I want to test the model with videos with backgrounds too. I will test the model with other videos and share the result with you.

[afnan248]

i have an issue in running the code. Can you help me in the process

Thanks for your quick response! Actually, I thought a white background would help us get better results from the model, but I want to test the model with videos with backgrounds too. I will test the model with other videos and share the result with you.