Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create albedo texture for a model from calibrated images #6806

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

ssheorey
Copy link
Member

@ssheorey ssheorey commented May 26, 2024

Create an albedo for the triangle mesh using calibrated images. The triangle mesh must have texture coordinates (texture_uvs triangle attribute). This works by back projecting the images onto the texture surface. Overlapping images are blended together in the resulting albedo. For best results, use images captured with exposure and white balance lock to reduce the chance of seams in the output texture.
Two BlendingMethod s for overlapping images are supported:

  • MAX: For each texel, pick the input pixel with the max weight from all overlapping images. This creates sharp textures but may have visible seams.
  • AVERAGE: The output texel value is the weighted sum of input texels. This creates smooth blending without seams, but the results may be blurry.

TODO:

  • Auto exposure / color balance between different input images.
  • Fine-tune input camera poses.

Type

  • Bug fix (non-breaking change which fixes an issue): Fixes #
  • New feature (non-breaking change which adds functionality). Resolves #
  • Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

This feature can be used to create more realistic 3D models with Open3D, e.g. RGBD reconstruction pipeline currently creates a 3D mesh with per-vertex colors only. This adds high resolution textures for a more photorealistic appearance, even with simplified / low poly meshes.

Checklist:

  • I have run python util/check_style.py --apply to apply Open3D code style
    to my code.
  • This PR changes Open3D behavior or adds new functionality.
    • Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
      updated accordingly.
    • I have added or updated C++ and / or Python unit tests OR included test
      results
      (e.g. screenshots or numbers) here.
  • I will follow up and update the code if CI fails.
  • For fork PRs, I have selected Allow edits from maintainers.

Example

Rendered images:

baluster_vase-150k-4096.mp4

Created albedo:
albedo

seg faults
Raycasting fails at vertices. (t_hit is inf)
Switch from vertices to triangle centers - Raycasting still fails (t_hit is inf)
Remove SelectVisible (implementation was incorrect).
Instead use visibility map for each texel directly.
Add pixel_foreshortening
Python test example by rendering images and using them to project albedo.
Copy link

update-docs bot commented May 26, 2024

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

Documentation
Cleanup
Add download for Baluster vase model in python example.
Needs tbb::feeder from tbb::parallel_for_each
@ssheorey ssheorey marked this pull request as draft June 4, 2024 00:36
Mostly works - slight issues
Image::Resize for Bool has bus error / seg fault.
highlights detection doesn't work.
@ssheorey ssheorey added this to the v0.19 milestone Oct 12, 2024
@ssheorey ssheorey added the status / to merge Looks good, merge after minor updates. label Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status / to merge Looks good, merge after minor updates.
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

1 participant