Processing new scenes from EgoExo

[richardrl]

Hi, great work!

I am wondering if you can explain in more detail how you get the colored scenes from EgoExo so we could try running your code overlayed on these scenes.

The pointclouds seem much higher quality than the default pointclouds in semidense_points.csv.gz, and from what I can tell there is no color associated with each point?

For example, this is the semidense points of sfu_cooking_009_5 from egoexo, which looks much lower quality than your scenes:

I noticed a file _cached_filtered_points.npz - perhaps you can explain how you filtered these pointclouds?

And do you build a Gaussian splat somehow?

[brentyi]

Hi @richardrl!

We unfortunately also don't have colors for points from EgoExo:

• The colors of EgoExo points in our results are computed here:
  

  egoallo/src/egoallo/vis_helpers.py

  Lines 158 to 166 in 25a95a9

  	point_cloud = server.scene.add_point_cloud(
  	"/aria_points",
  	points=points_data,
  	colors=np.cos(points_data + np.arange(3)) / 3.0
  	+ 0.7, # Make points colorful :)
  	point_size=0.01,
  	# point_size=0.1,
  	point_shape="sparkle",
  	)

• The scenes that we visualize with Gaussians are our own captures, which we process, reconstruct, and export using Nerfstudio: https://docs.nerf.studio/quickstart/custom_dataset.html#aria
  • I haven't tried running this process for EgoExo but probably it would work, just with lower fidelity than our (more intentionally) captured sequences.
• The scenes with the colored point clouds in our paper's teaser figure are from unprojecting the dense depth maps from Aria Digital Twin: https://www.projectaria.com/datasets/adt/.

For filtering the EgoExo points, you can check the logic for generating _cached_filtered_points.npz here:

egoallo/src/egoallo/data/aria_mps.py

Lines 36 to 64 in 25a95a9

	points_data = mps.read_global_point_cloud(str(points_path)) # type: ignore
	print("Loading + filtering points")
	assert points_path.exists()
	filtered_points_data = filter_points_from_confidence(
	points_data,
	threshold_invdep=0.0001,
	threshold_dep=0.005,
	)
	less_filtered_points_data = filter_points_from_confidence(
	points_data,
	threshold_invdep=0.001,
	threshold_dep=0.05,
	)
	filtered_points_data = np.array(
	[x.position_world for x in filtered_points_data]
	) # type: ignore
	less_filtered_points_data = np.array(
	[x.position_world for x in less_filtered_points_data]
	)
	filtered_points_npz_cache_path.parent.mkdir(exist_ok=True, parents=True)
	np.savez_compressed(filtered_points_npz_cache_path, points=filtered_points_data)
	print("Cached filtered points to", filtered_points_npz_cache_path)
	np.savez_compressed(
	less_filtered_points_npz_cache_path, points=less_filtered_points_data
	)
	print("Cached less filtered points to", less_filtered_points_npz_cache_path)