Skip to content

v0.1.0
Compare
Choose a tag to compare
@orperel orperel released this 03 Aug 18:08
· 144 commits to main since this release
v0.1.0
812713d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Highlights

First kaolin-wisp release.
Supports full optimization pipelines of neural radiance fields and signed distance functions.
Also includes an interactive renderer to visualize the optimization progress.

Features List

Core

Acceleration Structures

An Octree allows for fast location based queries and raymarching operations.
On NVIDIA RTX 3090, that brings NeRF training time down to anywhere between tens of seconds to single minutes for single objects like the bulldozer.
(The exact performance is affected by other blocks in the neural pipeline: tracer, feature structure, etc..)

Feature Grids

The initial version of wisp is shipped with 4 types of feature grids:

Additional Building Blocks

Support for Positional Encoding (Fourier), torch MLP Decoders.

Neural Fields

  • Neural Fields can be composed and configured with the various building blocks: occupancy & feature structures, embedders, decoders and so forth.
  • Wisp is shipped with implementations for NeuralRadianceFields and NeuralSDFs.
  • New types of neural fields can be registered for research purposes.

Tracers

The tracers framework supports types of channels (i.e: rgba, sdf)
2 implementations of differentiable ray tracers are included.

  • Radiance Fields
  • Sphere tracer for SDFs
    Both tracers keep samples as packed tensors and use the octree to accelerated performance.
    Raymarching backed by the octree supports 2 modes of sampling along the ray:
  • 'voxel' - fixed amount of samples will be generated per "octree cell"
  • 'ray' - fixed amount of samples will be generated between the near / far planes of the camera.

Core Structures

  • RenderBuffer - smart pixel buffers used by tracers. Support custom types of Channels and various blending and channel normalization modes.
  • Ray - batched ray packs for ray generation

Ray Generators

  • Ray generation uses kaolin's Camera module.
  • Default ray generators for perspective and ortho cameras are available.

Ops

A toolkit for loading, processing and evaluation of various modalities is included in this version (including images, meshes, sdfs and kaolin's Structured Point Clouds.

Data Loaders

wisp ships with data loaders support for:

  • Multiview images: NeRF standard dataset format, including extensions to the format introduced by Instant Neural Graphics Primitives).
  • Multiview images: RTMV dataset .
  • SDFs: OBJs + online samples (usually done once in the beginning of optimization)

Training

2 trainers are currently supported (for NeRF and SDFs).
Trainers can run with or without the interactive renderer.

Interactive Renderer

Wisp includes an interactive renderer with flexible support for new types of neural objects.
The renderer was built in mind to support future neural pipelines which may not yet exist.
Dependencies: pycuda, glumpy, glfw, imgui.

Scene Graph & Renderers

  • The renderer core fully supports neural fields and tracers currently available in wisp core (NeRF, SDFs traced via the packed tracers).
  • Neural fields are implemented as standalone renderers which produce a RenderBuffer.
    The renderer core supports multiple shape types by blending such buffers.
  • All objects are stored inside a scene graph. The scene graph can be manipulated to toggle objects visibility and adjust properties.

Debug Visualizations

  • Line primitives can be drawn on the canvas to display additional layers of information (i.e. Octree occupancy).
  • The optimization runs in parallel to the renderer, allowing for ad-hoc interaction with the optimized scene.

Canvas Modes

  • The canvas supports visualization of custom channels (rgb, depth..)
  • Camera can switch between perspective and ortho
  • Camera controllers: First person, Trackball, Turntable

Custom Apps & Gui

  • Wisp apps are highly customizable: the gui and canvas gizmos can be configured to display specific menus / canvas transients (such as the world grid).
  • The gui can be extended with new types of widgets.

Full Changelog: https://github.com/NVIDIAGameWorks/kaolin-wisp/commits/v0.1.0

Assets 2
Loading