v0.1.2

Highlights

• Easier installation: pycuda deprecated in favor of cuda-python, support for WSL added, some dependencies removed.
• Major refactor to the lib: cleanup of stale research code, added documentation and introduction of modular base classes such as BaseAS (acceleration structure, i.e. for raymarching), WispModule (any wisp module that can display properties in the renderer).
• Optimized hashgrid kernel and pruning
• Jupyter notebook support added

What's Changed

• Indention fix for main_interactive by @orperel in #76
• Add prestep and poststep, various fixes around trainer by @tovacinni in #77
• added default colors when none present in spc file by @orperel in #78
• Add f16 for hash kernel, add pruning controls by @tovacinni in #79
• Optimizations in data loader, other misc optimizations, and bug fixes by @tovacinni in #80
• Small patch to fix bug in hash grid resolutions by @tovacinni in #81
• Remove pidx from interpolate and raymarching output by @orperel in #82
• Updated grid + AS api by @orperel in #83
• Remove pidx from interpolate and raymarching output by @orperel in #84
• Updated grid + AS api by @orperel in #85
• Operel/mvdataset docker qol by @orperel in #86
• Updated trainer lifecycle by @orperel in #87
• Revert neural fields properties by @orperel in #90
• Operel/new configparser by @orperel in #91
• Make pruning more general to the type of BLAS by @tovacinni in #92
• Patch for base trainer, bug where self.writer is assumed to exist but is never initialized by @tovacinni in #93
• Hash grid backward interpolation kernel for positions by @tovacinni in #94
• New documentation + remove templates by @orperel in #95
• pytests for app pipelines by @orperel in #97
• fix WISP_HEADLESS doc error & quick links missing link by @orperel in #98
• BaseAS interface by @orperel in #99
• Introduce WispModule and add support for most modules by @orperel in #100
• Generic Widgets and @widget decorator by @orperel in #102
• Default RayTracedRenderer by @orperel in #103
• Replace pycuda with cuda-python calls by @orperel in #105
• Fix bug in iteration count by @tovacinni in #108
• Remove unneeded dependencies, make some others optional by @tovacinni in #109
• Add NVTX markers for profiling by @tovacinni in #110
• New datasets framework by @orperel in #111
• Performance optimization for Hashgrid and RFTracer by @hXl3s in #112
• OctreeAS raymarch 'voxel' dtype fix by @orperel in #114
• Dockerfile updates by @orperel in #115
• Fix "Start Training" button label by @orperel in #116
• OpenGL bug fix: load glumpy with set version by @orperel in #117
• Update kaolin base image in Dockerfile by @orperel in #119
• Fix default bottom level renderers args from tracers by @orperel in #121
• Mixed precision decorators for interactive renderer by @orperel in #122
• Make it possible to save checkpoints during interactive sessions by @shumash in #123
• run_tests.sh script by @orperel in #124
• Fixing checking if dataset supports depth by @DomaradzkiMaciej in #130
• Hashgrid PyTorch implementation by @alvaro-budria in #132
• [#131] Fix skimage SSIM win_size error by @jaeheungs in #133
• Fix issue witg scikit-image 0.20 by @Caenorst in #138
• remove channel_axis=2 by @Caenorst in #139
• Interactive Wisp checkpoint visualization in Jupyter notebooks. by @shumash in #140
• Lower mem footprint for Jupyter by @orperel in #144

New Contributors

• @hXl3s made their first contribution in #112
• @shumash made their first contribution in #123
• @DomaradzkiMaciej made their first contribution in #130
• @alvaro-budria made their first contribution in #132
• @jaeheungs made their first contribution in #133

Full Changelog: v0.1.1...v0.1.2

v0.1.1

Highlights

The second release of Wisp features:

• Support for installation with Docker
• Tracers now handle extra channels properly
• Object transformations allowing to scale, rotate and translate ray based neural fields
• Support for running Wisp on Windows (external contribution by @3a1b2c3)
• Integration with Weights&Biases, including 360 visualizations (external contribution by @soumik12345)
• Two demos: Visualizing NeRF latent channels and Browsing Structured Point Clouds.
• Various bug fixes
  and more..

What's Changed

• Fixed syntax error for tuple unpacking for Python 3.7 support by @joeylitalien in #7
• Adding channel interface to tracers by @tovacinni in #10
• fix interactive mode path in dockerfile and add Docker section in README by @Caenorst in #16
• replace master by main by @Caenorst in #18
• Cfujitsang/fix multiprocess by @Caenorst in #20
• Support tracers hierarchy in renderer factory by @orperel in #25
• Configurable mesh samples param by @orperel in #26
• Templates and cleaner api for renderer, apps and nefs by @orperel in #27
• Extra channels support for tracers by @tovacinni in #30
• run on windows, add point drawing by @3a1b2c3 in #28
• Cfujitsang/fix ci by @Caenorst in #36
• figures and README updates by @orperel in #44
• siggraph 2022 demo by @orperel in #45
• Deactivate interactive mode when using the gui by @orperel in #46
• support backprop when there is no ray intersections by @Caenorst in #47
• SPC fields + example by @orperel in #48
• fix triplanar script by @Caenorst in #49
• fix SDFTrainer validation() var typo by @NIRVANALAN in #50
• clean to main_nerf.py by @Caenorst in #51
• ERROR FIX: fix density rendering error for packed tracer by @chrischoy in #52
• fix readmes by @Caenorst in #55
• rollback main.py changes by @Caenorst in #56
• Object Transforms for ray traced objects by @orperel in #65
• Fix validation and add pandas logging by @tovacinni in #67
• Permutation matrices, bug fixes in transform, various methods to add … by @tovacinni in #68
• View Independent NeRF by @tovacinni in #69
• Weights & Biases support for wisp by @soumik12345 in #75

New Contributors

• @joeylitalien made their first contribution in #7
• @tovacinni made their first contribution in #10
• @3a1b2c3 made their first contribution in #28
• @NIRVANALAN made their first contribution in #50
• @chrischoy made their first contribution in #52

Full Changelog: v0.1.0...v0.1.1

v0.1.0

Highlights

First kaolin-wisp release.
Supports full optimization pipelines of neural radiance fields and signed distance functions.
Also includes an interactive renderer to visualize the optimization progress.

Features List

Core

Acceleration Structures

An Octree allows for fast location based queries and raymarching operations.
On NVIDIA RTX 3090, that brings NeRF training time down to anywhere between tens of seconds to single minutes for single objects like the bulldozer.
(The exact performance is affected by other blocks in the neural pipeline: tracer, feature structure, etc..)

Feature Grids

The initial version of wisp is shipped with 4 types of feature grids:

• Octree (from Neural Geometric Level of Detail)
• Hash grid (from Instant Neural Graphics Primitives)
• Triplane (from Convolutional Occupancy Networks, EG3D)
• Codebook (from VQAD)
  The hash grid is implemented as a CUDA kernel to further narrow the performance gap with the original NGP project.

Additional Building Blocks

Support for Positional Encoding (Fourier), torch MLP Decoders.

Neural Fields

• Neural Fields can be composed and configured with the various building blocks: occupancy & feature structures, embedders, decoders and so forth.
• Wisp is shipped with implementations for NeuralRadianceFields and NeuralSDFs.
• New types of neural fields can be registered for research purposes.

Tracers

The tracers framework supports types of channels (i.e: rgba, sdf)
2 implementations of differentiable ray tracers are included.

• Radiance Fields
• Sphere tracer for SDFs
  Both tracers keep samples as packed tensors and use the octree to accelerated performance.
  Raymarching backed by the octree supports 2 modes of sampling along the ray:
• 'voxel' - fixed amount of samples will be generated per "octree cell"
• 'ray' - fixed amount of samples will be generated between the near / far planes of the camera.

Core Structures

• RenderBuffer - smart pixel buffers used by tracers. Support custom types of Channels and various blending and channel normalization modes.
• Ray - batched ray packs for ray generation

Ray Generators

• Ray generation uses kaolin's Camera module.
• Default ray generators for perspective and ortho cameras are available.

Ops

A toolkit for loading, processing and evaluation of various modalities is included in this version (including images, meshes, sdfs and kaolin's Structured Point Clouds.

Data Loaders

wisp ships with data loaders support for:

• Multiview images: NeRF standard dataset format, including extensions to the format introduced by Instant Neural Graphics Primitives).
• Multiview images: RTMV dataset .
• SDFs: OBJs + online samples (usually done once in the beginning of optimization)

Training

2 trainers are currently supported (for NeRF and SDFs).
Trainers can run with or without the interactive renderer.

Interactive Renderer

Wisp includes an interactive renderer with flexible support for new types of neural objects.
The renderer was built in mind to support future neural pipelines which may not yet exist.
Dependencies: pycuda, glumpy, glfw, imgui.

Scene Graph & Renderers

• The renderer core fully supports neural fields and tracers currently available in wisp core (NeRF, SDFs traced via the packed tracers).
• Neural fields are implemented as standalone renderers which produce a RenderBuffer.
  The renderer core supports multiple shape types by blending such buffers.
• All objects are stored inside a scene graph. The scene graph can be manipulated to toggle objects visibility and adjust properties.

Debug Visualizations

• Line primitives can be drawn on the canvas to display additional layers of information (i.e. Octree occupancy).
• The optimization runs in parallel to the renderer, allowing for ad-hoc interaction with the optimized scene.

Canvas Modes

• The canvas supports visualization of custom channels (rgb, depth..)
• Camera can switch between perspective and ortho
• Camera controllers: First person, Trackball, Turntable

Custom Apps & Gui

• Wisp apps are highly customizable: the gui and canvas gizmos can be configured to display specific menus / canvas transients (such as the world grid).
• The gui can be extended with new types of widgets.

Full Changelog: https://github.com/NVIDIAGameWorks/kaolin-wisp/commits/v0.1.0