Skip to content

Commit

Permalink
deploy: 9c70035
Browse files Browse the repository at this point in the history
  • Loading branch information
liruilong940607 committed Sep 27, 2024
1 parent 4860cad commit 4aa91db
Show file tree
Hide file tree
Showing 81 changed files with 21,221 additions and 0 deletions.
4 changes: 4 additions & 0 deletions versions/1.4.0/.buildinfo
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: 3ac07230479e67752436930c2be446df
tags: 645f666f9bcd5a90fca523b33c5a78b7
Binary file added versions/1.4.0/.doctrees/apis/compression.doctree
Binary file not shown.
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/apis/strategy.doctree
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/apis/utils.doctree
Binary file not shown.
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/environment.pickle
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/examples/colmap.doctree
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/examples/image.doctree
Binary file not shown.
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/index.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/tests/eval.doctree
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/tests/profile.doctree
Binary file not shown.
Binary file added versions/1.4.0/.doctrees/tests/tests.doctree
Binary file not shown.
Binary file added versions/1.4.0/_images/square.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added versions/1.4.0/_images/training.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
780 changes: 780 additions & 0 deletions versions/1.4.0/_modules/gsplat/compression/png_compression.html

Large diffs are not rendered by default.

1,067 changes: 1,067 additions & 0 deletions versions/1.4.0/_modules/gsplat/cuda/_torch_impl.html

Large diffs are not rendered by default.

625 changes: 625 additions & 0 deletions versions/1.4.0/_modules/gsplat/cuda/_torch_impl_2dgs.html

Large diffs are not rendered by default.

2,364 changes: 2,364 additions & 0 deletions versions/1.4.0/_modules/gsplat/cuda/_wrapper.html

Large diffs are not rendered by default.

1,828 changes: 1,828 additions & 0 deletions versions/1.4.0/_modules/gsplat/rendering.html

Large diffs are not rendered by default.

705 changes: 705 additions & 0 deletions versions/1.4.0/_modules/gsplat/strategy/default.html

Large diffs are not rendered by default.

549 changes: 549 additions & 0 deletions versions/1.4.0/_modules/gsplat/strategy/mcmc.html

Large diffs are not rendered by default.

357 changes: 357 additions & 0 deletions versions/1.4.0/_modules/index.html

Large diffs are not rendered by default.

38 changes: 38 additions & 0 deletions versions/1.4.0/_sources/apis/compression.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
Compression
===================================

.. currentmodule:: gsplat

`gsplat` provides handy APIs for compressing and decompressing the Gaussian parameters,
which can significantly reduce the storage / streaming cost. For example, using :class:`PngCompression`,
1 million Gaussians that are stored in 236 MB can be compressed to 16.5 MB, with only 0.5dB
PSNR loss (29.18dB to 28.65dB).

Similar to the :doc:`strategy`, our compression APIs also has a specific expectation on
the format of the Gaussian parameters. It is designed to work with the Gaussians defined
as either a Dict of
`torch.Tensor <https://pytorch.org/docs/stable/tensors.html>`_
with at least the following keys: {"means", "scales", "quats", "opacities", "sh0", "shN"}.
On top of these attributes, an arbitrary number of extra attributes are supported.

The following code snippet is an example of how to use the compression approach in `gsplat`:

.. code-block:: python
from gsplat import PngCompression
splats: Dict[str, Tensor] = {
"means": Tensor(N, 3), "scales": Tensor(N, 3), "quats": Tensor(N, 4), "opacities": Tensor(N),
"sh0": Tensor(N, 1, 3), "shN": Tensor(N, 24, 3), "features1": Tensor(N, 128), "features2": Tensor(N, 64),
}
compression_method = PngCompression()
# run compression and save the compressed files to compress_dir
compression_method.compress(compress_dir, splats)
# decompress the compressed files
splats_c = compression_method.decompress(compress_dir)
Below is the API for the compression approaches we support in `gsplat`:

.. autoclass:: PngCompression
:members:
76 changes: 76 additions & 0 deletions versions/1.4.0/_sources/apis/rasterization.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
Rasterization
===================================

3DGS
------

.. currentmodule:: gsplat

Given a set of 3D gaussians parametrized by means :math:`\mu \in \mathbb{R}^3`, covariances
:math:`\Sigma \in \mathbb{R}^{3 \times 3}`, colors :math:`c`, and opacities :math:`o`, we first
compute their projected means :math:`\mu' \in \mathbb{R}^2` and covariances
:math:`\Sigma' \in \mathbb{R}^{2 \times 2}` on the image planes. Then we sort each gaussian such
that all gaussians within the bounds of a tile are grouped and sorted by increasing
depth :math:`z`, and then render each pixel within the tile with alpha-compositing.

Note, the 3D covariances are reparametrized with a scaling matrix
:math:`S = \text{diag}(\mathbf{s}) \in \mathbb{R}^{3 \times 3}` represented by a
scale vector :math:`s \in \mathbb{R}^3`, and a rotation matrix
:math:`R \in \mathbb{R}^{3 \times 3}` represented by a rotation
quaternion :math:`q \in \mathcal{R}^4`:

.. math::
\Sigma = RSS^{T}R^{T}
The projection of 3D Gaussians is approximated with the Jacobian of the perspective
projection equation:

.. math::
J = \begin{bmatrix}
f_{x}/z & 0 & -f_{x} t_{x}/z^{2} \\
0 & f_{y}/z & -f_{y} t_{y}/z^{2} \\
0 & 0 & 0
\end{bmatrix}
.. math::
\Sigma' = J W \Sigma W^{T} J^{T}
Where :math:`[W | t]` is the world-to-camera transformation matrix, and :math:`f_{x}, f_{y}`
are the focal lengths of the camera.

.. autofunction:: rasterization

2DGS
------

Given a set of 2D gaussians parametrized by means :math:`\mu \in \mathbb{R}^3`, two principal tangent vectors
embedded as the first two columns of a rotation matrix :math:`R \in \mathbb{R}^{3\times3}`, and a scale matrix :math:`S \in R^{3\times3}`
representing the scaling along the two principal tangential directions, we first transforms pixels into splats' local tangent frame
by :math:`(WH)^{-1} \in \mathbb{R}^{4\times4}` and compute weights via ray-splat intersection. Then we follow the sort and rendering similar to 3DGS.

Note that H is the transformation from splat's local tangent plane :math:`\{u, v\}` into world space

.. math::
H = \begin{bmatrix}
RS & \mu \\
0 & 1
\end{bmatrix}
and :math:`W \in \mathbb{R}^{4\times4}` is the transformation matrix from world space to image space.


Splatting is done via ray-splat plane intersection. Each pixel is considered as a x-plane :math:`h_{x}=(-1, 0, 0, x)^{T}`
and a y-plane :math:`h_{y}=(0, -1, 0, y)^{T}`, and the intersection between a splat and the pixel :math:`p=(x, y)` is defined
as the intersection bwtween x-plane, y-plane, and the splat's tangent plane. We first transform :math:`h_{x}` to :math:`h_{u}` and :math:`h_{y}`
to :math:`h_{v}` in splat's tangent frame via the inverse transformation :math:`(WH)^{-1}`. As the intersection point should fall on :math:`h_{u}` and :math:`h_{v}`, we have an efficient
solution:

.. math::
u(p) = \frac{h^{2}_{u}h^{4}_{v}-h^{4}_{u}h^{2}_{v}}{h^{1}_{u}h^{2}_{v}-h^{2}_{u}h^{1}_{v}},
v(p) = \frac{h^{4}_{u}h^{1}_{v}-h^{1}_{u}h^{4}_{v}}{h^{1}_{u}h^{2}_{v}-h^{2}_{u}h^{1}_{v}}
.. autofunction:: rasterization_2dgs
76 changes: 76 additions & 0 deletions versions/1.4.0/_sources/apis/strategy.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
Densification
===================================

.. currentmodule:: gsplat

In `gsplat`, we abstract out the densification and pruning process of the Gaussian
training into a strategy. A strategy is a class that defines how the Gaussian parameters
(along with their optimizers) should be updated (splitting, pruning, etc.) during the training.

An example of the training workflow using :class:`DefaultStrategy` is like:

.. code-block:: python
from gsplat import DefaultStrategy, rasterization
# Define Gaussian parameters and optimizers
params: Dict[str, torch.nn.Parameter] | torch.nn.ParameterDict = ...
optimizers: Dict[str, torch.optim.Optimizer] = ...
# Initialize the strategy
strategy = DefaultStrategy()
# Check the sanity of the parameters and optimizers
strategy.check_sanity(params, optimizers)
# Initialize the strategy state
strategy_state = strategy.initialize_state()
# Training loop
for step in range(1000):
# Forward pass
render_image, render_alpha, info = rasterization(...)
# Pre-backward step
strategy.step_pre_backward(params, optimizers, strategy_state, step, info)
# Compute the loss
loss = ...
# Backward pass
loss.backward()
# Post-backward step
strategy.step_post_backward(params, optimizers, strategy_state, step, info)
A strategy will inplacely update the Gaussian parameters as well as the optimizers,
so it has a specific expectation on the format of the parameters and the optimizers.
It is designed to work with the Guassians defined as either a Dict of
`torch.nn.Parameter <https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html>`_
or a
`torch.nn.ParameterDict <https://pytorch.org/docs/stable/generated/torch.nn.ParameterDict.html>`_
with at least the following keys: {"means", "scales", "quats", "opacities"}. On top of these attributes,
an arbitrary number of extra attributes are supported. Besides the parameters, it also
expects a Dict of `torch.optim.Optimizer <https://pytorch.org/docs/stable/optim.html>`_
with the same keys as the parameters, and each optimizer should correspond to only
one learnable parameter.

For example, the following is a valid format for the parameters and the optimizers
that can be used with our strategies:

.. code-block:: python
N = 100
params = torch.nn.ParameterDict{
"means": Tensor(N, 3), "scales": Tensor(N), "quats": Tensor(N, 4), "opacities": Tensor(N),
"colors": Tensor(N, 25, 3), "features1": Tensor(N, 128), "features2": Tensor(N, 64),
}
optimizers = {k: torch.optim.Adam([p], lr=1e-3) for k, p in params.keys()}
Below are the strategies that are currently implemented in `gsplat`:

.. autoclass:: DefaultStrategy
:members:

.. autoclass:: MCMCStrategy
:members:
46 changes: 46 additions & 0 deletions versions/1.4.0/_sources/apis/utils.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@

Utils
===================================

Below are the basic functions that supports the rasterization.

3DGS
-----

.. currentmodule:: gsplat

.. autofunction:: spherical_harmonics

.. autofunction:: quat_scale_to_covar_preci

.. autofunction:: proj

.. autofunction:: fully_fused_projection

.. autofunction:: isect_tiles

.. autofunction:: isect_offset_encode

.. autofunction:: world_to_cam

.. autofunction:: rasterize_to_pixels

.. autofunction:: rasterize_to_indices_in_range

.. autofunction:: accumulate

.. autofunction:: rasterization_inria_wrapper

2DGS
-----
.. currentmodule:: gsplat

.. autofunction:: fully_fused_projection_2dgs

.. autofunction:: rasterize_to_pixels_2dgs

.. autofunction:: rasterize_to_indices_in_range_2dgs

.. autofunction:: accumulate_2dgs

.. autofunction:: rasterization_2dgs_inria_wrapper
23 changes: 23 additions & 0 deletions versions/1.4.0/_sources/conventions/data_conventions.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
Data Conventions
===================================

.. currentmodule:: gsplat

Here we explain the various data conventions used in our repo.

Rotation Convention
-------------------
We represent rotations with four dimensional vectors :math:`q = (w,x,y,z)` such that the 3x3 :math:`SO(3)` rotation matrix is defined by:

.. math::
R = \begin{bmatrix}
1 - 2 \left( y^2 + z^2 \right) & 2 \left( x y - w z \right) & 2 \left( x z + w y \right) \\
2 \left( x y + w z \right) & 1 - 2 \left( x^2 + z^2 \right) & 2 \left( y z - w x \right) \\
2 \left( x z - w y \right) & 2 \left( y z + w x \right) & 1 - 2 \left( x^2 + y^2 \right) \\
\end{bmatrix}
View Matrix
---------------------------------
We refer to the `view matrix` :math:`W` as the world to camera frame transformation (referred to as `w2c` in some sources) that maps
3D world points :math:`(x,y,z)_{world}` to 3D camera points :math:`(x,y,z)_{cam}` where :math:`z_{cam}` is the relative depth to the camera center.
31 changes: 31 additions & 0 deletions versions/1.4.0/_sources/examples/colmap.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
Fit a COLMAP Capture
========================================

.. currentmodule:: gsplat

The :code:`examples/simple_trainer.py default` script allows you train a
`3D Gaussian Splatting <https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/>`_
model for novel view synthesis, on a COLMAP processed capture. This script follows the
exact same logic with the `official implementation
<https://github.com/graphdeco-inria/gaussian-splatting>`_ and we have verified it to be
able to reproduce the metrics in the paper, with much better training speed and memory
footprint. See :doc:`../tests/eval` for more details on the comparison.

Simply run the script under `examples/`:

.. code-block:: bash
CUDA_VISIBLE_DEVICES=0 python simple_trainer.py default \
--data_dir data/360_v2/garden/ --data_factor 4 \
--result_dir ./results/garden
It also supports a browser based viewer for real-time rendering, powered by
`Viser <https://github.com/nerfstudio-project/viser>`_ and
`nerfview <https://github.com/hangg7/nerfview>`_.

.. raw:: html

<video class="video" autoplay="" loop="" muted="" playsinline="", width="100%", height="auto">
<source src="../_static/viewer_garden_480p.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
25 changes: 25 additions & 0 deletions versions/1.4.0/_sources/examples/image.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
Fit a Single Image
===================================

.. currentmodule:: gsplat

The :code:`examples/image_fitting.py` script allows you to test the basic rasterization process
on a set of random gaussians and their differentiability on a single training image.

Simply run the script under `examples/`:

.. code-block:: bash
python image_fitting.py --height 256 --width 256 --num_points 2000 --save_imgs
to get a result similar to the one below:

.. image:: ../assets/square.gif
:alt: Gaussians overfit on a single image
:width: 256

You can also provide a path to your own custom image file using the ``--img_path`` flag:

.. code-block:: bash
python image_fitting.py --img_path PATH_TO_IMG --save_imgs
49 changes: 49 additions & 0 deletions versions/1.4.0/_sources/examples/large_scale.rst.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
Render a Large Scene
========================================

.. currentmodule:: gsplat

`gsplat` is designed with efficiency in mind so it's very suitable to render a large scene.
For example here we mimic a large scene by replicating the Garden scene into a 9x9 grid,
which results 30M Gaussians in total while `gsplat` still allows real-time rendering for it.

The main magic that allows this is a very simple trick: we disregard the Gaussians that are
far away from the camera, by applying a small threshold (e.g., 3 pixel) to the projected
Gaussian radius which is configurable in our :func:`rasterization` API as :code:`radius_clip`.

With `gsplat` as CUDA Backend:

.. raw:: html

<div style="position:relative; padding-bottom:56.25%; height:0; width:100%">
<iframe style="position:absolute; top:0; left:0; width:100%; height:100%" src="https://www.youtube.com/embed/rXyJwL7uJFQ?si=kQ10HcASlvhztTUA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
</div>

With `diff-gaussian-rasterization` as CUDA Backend:

.. raw:: html

<div style="position:relative; padding-bottom:56.25%; height:0; width:100%">
<iframe style="position:absolute; top:0; left:0; width:100%; height:100%" src="https://www.youtube.com/embed/kk3G3P68rkc?si=JJplV2ZY3PxE_5k7" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
</div>

Note: Similar to the `nerfstudio <https://docs.nerf.studio/>`_ viewer, our viewer automatically
switch to low resolution if the rendering is slow.

The code for this example can be found under `examples/`:

.. code-block:: bash
# First train a 3DGS model
CUDA_VISIBLE_DEVICES=0 python simple_trainer.py default \
--data_dir data/360_v2/garden/ --data_factor 4 \
--result_dir ./results/garden
# View it in a viewer with gsplat
python simple_viewer.py --scene_grid 5 --ckpt results/garden/ckpts/ckpt_6999.pt --backend gsplat
# Or, view it with inria's backend (requires to insteall `diff-gaussian-rasterization`)
python simple_viewer.py --scene_grid 5 --ckpt results/garden/ckpts/ckpt_6999.pt --backend inria
# Warning: a large `--scene_grid` might blow up your GPU memory.
Loading

0 comments on commit 4aa91db

Please sign in to comment.