-
Notifications
You must be signed in to change notification settings - Fork 316
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
4860cad
commit 4aa91db
Showing
81 changed files
with
21,221 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: 3ac07230479e67752436930c2be446df | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
780 changes: 780 additions & 0 deletions
780
versions/1.4.0/_modules/gsplat/compression/png_compression.html
Large diffs are not rendered by default.
Oops, something went wrong.
1,067 changes: 1,067 additions & 0 deletions
1,067
versions/1.4.0/_modules/gsplat/cuda/_torch_impl.html
Large diffs are not rendered by default.
Oops, something went wrong.
625 changes: 625 additions & 0 deletions
625
versions/1.4.0/_modules/gsplat/cuda/_torch_impl_2dgs.html
Large diffs are not rendered by default.
Oops, something went wrong.
2,364 changes: 2,364 additions & 0 deletions
2,364
versions/1.4.0/_modules/gsplat/cuda/_wrapper.html
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
Compression | ||
=================================== | ||
|
||
.. currentmodule:: gsplat | ||
|
||
`gsplat` provides handy APIs for compressing and decompressing the Gaussian parameters, | ||
which can significantly reduce the storage / streaming cost. For example, using :class:`PngCompression`, | ||
1 million Gaussians that are stored in 236 MB can be compressed to 16.5 MB, with only 0.5dB | ||
PSNR loss (29.18dB to 28.65dB). | ||
|
||
Similar to the :doc:`strategy`, our compression APIs also has a specific expectation on | ||
the format of the Gaussian parameters. It is designed to work with the Gaussians defined | ||
as either a Dict of | ||
`torch.Tensor <https://pytorch.org/docs/stable/tensors.html>`_ | ||
with at least the following keys: {"means", "scales", "quats", "opacities", "sh0", "shN"}. | ||
On top of these attributes, an arbitrary number of extra attributes are supported. | ||
|
||
The following code snippet is an example of how to use the compression approach in `gsplat`: | ||
|
||
.. code-block:: python | ||
from gsplat import PngCompression | ||
splats: Dict[str, Tensor] = { | ||
"means": Tensor(N, 3), "scales": Tensor(N, 3), "quats": Tensor(N, 4), "opacities": Tensor(N), | ||
"sh0": Tensor(N, 1, 3), "shN": Tensor(N, 24, 3), "features1": Tensor(N, 128), "features2": Tensor(N, 64), | ||
} | ||
compression_method = PngCompression() | ||
# run compression and save the compressed files to compress_dir | ||
compression_method.compress(compress_dir, splats) | ||
# decompress the compressed files | ||
splats_c = compression_method.decompress(compress_dir) | ||
Below is the API for the compression approaches we support in `gsplat`: | ||
|
||
.. autoclass:: PngCompression | ||
:members: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
Rasterization | ||
=================================== | ||
|
||
3DGS | ||
------ | ||
|
||
.. currentmodule:: gsplat | ||
|
||
Given a set of 3D gaussians parametrized by means :math:`\mu \in \mathbb{R}^3`, covariances | ||
:math:`\Sigma \in \mathbb{R}^{3 \times 3}`, colors :math:`c`, and opacities :math:`o`, we first | ||
compute their projected means :math:`\mu' \in \mathbb{R}^2` and covariances | ||
:math:`\Sigma' \in \mathbb{R}^{2 \times 2}` on the image planes. Then we sort each gaussian such | ||
that all gaussians within the bounds of a tile are grouped and sorted by increasing | ||
depth :math:`z`, and then render each pixel within the tile with alpha-compositing. | ||
|
||
Note, the 3D covariances are reparametrized with a scaling matrix | ||
:math:`S = \text{diag}(\mathbf{s}) \in \mathbb{R}^{3 \times 3}` represented by a | ||
scale vector :math:`s \in \mathbb{R}^3`, and a rotation matrix | ||
:math:`R \in \mathbb{R}^{3 \times 3}` represented by a rotation | ||
quaternion :math:`q \in \mathcal{R}^4`: | ||
|
||
.. math:: | ||
\Sigma = RSS^{T}R^{T} | ||
The projection of 3D Gaussians is approximated with the Jacobian of the perspective | ||
projection equation: | ||
|
||
.. math:: | ||
J = \begin{bmatrix} | ||
f_{x}/z & 0 & -f_{x} t_{x}/z^{2} \\ | ||
0 & f_{y}/z & -f_{y} t_{y}/z^{2} \\ | ||
0 & 0 & 0 | ||
\end{bmatrix} | ||
.. math:: | ||
\Sigma' = J W \Sigma W^{T} J^{T} | ||
Where :math:`[W | t]` is the world-to-camera transformation matrix, and :math:`f_{x}, f_{y}` | ||
are the focal lengths of the camera. | ||
|
||
.. autofunction:: rasterization | ||
|
||
2DGS | ||
------ | ||
|
||
Given a set of 2D gaussians parametrized by means :math:`\mu \in \mathbb{R}^3`, two principal tangent vectors | ||
embedded as the first two columns of a rotation matrix :math:`R \in \mathbb{R}^{3\times3}`, and a scale matrix :math:`S \in R^{3\times3}` | ||
representing the scaling along the two principal tangential directions, we first transforms pixels into splats' local tangent frame | ||
by :math:`(WH)^{-1} \in \mathbb{R}^{4\times4}` and compute weights via ray-splat intersection. Then we follow the sort and rendering similar to 3DGS. | ||
|
||
Note that H is the transformation from splat's local tangent plane :math:`\{u, v\}` into world space | ||
|
||
.. math:: | ||
H = \begin{bmatrix} | ||
RS & \mu \\ | ||
0 & 1 | ||
\end{bmatrix} | ||
and :math:`W \in \mathbb{R}^{4\times4}` is the transformation matrix from world space to image space. | ||
|
||
|
||
Splatting is done via ray-splat plane intersection. Each pixel is considered as a x-plane :math:`h_{x}=(-1, 0, 0, x)^{T}` | ||
and a y-plane :math:`h_{y}=(0, -1, 0, y)^{T}`, and the intersection between a splat and the pixel :math:`p=(x, y)` is defined | ||
as the intersection bwtween x-plane, y-plane, and the splat's tangent plane. We first transform :math:`h_{x}` to :math:`h_{u}` and :math:`h_{y}` | ||
to :math:`h_{v}` in splat's tangent frame via the inverse transformation :math:`(WH)^{-1}`. As the intersection point should fall on :math:`h_{u}` and :math:`h_{v}`, we have an efficient | ||
solution: | ||
|
||
.. math:: | ||
u(p) = \frac{h^{2}_{u}h^{4}_{v}-h^{4}_{u}h^{2}_{v}}{h^{1}_{u}h^{2}_{v}-h^{2}_{u}h^{1}_{v}}, | ||
v(p) = \frac{h^{4}_{u}h^{1}_{v}-h^{1}_{u}h^{4}_{v}}{h^{1}_{u}h^{2}_{v}-h^{2}_{u}h^{1}_{v}} | ||
.. autofunction:: rasterization_2dgs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
Densification | ||
=================================== | ||
|
||
.. currentmodule:: gsplat | ||
|
||
In `gsplat`, we abstract out the densification and pruning process of the Gaussian | ||
training into a strategy. A strategy is a class that defines how the Gaussian parameters | ||
(along with their optimizers) should be updated (splitting, pruning, etc.) during the training. | ||
|
||
An example of the training workflow using :class:`DefaultStrategy` is like: | ||
|
||
.. code-block:: python | ||
from gsplat import DefaultStrategy, rasterization | ||
# Define Gaussian parameters and optimizers | ||
params: Dict[str, torch.nn.Parameter] | torch.nn.ParameterDict = ... | ||
optimizers: Dict[str, torch.optim.Optimizer] = ... | ||
# Initialize the strategy | ||
strategy = DefaultStrategy() | ||
# Check the sanity of the parameters and optimizers | ||
strategy.check_sanity(params, optimizers) | ||
# Initialize the strategy state | ||
strategy_state = strategy.initialize_state() | ||
# Training loop | ||
for step in range(1000): | ||
# Forward pass | ||
render_image, render_alpha, info = rasterization(...) | ||
# Pre-backward step | ||
strategy.step_pre_backward(params, optimizers, strategy_state, step, info) | ||
# Compute the loss | ||
loss = ... | ||
# Backward pass | ||
loss.backward() | ||
# Post-backward step | ||
strategy.step_post_backward(params, optimizers, strategy_state, step, info) | ||
A strategy will inplacely update the Gaussian parameters as well as the optimizers, | ||
so it has a specific expectation on the format of the parameters and the optimizers. | ||
It is designed to work with the Guassians defined as either a Dict of | ||
`torch.nn.Parameter <https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html>`_ | ||
or a | ||
`torch.nn.ParameterDict <https://pytorch.org/docs/stable/generated/torch.nn.ParameterDict.html>`_ | ||
with at least the following keys: {"means", "scales", "quats", "opacities"}. On top of these attributes, | ||
an arbitrary number of extra attributes are supported. Besides the parameters, it also | ||
expects a Dict of `torch.optim.Optimizer <https://pytorch.org/docs/stable/optim.html>`_ | ||
with the same keys as the parameters, and each optimizer should correspond to only | ||
one learnable parameter. | ||
|
||
For example, the following is a valid format for the parameters and the optimizers | ||
that can be used with our strategies: | ||
|
||
.. code-block:: python | ||
N = 100 | ||
params = torch.nn.ParameterDict{ | ||
"means": Tensor(N, 3), "scales": Tensor(N), "quats": Tensor(N, 4), "opacities": Tensor(N), | ||
"colors": Tensor(N, 25, 3), "features1": Tensor(N, 128), "features2": Tensor(N, 64), | ||
} | ||
optimizers = {k: torch.optim.Adam([p], lr=1e-3) for k, p in params.keys()} | ||
Below are the strategies that are currently implemented in `gsplat`: | ||
|
||
.. autoclass:: DefaultStrategy | ||
:members: | ||
|
||
.. autoclass:: MCMCStrategy | ||
:members: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
|
||
Utils | ||
=================================== | ||
|
||
Below are the basic functions that supports the rasterization. | ||
|
||
3DGS | ||
----- | ||
|
||
.. currentmodule:: gsplat | ||
|
||
.. autofunction:: spherical_harmonics | ||
|
||
.. autofunction:: quat_scale_to_covar_preci | ||
|
||
.. autofunction:: proj | ||
|
||
.. autofunction:: fully_fused_projection | ||
|
||
.. autofunction:: isect_tiles | ||
|
||
.. autofunction:: isect_offset_encode | ||
|
||
.. autofunction:: world_to_cam | ||
|
||
.. autofunction:: rasterize_to_pixels | ||
|
||
.. autofunction:: rasterize_to_indices_in_range | ||
|
||
.. autofunction:: accumulate | ||
|
||
.. autofunction:: rasterization_inria_wrapper | ||
|
||
2DGS | ||
----- | ||
.. currentmodule:: gsplat | ||
|
||
.. autofunction:: fully_fused_projection_2dgs | ||
|
||
.. autofunction:: rasterize_to_pixels_2dgs | ||
|
||
.. autofunction:: rasterize_to_indices_in_range_2dgs | ||
|
||
.. autofunction:: accumulate_2dgs | ||
|
||
.. autofunction:: rasterization_2dgs_inria_wrapper |
23 changes: 23 additions & 0 deletions
23
versions/1.4.0/_sources/conventions/data_conventions.rst.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
Data Conventions | ||
=================================== | ||
|
||
.. currentmodule:: gsplat | ||
|
||
Here we explain the various data conventions used in our repo. | ||
|
||
Rotation Convention | ||
------------------- | ||
We represent rotations with four dimensional vectors :math:`q = (w,x,y,z)` such that the 3x3 :math:`SO(3)` rotation matrix is defined by: | ||
|
||
.. math:: | ||
R = \begin{bmatrix} | ||
1 - 2 \left( y^2 + z^2 \right) & 2 \left( x y - w z \right) & 2 \left( x z + w y \right) \\ | ||
2 \left( x y + w z \right) & 1 - 2 \left( x^2 + z^2 \right) & 2 \left( y z - w x \right) \\ | ||
2 \left( x z - w y \right) & 2 \left( y z + w x \right) & 1 - 2 \left( x^2 + y^2 \right) \\ | ||
\end{bmatrix} | ||
View Matrix | ||
--------------------------------- | ||
We refer to the `view matrix` :math:`W` as the world to camera frame transformation (referred to as `w2c` in some sources) that maps | ||
3D world points :math:`(x,y,z)_{world}` to 3D camera points :math:`(x,y,z)_{cam}` where :math:`z_{cam}` is the relative depth to the camera center. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
Fit a COLMAP Capture | ||
======================================== | ||
|
||
.. currentmodule:: gsplat | ||
|
||
The :code:`examples/simple_trainer.py default` script allows you train a | ||
`3D Gaussian Splatting <https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/>`_ | ||
model for novel view synthesis, on a COLMAP processed capture. This script follows the | ||
exact same logic with the `official implementation | ||
<https://github.com/graphdeco-inria/gaussian-splatting>`_ and we have verified it to be | ||
able to reproduce the metrics in the paper, with much better training speed and memory | ||
footprint. See :doc:`../tests/eval` for more details on the comparison. | ||
|
||
Simply run the script under `examples/`: | ||
|
||
.. code-block:: bash | ||
CUDA_VISIBLE_DEVICES=0 python simple_trainer.py default \ | ||
--data_dir data/360_v2/garden/ --data_factor 4 \ | ||
--result_dir ./results/garden | ||
It also supports a browser based viewer for real-time rendering, powered by | ||
`Viser <https://github.com/nerfstudio-project/viser>`_ and | ||
`nerfview <https://github.com/hangg7/nerfview>`_. | ||
|
||
.. raw:: html | ||
|
||
<video class="video" autoplay="" loop="" muted="" playsinline="", width="100%", height="auto"> | ||
<source src="../_static/viewer_garden_480p.mp4" type="video/mp4"> | ||
Your browser does not support the video tag. | ||
</video> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
Fit a Single Image | ||
=================================== | ||
|
||
.. currentmodule:: gsplat | ||
|
||
The :code:`examples/image_fitting.py` script allows you to test the basic rasterization process | ||
on a set of random gaussians and their differentiability on a single training image. | ||
|
||
Simply run the script under `examples/`: | ||
|
||
.. code-block:: bash | ||
python image_fitting.py --height 256 --width 256 --num_points 2000 --save_imgs | ||
to get a result similar to the one below: | ||
|
||
.. image:: ../assets/square.gif | ||
:alt: Gaussians overfit on a single image | ||
:width: 256 | ||
|
||
You can also provide a path to your own custom image file using the ``--img_path`` flag: | ||
|
||
.. code-block:: bash | ||
python image_fitting.py --img_path PATH_TO_IMG --save_imgs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
Render a Large Scene | ||
======================================== | ||
|
||
.. currentmodule:: gsplat | ||
|
||
`gsplat` is designed with efficiency in mind so it's very suitable to render a large scene. | ||
For example here we mimic a large scene by replicating the Garden scene into a 9x9 grid, | ||
which results 30M Gaussians in total while `gsplat` still allows real-time rendering for it. | ||
|
||
The main magic that allows this is a very simple trick: we disregard the Gaussians that are | ||
far away from the camera, by applying a small threshold (e.g., 3 pixel) to the projected | ||
Gaussian radius which is configurable in our :func:`rasterization` API as :code:`radius_clip`. | ||
|
||
With `gsplat` as CUDA Backend: | ||
|
||
.. raw:: html | ||
|
||
<div style="position:relative; padding-bottom:56.25%; height:0; width:100%"> | ||
<iframe style="position:absolute; top:0; left:0; width:100%; height:100%" src="https://www.youtube.com/embed/rXyJwL7uJFQ?si=kQ10HcASlvhztTUA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe> | ||
</div> | ||
|
||
With `diff-gaussian-rasterization` as CUDA Backend: | ||
|
||
.. raw:: html | ||
|
||
<div style="position:relative; padding-bottom:56.25%; height:0; width:100%"> | ||
<iframe style="position:absolute; top:0; left:0; width:100%; height:100%" src="https://www.youtube.com/embed/kk3G3P68rkc?si=JJplV2ZY3PxE_5k7" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe> | ||
</div> | ||
|
||
Note: Similar to the `nerfstudio <https://docs.nerf.studio/>`_ viewer, our viewer automatically | ||
switch to low resolution if the rendering is slow. | ||
|
||
The code for this example can be found under `examples/`: | ||
|
||
.. code-block:: bash | ||
# First train a 3DGS model | ||
CUDA_VISIBLE_DEVICES=0 python simple_trainer.py default \ | ||
--data_dir data/360_v2/garden/ --data_factor 4 \ | ||
--result_dir ./results/garden | ||
# View it in a viewer with gsplat | ||
python simple_viewer.py --scene_grid 5 --ckpt results/garden/ckpts/ckpt_6999.pt --backend gsplat | ||
# Or, view it with inria's backend (requires to insteall `diff-gaussian-rasterization`) | ||
python simple_viewer.py --scene_grid 5 --ckpt results/garden/ckpts/ckpt_6999.pt --backend inria | ||
# Warning: a large `--scene_grid` might blow up your GPU memory. | ||
Oops, something went wrong.