how to make feature dim arbitrary?

[CCCalcifer]

Hi, I encoounter an issue when I want to change the feature dim from 16 to 36 or 4. I found this is controlled by self.distill_feature_dim in ModelParams Class and self.distill_feature_dim must align with (self.sh_degree + 1) ** 2

But after I done above, and the feature dim of shs is equal to distill_feats

    rendered_image, rendered_feat, rendered_depth, rendered_gaussian_idx, radii = rasterizer(
        means3D = means3D,
        means2D = means2D,
        shs = shs,
        colors_precomp = colors_precomp,
        opacities = opacity,
        scales = scales,
        rotations = rotations,
        cov3D_precomp = cov3D_precomp,
        distill_feats = distill_feats)

but the feature dim of return value rendered_feat is still 16 whenever I change self.sh_degree to 1 or 5 (which self.distill_feature_dim is 4 or 36)

I wanna know how to change feature dim of rendered_feat?

[RogerQi]

Besides changing the Python code, you also need to modify the variable in CUDA rasterizer here:

feature-splatting-inria/submodules/diff-gaussian-rasterization/cuda_rasterizer/config.h

Line 16 in c902e40

#define NUM_FEAT_CHANNELS 16 // Default 768, CLIP ViT; has to be divisible by 2

[CCCalcifer]

Besides changing the Python code, you also need to modify the variable in CUDA rasterizer here:

feature-splatting-inria/submodules/diff-gaussian-rasterization/cuda_rasterizer/config.h

Line 16 in c902e40

#define NUM_FEAT_CHANNELS 16 // Default 768, CLIP ViT; has to be divisible by 2

Hi, thanks for your kind reply.
I try to change the value in file feature-splatting-inria/submodules/diff-gaussian-rasterization/cuda_rasterizer/config.h and recompile it use: sh build.sh

/*
 * Copyright (C) 2023, Inria
 * GRAPHDECO research group, https://team.inria.fr/graphdeco
 * All rights reserved.
 *
 * This software is free for non-commercial, research and evaluation use 
 * under the terms of the LICENSE.md file.
 *
 * For inquiries contact  [email protected]
 */

#ifndef CUDA_RASTERIZER_CONFIG_H_INCLUDED
#define CUDA_RASTERIZER_CONFIG_H_INCLUDED

#define NUM_COLOR_CHANNELS 3    // Default 3, RGB
#define NUM_FEAT_CHANNELS  36   // Default 768, CLIP ViT; has to be divisible by 2
#define BLOCK_X 16
#define BLOCK_Y 16

#define BLOCK_SIZE (BLOCK_X * BLOCK_Y)
#define NUM_WARPS (BLOCK_SIZE/32)

#define NUM_GAUSSIAN_LIMIT 100  // Trace the first 10 gaussians

#define FEATURE_SHARED_SIZE 36  // has to be divisible by 2

#endif

Meanwhile, I change the attribute in ModelParams class:

class ModelParams(ParamGroup): 
    def __init__(self, parser, sentinel=False):
        self.sh_degree = 5
        self.distill_feature_dim = (self.sh_degree + 1)  ** 2  # 36
        self._source_path = ""
        self._model_path = ""
        self._images = "images"
        self._resolution = -1
        self._white_background = False
        self.data_device = "cuda"
        self.eval = False
        super().__init__(parser, "Loading Parameters", sentinel)

but the output dim of rasterizer is still 16: rendered_feat.shape=torch.Size([16, 599, 800])

[RogerQi]

1. There is no need to change FEATURE_SHARED_SIZE. This is for CUDA coalescing to improve memory access efficiency and it has nothing to do with the number of feature channels.
2. Why is the distill_feature_dim correlated to sh_degree? These two parameters are supposed to be completely not related.
3. Are you sure you have run build.sh in the correct environment? Can you verify the output of the rasterizer by checking if the values returned immediately by the rasterizer wrapper here:

feature-splatting-inria/submodules/diff-gaussian-rasterization/diff_gaussian_rasterization/__init__.py

Line 104 in c902e40

num_rendered, color, rendered_feat_hwc, rendered_depth_hw, rendered_gaussian_idx_hwn, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)

matches the allocated buffer size here (which should match the value in CONFIG.h:

https://github.com/vuer-ai/feature-splatting-inria/blob/c902e40affe4d53512b25aaa6e4aa11e9d0c40ba/submodules/diff-gaussian-rasterization/rasterize_points.cu#L85C47-L85C64

[CCCalcifer]

1. There is no need to change FEATURE_SHARED_SIZE. This is for CUDA coalescing to improve memory access efficiency and it has nothing to do with the number of feature channels.
2. Why is the distill_feature_dim correlated to sh_degree? These two parameters are supposed to be completely not related.
3. Are you sure you have run build.sh in the correct environment? Can you verify the output of the rasterizer by checking if the values returned immediately by the rasterizer wrapper here:

feature-splatting-inria/submodules/diff-gaussian-rasterization/diff_gaussian_rasterization/__init__.py

Line 104 in c902e40

num_rendered, color, rendered_feat_hwc, rendered_depth_hw, rendered_gaussian_idx_hwn, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)

matches the allocated buffer size here (which should match the value in CONFIG.h:

https://github.com/vuer-ai/feature-splatting-inria/blob/c902e40affe4d53512b25aaa6e4aa11e9d0c40ba/submodules/diff-gaussian-rasterization/rasterize_points.cu#L85C47-L85C64

Thanks for your timely reply, I'll check it follow your instructions

[RogerQi]

Any updates on this?

[CCCalcifer]

I tried modify CONFIG.h and recompile by sh build.sh, and I successfully changed the feature dim this time. Thanks for your kind reply again~

[RogerQi]

That's good to know! I am closing the issue as it has been resolved.