How to use the hybrid kernel from the paper?

[benemer]

In Sec. 5.1 of the main paper "4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks", hybrid kernels are proposed to reduce the number of parameters by using a hypercubic kernel for the spatial and a hypercross kernel for the temporal dimension. However, it seems like the current version of the Minkowski Engine (0.5.4) does no longer support hybrid kernels. How can I use them?

---

import MinkowskiEngine as ME

kernel_size = 3
dimension = 4
list_region_types = [ME.RegionType.HYPER_CUBE, ME.RegionType.HYPER_CROSS]
for region_type in list_region_types:
    kernel_generator = ME.KernelGenerator(
        kernel_size=kernel_size,
        stride=1,
        dilation=1,
        is_transpose=False,
        region_type=region_type,
        region_offsets=None,
        expand_coordinates=False,
        axis_types=None,
        dimension=dimension,
    )
    print(
        "Kernel volume with region type {} is {}".format(
            str(region_type), kernel_generator.kernel_volume
        )
    )

axis_types = [
    ME.RegionType.HYPER_CUBE,
    ME.RegionType.HYPER_CUBE,
    ME.RegionType.HYPER_CUBE,
    ME.RegionType.HYPER_CROSS,
]
kernel_generator = ME.KernelGenerator(
    kernel_size=kernel_size,
    stride=1,
    dilation=1,
    is_transpose=False,
    region_type=ME.RegionType.HYBRID,
    region_offsets=None,
    expand_coordinates=False,
    axis_types=axis_types,
    dimension=dimension,
)
print("Kernel volume with hybrid kernel is {}.".format(kernel_generator.kernel_volume))

Output:

Kernel volume with region type RegionType.HYPER_CUBE is 81
Kernel volume with region type RegionType.HYPER_CROSS is 9
Traceback (most recent call last):
  File "test.py", line 35, in <module>
    region_type=ME.RegionType.HYBRID,
AttributeError: type object 'MinkowskiEngineBackend._C.RegionType' has no attribute 'HYBRID'

---

Expected behavior
As shown in the example code above, I would like to specify different kernel shapes for the spatial and temporal axis. According to the docs, this can be achieved by passing a list of RegionType as axis_types.

However, this results in

AssertionError: Axis types must be None when region_type is given

It is not possible to pass region_type=ME.RegionType.HYBRID since the hybrid RegionType is no longer available in Minkowski Engine 0.5.4 (see above).

How is the hybrid kernel of the paper implemented in the latest Minkowski Engine version?

---

Desktop:

• OS: Ubuntu 20.04
• Python version: 3.8.10
• Pytorch version: 1.10
• CUDA version: 10.1
• NVIDIA Driver version: 470.86
• Minkowski Engine version 0.5.4

[suyunzzz]

hello, have you solved this problem?
it seems that there are three RegionType in https://nvidia.github.io/MinkowskiEngine/common.html

i want to know if i use dim=3, i do not need use RegionType=HYBRID? HYBRID just for dim=4?

[benemer]

Hi, I was able to use the RegionType=HYBRID with ME version 4.3. However, a lot of other functionalities are not available here, therefore I would like to use the hybrid kernel with the latest version.

Regarding your question: The RegionType has nothing to do with the dimensions. The region type determines the shape of the kernel, see Fig. 3 of the paper for an example with dim=3, dim=2, and different kernel shapes. I think that the third region type CUSTOM allows to build a hybrid kernel of hypercross and hypercube sub-kernels. However, it seems like the CUSTOM region type is not yet available:

The custom region has not been ported to v0.5 yet. Sorry for the delay. I'll try to finish it in a few weeks.
Originally posted by @chrischoy in #355 (comment)

[Ltwicke]

Hello,

I too ran into the problem described in

• type object 'MinkowskiEngineBackend._C.RegionType' has no attribute 'HYBRID' in v0.5 #292
• [MinkowskiEngine 0.5.4][ME.RegionType.HYBRID vs ME.RegionType.CUSTOM] ME.RegionType.CUSTOM giving AttributeError: 'NoneType' object has no attribute 'numel' chrischoy/SpatioTemporalSegmentation#51
  and here

I feel like the main problem lies in he fact, that they explain nowhere, what the region_offset is exactly and how i can construct it. Does anyone know anything about that?

[izav-hub]

I'm having the same problems... Could anyone shed some light on the changes that need to be made to https://github.com/chrischoy/SpatioTemporalSegmentation in order to make it compatible with MinkowskiEngine v0.5? @chrischoy