Scene Semantic API returns a texture distorted when a device is moving or rotating.

[ForJobOk]

The feature is correctly working.

SimpleSegmentSample.mov

But, I faced a problem using Scene Semantic API when device is moving or rotating.
TryGetSemanticTexture and TryGetSemanticConfidenceTexture return a texture distorted like this capture.

SceneSemanticsAPI_TextureBug.1.mp4

Of course, we understand that violently moving or rotating the device is not supported.
However, it is prone to problems and can't be used at a production level unless there is a workaround.

Code

using System;
using System.Collections.Generic;
using Google.XR.ARCoreExtensions;
using UnityEngine;
using UnityEngine.UI;

public class SimpleUseSemantic : MonoBehaviour
{
    [SerializeField] private ARSemanticManager _semanticManager;
    [SerializeField] private Material _semanticMaterial;
    [SerializeField] private RawImage _segmentImage;
    [SerializeField] private RawImage _confidenceImage;
    [SerializeField] private List<LabelMapping> _mappings;

    [Serializable]
    public class LabelMapping
    {
        public SemanticLabel Label;
        public Color SemanticColor;
    }

    private Texture2D _semanticTexture;
    private Texture2D _confidenceTexture;

    private static readonly int LabelColorArray = Shader.PropertyToID("_LabelColorArray");

    private void Start()
    {
        var colorArray = _mappings.ConvertAll(mapping => mapping.SemanticColor).ToArray();
        _semanticMaterial.SetColorArray(LabelColorArray, colorArray);
    }

    void Update()
    {
        if (!_semanticManager.TryGetSemanticTexture(ref _semanticTexture))
        {
            _segmentImage.texture = null;
            return;
        }
        if (!_semanticManager.TryGetSemanticConfidenceTexture(ref _confidenceTexture))
        {
            _confidenceImage.texture = null;
            return;
        }

        _segmentImage.texture = _semanticTexture;
        _confidenceImage.texture = _confidenceTexture;
    }
}

Shader "Custom/SimpleUseSemantic"
{
    Properties
    {
        _MainTex ("Main Texture", 2D) = "white" {}
    }
    SubShader
    {
        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f
            {
                float2 uv : TEXCOORD0;
                float4 vertex : SV_POSITION;
            };

            sampler2D _MainTex;
            float4 _LabelColorArray[13];

            v2f vert(appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                o.uv = v.uv;
                return o;
            }

            float4 frag(v2f i) : SV_Target
            {
                float2 uv = float2(1.0 - i.uv.y, 1.0 - i.uv.x);
                float4 mainColor = tex2D(_MainTex, uv);
                int semanticLabel = round(mainColor.r * 255);
                semanticLabel = clamp(semanticLabel, 0, 12);
                return _LabelColorArray[semanticLabel];
            }
            ENDCG
        }
    }
}

Version

Unity 2022.3.24f1
ARCore Extensions 1.42.0
AR Foundation 5.1.3

[15kingben]

I don't think I fully understand the issue in the video. Is the problem the bits at the top and bottom which seem to be missing? Note that the Semantic Labels are only outdoor categories like trees, buildlings etc., so indoors the classification will not be very meaningful, and against a blank wall it will also not have much context to classify accurately. Do you have other examples where the performance is unacceptable?

[ForJobOk]

@15kingben
Thank you for your reply.

The problem occurs outside as well.
Also, the top and bottom bits that seem to be missing do not return 0 (Unlabeled).

[devbridie]

Could you share the shader code for the confidence image as well?

In this still from your video, the top part seems transparent; what values are those?

[devbridie]

Looked at this a little deeper and unfortunately this is working as intended:

• The semantics model runs at a lower rate than the camera system refresh rate. This means that not every camera image has a corresponding fresh Semantics image.
• When there is no fresh semantics image, the most recent semantics image is projected such that the pixels overlap with the expected translation and rotation of the device.
• This is causing the black bars you're observing at the top and the bottom: these are UNLABELED pixels that have "appeared" by means of panning since the last semantics update.

I will update the documentation to better reflect the state of these in between frames, but for now there aren't any changes related to this on the planning.

[ForJobOk]

GetSemanticLabelFraction has bug.
#204 (comment)