Enum Flags throw error if user already defined Everything/Nothing

[ramonvanschaik]

First of all, thanks for creating this great tool.

My issue arises when creating enum flags, I usually already include the everything and nothing in my own code. For example:

[System.Flags]
public enum COLLIDEWITH {
  Nothing = 0,
  Everything = ~0,
  Player = 1 << 1,
  Enemy = 1 << 2,
  Wall = 1 << 3,
}

Implementing an enum flag like above in a monobehaviour will prevent the inspector from being drawn and writes an error to the console:

IndexOutOfRangeException: Index was outside the bounds of the array.
UnityEditor.MaskFieldGUI.CalculateMaskValues (System.Int32 mask, System.Int32[] flagValues, System.Int32[]& optionMaskValues) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.MaskFieldGUI.GetMenuOptions (System.Int32 mask, System.String[] flagNames, System.Int32[] flagValues, System.String& buttonText, System.String& buttonMixedValuesText, System.String[]& optionNames, System.Int32[]& optionMaskValues, System.Int32[]& selectedOptions, System.Type enumType) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.MaskFieldGUI.DoMaskField (UnityEngine.Rect position, System.Int32 controlID, System.Int32 mask, System.String[] flagNames, System.Int32[] flagValues, UnityEngine.GUIStyle style, System.Int32& changedFlags, System.Boolean& changedToValue, System.Type enumType) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.EditorGUI.EnumFlagsField (UnityEngine.Rect position, UnityEngine.GUIContent label, System.Int32 enumValue, System.Type enumType, System.Boolean includeObsolete, UnityEngine.GUIStyle style) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.EditorGUI.EnumPopup (UnityEngine.Rect position, UnityEditor.SerializedProperty property, UnityEngine.GUIContent label) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.EditorGUI.DefaultPropertyField (UnityEngine.Rect position, UnityEditor.SerializedProperty property, UnityEngine.GUIContent label) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.PropertyHandler.OnGUI (UnityEngine.Rect position, UnityEditor.SerializedProperty property, UnityEngine.GUIContent label, System.Boolean includeChildren, UnityEngine.Rect visibleArea) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.PropertyHandler.OnGUI (UnityEngine.Rect position, UnityEditor.SerializedProperty property, UnityEngine.GUIContent label, System.Boolean includeChildren) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.PropertyHandler.OnGUILayout (UnityEditor.SerializedProperty property, UnityEngine.GUIContent label, System.Boolean includeChildren, UnityEngine.GUILayoutOption[] options) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.EditorGUILayout.PropertyField (UnityEditor.SerializedProperty property, UnityEngine.GUIContent label, System.Boolean includeChildren, UnityEngine.GUILayoutOption[] options) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEditor.EditorGUILayout.PropertyField (UnityEditor.SerializedProperty property, System.Boolean includeChildren, UnityEngine.GUILayoutOption[] options) (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
MyBox.Internal.FoldoutAttributeHandler.Body () (at ./Library/PackageCache/com.domybest.mybox@3950fccd96/Attributes/FoldoutAttribute.cs:105)
MyBox.Internal.FoldoutAttributeHandler.OnInspectorGUI () (at ./Library/PackageCache/com.domybest.mybox@3950fccd96/Attributes/FoldoutAttribute.cs:75)
MyBox.Internal.UnityObjectEditor.OnInspectorGUI () (at ./Library/PackageCache/com.domybest.mybox@3950fccd96/Tools/Internal/UnityObjectEditor.cs:34)
UnityEditor.UIElements.InspectorElement+<>c__DisplayClass76_0.<CreateInspectorElementUsingIMGUI>b__0 () (at <50e19b0ed93a4256b2e454b4acec2d48>:0)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

My guess my code seems to clash with something in MyBox, I guess having to do with #47. I found out I could fix it by not declaring everything and nothing anymore, which is okay for me. Not sure if this can be fixed, otherwise a more specific error telling the user not to declare everything and nothing themselves might be helpful, because it took me some time before I figured out what was wrong.

[Deadcows]

Whoa, that's strange! In fact, in #47, I don't do anything with MyBox — I just provide a brief explanation to the guy about how the Flags work. Unity automatically adds Nothing and Everything internally, and it seems we've encountered a bug in Unity here. The issue arises when an enum field with Flags and Nothing or Everything is drawn via EditorGUILayout.PropertyField, and I can't see any workaround for it. EditorGUILayout.PropertyField is used to redraw the entire inspector for each behavior to allow FoldoutAttribute and ButtonMethod to function properly.

No worries, though! This behavior is optional. Just toggle off "Inspector override" and it should work fine.

Let me know if it's crucial for you to keep Foldout or ButtonMethod in your project. I'll work on disabling the Inspector override per MB or SO where you are using the enums in question — that's the best solution I can offer for now. In the meantime, I'll put together an issue report for the Unity team to investigate 👍🏻

[ramonvanschaik]

Thanks! ButtonMethod is crucial for me but I've worked around it by removing all declarations of Nothing in my code (it turned out that Everything was allowed, it' just drawn twice).

[ramonvanschaik]

I couldn't resist trying for a hotfix while the Unity bug still exists. What I came up with was to change part of the Body method of FoldoutAttribute.cs to something like this:

for (var i = 1; i < _props.Count; i++)
{
    if (_props[i].propertyType==SerializedPropertyType.Enum) {
        _props[i].enumValueFlag = EditorGUILayout.MaskField(_props[i].enumValueFlag, _props[i].enumNames);
    } else {
        EditorGUILayout.PropertyField(_props[i], true);
    }
}

Still some work to be done there as it doesn't differentiate between flags and enums yet. However, I can imagine that as a principal you don't want hotfixes for Unity bugs in the code, as it makes the code prone for future bugs. But if you think it's a path worth pursuing, just let me know and I'll see if I can create a PR.

[Deadcows]

Oh my, that's the reason I shouldn't work late at night 😅 The idea to try specific Enum and Mask Fields instead of PropertyField is evaded from me. Glad that it works! PR is welcomed 👐🏻