Skip to content

Home

Jump to bottom
Antoine Lelievre edited this page Jan 13, 2021 · 6 revisions

Upload Photo

Article

Mastering UnityEditor Handles

Learn to use builtin handles, improve them and create completely custom handles

Handles overview

====================

box collider handle

Position Handle

Handles in unity are a very powerful tool to accelerate the development of your game and enhance your editor scripts, they are widely used by unity's default components to give fast and accurate control over different things like colliders bounds or the radius of a point light. Even the transform tools provided by the editor to move/rotate/scale objects are handles, it gives you an idea of how useful they are!

And now, with this article you'll learn to create your own handle, change the control you have over an object in the scene view and add a lot of useful debug tools to your projects. It'll give you the possibility to develop the perfect tool that fits your needs whether it is to create an "attack pattern editor" or a "3D shape editor', we'll also see two examples of custom handles I created for this article (available here): the curve handle and the snap move tool. With that said, let's get started with some theory and docs!

How to use Handles?

You can use handles in every unity scripts unless you have UnityEditor included. It will be easier on the custom editor which inherit from Editor and so have the OnSceneGUI message where you can put all your Handles calls. But it's possible to draw to implement our own OnSceneGUI by subscribing to the SceneView.onSceneGUIDelegate (not documented) event like this:

//OnSceneGUI for [EditorWindow](https://docs.unity3d.com/ScriptReference/EditorWindow.html) void OnEnable() {
  SceneView.onSceneGUIDelegate += OnSceneGUI;
} void OnDisable() {
  SceneView.onSceneGUIDelegate -= OnSceneGUI;
} void OnSceneGUI(SceneView sv) { //Draw your handles here }

This method has another advantage: it gives you access to the current SceneView but unfortunately, this class is totally undocumented but it contains really powerful functions to control the current scene view camera (LookAt, Move to, 2d mode, align view, ....) if you're interested you can check the source code of this class on the official Unity C# reference repository.

Once you are in a good context to call your handles, you just have to choose which handle you want to use in the list of Unity Handles. There are some additional settings that cannot be passed as parameters like the handle colors, or if it's lit, these parameters can be set with properties in the Handle class: Handle color, Handle lighting. Some of the Handle functions will take a CapFunction in parameter, the purpose of these functions is to manage the visual and "grab" parts of the Handle, so you can have one handle control with multiple visuals (sphere, dot, capsule, ...). By default, there is a couple of Cap functions (they all end with "HandleCap") that you can use in the Handles class.

Cap and Handles functions

Inside the Handles class, you have three groups of functions (plus some utils): Draw, Cap, and Handle functions.

  • About the Draw function, as their name says, these functions draw something to the current Handle camera, you can use them to debug or improve your visualization tools but they can't interact with user input.

  • Cap functions are used to manage both the visual and control parts of the handle. They act during two event passes: Repaint and Layout, during the repaint pass the cap function draws what it has to draw, and in the layout, it calls the AddControl function to record the distance from the mouse to the visual.

  • And finally, Handle functions provide control over the cap function given in parameter (it generates a controlId, catch event relative to the handle, and then call the cap function with its control id). For example, the FreeMoveHandle will allow his cap function to be dragged in 3D inside the scene view.

The size parameter of HandleCap functions can be dynamic or static (like when you switch off 3D icons in the scene view). A dynamic size in comparison with a static size (which is a simple value), is calculated HandleUtility.GetHandleSize which takes a 3D position and returns a size based on how far is the camera from the position so your handle will appear to be the same size if you're near or far from it.

Better uses of builtin Handles

Arc Handle

Box Bounds Handle

Joint Angular Limit Handle

I didn't found a better way to introduce this section so it will just be a list of cool things to know when you want to use Handles :)

Some of the Draw and Cap function in the Handle class does not let you change the scale of the visual but there is a simple way to change it: DrawingScope, this function will push a matrix and a color in parameter and use them in Handles.matrix and Handles.color so you just have to put a rotation matrix in the DrawingScope and then draw every Handle you need.

In Unity, there are some Handles that are not in the Handles class but in the IMGUI.ControlsOther class, these are powerful Handles and can be easily implemented (clickable images beside).

You can write text to the current camera with Label, this function support styling so you can customize the font, color, or size of your text but it does not support 3D text.

HandleUtility.Repaint can be used to Repaint the current camera if you have a dynamic Handle which moves without users' event.

If you need to raycast from your mouse, HandleUtility.GUIPointToWorldRay can be used to get the current ray from the mouse position.

Creating custom Handles

===========================

Here we are, the core of the article and certainly the most important part for you if you want to create good editor tools :) In this section I'll support my explanation with some code, I recommend you to take a look at the project I create for this article, it is fully available here: MasteringUnityHandles. It contains a simple Free2DMoveHandle, KeyframeHandle, and a CurveHandle (thumbnail of the article), moreover, there is an editor window (in Window/Handles Examples tab) which list some Handles draw functions, all IMGUI handles and my custom handles.

In this section, I'll take for example the Free2DMoveHandle I created, it's a rather simple handle so I won't be too complex to be a start point. The purpose of this Handle is to allow a point to move freely on a plane in 2D and display a texture or a square facing the camera at the position of the point.

Custom controls

First, for our Handle, we need a controlId. The controlId will be used to tell which Handle we are controlling so it's essential if we don't want to break everything to make good use of the controlId. We use GetControlId to generate a new controlId, it needs to be called every frame so at the beginning of our Handle function we put this:

int controlId = EditorGUIUtility.GetControlID(controlIdHash, FocusType.Keyboard);

The first parameter is a hint to help the id generation, it's a good practice to put a hash of a string containing the name of your handle in this parameter "Free2DMoveHandle".GetHashCode(). The second parameter is a FocusType which specifies if the control can catch the keyboard focus (by pressing tab for example) or not.

Now that we have our controlId let's do a switch to handle our events.

switch (Event.current.type)
{ case EventType.MouseDown: //check nearest control and set hotControl/keyboardControl break ; case EventType.MouseUp: //check if i'm controlled and set hotControl/keyboardControl to 0 break ; case EventType.MouseDrag: //if i'm controlled, move the point break ; case EventType.Repaint: //draw point visual break ; case EventType.Layout: //register distance from mouse to my point break ;
}

On MouseDown we use HandleUtility.nearestControl to check the nearest controlId in the scene. This field was updated during the last Layout pass using the lowest value recorded in AddControl, this guarantees that only one Handle will be a controller at the same time. It also lets you know when the mouse is near your handle which is cool for an over effect. So we check if the nearest control is our control and if we pressed the left mouse button. Then I assigned both hotControl and keyboardControl to my controlId to tell others Handles that I have the focus of the user.

if (HandleUtility.nearestControl == controlId && e.button == 0)
{
	GUIUtility.hotControl = controlId;
	GUIUtility.keyboardControl = controlId;
	e.Use();
}

As written in the doc, hotControl ensures that no other objects will take control until you reset it to 0. I choose to assign keyboardControl too because I wanted the focus to persist after we release the mouse so the keyboardControl field can keep track of my controlId and I can check if my object is selected if hotControl or keyboardControl is equal to my controlId.

Then on MouseUp we have to reset hotControl if we are controlled:

if (GUIUtility.hotControl == controlId && e.button == 0)
{
	GUIUtility.hotControl = 0;
	e.Use();
}

The last event to manage for controls is Layout I'll talk about the Repaint event in the next part. During this event, we need to call AddControl to record the distance between the mouse and our handle. To get this distance there are some utils in the HandleUtility class to help us, here I used the DistanceToRectangle function, it just takes the world position of the rectangle, it's rotation and it's size to calculate the closest distance between the mouse and this rect for us. For the rotation of the rectangle, I used the rotation of the current camera rotation so the distance will fit the visual. To get my world position I have to take the position I got on my 2D plane (position) and multiply it by a TRS matrix that I calculate before.

Vector3 pointWorldPos = matrix.MultiplyPoint3x4(position);
float distance = HandleUtility.DistanceToRectangle(pointWorldPos, Camera.current.transform.rotation, size); HandleUtility.AddControl(controlId, distance);

Custom visual

During the Repaint pass, you have to draw the visual of your handle, to do that you have multiple choices, it depends on what you want to draw. If you want a geometry that changes when the user interacts with the handle you must use legacy Opengl instant mode (it's an old deprecated way to draw objects during the render loop but unity still use it inside the editor maybe because it's simpler to write ?), an example of procedural geometry with an existing Handle is the Arc handle, at runtime it generates the vertices of the arc depending on his angle and then draws it. On the other side if you you have static geometry you can simply use a mesh loaded with Resources.Load or EditorGUIUtility.Load and draw it with Graphics.DrawMeshNow. The mesh technique is obviously faster than with legacy Opengl (by almost a 10x factor) even by recreating the mesh from scratch each repaint frames (i made a little bench with 30k triangles in my project) so you can also use it for procedural geometry but it can be really boring to write procedural mesh generation with the current mesh API so I'll show you the legacy way for procedural geometry.

For our quad, we are going to use legacy OpenGL to draw a quad since even if it's slower than drawing a mesh it remains instant so we don't have to care about that. The first thing we need to do is to generate the top right corner position of our quad, this is pretty simple since our quad is facing the camera:

Vector3 worldPos = matrix.MultiplyPoint3x4(position); Vector3 camRight = Camera.current.transform.right * size; Vector3 camUp = Camera.current.transform.up * size;

We also got the world position of our handle like in the Layout pass. Then with these data we can draw out quad:

GL.Begin(GL.QUADS);
{
	GL.Vertex(worldPos + camRight + camUp);
	GL.Vertex(worldPos + camRight - camUp);
	GL.Vertex(worldPos - camRight - camUp);
	GL.Vertex(worldPos - camRight + camUp);
}
GL.End();

Now there is just the last step before you can see your handle: set it a material. In fact, if you just run your code like this you'll see nothing because the material assigned to this geometry comes from the last object drawn by the editor, so you have to assign your material before the GL.End. To do that you need a Material (it can be loaded from an asset or it can be created at runtime) once you have handled the material you need to call the SetPass function to bind the material. We generally put 0 in the pass count parameter but if you're using a more complex material you must want to use another number. In my project I've created a static class that store all my materials for Handles so I can access them like this:

HandlesMaterials.textured.SetPass(0);

If you want to add a texture like me, you'll need UVs in your geometry, this can be done by adding GL.TexCoord2(uv.x, uv.y) before each GL.vertex to set UV per vertices. Then you'll have to specify your texture at your material before calling SetPass, and it's obvious but your material must have a Texture2D field in its properties. In some other cases, you only want to draw colors and don't care about texture, in my curve Handle it's the case, to do it you have to support vertex coloring in your material which allows your shader to get the per-vertex colors specified with GL.Color(color) and then blend between them. I've written a shader that does it available here if anyone is interested.

procedural geometry / mesh / material / vertex coloring

Free2DMoveHandle Finish up

There are few things left to do to get our handle to work, indeed we didn't manage the MouseDrag event in our switch yet. During this pass, We have to cast a ray from the mouse, intersect it with our 2D plane, and move the position to this new intersection. Getting the mouse ray:

HandleUtility.GUIPointToWorldRay(Event.current.mousePosition);

Then we create a plane with our current matrix as rotation, and intersect with our ray:

float dist;

Plane plane = new Plane(matrix * Vector3.forward, matrix.MultiplyPoint(Vector3.zero));
plane.Raycast(ray, out dist); if (dist > 0)
	position = ray.GetPoint(dist);

We also have to check if the hit distance is positive before assigning our position so the position will not move if the ray does not hit the plane.

Another last thing you have to do during the MouseDrag pass is to tell the GUI system that something has been changed because the user interacted with your handle this is very important if you want your handle to support Undo/Redo. Here is my simple implementation of this:

if (Event.current.delta != Vector2.zero)
	GUI.changed = true;

With this little piece of code, when the mouse is dragging our handle, the GUI state is set to "changed".

And here we are, it's finished! You have a fully functional 2D move handle. Now I'll show you how to use my implementation of this handle (you can also find it in the HandlesExtended.cs file):

static Free2DMoveHandle	free2DMoveHandle = new Free2DMoveHandle(); public static void Free2DMoveHandle(ref Vector2 position, float size, Quaternion rotation) {
	free2DMoveHandle.matrix = Matrix4x4.TRS(Vector3.zero, rotation, Vector3.one);
	free2DMoveHandle.faceCamera = true;
	free2DMoveHandle.DrawHandle(ref position, size);
}

First, we create an instance of our Handle, then we set the rotation of the plane, by default without rotation this plane is the XY plane so you must take this into account when you rotate the plane. Then we set faceCamera to true, I've implemented another mode in my handle which faces the current plane, I thought that can be useful in some cases. And finally, we call our handle draw function which will directly go into the switch we created before.

Custom Cap Handle function

Now that you know how to recreate a basic handle you can improve existing handles visuals, maybe you just want to create a strange form with existing controls (for example a Cylinder you can move around ?) then you might want to use FreeMoveHandle but with a custom Cap function that draws a cylinder. Let's give it a try!

First, looking at the CapFunction delegate in the Handles class we need our function to have the same parameters and then we need to manage Repaint and Layout events:

static void CylinderHandleCap(int controlId, Vector3 position, Quaternion rotation, float size, EventType eventType) { if (eventType == EventType.Repaint)
	{
		HandlesMaterials.vertexColor.SetPass(0);
		Graphics.DrawMeshNow(cylinderMesh, position, Quaternion.identity);
	} else if (eventType == EventType.Layout)
	{ float cylinderHeight = 3.5f;
		Vector3 startPosition = position + Vector3.up * (cylinderHeight / 2); float distance = 1e20f; for (int i = 0; i < 9; i++)
		{
			distance = Mathf.Min(distance, HandleUtility.DistanceToCircle(startPosition, size / 2));
			startPosition -= Vector3.up * cylinderHeight / 8;
		}
			
		HandleUtility.AddControl(controlId, distance);
	}
}

During the Repaint pass, we set the material for our mesh and draw our cylinder mesh at the Handle position, unfortunately, the rotation in the parameter is not the rotation passed to the FreeMoveHandle, it's the current camera rotation so if you use this rotation to draw the mesh, it will always face the camera that why I used Quaternion.identity instead.

The Layout part may seem a bit strange but it's because evaluating a distance between a point and a rotating cylinder is very complex and annoying so I used multiple distances from a circle and combined them with Mathf.Min to create a distance to the cylinder and it works pretty well.

And here we are, a functional cylinder custom cap, you just have to put this function in the parameter of FreeMoveHandle or any other Handle functions and it will display a movable cylinder.

Undo and Redo

This is a part I wanted to talk about, undo/redo is very important when creating a new tool in Unity (or any other software) and can ruin the user experience if not/bad implemented. Unfortunately, in unity, there is nearly no way to implement a custom Undo/Redo system and we have to use the unity implementation of this system. The problem with this is that the unity undo only supports unity objects so our curve handle or free 2D move handle can't benefit from it (because AnimationCurve or Vector2 does not inherit from Object). So as standalone our handles can't benefit from the editor system, we have to link them to an existing object to be saved. Indeed handles never implements an undo/redo system but the script that draws the handle does and so when you change the object data with a handle, the script detects if the handle has been changed with GUI.changed and records the object modification with Undo.RecordObject. You can see an implementation of this in the CurveScriptableObjectEditor.cs file in my project, and here is a simple implementation of this with my curveEditor:

AnimationCurve modifiedCurve = curveHandle.DrawHandle(curve); if (GUI.changed)
{
	Undo.RecordObject(target, "Changed curve");
	curve = modifiedCurve;
}

Furthermore, you'll notice that I assigned my curve to the modified curve after recording the object, this is very important to do that in this order, if you don't you'll get inconsistencies in the undo/redo behavior.

If you go to my project in the MasterUnityHandles/ExamplesScenes/curves folder you'll find some scriptable objects with curves within and a handle in the scene view for modifying them. As you can see, the Undo/Redo works perfectly on the curve thanks to this little piece of code placed in the editor script of the scriptable objects.

Example: curve handle

To complement this article, I decided to add two different examples of custom handles implementation (wit full sources available in my project). I'll try to explain every part of the code that I find interesting without being redundant with the explanations provided above. The first example we're reviewing is the Curve Handle:

The Curve Handle work with two other Handles I created for this curve Handle: the free 2D move handle and the keyframe handle. We saw the first in detail so I'll skip to the keyframe Handle.

Before to start a bit of context: the pointHandle and tangentHandle fields are Free2DMoveHandles and are here to draw the center of the keyframe and his tangents, then the e field is Event.current coming from the CustomHandle base class. In the DrawHandle function, the first thing you can notice is that the right mouse down event calls a function to display a context menu, in the condition you can also see that I check the nearestControl with my center point control id (that's why they are public) to know when my mouse is over the center point.

if (e.type == EventType.MouseDown)
{ //we add the context menu when right clicking on the point Handle: if (HandleUtility.nearestControl == pointHandle.controlId && e.button == 1)
		KeyframeContextMenu(keyframe);
}

context menu in the scene view

If this condition is true, then KeyframeContextMenu is called. This function is pretty simple and just create a GenericMenu, add items if there is a curve linked to the keyframe and show it (pic beside).

void KeyframeContextMenu(Keyframe keyframe)
{
	GenericMenu	menu = new GenericMenu(); if (curve != null)
	{
		int	keyframeIndex = curve.keys.ToList().FindIndex(k => k.Equal(keyframe)) - 1; //add keyframes actions as [MenuItems](https://docs.unity3d.com/ScriptReference/GenericMenu.AddItem.html) menu.ShowAsContext();
	}
}

The rest of the handle logic is managed in the DrawKeyframeHandle function which draws and assigns the position and tangents of our keyframe. The hardest part of this function was to find how the values inside inTangent and outTangent works and how to display them. With this difficulty solved, the rest of the function is pretty straightforward: draw the tangent lines, call the handle center point, then handle for tangents, and finally set keyframe values back.

That's everything for the KeyframeHandle, now let's see the CurveHandle class. In this class, nothing new except the system to add a point to the curve when your mouse is near the edge of the curve, for this system I decided not to use the standard Handle event management with AddControl but to implement my own (there is no reason for that but I found that it might be useful in some cases). So let's jump to the interesting function: DrawCurve which in addition to drawing the curve, detect if the mouse is near the edge of the curve:

//draw curve GL.Begin(GL.QUADS);
{ for (int i = 0; i < curveSamples; i++)
	{ float f0 = (float)i / (float)curveSamples; float f1 = (float)(i + 1) / (float)curveSamples;
		DrawCurveQuad(curve, f0, f1);
	}
}
GL.End();

In the first part of this function we sample the curve with curveSamples number of samples to draw a quad (inside DrawCurveQuad) from the time and values evaluated with the curve:

Vector3 topLeft = new Vector3(f0 * width, curve.Evaluate(f0) * height, 0);
Vector3 topRight = new Vector3(f1 * width, curve.Evaluate(f1) * height, 0); //check if the mouse is near frmo the curve edge: float dst = HandleUtility.DistancePointToLineSegment(currentMouseWorld, topLeft, topRight); if (dst < mouseCurveEdgeDst)
	mouseCurveEdgeDst = dst; if (dst < mouseOverEdgeDstThreshold)
	mouseOverCurveEdge = true;

In this piece of code, we take the two values of the curve at time f0 and f1 and we use DistancePointToLineSegment to get the distance from the current mouse position to the top segment of the curve. Then we keep the smallest distance and if it's under a threshold we set a boolean to tell that the mouse is near the edge.

The rest of the code is pretty simple, there is a lot of legacy OpenGL to draw the curve and its edges plus some code to call other handles (like Keyframe Handles or labels). The usage of this Handle is also straightforward and has already been explained in the Undo/Redo section so I'll jump directly to the next example.

Another full example: Snap Handle tool for GameObjects

Snap tool (cylinders) added to the default move tool

And once again, the source code of this example is available here: SnapHandleEditor.

In this example, we'll see how to improve the default Move tool by adding a little cylinder on each axis which helps to snap the position of the object by 0.25 if action is a hold or 0.1 if action+shift.

To do that, first, we need a script that runs when a GameObject is selected. The first logical idea is to use a custom editor on the Transform component, from them we can draw Handles and access the GameObject transform, but this is not what we are going to use, instead we'll use a simple script with the InitializeOnLoad attribute. This attribute will run our class in the editor without begin attached to anything and this is a real benefit because in contrast to the custom editor, we're not going to override any default inspectors (which would break the unity's custom editor for Transform).

[InitializeOnLoad] public class SnapHandleEditor { static SnapHandleEditor() {
		SceneView.onSceneGUIDelegate += OnSceneGUI;
	} static void OnSceneGUI(SceneView sv) {
	
	}
}
`

To explain a little bit more, what does InitializeOnLoad is, when the unity context is ready, to call the static constructor of each class marked with this attribute, so inside our constructor we can access every static unity fields like in the example onSceneGUIDelegate which allow our OnSceneGUI to be bound to the scene view. With this class as a base we can now begin to code the core of our system:

```CSharp
if (Tools.current != Tool.Move) return ; if (Selection.activeGameObject == null) return ;
	
transform = Selection.activeGameObject.transform;

We start with some checks: first, we check if the current tool selected in the editor is the Move tool otherwise we won't display our handles, then we check if the currently selected object is a GameObject and is not null, finally we assign our local transform to the selected object's transform. then I decided to create a function to draw one handle for an axis:

static void DrawAxisHandle(Quaternion rotation, Vector3 direction, Color color, int index) { var e = Event.current; float snapUnit = 0; float size = HandleUtility.GetHandleSize(transform.position) * snapHandleSize; float dist = size * snapHandleDistance; if (EditorGUI.actionKey)
		snapUnit = (e.shift) ? shiftSnapUnit : commandSnapUnit;

	currentRotation = rotation;
	currentColor = color;

	Vector3 addPos = direction * dist;
	Vector3 newPosition = Handles.Slider(transform.position + addPos, direction, size, SnapHandleCap, snapUnit) - addPos;

	transform.position = newPosition;
}

This function takes a rotation for the cylinder to draw, a direction which is the axis, and a color for the axis color. The first thing we do here is to calculate the size of the handle with GetHandleSize so the size of the cylinders is not dependent on the camera distance. Then we compute the snap value, unfortunately, the Slider snapping is only active when EditorGUI.actionKey holds so I only assign snapUnit when it is true if we press shift and actionKey it changes the snapUnit to 0.1 (default is 0.25). The next step is to set our current rotation and color, I've used external variables to store the rotation and the color because to draw the cylinder I'm using a custom Cap function and I can't pass additional parameters to them. Then we call the Slider handle with our custom cap function and finally, we set the new position of the GameObject.

For the custom cap function it looks like the one we code above:

static void SnapHandleCap(int controlId, Vector3 position, Quaternion rotation, float size, EventType eventType) { if (eventType == EventType.Repaint)
	{ //we set the material color or preselected color if mouse is near from our handle Color color = (HandleUtility.nearestControl == controlId || GUIUtility.hotControl == controlId) ? Handles.preselectionColor : currentColor;
		HandlesMaterials.overlayColor.SetColor("_Color", color); //we draw the cylinder with overlay material HandlesMaterials.overlayColor.SetPass(0);
		Matrix4x4 trs = Matrix4x4.TRS(position, transform.rotation * currentRotation, Vector3.one * size);
		Graphics.DrawMeshNow(snapToolMesh, trs);
	} else if (eventType == EventType.Layout)
	{ float distance = HandleUtility.DistanceToCircle(position, size);
		HandleUtility.AddControl(controlId, distance);
	}
}

The only different thing here is the material used to draw the mesh, it's a custom material with ZTest disabled so our cylinder will always be in front of everything exactly like the move handle.

This concludes our last example, pretty short but enough to understand how to improve or add custom behaviors to the unity's default tool with handles. I hope this article will be useful for your future creations, keep creating awesome tools, and see you in another post.

Clone this wiki locally