Skip to content

Latest commit

 

History

History
372 lines (284 loc) · 16.4 KB

ui.md

File metadata and controls

372 lines (284 loc) · 16.4 KB
Raw
UI

The UI add-on

The easiest way to create a basic UI in TDW is via the UI add-on:

from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.add_ons.ui import UI
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH

c = Controller()
camera = ThirdPersonCamera(avatar_id="a",
                           position={"x": 1, "y": 2.5, "z": 0},
                           look_at={"x": 0, "y": 0, "z": 0})
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("ui")
print(f"Images will be saved to: {path}")
capture = ImageCapture(path=path, avatar_ids=["a"])
ui = UI()
c.add_ons.extend([camera, capture, ui])
c.communicate(TDWUtils.create_empty_room(12, 12))
ui.attach_canvas_to_avatar(avatar_id="a")
ui.add_text(text="hello world",
            position={"x": 0, "y": 0},
            font_size=36)
c.communicate({"$type": "terminate"})

Result:

Limitations

TDW's UI API is deliberately limited. Unity has a powerful UI API, but it was designed assuming that the developer would have access to Unity Editor. Without visual aids, it is extremely difficult to use, or even to explain. TDW's implementation of the Unity UI API has only as subset of the full functionality; it is not designed for a good-looking video game-like interface, and more for displaying rudimentary metrics or messages. There is also no user input such as button presses, although this may be implemented in the future.

Canvases, avatars, and VR rigs

In Unity, UI elements ("RectTransforms") must be attached to a canvas. There can be more than one canvas in the scene.

In TDW, the UI add on has an optional canvas_id in its constructor (default value is 0). When the add-on initializes, it automatically sends add_ui_canvas. The UI add-on will then automatically append its canvas ID to all subsequent commands.

To add multiple canvases to the scene, simply add multiple UI add-ons:

from tdw.add_ons.ui import UI
from tdw.controller import Controller

ui_0 = UI(canvas_id=0)
ui_1 = UI(canvas_id=1)
c = Controller()
c.add_ons.extend([ui_0, ui_1])

In practice, the only reason to add multiple UI canvases is if there are multiple avatars in the scene.

By default, a canvas is in "overlay" mode. It is rendered separately from TDW's camera passes. In order for the canvas to actually appear in image output data it must be "attached" to an avatar. To do this, create an avatar (i.e. ThirdPersonCamera) and then call ui.attach_canvas_to_avatar(avatar_id). This function automatically sends attach_ui_canvas_to_avatar.

ui.attach_canvas_to_avatar() also has an additional optional parameter focus_distance. This should be set to the default (2.5) or higher, otherwise the UI will look blurry. This is mostly important for VR, where it is possible that the ideal focus distance might differ between headsets.

For VR rigs, call ui.attach_canvas_to_vr_rig() instead of ui.attach_canvas_to_avatar().

Anchors, pivots, and positions

UI elements are positioned using local screen space positions, parameterized as Vector2 objects, e.g. {"x": 0, "y": 0}. "x" is the horizontal value, and "y" is the vertical value.

Positions can reflect the "true" screen position but it is often convenient to apply offsets using anchors and pivots.

An anchor is position offset factor Vector2 where each value is between 0 and 1. By default, the anchor of all UI elements is {"x": 0.5, "y": 0.5}, meaning that there is no offset; position {"x": 0, "y": 0} is in the center of the screen. But, if the anchor is {"x": 0, "y": 1}, then position {"x": 0, "y": 0} is actually the top-left corner of the screen.

A pivot is the UI element's pivot point as a Vector2 factor where each value is between 0 and 1. By default, the pivot of all UI elements is {"x": 0.5, "y": 0.5} meaning that the pivot is in the center of the object.

You can set the anchor, pivot, and position of a UI element to easily snap it to sides or corners of the screen without actually knowing the dimensions of the screen.

In this example, we'll add text to the top-left corner of the screen. Note that in both this example and the previous "hello world" world example, position={"x": 0, "y": 0} but we've adjusted the anchor and pivot such that the top-left corner of the text will be moved to the top-left corner of the screen.

from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.add_ons.ui import UI
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH

c = Controller()
camera = ThirdPersonCamera(avatar_id="a",
                           position={"x": 1, "y": 2.5, "z": 0},
                           look_at={"x": 0, "y": 0, "z": 0})
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("anchors_and_pivots")
print(f"Images will be saved to: {path}")
capture = ImageCapture(path=path, avatar_ids=["a"])
ui = UI()
c.add_ons.extend([camera, capture, ui])
c.communicate(TDWUtils.create_empty_room(12, 12))
ui.attach_canvas_to_avatar(avatar_id="a")
ui.add_text(text="hello world",
            position={"x": 0, "y": 0},
            anchor={"x": 0, "y": 1},
            pivot={"x": 0, "y": 1},
            font_size=36)
c.communicate({"$type": "terminate"})

Text

Add text to the screen via ui.add_text(text), which sends add_ui_text. See above for some minimal examples.

Set the color of the text with the optional color parameter:

from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.add_ons.ui import UI
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH

c = Controller()
camera = ThirdPersonCamera(avatar_id="a",
                           position={"x": 1, "y": 2.5, "z": 0},
                           look_at={"x": 0, "y": 0, "z": 0})
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("text_color")
print(f"Images will be saved to: {path}")
capture = ImageCapture(path=path, avatar_ids=["a"])
ui = UI()
c.add_ons.extend([camera, capture, ui])
c.communicate(TDWUtils.create_empty_room(12, 12))
ui.attach_canvas_to_avatar(avatar_id="a")
ui.add_text(text="hello world",
            position={"x": 0, "y": 0},
            anchor={"x": 0, "y": 1},
            pivot={"x": 0, "y": 1},
            font_size=36,
            color={"r": 1, "g": 0, "b": 0, "a": 1})
c.communicate({"$type": "terminate"})

Result:

ui.add_text() returns the ID of the UI text element.

Dynamically set the text of an existing UI element by calling ui.set_text(ui_id, text):

from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.add_ons.ui import UI
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH

c = Controller(launch_build=False)
camera = ThirdPersonCamera(avatar_id="a",
                           position={"x": 1, "y": 2.5, "z": 0},
                           look_at={"x": 0, "y": 0, "z": 0})
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("set_text")
print(f"Images will be saved to: {path}")
capture = ImageCapture(path=path, avatar_ids=["a"])
ui = UI()
c.add_ons.extend([camera, capture, ui])
c.communicate(TDWUtils.create_empty_room(12, 12))
ui.attach_canvas_to_avatar(avatar_id="a")
ui_id = ui.add_text(text="hello world",
                    position={"x": 0, "y": 0},
                    anchor={"x": 0, "y": 1},
                    pivot={"x": 0, "y": 1},
                    font_size=36,
                    color={"r": 1, "g": 0, "b": 0, "a": 1})
c.communicate([])
ui.set_text(ui_id=ui_id, text="new text")
c.communicate({"$type": "terminate"})

Result:

Images

Add UI images via ui.add_image(image, position, size), which sends add_ui_image.

Mandatory parameters:

  • The image parameter can be a string (a filepath), a Path object (a filepath), bytes (the image byte data), or PIL image. If image is a filepath, then it must be valid on the computer running the controller.
  • The position parameter is the position of the image; see above for how to set this.
  • The size parameter is the actual pixel size of the images as a Vector2.

Optional parameters:

  • The rgba parameter tells the build whether to expect RGBA data or RGB data.
  • The scale_factor parameter can be set to resize the image.
  • See above for how anchor and pivot work.
  • color is the same as in text; an RGBA dictionary with values ranging from 0 to 1. color will tint an image; by default, it is white (no tint).
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.add_ons.ui import UI
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH

c = Controller()
camera = ThirdPersonCamera(avatar_id="a",
                           position={"x": 1, "y": 2.5, "z": 0},
                           look_at={"x": 0, "y": 0, "z": 0})
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("ui_image")
print(f"Images will be saved to: {path}")
capture = ImageCapture(path=path, avatar_ids=["a"])
ui = UI()
c.add_ons.extend([camera, capture, ui])
c.communicate(TDWUtils.create_empty_room(12, 12))
ui.attach_canvas_to_avatar(avatar_id="a")
ui.add_image(image="test.jpg",
             rgba=False,
             size={"x": 128, "y": 128},
             position={"x": 0, "y": 0})
c.communicate({"$type": "terminate"})

Result:

Transform a UI element

Create a cutout

Call ui.add_cutout(base_id, image, position) to cut a transparent hole in another UI element. This function sends add_ui_cutout.

Mandatory parameters:

  • base_id is the ID of the image that will have a hole in it. This can be added on the same frame as the cutout image but it must be added prior to the cutout image.
  • The image parameter can be a string (a filepath), a Path object (a filepath), bytes (the image byte data), or PIL image. If image is a filepath, then it must be valid on the computer running the controller.
  • The position parameter is the position of the image; see above for how to set this.
  • The size parameter is the actual pixel size of the images as a Vector2.

Optional parameters:

  • The scale_factor parameter can be set to resize the image.
  • See above for how anchor and pivot work.

This example adds a cube, a base image, and a cutout image, and then moves the cutout around the screen:

from PIL import Image, ImageDraw
from tdw.controller import Controller
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.ui import UI
from tdw.add_ons.image_capture import ImageCapture
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH


c = Controller()
# Add the UI add-on and the camera.
camera = ThirdPersonCamera(position={"x": 0, "y": 0, "z": -1.2},
                           avatar_id="a")
ui = UI()
c.add_ons.extend([camera, ui])
ui.attach_canvas_to_avatar(avatar_id="a")
screen_size = 512
commands = [{"$type": "create_empty_environment"},
            {"$type": "set_screen_size",
             "width": screen_size,
             "height": screen_size}]
# Add a cube slightly off-center.
commands.extend(Controller.get_add_physics_object(model_name="cube",
                                                  library="models_flex.json",
                                                  object_id=0,
                                                  position={"x": 0.25, "y": 0, "z": 1},
                                                  rotation={"x": 30, "y": 10, "z": 0},
                                                  kinematic=True))
c.communicate(commands)

# Enable image capture.
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("ui_mask")
print(f"Images will be saved to: {path}")
capture = ImageCapture(path=path, avatar_ids=["a"])
c.add_ons.append(capture)

# Create the background UI image.
bg_size = screen_size * 2
base_id = ui.add_image(image=Image.new(mode="RGBA", size=(bg_size, bg_size), color=(0, 0, 0, 255)),
                       position={"x": 0, "y": 0},
                       size={"x": bg_size, "y": bg_size})

# Create the cutout image.
diameter = 256
mask = Image.new(mode="RGBA", size=(diameter, diameter), color=(0, 0, 0, 0))
# Draw a circle.
draw = ImageDraw.Draw(mask)
draw.ellipse([(0, 0), (diameter, diameter)], fill=(255, 255, 255, 255))
x = 0
y = 0
# Add the cutout.
cutout_id = ui.add_cutout(image=mask, position={"x": x, "y": y}, size={"x": diameter, "y": diameter}, base_id=base_id)
c.communicate([])

# Move the cutout.
for i in range(100):
    x += 4
    y += 3
    ui.set_position(ui_id=cutout_id, position={"x": x, "y": y})
    c.communicate([])
c.communicate({"$type": "terminate"})

Result:

Set a UI element's depth (z value)

Call ui.set_depth(id, depth) to set a UI element's depth relative to its parent canvas. This function sends set_ui_element_depth.

depth is measured in meters from the parent canvas.

If the canvas is attached to a camera (see ui.attach_canvato_avatar(avatar_id, focus_distance, plane_distance) or to a VR rig's camera (see ui.attach_canvas_to_vr_rg(plane_distance)), the canvas' distance from the camera is plane_distance, and so the true local z value is plane_distance + depth.

Destroy UI elements

Destroy a specific UI element via ui.destroy(ui_id), which sends destroy_ui_element.

Destroy all UI elements, and optionally the canvas, via ui.destroy_all().

If the canvas is attached to the avatar and you're resetting a scene and/or destroying the avatar, call ui.destroy_all(destroy_canvas=True) which tells the add-on to send destroy_ui_canvas.

Next: UI Widgets

Return to the README

Example controllers:

Python API:

Command API: