Replicant arm articulation is a complex topic. This document covers basic arm articulation actions. Part 2 covers grasping and dropping. Part 3 covers more advanced examples that use some additional optional parameters. Part 4 describes a relatively complex use-case: stacking objects on top of each other.
Unlike movement and animations, which are controlled by pre-recorded animation data, Replicant arm articulation is procedural. It uses inverse kinematic (IK) and the FinalIK Unity asset to solve an end pose, given a target position that a hand is reaching for.
The Replicant can reach for a target position or object via reach_for(target, arm).
To reach for a target position, set target to an x, y, z, dictionary or numpy array, and arm to an Arm value:
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.replicant import Replicant
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
c = Controller()
replicant = Replicant()
camera = ThirdPersonCamera(position={"x": 0, "y": 1.5, "z": 2.1},
look_at=replicant.replicant_id,
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_reach_for_position")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=[camera.avatar_id], path=path)
c.add_ons.extend([replicant, camera, capture])
c.communicate(TDWUtils.create_empty_room(12, 12))
replicant.reach_for(target={"x": 0.8, "y": 0.9, "z": 0.3}, arm=Arm.right)
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
c.communicate({"$type": "terminate"})Result:
The Replicant can reach with both hands at the same time. To do this, set arm to a list of arms:
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.replicant import Replicant
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
c = Controller()
replicant = Replicant()
camera = ThirdPersonCamera(position={"x": 0, "y": 1.5, "z": 2.1},
look_at=replicant.replicant_id,
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_reach_for_position_both_hands")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=[camera.avatar_id], path=path)
c.add_ons.extend([replicant, camera, capture])
c.communicate(TDWUtils.create_empty_room(12, 12))
replicant.reach_for(target={"x": 0.8, "y": 0.9, "z": 0.3}, arm=[Arm.left, Arm.right])
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
c.communicate({"$type": "terminate"})
To reach for a target object, set target to an object ID integer:
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.replicant import Replicant
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
c = Controller()
replicant = Replicant()
camera = ThirdPersonCamera(position={"x": 0, "y": 1.5, "z": 2.5},
look_at=replicant.replicant_id,
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_reach_for_object")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=[camera.avatar_id], path=path)
c.add_ons.extend([replicant, camera, capture])
# Set the object ID.
object_id = Controller.get_unique_id()
commands = [TDWUtils.create_empty_room(12, 12)]
commands.extend(Controller.get_add_physics_object(model_name="basket_18inx18inx12iin_wicker",
object_id=object_id,
position={"x": -0.1, "y": 0, "z": 0.6}))
c.communicate(commands)
replicant.reach_for(target=object_id, arm=Arm.right)
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
c.communicate({"$type": "terminate"})Result:
Notice that in the previous example, the Replicant does not reach for the center of the object. This is intentional.
Certain models in TDW have "affordance points" that are automatically attached to them upon instantiation. Affordance points don't have meshes, colliders, etc.; they are just an transform with a position and rotation. Affordance points are parented to objects, meaning that when the object moves and rotates, the empty objects will follow within the local coordinate space.
When you call replicant.reach_for(target, arm), the Replicant will always try to reach for the affordance point closest to the hand. If there are no affordance points, the Replicant will reach for the bounds position closest to the hand.
In the above example, basket_18inx18inx12iin_wicker has 4 affordance points, one on each side of the lid. This means that the Replicant will always try to reach for one of the sides of the basket.
Right now, only a small amount of models in TDW have affordance points. To get a list of objects with affordance positions, review model_record.empty_objects:
from tdw.librarian import ModelLibrarian
lib = ModelLibrarian()
for record in lib.records:
if record.do_not_use:
continue
if len(record.affordance_points) > 0:
print(record.name)As with target positions, it is possible to reach for target objects with both hands. Each hand will aim for the nearest affordance point or bounds position, but will never reach for the same point in space:
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.replicant import Replicant
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
c = Controller()
replicant = Replicant()
camera = ThirdPersonCamera(position={"x": 0, "y": 1.5, "z": 2.5},
look_at=replicant.replicant_id,
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_reach_for_object_two_hands")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=[camera.avatar_id], path=path)
c.add_ons.extend([replicant, camera, capture])
# Set the object ID.
object_id = Controller.get_unique_id()
commands = [TDWUtils.create_empty_room(12, 12)]
commands.extend(Controller.get_add_physics_object(model_name="basket_18inx18inx12iin_wicker",
object_id=object_id,
position={"x": -0.1, "y": 0, "z": 0.6}))
c.communicate(commands)
replicant.reach_for(target=object_id, arm=[Arm.left, Arm.right])
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
c.communicate({"$type": "terminate"})Result:
The action succeeds if, when it ends, the hand is near the target (see below, arrived_at).
The action has several fail states:
- If, at the start of the action, the target is too far away, the action immediately fails with
ActionStatus.cannot_reach. - If, at the end of the action, the target is too far away, the action ends with
ActionStatus.failed_to_reach. - If, during the action, the Replicant collides with an object, the action ends with
ActionStatus.collision. To suppress this behavior, setreplicant.collision_detection.objects = Falseor add an object ID toreplicant.collision_detection.exclude_objects
In this example, the Replicant will walk to an object, pick it up, walk away, and drop the object. Notice that we add the trunk's ID to collision_detection.exclude_objects. This is because the Replicant's lower arm will collide with the trunk while reaching for the mug. Unlike most of our examples controllers, this example is a subclass of Controller. This is much closer to how you should structure your controllers. We've written a do_action() function to handle the basic Replicant action loop.
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.add_ons.replicant import Replicant
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
class GoToReachFor(Controller):
def __init__(self, port: int = 1071, check_version: bool = True, launch_build: bool = True):
super().__init__(port=port, check_version=check_version, launch_build=launch_build)
# Set the replicant and the object IDs here because we need to reference them elsewhere.
self.replicant = Replicant()
self.trunk_id = 1
self.mug_id = 2
def do_action(self) -> None:
while self.replicant.action.status == ActionStatus.ongoing:
self.communicate([])
self.communicate([])
def run(self) -> None:
camera = ThirdPersonCamera(position={"x": -1.5, "y": 1.175, "z": 5.25},
look_at={"x": 0.5, "y": 1, "z": 0},
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_go_to_reach")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=["a"], path=path)
self.add_ons.extend([self.replicant, camera, capture])
# Create the room.
commands = [TDWUtils.create_empty_room(12, 12)]
commands.extend(Controller.get_add_physics_object(model_name="trunck",
object_id=self.trunk_id,
position={"x": 0, "y": 0, "z": 3},
kinematic=True))
commands.extend(Controller.get_add_physics_object(model_name="coffeemug",
object_id=self.mug_id,
position={"x": 0, "y": 0.9888946, "z": 3}))
self.communicate(commands)
self.replicant.move_to(target=self.trunk_id)
self.do_action()
# Ignore the trunk.
self.replicant.collision_detection.exclude_objects.append(self.trunk_id)
self.replicant.reach_for(target=self.mug_id, arm=Arm.right)
self.do_action()
print(self.replicant.action.status)
self.communicate({"$type": "terminate"})
if __name__ == "__main__":
c = GoToReachFor()
c.run()Result:
Output:
ActionStatus.success
If target is a position (a dictionary or a numpy array, as opposed to a position), it defaults to a world space position. It's often useful, however, to set absolute=False, which defines target as being relative to the Replicant's position and rotation:
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.replicant import Replicant
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
def do_action():
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
c = Controller()
replicant = Replicant()
camera = ThirdPersonCamera(position={"x": 0, "y": 1.5, "z": 3.5},
look_at=replicant.replicant_id,
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_reach_for_relative")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=[camera.avatar_id], path=path)
c.add_ons.extend([replicant, camera, capture])
for rotation in [0, 30, -45]:
replicant.reset()
camera.initialized = False
capture.initialized = False
c.communicate([{"$type": "load_scene",
"scene_name": "ProcGenScene"},
TDWUtils.create_empty_room(12, 12)])
# Turn the Replicant.
replicant.turn_by(rotation)
do_action()
# Reach for a relative target with the right hand.
replicant.reach_for(target={"x": 0.6, "y": 1.5, "z": 0.3}, arm=Arm.right, absolute=False)
do_action()
c.communicate({"$type": "terminate"})Result:
duration is an optional parameter that controls the speed in seconds of the arm motion:
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.replicant import Replicant
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
c = Controller()
replicant = Replicant()
camera = ThirdPersonCamera(position={"x": 2, "y": 3, "z": 2.53},
look_at=replicant.replicant_id,
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_reach_for_object_slow")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=[camera.avatar_id], path=path)
c.add_ons.extend([replicant, camera, capture])
# Set the object ID.
object_id = Controller.get_unique_id()
commands = [TDWUtils.create_empty_room(12, 12)]
commands.extend(Controller.get_add_physics_object(model_name="basket_18inx18inx12iin_wicker",
object_id=object_id,
position={"x": -0.1, "y": 0, "z": 0.6}))
c.communicate(commands)
replicant.reach_for(target=object_id,
arm=[Arm.left, Arm.right],
duration=1.5)
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
c.communicate({"$type": "terminate"})Result:
duration is measured in seconds. If the simulation is running faster than real life, this will appear too slow. To handle this, the ReachFor action dynamically scales the duration value in proportion to the actual framerate: duration *= 60 / (1 / framerate). This is usually desirable, but can be suppressed by setting scale_duration=False.
arrived_at controls the distance that defines a successful action. If, at the end of the action, the hand is this distance or less from the target position (a position, affordance point, or bounds position), the action is successful.
max_distance controls the maximum distance from the hand to the target. If at the start of the action the target is too far away, the action will immediately fail with ActionStatus.cannot_reach. You can extend this maximum distance, but the Replicant may behave very strangely:
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.replicant import Replicant
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
c = Controller()
replicant = Replicant()
camera = ThirdPersonCamera(position={"x": 3.2, "y": 3, "z": 3.5},
look_at=replicant.replicant_id,
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_reach_too_far")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=[camera.avatar_id], path=path)
c.add_ons.extend([replicant, camera, capture])
c.communicate([TDWUtils.create_empty_room(12, 12)])
target = {"x": 3, "y": 0.9, "z": 6}
# The target is too far away.
replicant.reach_for(target=target, arm=Arm.right)
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
print(replicant.action.status)
# Try to reach the target anyway.
replicant.reach_for(target=target,
arm=Arm.right,
max_distance=8)
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
print(replicant.action.status)
c.communicate({"$type": "terminate"})Result:
Output:
ActionStatus.cannot_reach
ActionStatus.failed_to_reach
Reset a Replicant's arm to its neutral position by calling replicant.reset_arm(arm):
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from tdw.add_ons.replicant import Replicant
from tdw.add_ons.third_person_camera import ThirdPersonCamera
from tdw.add_ons.image_capture import ImageCapture
from tdw.replicant.action_status import ActionStatus
from tdw.replicant.arm import Arm
from tdw.backend.paths import EXAMPLE_CONTROLLER_OUTPUT_PATH
c = Controller()
replicant = Replicant()
camera = ThirdPersonCamera(position={"x": 2, "y": 3, "z": 2.53},
look_at=replicant.replicant_id,
avatar_id="a")
path = EXAMPLE_CONTROLLER_OUTPUT_PATH.joinpath("replicant_reset_arm")
print(f"Images will be saved to: {path}")
capture = ImageCapture(avatar_ids=[camera.avatar_id], path=path)
c.add_ons.extend([replicant, camera, capture])
c.communicate(TDWUtils.create_empty_room(12, 12))
replicant.reach_for(target={"x": 0.8, "y": 0.9, "z": 0.3}, arm=Arm.right)
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
replicant.reset_arm(arm=Arm.right)
while replicant.action.status == ActionStatus.ongoing:
c.communicate([])
c.communicate([])
c.communicate({"$type": "terminate"})Result:
The arm parameter can be a single value such as Arm.right or a list of values, such as [Arm.left, Arm.right], in which case both arms are reset.
duration is an optional parameter that controls the speed in seconds of the arm motion; it works the same way as it does in reach_for(target, arm).
The scale_duration parameter works the same way that it does in reach_for(target, arm).
The action succeeds when the arm(s) finish resetting. The action can end in ActionStatus.collision. The collision detection rules for reset_arm(arm) are the same as those for reach_for(target, arm); see above for more information.
The action can't end in ActionStatus.cannot_reach or ActionStatus.failed_to_reach because the Replicant isn't reaching for a target.
replicant.reach_for(target, arm) sets replicant.action to an ReachFor action.
In addition to the usual Action initialization commands, ReachFor sends replicant_reach_for_position , replicant_reach_for_relative_position, or replicant_reach_for_object, depending on the type of target and the value of the absolute parameter.
The action continues until there is a collision or until replicant.dynamic.output_action_status != ActionStatus.ongoing (meaning that the build has signaled that the animation ended).
replicant.reset_arm(arm) sets replicant.action to an ResetArm action.
In addition to the usual Action initialization commands, ResetArm sends replicant_reset_arm.
The action continues until there is a collision or until replicant.dynamic.output_action_status == ActionStatus.success (meaning that the build has signaled that the animation ended).
Next: Arm articulation, pt. 2: Grasp and drop objects
Example controllers:
- reach_for_position.py Reach for a target position.
- reach_for_object.py Reach for a target object.
- reach_for_relative.py Reach for a relative target position.
- reach_too_far.py Reach for a target that is too far away.
- reset_arm.py Reach for a target position and then reset the arm.
Command API:
replicant_reach_for_positionreplicant_reach_for_relative_positionreplicant_reach_for_objectreplicant_reset_arm
Python API: