Architecture

This page describes the ARCOR2 system architecture. The following picture shows runtime dependencies in the system:

The framework is divided into a set of independent services (backend) and a user interface (frontend). The main service of the system is ARServer, which acts as a central point for user interfaces and mediates communication with other services. So far, only one implementation of a user interface has been developed, a tablet-based app providing full functionality (AREditor), however, the intention is to allow the involvement of several simpler, complementary interfaces providing only some aspect of functionality, e.g. RGB LED strip indicating system status or hand tracking-based interface for controlling a robot. Therefore, the server must be able to deal with multiple connected interfaces even in single-user scenarios.

Interfaces are connected to ARServer through Websockets, which allows bidirectional communication. The ARServer holds the system state, while interfaces can manipulate it using a set of RPCs and receive notifications on changes. It is assumed, that each workplace runs its own instance of ARServer and therefore, the server maintains only one session for all users: if one user opens a project, the same project is shown to other connected users as well. In order to support efficient and safe collaboration between users, there is a locking mechanism that prevents multiple users to manipulate the same element (e.g. control the robot).

The ARServer also serves as a proxy between Python code and AR environment, which is code-agnostic. It analyzes code of Object Types in order to extract available Actions and their parameters and creates JSON metadata, that is available to user interfaces. The code analysis takes advantage of PEP 484 type hints in order to extract e.g. parameter types and matching nodes of Abstract Syntax Tree (AST) in order to e.g. inspect value ranges, that are defined using assertions or check if a method (feature) is implemented.

The scene or project opened within the server could be either offline or online. In the online state, instances for all objects are created meaning that e.g. connection to a robot is made. In an online state, any action instance added to a project may be executed, which simplifies a debugging process. However, it is also possible to work offline, where just functionalities as controlling a robot are not available.

The roles of other services are:

• Project Service provides persistent storage for workplace-relevant data: scenes, projects, Object Types, models, etc.
• Scene Service, used e.g. in cases where underlying implementation is based on ROS, is responsible for the management of collision objects.
• The Build Service creates for a given project a self-contained executable package. The logic could be defined within the AR environment or provided in a standalone file. When generating logic from its JSON representation, it is first assembled in a form of AST and then compiled into Python code. Moreover, a set of supplementary classes e.g. simplifying work with Action Points are generated.
  • The package is self-contained only to a certain extent: it contains scene and project JSONs, Main Script, all necessary Object Types, and associated collision models. However, meshes (e.g. dae or stl files) are not included at the moment (can be downloaded by the Scene Service from Project Service if needed).
• Execution Service manages execution packages created by the Build Service. The most important functionality is running the package when the service streams events regarding execution state (e.g. which action with what parameters is being executed) to ARServer. The execution can also be paused or resumed when needed.
• Calibration Service provides a method to perform camera pose estimation based on ArUco marker detection and a method to adjust the pose of the robot using an RGBD camera.

Please note:

• Project and Scene services are being developed by project partner (Kinali). This repository contains mocks of these services that may be used for testing purposes.
• Build or Execution services do not have any code or runtime dependency on the ARServer, which acts as their client and provides an additional layer above them with AR-related services. If you don't need AR-related services, you can still utilize Build and Execution, accessing their API directly.
• There exists a REST API wrapper for the Execution Service (Execution REST Proxy), which might be used in simple cases, where real-time notification about events is not needed.

Integration of Devices

When integrating a new device (e.g. a robot), there are more ways how it could be done.

Object Type Only

The easiest way is to implement everything within an Object Type (OT). This is fully sufficient if the device already has some kind of API and all the necessary functionality. If you need some third-party dependency, this is still viable, however, you won't be able to use default docker images as the additional dependency won't be available there. You could still use services outside of docker, or build custom images with additional libraries. In cases where necessary dependency is a tiny library, it might make sense to simply copy its code to an OT or mixin. When OT connects to some hardware (e.g. opens a serial port), there is a drawback, that all the hardware has to be connected to the computer where Execution Service is running (or you need to do some fancy tunneling over the network).

We adopted this approach for the ABB YuMi robot (see arcor2_yumi). There is a simple native program within a robot (written in RAPID language), that opens a TCP port and listens for external commands in a custom protocol. The corresponding OT implements the protocol and connects to the robot's TCP port. Moreover, some commands are issued through the standard REST API of the robot. In this way, we relied on the robot's motion planning capabilities. Another approach (belonging to the next section) could be for instance to run a ROS container and control the robot using it. Then, OT would not communicate directly with the robot, but with the ROS container.

Object Type + Service

When there is a need to make something computationally expensive, connect a lot of distributed hardware to more computers, use additional dependencies, maybe even a different operating system, then it makes sense to develop a new service with all the functionality and its own API and OT that will make this functionality accessible through its actions.

For instance, for Dobot Magician/M1, we have developed a separate service and corresponding OT. The same approach was chosen for Kinect Azure (which needs some special dependency), where we also developed service and OT. Communication between services and OT could be for instance based on REST API, for which we have a handy client supporting type annotations and dataclasses (based on requests), using WebSockets, plain TCP/UDP sockets, or whatever is available in the standard Python library.

Please note:

• A service might or might not be dependent on the arcor2 ecosystem. In order to get familiar with how services within the ecosystem are created, please see Services Development.
• If the robot motion should be controlled by e.g. a ROS-based container, it would be necessary to develop a custom Scene Service (should be quite easy!), that will pass collision objects to the ROS container.
• Services have to be started beforehand or somehow from the OT - ARCOR2 does not handle the life cycle of services. It only initializes / tears down OTs.