Skip to content

Services Development

Jump to bottom
Zdeněk Materna edited this page Jan 6, 2022 · 7 revisions

This page gives some basic information for developers intending to fix, improve, extend existing services or create completely new ones, while taking advantage of the existing stuff. However, a service might be certainly also completely independent of the arcor2 ecosystem, developed in a different repository, in a different language, whatever, and just integrated into it with the help of the appropriate Object Type.

Within the arcor2 ecosystem, there exist basically two types of services:

Common Conventions

For each service, there usually exist two Python packages:

  • arcor2_whatever_service - the main one, defining necessary libraries, executable scripts, etc.
  • arcor2_whatever_service_data - defines stuff that can be imported from other packages without making them dependent on the service, typically its API (at least necessary dataclasses, ideally client). Should have minimal dependencies.

Take arcor2_calibration and arcor2_calibration_data as an example.

If the service (executable script) can take any parameters, it should be possible to specify them using command line arguments (handy during development), or environment variables (handy for Docker images). For more complex configuration, we suggest the YAML format.

Websocket-based

In order to simplify the implementation of these services, there is the server coroutine (defined in the ws_server module). It helps with the handling of the protocol described below. The (simplified) minimal example of the server able to handle one RPC might look like this:

async def my_rpc_cb(req: MyRpc.Request, ui: WsClient) -> None:
    ...
    return None  # in fact, a MyRpc.Response with default values will be sent

await websockets.server.serve(
        functools.partial(ws_server.server, rpc_dict={MyRpc.__name__: (MyRpc, my_rpc_cb)}),
        "0.0.0.0",
        1234,
    )

Internally, the server coroutine creates a task for each RPC call and handles all possible errors.

Protocol

The protocol uses two basic patterns. RPCs and events. The basic structure of both is defined by the corresponding dataclasses: Event and RPC (respectively the nested Request and Response classes). Events are used by services to notify their clients about asynchronous events. The base class contains mainly event type event. Specific events are created by deriving from Event like this, where typically a specific data field is added:

@dataclass
class ProjectException(Event):
    @dataclass
    class Data(JsonSchemaMixin):
        message: str
        type: str
        handled: bool = False

    data: Data

Please notice the nested structure of dataclasses. The serialized data (.to_json()) will look like this:

{"event": "ProjectException", "data": {"message": "Hola!", "type": "WhatEver", "handled": false}}

Optional fields as change_type and parent_id are used in specific cases - for instance when ARServer notifies clients that a new Action Point with a parent has been added - the change_type is set to ADD and parent_id to UID of the parent.

An example of the definition of the specific RPC could be:

class Version(RPC):
    @dataclass
    class Request(RPC.Request):
        pass

    @dataclass
    class Response(RPC.Response):
        @dataclass
        class Data(JsonSchemaMixin):
            version: str = ""

        data: Optional[Data] = None

In this case, the request is empty - has no data. The response then contains the string version. The purpose of class Version is to encapsulate Request and Response and to define RPC's unique type. The communication is then as follows... The client sends:

{"request": "Version", "id": 1234}

And server responds with:

{"response": "Version", "id": 1234, "result": true, "data": {"version": "0.1.2"}}

However, the request might also fail (highly unlikely in this particular case) and in that case, the data part will be missing:

{"response": "Version", "id": 1234, "result": false, "messages": ["Can't get version, sorry."]}

Very often, the pattern is that a client sends a request to a server (e.g. AddActionPoint.Request), the server responds (AddActionPoint.Response) and then broadcasts the event (ActionPointChanged) to all clients to notify them about the change that just happened. However, it is not guaranteed that notification will arrive after response to the request.

Flask (REST) ones

The services are taking advantage of the Flask framework and several other libraries, which make us able to define models as dataclasses and to specify OpenAPI of the services within the code (as a docstring) in a quite comfortable way. There exist some helper functions within the module arcor2.flask. The really minimal example of such service, defining its own dataclass that is used in some API could be:

@dataclass  # should be placed in the _data package
class MyData(JsonSchemaMixin):
    int_property: int
    str_property: str


app = create_app(__name__)

@app.route("/test", methods=["GET"])
def get_my_data() -> RespT:
    """Test path.
    ---
    get:
      summary: Call without parameters.
      responses:
        200:
          description: Ok
          content:
                application/json:
                  schema:
                    $ref: MyData
    """
    return MyData(1, "whatever").to_json(), 200


def main() -> None:

    parser = argparse.ArgumentParser(description="MyService")
    parser.add_argument("-s", "--swagger", action="store_true", default=False)

    args = parser.parse_args()

    run_app(
        app,
        "MyService",
        "0.1.0",  # service version
        "0.6.0",  # API version
        1234,  # port
        [MyData],
        print_spec=args.swagger,
    )

Such service will run on port 1234 and will have the SwaggerUI available. If necessary, a YAML file with its OpenAPI could be printed out when started with the --swagger argument.

Please note: for each service, there should be a check for validity of its OpenAPI (it is quite easy to make a typo in docstring!) and an integration test validating API functionality (making real calls, to make sure that it really works as intended).

Dataclasses

All dataclasses are using JsonSchemaMixin from dataclasses-jsonschema library which gives them methods as .to_json() and .from_json() and also ability to generate json schema / OpenAPI. The library is highly useful, however, some magic tricks in the runtime were needed to make it suit arcor2 needs. First of all, from_json and to_json are monkey patched to use orjson instead of standard json library. Then, resolve_schema_refs is patched in order to solve the issue with arrays in OpenAPI. There was also issue with lazy compilation of json schemas in asyncio applications (done by fastjsonschema) so it can be forced by calling compile_json_schemas. See init.py of arcor2.data. The last magic happens when ARServer is asked to generate OpenAPI (containing models for RPCs and events) - by default, the dataclasses-jsonschema library does not handle nested classes properly, so there is a generate_openapi function, that manipulates __name__ attribute of nested classes to make it "fully qualified", e.g. __name__ of Data (nested inside Version.Response) becomes VersionResponseData which is then used as a name for the model in the output OpenAPI.

Clone this wiki locally