• Introduction
• Features
• System Dashboard
• Test System Architecture
• Design of Robot Test Server
• Set Up The Test Environment
• Run Tests
• Configurations of the Auto Test System
• Support The Robot Test Cases In Markdown
• Hacks To The Robot
• RESTful API Document of Web Server

Introduction

This is a distributed test automation framework with a centralized management UI. We are not intent to invent a new test framework or language here, so we choose Robot Framework and Robot Framework Remote Server as the test infrastructure, and then add the upper layer applications to make it easier to use.

Features

1. Provide a web server to serve all test scripts and test data in a centralized place to facilitate the update.
2. Automatically download test libraries to test endpoints which run the actual tests.
3. Support test cases written in markdown to make it more legible to test users.
4. Provide a Web UI to help schedule tests in priority and review the test reports in a diverse charts.

System Dashboard

Test System Architecture

A typical architecture for a Robot Remote Server is described in the official website.

We improves it by adding a daemon process on the Test Endpoint (aka. Robot Remote Server). The daemon, which is a special test library, will listen on the test endpoint so that it can accept any new test request from the Robot Server without needing to run a corresponding test library every time as we usually do in Robot Remote Server tutorial.

When a test request is sent to the Test Endpoint, a corresponding test library will be downloaded by the daemon process from the Robot Server which is also hosted as a web server. The downloaded test library will be served in the same address where a test library did before, namely we always serve the test libraries at a fixed address.

It's recommended to deploy Robot Server and Test Endpoint on the separated machines. For our case, we use Raspberry Pi 3 to reduce the deployment cost, but more performance PC would work as well.

Design of Robot Test Server

Set Up The Test Environment

1. Install python 3.7.x and pip.

2. Install poetry

   pip install --user poetry

3. Set up test endpoint environment

   cd robot-test-endpoint
   poetry install

4. Set up Web server and test runner environment

   cd webrobot
   poetry install
   poetry run task patch

   The last step is to patch installed robot framework package, details please see sections Support the robot test cases in markdown and Hacks to the robot.

5. Set up the MongoDB database

   Install MongoDB from the official website, please be noted to choose the community version instead of cloud based version.

Run Tests

1. Run the web server

   cd webrobot
   poetry run task server

2. Run the daemon process of a test endpoint

   cd robot-test-endpoint
   poetry run task daemon

3. (Work in progress) Run a test by using the restful API

   http POST http://127.0.0.1:5000/task/ test_suite:=\"demo-test\" endpoint_list:=[\"127.0.0.1:8270\"] variables:={\"echo_message\":\"bye\"} tester:=\"[email protected]\"
   

   http is a handy http client tool provided by python, "pip install httpie". Alternatively you can use curl. For more complicate test arguments, we can put them to a file and load it to httpie as follows.

   http POST http://127.0.0.1:5000/task/ < task.json
   

   While contents in task.json look like:

   {
     "test_suite": "demo-test",
     "endpoint_list": ["127.0.0.1:8270"],
     "variables": {
       "echo_message": "bye"
     },
     "testcases": ["hello world"],
     "tester": "[email protected]"
   }
   

   For more complex operations like uploading files, please refer to RESTful API of Web Server.

   There is a script to help relieve the pain of debugging RESTful APIs, please note that it's not for production environment.

   cd webrobot
   pipenv run start http://127.0.0.1:5000 demo-test -e 127.0.0.1:8270 --file firmware.bin --tester [email protected] -v echo_message bye -t hello_world
   

Notice:

1. For a test in action, please check out wifi-basic-test.md.

2. The robot server and test endpoint run on the same PC by default, if you want to deploy the them on the different PCs respectively, change the IP addresses in the robot server's config script (config.py) and test endpoint's config file (config.yml). Don't forget to configure the firewall to let through the communication on the TCP port 8270/8271.

3. The example-test-scripts directory is just for demonstration so that you can play around with the demo tests out of box. For production environment, you will probably have your own test assets. Same for robot-test-endpoint, implement your own work of test endpoint along with the test scripts in a stand-alone repository as they're coupled to work together. Please remember to modify configuration variables in the config.py to point to the right places after setting up your test asset repository.

   By this way you can keep tracking the latest code of auto test framework without the pain of messing with the code here by the frequent changes of test scripts.

Configurations of the Auto Test System

1. Test Endpoint

   It's as known as the Robot Remote Server. There is a config.yml to describe any SUT dependent details, such as serial port interfaces, robot server port, test library serving port, etc.

2. Web server and test runner

   All configurations are store in the config.py file, such as mongodb URI, robot scripts root directory, etc.

Support The Robot Test Cases In Markdown

For how to write a robot test case, please check out the official Quick Start Tutorial and User Manuel.

In an effort to support test cases written in a markdown file, we patched an unofficial work from here to make robot recognize markdown files and then read all code blocks marked as robotframework.

While we go a bit further by Adding Support of Tables in Markdown. Robot will also read all tables that start with any robot keyword in the table header in a markdown file.

After that, we can execute a test suite in the markdown file as usual.

robot demo-test.md

Hacks To The Robot

1. robot will cache test libraries if they have been imported before, we disabled it in _import_library in importer.py to support reloading test libraries in order to get the latest test library downloaded by daemon process on the test endpoint. Change details please see patch/robot.diff.

RESTful API Document of Web Server

Please visit automatically generated swagger doc