Behavior Driven Test Automation

Overview

This solution implements Behavior-driven Development (abbreviated BDD) principles to allow users to use tests written in a ubiquitous language style to validate the web/native applications and web/native platform and offers a convenient and efficient way to develop, manage, execute the test cases, and easy to combine with high level test framework

Background

Application Test Automation Challenges

Hard to automate System level operations such as opening the Settings menu or the app launcher in a test device
Hard to implement cross-app functional UI automation testing across system and installed apps
Hard to apply image recognition or platform native UI automation solution to web application
Multi-formity of web/native UI layout and elements requires a more convenient test design and development method
Existing browser test automation techniques and tools have gap on support on web applications or web runtime

WebDriver

WebDriver is a W3C standard: The WebDriver API is defined by a wire protocol and a set of interfaces to discover and manipulate DOM elements on a page, and to control the behavior of the containing browser from a separate controlling process
Most popular instance of WebDriver specification is selenium2.0
Enable WebDriver on Crosswalk

UI Automator

The UI Automator testing framework provides a set of APIs to build UI tests that perform interactions on user apps and system apps. The UI Automator APIs allows you to perform operations such as opening the Settings menu or the app launcher in a test device.
uiautomator is the most popular Python wrapper of Android UI Automator testing framework. It works on Android 4.1+ simply with Android device attached via adb, no need to install anything on Android device.

BDD

An agile software development technique that encourages collaboration between developers, QA and non-technical or business participants in a software project.
Our solution implements BDD principles to allow users to use tests written in a ubiquitous language style to validate the web applications and web platform and offers a convenient and efficient way to develop, manage, execute the test cases, and easy to combine with high level test framework

Solution

image-name

"behave"

"behave is behaviour-driven development, Python style", and the page of "comparisons with other tools" shows why we select "behave" as the default BDD tool.

ATIP - a new "behave" binding library

ATIP(Application Test in Python), a “behave” binding library as the bridge between application and BDD too “behave”, and use WebDriver and platform interfaces to implement detailed BDD steps for application. ATIP resources host on Crosswalk Test Suite Project

Features Status

Features Done

Basic WebDriver based steps
Basic Android UI Automator based steps
Environment initialization from config file
Environment initialization from test tools/system environment vars
Support Crosswalk on Tizen and Android
Crosswalk application launching

Features in Plan

Add more WebDriver based UI automation steps
Add more platform native application UI automation support
Provide APIs level interfaces from ATIP so that generic script can import ATIP too
More platforms support

Tests Development

Before below sections, you'd better already pretty familiar with the tests developing with "behave": "behave" docs. A typical "behave" based tests source layout as below:

 tests/
  ├── environment.py
  ├── steps
  │   └── steps.py
  └── test.feature

"test.feature"

Feature file, one tests folder can contains more than one feature file which has a natural language format describing a feature or part of a feature with representative examples of expected outcomes, e.g.

 Feature: api tests
      Scenario: api test 001
           When launch "haha"
            And I go to "http://www.google.com"
            And I wait for 1 seconds
           Then I should see "Google"

"steps.py"

Python steps implementations file. In ATIP usage, this file will import the steps from ATIP. Of course, you can write your own steps to this "steps.py" file too. The build-in ATIP steps as following:

"web" steps(Webdriver API based steps)

@step(u'I launch "{app_name}" with "{apk_pkg_name}" and "{apk_activity_name}"')
@step(u'switch to "{app_name}"')
@step(u'I go to "{url}"')
@step(u'I reload')
@step(u'I go back')
@step(u'I go forward')
@step(u'The current URL should be "{text}"')
@step(u'I should see title "{text}"')
@step(u'I should see "{text}"')
@step(u'I should not see "{text}"')
@step(u'I should see "{text}" in {timeout:d} seconds')
@step(u'I should not see "{text}" in {timeout:d} seconds')
@step(u'I should see "{text}" in "{key}" area')
@step(u'I press "{key}"')
@step(u'press "{key_c}" in "{key_p}"')
@step(u'I click "{key}"')
@step(u'click "{key_c}" in "{key_p}"')
@step(u'I click coords {x:d} and {y:d} of "{key}"')
@step(u'I fill in "{key}" with "{text}"')
@step(u'I check "{key}"')
@step(u'I uncheck "{key}"')
@step(u'I should see an alert')
@step(u'I should not see an alert')
@step(u'I accept the alert')
@step(u'I should see an alert with text "{text}"')

"android" steps(Android platform specific steps which is implemented by uiautomator)

@step(u'I launch "{app_name}" with "{apk_pkg_name}" and "{apk_activity_name}" on android')
@step(u'I scroll to end')
@step(u'I fling "{orientation}" goto "{direction}"')
@step(u'I swipe object "{key}" to "{orientation}"')
@step(u'I force to run all watchers')
@step(u'I remove all watchers')
@step(u'I register watcher "{watcher_name}" when "{when_text}" click "{click_text}"')
@step(u'I register watcher2 "{watcher_name}" when "{when_text1}" and "{when_text2}" click "{click_text}"')
@step(u'I should see text "{text_name}"')
@step(u'I should see image "{image_name}"')
@step(u'I should see web "{web_desc}"')
@step(u'I should see view "{view_desc}"')
@step(u'I should see "{class_name}" on the "{relative}" side of text "{text_name}"')
@step(u'I should see "{class_name}" on the "{relative}" side of view "{view_desc}"')
@step(u'I should see "{class_target}" on the "{relative}" side of any "{class_name}" "{value_name}"')
@step(u'I should not see text "{text_name}"')
@step(u'I should not see image "{image_name}"')
@step(u'I should not see web "{web_desc}"')
@step(u'I should not see view "{view_desc}"')
@step(u'I should not see "{class_name}" on the "{relative}" side of text "{text_name}"')
@step(u'I should not see "{class_name}" on the "{relative}" side of view "{view_desc}"')
@step(u'I should not see "{class_target}" on the "{relative}" side of any "{class_name}" "{value_name}"')
@step(u'I wait object "{key}" exist for "{time_out}"')
@step(u'I wait object "{key}" gone for "{time_out}"')
@step(u'I click button "{button_name}"')
@step(u'I click other "{class_name}" by "{which_key}" "{which_value}"')
@step(u'I click object "{key}"')
@step(u'I edit text "{edit_text}" to input "{text}"')
@step(u'I edit index {n:d} text to input "{text}"')
@step(u'I compare text "{text_name}" info "{what}" with "{except_result}"')
@step(u'I compare view "{view_desc}" info "{what}" with "{except_result}"')
@step(u'I compare object "{key1}" equal "{key2}" on info "{what}"')
@step(u'I compare object "{key1}" unequal "{key2}" on info "{what}"')
@step(u'I save text object "{text_name}" to temporary value "{key}"')
@step(u'I save view object "{view_desc}" to temporary value "{key}"')
@step(u'I save any object "{class_name}" "{value_name}" to temporary value "{key}"')
@step(u'I save "{class_name}" on the "{relative}" side of text "{text_name}" to temporary value "{key}"')
@step(u'I save "{class_name}" on the "{relative}" side of view "{view_desc}" to temporary value "{key}"')
@step(u'I save "{class_target}" on the "{relative}" side of any "{class_name}" "{value_name}" to temporary value "{key}"')
@step(u'I process text object "{text_name}"')
@step(u'I process view object "{view_desc}"')
@step(u'I process any object "{class_name}" "{value_name}"')
@step(u'I process "{class_name}" on the "{relative}" side of text "{text_name}"')
@step(u'I process "{class_name}" on the "{relative}" side of view "{view_desc}"')
@step(u'I process "{class_target}" on the "{relative}" side of any "{class_name}" "{value_name}"')
@step(u'I reload process result to temporary value "{key}"')

"common" steps(common operations cross different platforms)

@step(u'I wait for {timeout:d} seconds')
@step(u'call "{js}" scripts') - TBD
@step(u'call PIL to handle "{image_file}"') - TBD
@step(u'launch "{app_name}"')
@step(u'I turn on device')
@step(u'I turn off device')
@step(u'I set orientation "{orientation}"')
@step(u'I take screenshot as "{name}"')
@step(u'I open notification')
@step(u'I open quick settings')
@step(u'I press "{key}" hardware key')
@step(u'I open wifi')
@step(u'I close wifi')
@step(u'I open airplane mode')
@step(u'I close airplane mode')

"environment.py"

Environmental controls. In ATIP usage, you can write the environment initial processes on this file. ATIP provides two templates of "environment.py" for tests developers. One is for WebDriver backend, the other one is for uiautomator backend. These templates support running tests independently by "behave" tool.

In WebDriver testing, the template need to know the some test vars, e.g. which device be tested? which test platform? Some WebDriver vars. the vars can be got by following ways:

By environment vars:

TEST_PLATFORM
DEVICE_ID
CONNECT_TYPE
WEBDRIVER_VARS

In uiautomator testing, only two environment vars are required, "TEST_PLATFORM" and "DEVICE_ID". This allows running uiautomator tests independently without WebDriver.

ATIP source provides a script "tools/set_env.sh" which can help you to setup those environment vars. Especially, the environment.py template can get those environment vars from Testkit-lite tool automatically:

A JSON config file which named "bdd.json for environment vars sharing, a template provided for reference: "atip/tools/bdd.*.json"

##Example image-name

Run Tests