Behavior Driven Test Automation

Overview

This solution implements Behavior-driven Development (abbreviated BDD) principles to allow users to use tests written in a ubiquitous language style to validate the web/native applications and web/native platform and offers a convenient and efficient way to develop, manage, execute the test cases, and easy to combine with high level test framework

Background

Application Test Automation Challenges

Hard to apply image recognition or platform native UI automation solution to web application
Multi-formity of web UI layout and elements requires a more convenient test design and development method
Existing browser test automation techniques and tools have gap on support on web applications or web runtime

WebDriver

WebDriver is a W3C standard: The WebDriver API is defined by a wire protocol and a set of interfaces to discover and manipulate DOM elements on a page, and to control the behavior of the containing browser from a separate controlling process
Most popular instance of WebDriver specification is selenium2.0
Enable WebDriver on Crosswalk

UI Automator

The UI Automator testing framework provides a set of APIs to build UI tests that perform interactions on user apps and system apps. The UI Automator APIs allows you to perform operations such as opening the Settings menu or the app launcher in a test device.
uiautomator is a Python wrapper of Android UI Automator testing framework. It works on Android 4.1+ simply with Android device attached via adb, no need to install anything on Android device.

BDD

An agile software development technique that encourages collaboration between developers, QA and non-technical or business participants in a software project.
Our solution implements BDD principles to allow users to use tests written in a ubiquitous language style to validate the web applications and web platform and offers a convenient and efficient way to develop, manage, execute the test cases, and easy to combine with high level test framework

Solution

image-name

"behave"

"behave is behaviour-driven development, Python style", and the page of "comparisons with other tools" shows why we select "behave" as the default BDD tool.

ATIP - a new "behave" binding library

ATIP(Application Test in Python), a “behave” binding library as the bridge between application and BDD too “behave”, and use WebDriver and platform interfaces to implement detailed BDD steps for application. ATIP resources host on Crosswalk Test Suite Project

Features Status

Features Done

Basic WebDriver based steps
Environment initialization from config file
Environment initialization from test tools/system environment vars
Support Crosswalk on Tizen and Android
Crosswalk application launching

Features in Plan

Support more pre-defined scenarios in ATIP
Native and Hybrid application support in ATIP
Multiple-application support in ATIP
Support both BDD steps and Python APIs
Adapt to W3C WebDriver specification update

Tests Development

Before below sections, you'd better already pretty familiar with the tests developing with "behave": "behave" docs. A typical "behave" based tests source layout as below:

 tests/
  ├── environment.py

  ├── steps

  │   └── steps.py

  └── test.feature

"test.feature"

Feature file, one tests folder can contains more than one feature file which has a natural language format describing a feature or part of a feature with representative examples of expected outcomes, e.g.

 Feature: api tests

      Scenario: api test 001

           When launch "haha"

           And I go to "http://www.google.com"

           And I wait for 1 seconds

"steps.py"

Python steps implementations file. In ATIP usage, this file will import the steps from ATIP. Of course, you can write your own steps to this "steps.py" file too. The build-in ATIP steps as following:

@step(u'I wait for {timeout:d} seconds')
@step(u'launch "{app_name}"')
@step(u'switch to "{app_name}"')
@step(u'I go to "{url}"')
@step(u'I reload')
@step(u'I go back')
@step(u'I go forward')
@step(u'The current URL should be "{text}"')
@step(u'I should see title "{text}"')
@step(u'I should see "{text}"')
@step(u'I should not see "{text}"')
@step(u'I should see "{text}" in {timeout:d} seconds')
@step(u'I should not see "{text}" in {timeout:d} seconds')
@step(u'I should see "{text}" in "{key}" area')
@step(u'I press "{key}"')
@step(u'I click "{key}"')
@step(u'I click coords {x:d} and {y:d} of "{key}"')
@step(u'I fill in "{key}" with "{text}"')
@step(u'I check "{key}"')
@step(u'I uncheck "{key}"')
@step(u'I should see an alert')
@step(u'I should not see an alert')
@step(u'I accept the alert')
@step(u'I should see an alert with text "{text}"')

"environment.py"

Environmental controls. In ATIP usage, you can write the environment initial processes on this file. ATIP provides a template of "environment.py" for tests developers. This template supports running tests independently by "behave" tool.

In Crosswalk testing, the template need to know the some test vars, e.g. which device be tested? which test platform? Some WebDriver vars. the vars can be got by following ways:

By environment vars:

TEST_PLATFORM
DEVICE_ID
CONNECT_TYPE
WEBDRIVER_VARS

ATIP source provides a script "tools/set_env.sh" which can help you to setup those environment vars. Especially, the environment.py template can get those environment vars from Testkit-lite tool automatically:

A JSON config file which named "webdriver.json for environment vars sharing, a template provided for reference: "atip/tools/webdriver.*.json"

##Example image-name

Run Tests