Skip to content

Home

Jump to bottom Edit New page
jake9050 edited this page Jul 24, 2017 · 14 revisions

Introduction

This repository contains all there is to know about the automated tests The tests rely on the python nose framework All testsuites, testfiles, testmodules, testclasses and testcases must be compliant to the definition nose relies on.

The integration tests can be installed like this:

  • Ubuntu: apt-get install openvstorage-test

Structure

The topdirectories within this repo are CI and PACKAGING

  • ci: The ci directory contains all logic regarding the autotests itself:
    • api_lib: (Generic) libraries for setup and validation, which on their turn consist out of 3 subdirectories: backend (Ceph, Swift, ALBa, ...), framework (vDisk, vMachine, vPool,...), store (Arakoon, Memcache, Pyrakoon, ...)
    • config: Configuration files used by the integration tests
    • scenario_helpers: Test helper libraries
    • scenarios: Actual test runs
  • packaging: The packaging directory contains logic to create the openvstorage-test package

Usage

ipython from ci.autotests import AutoTests

Listing tests

AutoTests.list_tests() AutoTests.list_tests('ci.scenarios.arakoon)

Example output of above command:

In [1]: from ci.autotests import AutoTests

In [2]: AutoTests.list_tests()
Out[2]: 
['ci.scenarios.arakoon.ovs_4509_validate_arakoon_collapse_test',
 'ci.scenarios.arakoon.ar_0002_arakoon_cluster_validation_test',
 'ci.scenarios.arakoon.ovs_3671_validate_archiving_of_existing_arakoon_data_on_create_and_extend_test',
 'ci.scenarios.edge.reroute',
 'ci.scenarios.storagerouter.tdr_0001_add_append_remove_role_test',
 'ci.scenarios.healthcheck.healthcheck_run_test',
 'ci.scenarios.system.test_basic_logrotate',
 'ci.scenarios.hypervisor.automated_ha',
 'ci.scenarios.backend.be_0002_add_update_remove_preset_test',
 'ci.scenarios.backend.be_0001_add_and_remove_backend_test',
 'ci.scenarios.vmachine.check_scrubbing_test',
 'ci.scenarios.vmachine.perform_fio_test',
 'ci.scenarios.installation.services_check_test',
 'ci.scenarios.installation.ssh_check_test',
 'ci.scenarios.vPool.mds_regression',
 'ci.scenarios.vPool.add_remove_alba_vpool_test',
 'ci.scenarios.vDisk.offline_migrate_test',
 'ci.scenarios.vDisk.rapid_creation_same_device',
 'ci.scenarios.vDisk.regress_template_memleak_test',
 'ci.scenarios.vDisk.rollback_vdisk_test',
 'ci.scenarios.vDisk.deployment_vdisk_test',
 'ci.scenarios.vDisk.basic_dtl_vdisk_test',
 'ci.scenarios.vDisk.data_corruption_reg_test',
 'ci.scenarios.vDisk.validate_clone_disk_test',
 'ci.scenarios.vDisk.validate_template_disk_test',
 'ci.scenarios.vDisk.advanced_dtl_vdisk_test']

Running tests

To run a test, simply copy/paste a testcase from above output and run it

Autotests.run('ci.scenarios.vDisk.advanced_dtl_vdisk_test')

Its also possible to run entire testclasses, testmodules or testsuites by just copy/pasting up until the desired depth
AutoTests.run('ci.scenarios.installation') AutoTests.run('ci.scenarios.healthcheck')

Implementing tests

  • Separate setup, validation and autotests through the default QA structure:
ci.tests
ci.lib.validation
ci.lib.setup

ci.lib.validation.store
ci.lib.validation.backend
ci.lib.validation.framework

ci.lib.setup.store
ci.lib.setup.backend
ci.lib.setup.framework

  • When creating autotests, you add your test to ci.tests. Because we are working with NOSE, all autotest methods need to end with _test() and files need to end with _test.py

  • When you contribute your autotest:

    • Make a new branch
    • Adjust your autotest
    • Run tests (master and your own in parallel and check if everything succeeded ==> this is not to break the master branch and falsify the results.)
    • Validate by yourself
    • Create pull request
    • Validate by 3rd party
    • Merge
    • Done!

  • Make your tests as modular and independent as possible. (Don't depend on other autotests only on the libraries)

  • Make your code as generic and flexible as possible (Tests need to PASS on hyper-scale and hyperconverged)

  • Make it possible to work with external settings/configuration

Nose basic guidelines

The tests rely on the python-nose framework
Basic guidelines on how the tests should be defined and on the setup and teardown (http://nose.readthedocs.org/en/latest/writing_tests.html)

Layout/structure used by ovs:

integrationtests
 \_ ci
     \_ scenarios
          \_ api
               \_ __init__.py
               \_ extended_test.py
          \_ arakoon
               \_ __init__.py
               \_ ovs_4509_validate_arakoon_collapse_test/__init__.py
               \_ ovs_4509_validate_arakoon_collapse_test/main.py
          \_ backend
               \_ __init__.py
               \_ be_0001_add_and_remove_backend_test/__init__.py
               \_ be_0001_add_and_remove_backend_test/main.py
          \_ ...

The directories arakoon, backend all contain an __init__.py file
For each test a new directory is created containing:

  • init.py
  • main.py

Nose requirements:

  • directory ends with _test
  • example test structure:
# Copyright (C) 2016 iNuron NV
#
# This file is part of Open vStorage Open Source Edition (OSE),
# as available from
#
#      http://www.openvstorage.org and
#      http://www.openvstorage.com.
#
# This file is free software; you can redistribute it and/or modify it
# under the terms of the GNU Affero General Public License v3 (GNU AGPLv3)
# as published by the Free Software Foundation, in version 3 as it comes
# in the LICENSE.txt file of the Open vStorage OSE distribution.
#
# Open vStorage is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY of any kind.


class ArakoonValidation(CIConstants):

    CASE_TYPE = 'FUNCTIONAL'
    TEST_NAME = "ci_scenario_arakoon_validation"
    LOGGER = LogHandler.get(source="scenario", name="ci_scenario_arakoon_validation")

    def __init__(self):
        pass

    def main(blocked):
        """
        Run all required methods for the test
        :param blocked: was the test blocked by other test?
        :return: results of test
        :rtype: dict
        """
        _ = blocked
        return ArakoonValidation.validate_cluster()

    @staticmethod
    def validate_cluster(cluster_name='ovsdb'):
        """
        Validate if the chosen cluster is
         * deployed on all required nodes
         * running on all required nodes
         * working correctly on all required nodes

        :param cluster_name: name of a existing arakoon cluster (DEFAULT=ovsdb)
        :type cluster_name: str
        :return:
        """

        ### TEST LOGIC ###

def run(blocked=False):
    """
    Run a test

    :param blocked: was the test blocked by other test?
    :return: results of test
    :rtype: dict
    """
    return ArakoonValidation().main(blocked)

if __name__ == "__main__":
    run()

Definitions

ci.tests.arakoon is considered a test package
ci.tests.arakoon.extend_promote_demote_test is considered a test module
ci.tests.arakoon.extend_promote_demote_test.ArakoonValidation is considered a test class
ci.tests.arakoon.extend_promote_demote_test.ArakoonValidation.main runs actual test case

Setup and teardown

Test environments are installed and configured afterwards with a certain configuration used during all test runs. This is handled by the Workflow in the setup-runner repo (which is a private to OpenvStorage)

Test runs

Test runs are handled by autotests.py which also stores test results in testrail

Add a custom footer
Add a custom sidebar
Clone this wiki locally