Skip to content

Commit

Permalink
a bit more
Browse files Browse the repository at this point in the history
  • Loading branch information
cecille committed Jan 21, 2024
1 parent a0e49f3 commit 7b5d182
Show file tree
Hide file tree
Showing 3 changed files with 58 additions and 16 deletions.
52 changes: 37 additions & 15 deletions docs/testing/integration_tests.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,46 @@
# Integration and Certification Tests

Integration tests use a server and a controller or controllers to test the behavior of a device. Certification tests are all integration tests. For certified products, the device under test (DUT) is tested against one of the SDK controller implementations (either chip-tool or the python-based controller, depending on the test type). For software component certification, the software component is tested against a sample device built from the SDK.

Certification tests require an accompanying certification test plan in order to be used in the certification testing process. More information about test plans can be found in the [test plans repository](https://github.com/CHIP-Specifications/chip-test-plans/tree/master/docs). Integration testing can also be used outside of the certification testing program to test device behavior in the SDK. Certification tests are all run in the [CI](./ci_testing).
Integration tests use a server and a controller or controllers to test the
behavior of a device. Certification tests are all integration tests. For
certified products, the device under test (DUT) is tested against one of the SDK
controller implementations (either chip-tool or the python-based controller,
depending on the test type). For software component certification, the software
component is tested against a sample device built from the SDK.

Certification tests require an accompanying certification test plan in order to
be used in the certification testing process. More information about test plans
can be found in the
[test plans repository](https://github.com/CHIP-Specifications/chip-test-plans/tree/master/docs).
Integration testing can also be used outside of the certification testing
program to test device behavior in the SDK. Certification tests are all run in
the [CI](./ci_testing).

There are two main integration test types:

- [YAML](./yaml.md)
- [Python framework](./python.md)
- [YAML](./yaml.md)
- [Python framework](./python.md)

YAML is a human-readable serialization language that uses structured tags to define test steps. Tests are defined in YAML, and parsed and run through a runner that is backed by the chip-tool controller.
YAML is a human-readable serialization language that uses structured tags to
define test steps. Tests are defined in YAML, and parsed and run through a
runner that is backed by the chip-tool controller.

The Python framework tests are written in python and use the [Mobly](https://github.com/google/mobly) test framework to execute tests.
The Python framework tests are written in python and use the
[Mobly](https://github.com/google/mobly) test framework to execute tests.

## Which test framework to use
Both types of tests can be run through the Test Harness for certification testing, locally for the purposes of development and in the CI for the SDK. The appropriate test framework to use is whatever lets you automate your tests in a way that is understandable, readable, and has the features you need

* YAML
* pros: more readable, simpler to write, easy for ATLs to parse and understand
* cons: conditionals are harder (not all supported), no branch control, schema not well documented
* python
* pros: full programming language, full control API with support for core (certs, commissioning, etc), less plumbing if you need to add features, can use python libs
* cons: more complex, can be harder to read

Both types of tests can be run through the Test Harness for certification
testing, locally for the purposes of development and in the CI for the SDK. The
appropriate test framework to use is whatever lets you automate your tests in a
way that is understandable, readable, and has the features you need

- YAML
- pros: more readable, simpler to write, easy for ATLs to parse and
understand
- cons: conditionals are harder (not all supported), no branch control,
schema not well documented
- python
- pros: full programming language, full control API with support for core
(certs, commissioning, etc), less plumbing if you need to add features,
can use python libs
- cons: more complex, can be harder to read
3 changes: 3 additions & 0 deletions docs/testing/python.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Python framework tests

This file is a placeholder for python framework test information.

NOTE: be sure to include information about how you need to commission with the
python controller, not chip-tool and how to do that in the scripts
19 changes: 18 additions & 1 deletion docs/testing/yaml.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,14 @@ commands over a socket to chip-tool (running in interactive mode).

### Running locally

#### Commissioning the DUT

All YAML tests assume that the DUT has previously been commissioned before
running. DUTs should be commissioned using chip-tool. Use the same KVS file when
running the test.

#### Running the tests

There are several options for running tests locally. Because the YAML runner
uses python, it is necessary to compile and install the chip python package
before using any YAML runner script.
Expand Down Expand Up @@ -330,14 +338,23 @@ with --
```

#### Factory resetting the DUT

On the host machine, you can simulate a factory reset by deleting the KVS file.
If you did not specify a location for the KVS file when starting the
application, the KVS file will be in /tmp as chip_kvs

### Running in the CI

- YAML tests added to the certification directory get run automatically
- src/app/tests/suites/certification/
- PICS file: src/app/tests/suites/certification/ci-pics-values
- If you DON’T want to run a test in the CI
- (ex under development), add it to \_GetInDevelopmentTests in
scripts/tests/chiptest/**init**.py
`scripts/tests/chiptest/__init__.py`

Please see [CI testing](./ci_testing.md) for more information about how to set
up examples apps, PICS and PIXIT values for use in the CI.

### Running in the TH

Expand Down

0 comments on commit 7b5d182

Please sign in to comment.