From 7b5d18254b2862a34d9e943f9aa9643b29cc63e1 Mon Sep 17 00:00:00 2001 From: Cecille Freeman Date: Sun, 21 Jan 2024 17:30:41 -0500 Subject: [PATCH] a bit more --- docs/testing/integration_tests.md | 52 ++++++++++++++++++++++--------- docs/testing/python.md | 3 ++ docs/testing/yaml.md | 19 ++++++++++- 3 files changed, 58 insertions(+), 16 deletions(-) diff --git a/docs/testing/integration_tests.md b/docs/testing/integration_tests.md index 0939c5883387be..5db1d411322170 100644 --- a/docs/testing/integration_tests.md +++ b/docs/testing/integration_tests.md @@ -1,24 +1,46 @@ # Integration and Certification Tests -Integration tests use a server and a controller or controllers to test the behavior of a device. Certification tests are all integration tests. For certified products, the device under test (DUT) is tested against one of the SDK controller implementations (either chip-tool or the python-based controller, depending on the test type). For software component certification, the software component is tested against a sample device built from the SDK. - -Certification tests require an accompanying certification test plan in order to be used in the certification testing process. More information about test plans can be found in the [test plans repository](https://github.com/CHIP-Specifications/chip-test-plans/tree/master/docs). Integration testing can also be used outside of the certification testing program to test device behavior in the SDK. Certification tests are all run in the [CI](./ci_testing). +Integration tests use a server and a controller or controllers to test the +behavior of a device. Certification tests are all integration tests. For +certified products, the device under test (DUT) is tested against one of the SDK +controller implementations (either chip-tool or the python-based controller, +depending on the test type). For software component certification, the software +component is tested against a sample device built from the SDK. + +Certification tests require an accompanying certification test plan in order to +be used in the certification testing process. More information about test plans +can be found in the +[test plans repository](https://github.com/CHIP-Specifications/chip-test-plans/tree/master/docs). +Integration testing can also be used outside of the certification testing +program to test device behavior in the SDK. Certification tests are all run in +the [CI](./ci_testing). There are two main integration test types: -- [YAML](./yaml.md) -- [Python framework](./python.md) +- [YAML](./yaml.md) +- [Python framework](./python.md) -YAML is a human-readable serialization language that uses structured tags to define test steps. Tests are defined in YAML, and parsed and run through a runner that is backed by the chip-tool controller. +YAML is a human-readable serialization language that uses structured tags to +define test steps. Tests are defined in YAML, and parsed and run through a +runner that is backed by the chip-tool controller. -The Python framework tests are written in python and use the [Mobly](https://github.com/google/mobly) test framework to execute tests. +The Python framework tests are written in python and use the +[Mobly](https://github.com/google/mobly) test framework to execute tests. ## Which test framework to use -Both types of tests can be run through the Test Harness for certification testing, locally for the purposes of development and in the CI for the SDK. The appropriate test framework to use is whatever lets you automate your tests in a way that is understandable, readable, and has the features you need - -* YAML - * pros: more readable, simpler to write, easy for ATLs to parse and understand - * cons: conditionals are harder (not all supported), no branch control, schema not well documented -* python - * pros: full programming language, full control API with support for core (certs, commissioning, etc), less plumbing if you need to add features, can use python libs - * cons: more complex, can be harder to read \ No newline at end of file + +Both types of tests can be run through the Test Harness for certification +testing, locally for the purposes of development and in the CI for the SDK. The +appropriate test framework to use is whatever lets you automate your tests in a +way that is understandable, readable, and has the features you need + +- YAML + - pros: more readable, simpler to write, easy for ATLs to parse and + understand + - cons: conditionals are harder (not all supported), no branch control, + schema not well documented +- python + - pros: full programming language, full control API with support for core + (certs, commissioning, etc), less plumbing if you need to add features, + can use python libs + - cons: more complex, can be harder to read diff --git a/docs/testing/python.md b/docs/testing/python.md index 103eabf275171b..3af81e2394fe85 100644 --- a/docs/testing/python.md +++ b/docs/testing/python.md @@ -1,3 +1,6 @@ # Python framework tests This file is a placeholder for python framework test information. + +NOTE: be sure to include information about how you need to commission with the +python controller, not chip-tool and how to do that in the scripts diff --git a/docs/testing/yaml.md b/docs/testing/yaml.md index f1219f901cb103..8caf55d4105747 100644 --- a/docs/testing/yaml.md +++ b/docs/testing/yaml.md @@ -287,6 +287,14 @@ commands over a socket to chip-tool (running in interactive mode). ### Running locally +#### Commissioning the DUT + +All YAML tests assume that the DUT has previously been commissioned before +running. DUTs should be commissioned using chip-tool. Use the same KVS file when +running the test. + +#### Running the tests + There are several options for running tests locally. Because the YAML runner uses python, it is necessary to compile and install the chip python package before using any YAML runner script. @@ -330,6 +338,12 @@ with -- ``` +#### Factory resetting the DUT + +On the host machine, you can simulate a factory reset by deleting the KVS file. +If you did not specify a location for the KVS file when starting the +application, the KVS file will be in /tmp as chip_kvs + ### Running in the CI - YAML tests added to the certification directory get run automatically @@ -337,7 +351,10 @@ with -- - PICS file: src/app/tests/suites/certification/ci-pics-values - If you DON’T want to run a test in the CI - (ex under development), add it to \_GetInDevelopmentTests in - scripts/tests/chiptest/**init**.py + `scripts/tests/chiptest/__init__.py` + +Please see [CI testing](./ci_testing.md) for more information about how to set +up examples apps, PICS and PIXIT values for use in the CI. ### Running in the TH