diff --git a/docs/testing/python.md b/docs/testing/python.md index e69de29bb2d1d6..103eabf275171b 100644 --- a/docs/testing/python.md +++ b/docs/testing/python.md @@ -0,0 +1,3 @@ +# Python framework tests + +This file is a placeholder for python framework test information. diff --git a/docs/testing/unit_testing.md b/docs/testing/unit_testing.md index e69de29bb2d1d6..e62940f15a9c7c 100644 --- a/docs/testing/unit_testing.md +++ b/docs/testing/unit_testing.md @@ -0,0 +1,3 @@ +# Unit testing + +This doc is a placeholder for an guide around unit tests. diff --git a/docs/testing/yaml.md b/docs/testing/yaml.md index c02c01bf884330..8ec437317f4615 100644 --- a/docs/testing/yaml.md +++ b/docs/testing/yaml.md @@ -1,36 +1,51 @@ # YAML tests -YAML is a structured, human-readable data-serialization language. Much like json or proto, YAML refers to the structure and parser, and the schema used for any particular application is defined by the application. +YAML is a structured, human-readable data-serialization language. Much like json +or proto, YAML refers to the structure and parser, and the schema used for any +particular application is defined by the application. -In Matter, we use YAML for describing tests and test steps. A YAML parser and runner is then used to translate the YAML instructions into actions used to interact with the device under test (DUT). +In Matter, we use YAML for describing tests and test steps. A YAML parser and +runner is then used to translate the YAML instructions into actions used to +interact with the device under test (DUT). -The main runner we use for testing in Matter parses the YAML instructions into chip-tool commands. +The main runner we use for testing in Matter parses the YAML instructions into +chip-tool commands. -The schema description for the Matter test YAML is available here: [YAML Schema](./yaml_schema.md) +The schema description for the Matter test YAML is available here: +[YAML Schema](./yaml_schema.md) ## Writing YAML tests -Most YAML tests are written for certification. These follow a standard format that is used to display the test easily in the test harness. +Most YAML tests are written for certification. These follow a standard format +that is used to display the test easily in the test harness. ### Placeholder for anatomy of a yaml test - need diagram ### Placeholder for anatomy of a test step - need diagram ### Common actions + #### Sending a cluster command + The following shows a test step sending a simple command with no arguments. + ``` - label: "This label gets printed" cluster: "On/Off" command: "On" ``` -* label - label to print before performing the test step -* cluster - name of the cluster to send the command to -* command - name of the command to send -This send the On command to the On/Off cluster on the DUT. For most tests, the nodeID of the DUT and endpoint for the cluster are defined in the top-level config section of the file and applied to every test step. However, these can also be overwritten in the individual test steps. +- label - label to print before performing the test step +- cluster - name of the cluster to send the command to +- command - name of the command to send + +This send the On command to the On/Off cluster on the DUT. For most tests, the +nodeID of the DUT and endpoint for the cluster are defined in the top-level +config section of the file and applied to every test step. However, these can +also be overwritten in the individual test steps. The following shows how to send a command with arguments: + ``` - label: "This label gets printed before the test step" command: "MoveToColor" @@ -47,24 +62,35 @@ The following shows how to send a command with arguments: - name: "OptionsOverride" value: 0 ``` -* label - label to print before performing the test step -* command - name of the command to send -* argument - this is a list parameter that takes either a "value" or "values" tag. Commands with arguments all use structured fields, which require the "values" tag with a list. Each of the fields is represented by a "name" and "value" pair -In this command, the cluster: tag is elided. The cluster for the entire test can be set in the config section at the top of the test. This can be overwritten for individual test steps (as above). +- label - label to print before performing the test step +- command - name of the command to send +- argument - this is a list parameter that takes either a "value" or "values" + tag. Commands with arguments all use structured fields, which require the + "values" tag with a list. Each of the fields is represented by a "name" and + "value" pair + +In this command, the cluster: tag is elided. The cluster for the entire test can +be set in the config section at the top of the test. This can be overwritten for +individual test steps (as above). #### Reading and writing attributes -Reading and writing attributes is represented in the Matter test YAML schemas as a special command that requires an additional "attribute" tag. +Reading and writing attributes is represented in the Matter test YAML schemas as +a special command that requires an additional "attribute" tag. + +The following YAML would appear as a test step, and shows how to read an +attribute. -The following YAML would appear as a test step, and shows how to read an attribute. ``` - label: "TH reads the ClusterRevision from DUT" command: "readAttribute" attribute: "ClusterRevision" ``` -The following YAML would appear as a test step and shows how to write an attribute. Commands to write attributes always require an argument: tag. +The following YAML would appear as a test step and shows how to write an +attribute. Commands to write attributes always require an argument: tag. + ``` - label: "Write example attribute" command: "writeAttribute" @@ -74,9 +100,14 @@ The following YAML would appear as a test step and shows how to write an attribu ``` #### Parsing Responses -After sending a command or read or write attribute request, you may want to verify the response. This is done using the "response" tag with various sub-tags. -The following shows a simple response parsing with two (somewhat redundant) checks. +After sending a command or read or write attribute request, you may want to +verify the response. This is done using the "response" tag with various +sub-tags. + +The following shows a simple response parsing with two (somewhat redundant) +checks. + ``` - label: "TH reads the ClusterRevision from DUT" command: "readAttribute" @@ -86,15 +117,15 @@ The following shows a simple response parsing with two (somewhat redundant) chec constraints: minValue: 1 ``` + The following tags can be used to parse the response -|Example | Description | -| :--- | :--- | -| response:
 value: [1, 2, 3, 4] | must match exactly. Variables and saveAs values allowed | +| Example | Description | +| :---------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------- | +| response:
 value: [1, 2, 3, 4] | must match exactly. Variables and saveAs values allowed | | response:
 values:
  - name: response_field
     value: 1 | Must match exactly
Use for commands that return command responses with named fields | -| response:
  error: CONSTRAINT_ERROR | expect an error back (omit for success)
Variables and saveAs values will not work. | -| response:
 constraints: | more complex checks - see [Schema](./yaml_schema.md) for a complete description | - +| response:
  error: CONSTRAINT_ERROR | expect an error back (omit for success)
Variables and saveAs values will not work. | +| response:
 constraints: | more complex checks - see [Schema](./yaml_schema.md) for a complete description | #### Lists and structs @@ -105,6 +136,7 @@ Lists: `[1,2,3,4,5]` structs: `{field1:value, field2:value}` lists of structs: + ``` [ @@ -115,16 +147,20 @@ lists of structs: ] ``` -Note that structs are different than command and command response fields, which are represented using name:, value: tags. - - +Note that structs are different than command and command response fields, which +are represented using name:, value: tags. #### Pseudo clusters -Tests often require functionality that is not strictly cluster-based. Some of this functionality is supported in YAML using pseudo-clusters. These clusters accept command: tags like the DUT clusters to control the psudo-cluster functionality. + +Tests often require functionality that is not strictly cluster-based. Some of +this functionality is supported in YAML using pseudo-clusters. These clusters +accept command: tags like the DUT clusters to control the psudo-cluster +functionality. Some of the more common functionality is shown below: -Establishing a connection to the DUT. This is the first step in nearly every test. +Establishing a connection to the DUT. This is the first step in nearly every +test. ``` - label: "Establish a connection to the DUT" @@ -137,6 +173,7 @@ Establishing a connection to the DUT. This is the first step in nearly every tes ``` Wait for a user action: + ``` - label: "Do a simple user prompt message. Expect 'y' to pass." cluster: "LogCommands" @@ -150,6 +187,7 @@ Wait for a user action: ``` Wait for a time: + ``` - label: "Wait for 5S" cluster: "DelayCommands" @@ -160,16 +198,20 @@ Wait for a time: value: 5000 ``` - -A full description of the available pseudo-clusters and their commands is available below: +A full description of the available pseudo-clusters and their commands is +available below: TODO: generate a script to parse the pseudo cluster xml into something readable - #### Config variables and saveAs: -Certain tags can use variables that are either declared in the config: section or saved from other steps. Variables that are declared in the config can be overwritten on the command line when running locally or through the config file in the test harness. -To declare config variables in the config section, use a label with the desired name, then provide the type and defaultValue tags as sub-tags. +Certain tags can use variables that are either declared in the config: section +or saved from other steps. Variables that are declared in the config can be +overwritten on the command line when running locally or through the config file +in the test harness. + +To declare config variables in the config section, use a label with the desired +name, then provide the type and defaultValue tags as sub-tags. ``` config: @@ -200,6 +242,7 @@ Variables can also be saved from responses: ``` Variables can then be used in later steps: + ``` - label: "Send Test Add Arguments Command" command: "TestAddArguments" @@ -215,25 +258,40 @@ Variables can then be used in later steps: value: TestAddArgumentDefaultValue ``` -Tags where variables can be used are noted in the [schema description](./yaml_schema.md). - +Tags where variables can be used are noted in the +[schema description](./yaml_schema.md). Config variables can be used to implement PIXIT values in tests. #### Gating tests and steps: PICS, TestEqualities and runIf -The PICS tag can be used to unconditionally gate a test step on the PICS value in the file. + +The PICS tag can be used to unconditionally gate a test step on the PICS value +in the file. The PICS tag can handle standard boolean operations on pics (!, ||, &&, ()). -A PICS tag at the top level of the file can be used to gate the entire test in the test harness. Note that full-test gating is not currently implemented in the local runner or in the CI. +A PICS tag at the top level of the file can be used to gate the entire test in +the test harness. Note that full-test gating is not currently implemented in the +local runner or in the CI. -Some test steps need to be gated on values from earlier in the test. In these cases, PICS cannot be used. Instead, the runIf: tag can be used. This tag requires a boolean value. To convert values to booleans, the TestEqualities function can be use. See [TestEqualities](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/TestEqualities.yaml) for an example of how to use this pseudo-cluster. +Some test steps need to be gated on values from earlier in the test. In these +cases, PICS cannot be used. Instead, the runIf: tag can be used. This tag +requires a boolean value. To convert values to booleans, the TestEqualities +function can be use. See +[TestEqualities](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/TestEqualities.yaml) +for an example of how to use this pseudo-cluster. ## Running YAML tests -YAML scripts are parsed and run using a python-based runner program that parses the file, then translates the tags into chip-tool commands, and sends those commands over a socket to chip-tool (running in interactive mode). + +YAML scripts are parsed and run using a python-based runner program that parses +the file, then translates the tags into chip-tool commands, and sends those +commands over a socket to chip-tool (running in interactive mode). ### Running locally -There are several options for running tests locally. Because the YAML runner uses python, it is necessary to compile and install the chip python package before using any YAML runner script. + +There are several options for running tests locally. Because the YAML runner +uses python, it is necessary to compile and install the chip python package +before using any YAML runner script. ``` ./scripts/build_python.sh -i py @@ -246,23 +304,28 @@ Compile chip-tool: ./scripts/build/build_examples.py --target linux-x64-chip-tool build ``` -NOTE: use the target appropriate to your system +NOTE: use the target appropriate to your system -[chiptool.py](https://github.com/project-chip/connectedhomeip/blob/master/scripts/tests/yaml/chiptool.py) can be used to run tests against a commissioned DUT (commissioned by chip-tool). This will start an interactive instance of chip-tool automatically. +[chiptool.py](https://github.com/project-chip/connectedhomeip/blob/master/scripts/tests/yaml/chiptool.py) +can be used to run tests against a commissioned DUT (commissioned by chip-tool). +This will start an interactive instance of chip-tool automatically. ``` ./scripts/tests/yaml/chiptool.py tests Test_TC_OO_2_1 --server_path ./out/linux-x64-chip-tool/chip-tool ``` + NOTE: substitute the appropriate test name and chip-tool path as appropriate. A list of available tests can be generated using: + ``` ./scripts/tests/yaml/chiptool.py list ``` -Config variables can be passed to chiptool.py after the script by separating with -- +Config variables can be passed to chiptool.py after the script by separating +with -- ``` ./scripts/tests/yaml/chiptool.py tests Test_TC_OO_2_1 --server_path ./out/linux-x64-chip-tool/chip-tool -- nodeId 0x12344321 @@ -271,12 +334,14 @@ Config variables can be passed to chiptool.py after the script by separating wit ### Running in the CI -* YAML tests added to the certification directory get run automatically - * src/app/tests/suites/certification/ - * PICS file: src/app/tests/suites/certification/ci-pics-values -* If you DON’T want to run a test in the CI - * (ex under development), add it to _GetInDevelopmentTests in scripts/tests/chiptest/__init__.py +- YAML tests added to the certification directory get run automatically + - src/app/tests/suites/certification/ + - PICS file: src/app/tests/suites/certification/ci-pics-values +- If you DON’T want to run a test in the CI + - (ex under development), add it to \_GetInDevelopmentTests in + scripts/tests/chiptest/**init**.py ### Running in the TH -TODO: Do we have a permanent link to the most up to date TH documentation? If so, add here. \ No newline at end of file +TODO: Do we have a permanent link to the most up to date TH documentation? If +so, add here.