Automated python test selection and integration into TH

[cecille]

During 1.3, we tested some test mechanisms that allow tests to be run on every device, and mark themselves as skipped if they do not apply to the device endpoint under test. Due to some differences between the CI testing and the test harness, we found that although these tests are easy and fast to run on a local desktop or on the CI, they are quite expensive to run through the test harness. We still believe the test self-selection approach is good, but we need more up-front test harness integration to make this practical at certification.

We would like to start by creating and refining a standalone script with the following properties:

• runs all self-selecting tests against all endpoints on the device automatically
• outputs logs by endpoint, separated into pass / fail / skip, test output should be in the format and naming convention required by the ATLs (will require consultation with the ATL stakeholders)
• test selection options - either run a test list (possibly with tests from multiple files), all self-selecting tests, or all previously failed tests (allows easy re-run)
• automated commissioning handling, such that the same script could be used with devices that are commissioned or uncommissioned
• automated detection of commissioning method (on-network, thread, wifi) and obvious warnings about missing parameters for commissioning
• dry-run capabilities to understand what tests WOULD be run

Then the project can move on to test harness integration to allow the test harness to expose this in a friendly way for certification.

We can then work on moving tests to be PICS-free and auto-selecting in order to improve testing time and reliability. This would involve removing PICS guards from tests and changing them to use the self-selection decorators. We have some work on this being done already by one of the CSA contractors, so we’ll probably need to meet in the middle there.