diff --git a/.github/.wordlist.txt b/.github/.wordlist.txt index 543786c1473460..b891546f940e53 100644 --- a/.github/.wordlist.txt +++ b/.github/.wordlist.txt @@ -585,6 +585,8 @@ GenericWiFiConfigurationManagerImpl GetDeviceId GetDeviceInfo GetDns +getter +getters GetInDevelopmentTests GetIP getManualTests @@ -907,6 +909,7 @@ Multicast multilib Multiprotocol multithreaded +mutex mutexes mv MX @@ -936,6 +939,7 @@ nfds NitricOxideConcentrationMeasurement NitrogenDioxideConcentrationMeasurement nl +nltest NLUnitTest NLUnitTests nmcli @@ -1499,6 +1503,7 @@ utils UUID ux validator +valgrind vcom VCP Vectorcall diff --git a/docs/testing/img/integration_tests.png b/docs/testing/img/integration_tests.png new file mode 100644 index 00000000000000..dd83695cf20163 Binary files /dev/null and b/docs/testing/img/integration_tests.png differ diff --git a/docs/testing/img/plant_uml_source.txt b/docs/testing/img/plant_uml_source.txt new file mode 100644 index 00000000000000..991d9c5e5aa2fa --- /dev/null +++ b/docs/testing/img/plant_uml_source.txt @@ -0,0 +1,55 @@ +@startuml +class AttributeAccessInterface { + mEndpointId : Optional + mClusterId : ClusterId + Read(...) + Write(...) +} + +class CommandHandlerInterface { + InvokeCommand(...) +} + +class ClusterServer { + Read(...) + Write(...) + InvokeCommand(...) + RegisterEndpoint(...) + mEndpoints: ClusterLogic[] +} + +class MatterContext { + LogEvent() + MarkAttributeDirty() + mPersistentStorageDelegate + mOtherFakeableThings +} + +class ClusterLogic { + Init(...) + GetXAttribute(...) + SetXAttribute(...) + HandleXCommand(...) + mStateVariables +} + +class ClusterDriver { + OnClusterStateChange(...) + RegisterListener(...) + mListener +} + +AttributeAccessInterface <|-- ClusterServer +CommandHandlerInterface <|-- ClusterServer +ClusterServer "1" *-- "Many" ClusterLogic +ClusterLogic "1" *-- "1" ClusterDriver +ClusterLogic "1" *-- "1" MatterContext + +hide ClusterServer members +hide AttributeAccessInterface members +hide CommandHandlerInterface members +hide ClusterLogic members +hide ClusterDriver members +hide MatterContext members + +@enduml diff --git a/docs/testing/img/unit_testable_clusters.png b/docs/testing/img/unit_testable_clusters.png new file mode 100644 index 00000000000000..61f6c51f59b765 Binary files /dev/null and b/docs/testing/img/unit_testable_clusters.png differ diff --git a/docs/testing/img/unit_testable_clusters_all_classes.png b/docs/testing/img/unit_testable_clusters_all_classes.png new file mode 100644 index 00000000000000..6cd8f903b60ae8 Binary files /dev/null and b/docs/testing/img/unit_testable_clusters_all_classes.png differ diff --git a/docs/testing/img/unit_testable_clusters_context.png b/docs/testing/img/unit_testable_clusters_context.png new file mode 100644 index 00000000000000..3a42eca37be258 Binary files /dev/null and b/docs/testing/img/unit_testable_clusters_context.png differ diff --git a/docs/testing/img/unit_testable_clusters_driver.png b/docs/testing/img/unit_testable_clusters_driver.png new file mode 100644 index 00000000000000..bf97434fb5797b Binary files /dev/null and b/docs/testing/img/unit_testable_clusters_driver.png differ diff --git a/docs/testing/img/unit_testable_clusters_logic.png b/docs/testing/img/unit_testable_clusters_logic.png new file mode 100644 index 00000000000000..9ebed283777b4e Binary files /dev/null and b/docs/testing/img/unit_testable_clusters_logic.png differ diff --git a/docs/testing/img/unit_testable_clusters_server.png b/docs/testing/img/unit_testable_clusters_server.png new file mode 100644 index 00000000000000..d8632e73a32338 Binary files /dev/null and b/docs/testing/img/unit_testable_clusters_server.png differ diff --git a/docs/testing/img/unit_tests.png b/docs/testing/img/unit_tests.png new file mode 100644 index 00000000000000..95b29b415c967c Binary files /dev/null and b/docs/testing/img/unit_tests.png differ diff --git a/docs/testing/index.md b/docs/testing/index.md index 0b9bbf4c494a38..d8ebe92da46f40 100644 --- a/docs/testing/index.md +++ b/docs/testing/index.md @@ -11,16 +11,33 @@ in the SDK. * ``` -## Unit testing +## Integration and Certification tests -- [Unit tests](./unit_testing.md) +Integration tests test the entire software stack using the same mechanisms as a +standard controller. -## Integration and Certification tests +![](./img/integration_tests.png) + +The certification tests are all integration tests, since they run against the +product as a black box. - [Integration and Certification tests](./integration_tests.md) - [YAML](./yaml.md) - [Python testing framework](./python.md) - [Enabling tests in the CI](./ci_testing.md) +- [Integration test utilities](./integration_test_utilities.md) + +## Unit testing + +Unit tests run on small pieces (“units”) of business logic. They do not use an +external controller and instead test at the public interface of the class or +function. For clusters, this requires an API that separates the cluster logic +from the global ember and message delivery layers. + +![](./img/unit_tests.png) + +- [Unit tests](./unit_testing.md) +- [Designing clusters for unit testing and portability](./unit_testing_clusters.md) ## PICS and PIXIT diff --git a/docs/testing/integration_test_utilities.md b/docs/testing/integration_test_utilities.md new file mode 100644 index 00000000000000..e50ad511caa28b --- /dev/null +++ b/docs/testing/integration_test_utilities.md @@ -0,0 +1,134 @@ +# Integration Test utilities + +There are several test utilities that can be used to simulate or force behavior +on devices for the purposes of testing. + +When using any of these utilities it is important to inject the errors at the +point where they are running through the MOST code that they can. + +If the cluster uses the [ClusterLogic](./unit_testing_clusters.md) pattern, this +means injecting errors as close as possible to the driver layer, rather than +catching errors in the server. + +## TestEventTriggers + +TestEventTriggers are used to test interactions on the DUT that are difficult to +perform during certification testing (ex. triggering a smoke alarm) + +**These should be used sparingly!** + +TestEventTriggers are started though a command in the General Diagnostics +cluster. The command takes a “test key” and a “trigger code” to request that a +device to perform a specific action. Currently most devices use a default key, +but it can be overridden by a specific device if required. + +**TestEventTriggers need to be turned off outside of certification tests** + +To use these triggers: + +- Derive from + [TestEventTriggerHandler](https://github.com/project-chip/connectedhomeip/blob/master/src/app/TestEventTriggerDelegate.h) +- Implement HandleEventTrigger function +- Register with TestEventTriggerDelegate::AddHandler + +Please see +[EnergyEvseTestEventTriggerHandler](https://github.com/project-chip/connectedhomeip/blob/master/src/app/clusters/energy-evse-server/EnergyEvseTestEventTriggerHandler.h) +for a good example. + +## NamedPipes + +NamedPipes are used to trigger actions on Linux applications. These can be used +in the CI, and are normally used to simulate manual actions for CI integration. +Any required manual action in a test (ex. push a button) should have a +corresponding NamedPipe action to allow the test to run in the CI. + +In python tests, the app-pid required to access the named pipe can be passed in +as a flag (--app-pid). + +NamedPipes are implemented in +[NamedPipeCommands.h](https://github.com/project-chip/connectedhomeip/blob/master/examples/platform/linux/NamedPipeCommands.h) + +To use NamedPipes + +- Derive from NamedPipeCommandDelegate +- Implement the OnEventCommandReceived(const char \* json) function +- Instantiate and start a NamedPipeCommands object to receive commands and + pass in the NamedPipeCommandDelegate and a file path base name +- (while running) Write to the file (baseName_pid) to trigger the actions + +For a good example, see Air Quality: + +- [Delegate](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/AirQualitySensorAppAttrUpdateDelegate.cpp) +- [main](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/main.cpp) +- [README](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/README.md) + +[RVC Clean Mode](https://github.com/project-chip/connectedhomeip/blob/master/src/python_testing/TC_RVCCLEANM_2_1.py) +gives an example of how to use named pipes in testing. + +## Fault Injection + +Fault injection is used to inject conditional code paths at runtime, e.g. +errors. This is very useful for things like client testing, to check error +handling for paths that are difficult to hit under normal operating conditions. + +The fault injection framework we are currently using is nlFaultInjection.The +framework uses a macro for injecting the code paths, and the macro is set to a +no-op if the option is turned off at compile time. The build option to turn on +fault inject is `chip_with_nlfaultinjection`. + +Fault injection has been plumbed out through a manufacturer-specific +[fault injection cluster](#fault-injection-cluster) that is available in the +SDK. This allows fault injection to be turned on and off using standard cluster +mechanisms during testing. For certification, operating these using a secondary, +non-DUT controller is recommended. For a good example of this, please see +[TC-IDM-1.3](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/interactiondatamodel.adoc#tc-idm-1-3-batched-commands-invoke-request-action-from-dut-to-th-dut_client). + +The nlFaultInjection allows the application to define multiple managers. In the +SDK, we have managers for System, inet and CHIP. CHIP should be used for +anything above the system layer (basically all new cluster development). The +CHIP fault manager is available at +[lib/support/CHIPFaultInjection.h](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/CHIPFaultInjection.h). + +To add new fault injection code paths: + +- Add new IDs (aFaultID) to the enum in + [CHIPFaultInjection](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/CHIPFaultInjection.h) +- add CHIP_FAULT_INJECT(aFaultID, aStatements) at the point where the fault + injection should occur + +### Fault Injection example + +``` +CHIP_ERROR CASEServer::OnMessageReceived(Messaging::ExchangeContext * ec, + const PayloadHeader & payloadHeader, + System::PacketBufferHandle && payload) +{ + MATTER_TRACE_SCOPE("OnMessageReceived", "CASEServer"); + + bool busy = GetSession().GetState() != CASESession::State::kInitialized; + CHIP_FAULT_INJECT(FaultInjection::kFault_CASEServerBusy, busy = true); + if (busy) + { +… +``` + +### Fault Injection cluster + +The Fault injection cluster is a manufacturer-specific cluster, available in the +SDK (0xFFF1FC06). + +- [server](https://github.com/project-chip/connectedhomeip/blob/master/src/app/clusters/fault-injection-server/fault-injection-server.cpp) +- [xml](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap-templates/zcl/data-model/chip/fault-injection-cluster.xml) + +Example apps can be compiled with this cluster for client-side certification and +integration tests. + +To use this cluster to turn on a fault, use the FailAtFault command: + +- Type: FaultType - use FaultType::kChipFault (0x03) +- Id: int32u - match the ID you set up for your fault +- NumCallsToSkip: int32u - number of times to run normally +- NumCallsToFail: int32u - number of times to hit the fault injection + condition after NumCallsToSkip +- TakeMutex: bool - controls access to the fault injection manager for + multi-threaded systems. False is fine. diff --git a/docs/testing/unit_testing.md b/docs/testing/unit_testing.md index e62940f15a9c7c..c539420f397e92 100644 --- a/docs/testing/unit_testing.md +++ b/docs/testing/unit_testing.md @@ -1,3 +1,122 @@ # Unit testing -This doc is a placeholder for an guide around unit tests. +## Why? + +- MUCH faster than integration tests. +- Runs as part of build process. +- Allows testing specific error conditions that are difficult to trigger under + normal operating conditions. + - e.g. out of memory errors etc. +- Allows testing different device compositions without defining multiple + example applications. + - e.g. feature combinations not in example apps. + +## Unit testing in the SDK - nlUnitTest + +The following example gives a small demonstration of how to use nlUnitTest to +write a unit test + +``` +#include +#include +#include + +class YourTestContext : public Test::AppContext { + ... +}; + + +static void TestName(nlTestSuite * apSuite, void * apContext) { + // If you register the test suite with a context, cast + // apContext as appropriate + YourTestContext * ctx = static_cast(aContext); + + // Do some test things here, then check the results using NL_TEST_ASSERT + NL_TEST_ASSERT(apSuite, ) +} + +static const nlTest sTests[] = +{ + NL_TEST_DEF("TestName", TestName), // Can have multiple of these + NL_TEST_SENTINEL() // If you forget this, you’re going to have a bad time +}; + +nlTestSuite sSuite = +{ + "TheNameOfYourTestSuite", // Test name + &sTests[0], // The list of tests to run + TestContext::Initialize, // Runs before all the tests (can be nullptr) + TestContext::Finalize // Runs after all the tests (can be nullptr) +}; + +int YourTestSuiteName() +{ + return chip::ExecuteTestsWithContext(&sSuite); // or “without” +} + +CHIP_REGISTER_TEST_SUITE(YourTestSuiteName) +``` + +Each test gets an nlTestSuite object (apSuite) that is passed into the test +assertions, and a void\* context (apContext) that is yours to do with as you +require. + +The apContext should be derived from +[Test::AppContext](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/AppTestContext.h) + +See +[TestSpan.cpp](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/TestSpan.cpp) +for a great example of a good unit test. + +## nlUnitTest - Compiling and running + +- Add to src/some_directory/tests/BUILD.gn + - chip_test_suite_using_nltest("tests") + - See for example + [src/lib/support/tests/BUILD.gn](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/BUILD.gn) +- [./gn_build.sh](https://github.com/project-chip/connectedhomeip/blob/master/gn_build.sh) + will build and run all tests + - CI runs this, so any unit tests that get added will automatically be + added to the CI +- Test binaries are compiled into: + - out/debug//tests + - e.g. out/debug/linux_x64_clang/tests +- Tests are run when ./gn_build.sh runs, but you can run them individually in + a debugger from their location. + +## Debugging unit tests + +- After running ./gn_build.sh, test binaries are compiled into + - out/debug//tests + - e.g. out/debug/linux_x64_clang/tests +- Individual binaries, can be run through regular tools: + - gdb + - valgrind + - Your favorite tool that you tell everyone about. + +## Utilities + +We have a small number of unit testing utilities that should be used in unit +tests. + +Consider adding more utilities for general use if you require them for your +tests. + +### Mock clock + +The mock clock is located in +[src/system/SystemClock.h](https://github.com/project-chip/connectedhomeip/blob/master/src/system/SystemClock.h) +as `System::Clock::Internal::MockClock`. + +To use the mock clock, use the `chip::System::SystemClock()` function as normal. +In the test, instantiate a MockClock and use the `SetSystemClockForTesting` to +inject the clock. The Set and Advance functions in the MockClock can then be +used to set exact times for testing. This allows testing specific edge +conditions in tests, helps reduce test flakiness due to race conditions, and +reduces the time required for testing as tests no long require real-time waits. + +### TestPersistentStorageDelegate + +The TestPersistentStorageDelegate is an in-memory version of storage that easily +allows removal of keys, presence checks, etc. It is available at +[src/lib/support/TestPersistentStorageDelegate.h](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/TestPersistentStorageDelegate.h) diff --git a/docs/testing/unit_testing_clusters.md b/docs/testing/unit_testing_clusters.md new file mode 100644 index 00000000000000..b2806c8084b6ee --- /dev/null +++ b/docs/testing/unit_testing_clusters.md @@ -0,0 +1,172 @@ +# Designing Clusters for Testing and Portability + +## Unit Testable, Modular Cluster Design + +When designing new clusters, consider the following approach: + +- Separate the cluster logic from the on-the-wire data model + - Server vs. ClusterLogic + - Makes the cluster logic unit-testable without generating TLV. +- Separate the basic cluster logic from code that is platform- or + device-specific. + - ClusterLogic uses a ClusterDriver + - Makes the cluster logic portable between platforms / manufacturers + - Removes necessity of overriding global singleton functions like + PostAttributeChangeCallback. + +General approach: + +![](./img/unit_testable_clusters.png) + +Class proposal: + +![](./img/unit_testable_clusters_all_classes.png) + +### ClusterServer + +![](./img/unit_testable_clusters_server.png) + +The ClusterServerClass is a **Very** light wrapper over ClusterLogic. It +translates Interaction Model wire format handling into API calls for cluster +logic methods. + +This class implements both the AttributeAccessInterface and the CommandHandler +interfaces so ClusterLogic properly handles data dependencies between commands +and attributes. + +An example code snippet showing the translation of the TLV into API calls to the +ClusterLogic class: + +``` +CHIP_ERROR DiscoBallServer::Read(const ConcreteReadAttributePath & aPath, + AttributeValueEncoder & aEncoder) +{ + DiscoBallClusterLogic * cluster = FindEndpoint(aPath.mEndpointId); + VerifyOrReturnError(cluster != nullptr, CHIP_IM_GLOBAL_STATUS(UnsupportedEndpoint)); + + switch (aPath.mAttributeId) + { + case Clusters::DiscoBall::Attributes::Run::Id: + return aEncoder.Encode(cluster->GetRunAttribute()); + … + } +} +``` + +### ClusterLogic + +![](./img/unit_testable_clusters_logic.png) + +The ClusterLogic class is for all the code that is SHARED between platforms. It +does NOT include any TLV parsing or direct calls to Ember/IM/LogEvent etc. + +The class should include attribute getters/setters and handlers for all +commands. + +The class receives “plain data” Matter requests from ClusterServer class, +performs required common actions, and calls driver class to perform platform- or +hardware-specific actions. It also receives driver updates (e.g. +application-driven value changes) from the ClusterDriver class and updates state +as appropriate. + +The class should handle spec-requirements for: + + - Range checking (CONSTRAINT_ERROR) + - Attribute and metadata storage (persistent or in-memory) + - Data dependencies between commands and attributes + - Event generation / dirty attributes (callback to server) + - Calling driver when platform or hardware interactions are required + +API recommendation: + + - Maintain all cluster state in a separate data-only class. + - Provider getters/setters for application logic to use. + - Implement handlers for all commands, conditional on features. + - Let the caller provide (inject) dependencies. Avoid explicit memory + management. + +### ClusterDriver + +Implements hardware or platform-specific actions required on cluster +interactions or when application wants to report state changes. + +![](./img/unit_testable_clusters_driver.png) + +### MatterContext + +The ClusterLogic class must not directly use global resource because they cannot +be isolated for testing. Instead, the MatterContext holds pointers to Matter +stack objects, which can be be injected / faked for testing. This includes - +Wrapper over IM Engine interface functions for marking attributes dirty, and +logging events. - Storage - Anything you would normally access with +Server::GetInstance() + +![](./img/unit_testable_clusters_context.png) + +### ClusterDriver + +The ClusterDriver is called by the ClusterLogic class and is used to translate +attribute changes and commands into application actions. It also reports +external changes back to the ClusterLogic class. + +The API design for this class will vary by the cluster, but it is generally +recommended to use a generic API where possible, so the API ports easily to +other platforms, for example an attribute changed callback with the changes +listed. It is important to be careful about the design and revisit this early if +issues arise. + +## Unit testing with the modular cluster design + +### ClusterLogic class + +The unit test instantiates the ClusterLogic, provides MatterContext and +ClusterDriver instance with fakes/mocks for testing. + +Unit test against the API, and check the fakes/mocks to ensure they are being +called as appropriate. + +As with all unit tests, prefer testing for behavior rather than implementation +details. + +Important tests to consider: + +- Initialization and initial attribute correctness. +- Errors for out-of-range on all attribute setters and command handlers. +- All spec-defined error conditions, especially ones that are difficult to + trigger. +- Data dependencies between commands and attributes. +- Incoming actions in different states (stopped, running, etc). +- Calls out to storage for persistent attributes. +- Calls out to driver for changes as appropriate. +- Driver error reporting. +- Event generation and dirty attribute marking, including attributes that are + changed from the driver side. +- Others - very dependent on the cluster. + +# Unit testing ClusterServer + +- Best to have the lightest wrapping possible + - If the wrapper is light, the code can be covered by integration or unit + tests, or a combination. + - Correctness can mostly be validated by inspection if it’s trivial. +- Important tests + - Errors when ClusterLogic instances aren’t properly registered. + - Flow through to ClusterLogic for all reads/writes/commands. +- Can unit test this class by generating the TLV / path for input, parsing the + TLV output. + +# Unit testing existing clusters + +- Important for clusters where there are multiple configurations that cannot + easily be represented with example apps +- Option 1 + - Refactor the cluster logic to be unit-testable. +- Option 2 + - Test at AttributeAccessInterface boundary. + - Instantiate or access the cluster server instance in the test. + - Read / Write TLV and use TLV encode/decode functions to verify + correctness. + - See TestPowerSourceCluster for an example of how to do this +- Additional test coverage on clusters, especially for hard to trigger + conditions, is important. However, **don’t let perfection be the enemy of + progress** .