Docs: Add documentation from SDK testing session at the Singapore mem…

…ber meeting (project-chip#32831)

* as converted

* Moving to individual files, adding plantuml source

* spelling

* remove syntax markers, apparently this is unsupported

* why do we not have a coding dictionary

* Apply suggestions from code review

Co-authored-by: Rob Bultman <[email protected]>

* Restyled by prettier-markdown

---------

Co-authored-by: Rob Bultman <[email protected]>
Co-authored-by: Restyled.io <[email protected]>

.github/.wordlist.txt

@@ -585,6 +585,8 @@ GenericWiFiConfigurationManagerImpl
 GetDeviceId
 GetDeviceInfo
 GetDns
+getter
+getters
 GetInDevelopmentTests
 GetIP
 getManualTests
@@ -907,6 +909,7 @@ Multicast
 multilib
 Multiprotocol
 multithreaded
+mutex
 mutexes
 mv
 MX
@@ -936,6 +939,7 @@ nfds
 NitricOxideConcentrationMeasurement
 NitrogenDioxideConcentrationMeasurement
 nl
+nltest
 NLUnitTest
 NLUnitTests
 nmcli
@@ -1499,6 +1503,7 @@ utils
 UUID
 ux
 validator
+valgrind
 vcom
 VCP
 Vectorcall

docs/testing/img/plant_uml_source.txt

@@ -0,0 +1,55 @@
+@startuml
+class AttributeAccessInterface {
+  mEndpointId : Optional<EndpointId>
+  mClusterId : ClusterId
+  Read(...)
+  Write(...)
+}
+
+class CommandHandlerInterface {
+  InvokeCommand(...)
+}
+
+class ClusterServer {
+  Read(...)
+  Write(...)
+  InvokeCommand(...)
+  RegisterEndpoint(...)
+  mEndpoints: ClusterLogic[]
+}
+
+class MatterContext {
+  LogEvent()
+  MarkAttributeDirty()
+  mPersistentStorageDelegate
+  mOtherFakeableThings
+}
+
+class ClusterLogic {
+  Init(...)
+  GetXAttribute(...)
+  SetXAttribute(...)
+  HandleXCommand(...)
+  mStateVariables
+}
+
+class ClusterDriver {
+  OnClusterStateChange(...)
+  RegisterListener(...)
+  mListener
+}
+
+AttributeAccessInterface  <|-- ClusterServer
+CommandHandlerInterface <|-- ClusterServer
+ClusterServer "1" *-- "Many" ClusterLogic
+ClusterLogic "1" *-- "1" ClusterDriver
+ClusterLogic "1" *-- "1" MatterContext
+
+hide ClusterServer members
+hide AttributeAccessInterface members
+hide CommandHandlerInterface members
+hide ClusterLogic members
+hide ClusterDriver members
+hide MatterContext members
+
+@enduml

docs/testing/index.md

@@ -11,16 +11,33 @@ in the SDK.
 *
 ```
 
-## Unit testing
+## Integration and Certification tests
 
--   [Unit tests](./unit_testing.md)
+Integration tests test the entire software stack using the same mechanisms as a
+standard controller.
 
-## Integration and Certification tests
+![](./img/integration_tests.png)
+
+The certification tests are all integration tests, since they run against the
+product as a black box.
 
 -   [Integration and Certification tests](./integration_tests.md)
 -   [YAML](./yaml.md)
 -   [Python testing framework](./python.md)
 -   [Enabling tests in the CI](./ci_testing.md)
+-   [Integration test utilities](./integration_test_utilities.md)
+
+## Unit testing
+
+Unit tests run on small pieces (“units”) of business logic. They do not use an
+external controller and instead test at the public interface of the class or
+function. For clusters, this requires an API that separates the cluster logic
+from the global ember and message delivery layers.
+
+![](./img/unit_tests.png)
+
+-   [Unit tests](./unit_testing.md)
+-   [Designing clusters for unit testing and portability](./unit_testing_clusters.md)
 
 ## PICS and PIXIT
 

docs/testing/integration_test_utilities.md

@@ -0,0 +1,134 @@
+# Integration Test utilities
+
+There are several test utilities that can be used to simulate or force behavior
+on devices for the purposes of testing.
+
+When using any of these utilities it is important to inject the errors at the
+point where they are running through the MOST code that they can.
+
+If the cluster uses the [ClusterLogic](./unit_testing_clusters.md) pattern, this
+means injecting errors as close as possible to the driver layer, rather than
+catching errors in the server.
+
+## TestEventTriggers
+
+TestEventTriggers are used to test interactions on the DUT that are difficult to
+perform during certification testing (ex. triggering a smoke alarm)
+
+**These should be used sparingly!**
+
+TestEventTriggers are started though a command in the General Diagnostics
+cluster. The command takes a “test key” and a “trigger code” to request that a
+device to perform a specific action. Currently most devices use a default key,
+but it can be overridden by a specific device if required.
+
+**TestEventTriggers need to be turned off outside of certification tests**
+
+To use these triggers:
+
+-   Derive from
+    [TestEventTriggerHandler](https://github.com/project-chip/connectedhomeip/blob/master/src/app/TestEventTriggerDelegate.h)
+-   Implement HandleEventTrigger function
+-   Register with TestEventTriggerDelegate::AddHandler
+
+Please see
+[EnergyEvseTestEventTriggerHandler](https://github.com/project-chip/connectedhomeip/blob/master/src/app/clusters/energy-evse-server/EnergyEvseTestEventTriggerHandler.h)
+for a good example.
+
+## NamedPipes
+
+NamedPipes are used to trigger actions on Linux applications. These can be used
+in the CI, and are normally used to simulate manual actions for CI integration.
+Any required manual action in a test (ex. push a button) should have a
+corresponding NamedPipe action to allow the test to run in the CI.
+
+In python tests, the app-pid required to access the named pipe can be passed in
+as a flag (--app-pid).
+
+NamedPipes are implemented in
+[NamedPipeCommands.h](https://github.com/project-chip/connectedhomeip/blob/master/examples/platform/linux/NamedPipeCommands.h)
+
+To use NamedPipes
+
+-   Derive from NamedPipeCommandDelegate
+-   Implement the OnEventCommandReceived(const char \* json) function
+-   Instantiate and start a NamedPipeCommands object to receive commands and
+    pass in the NamedPipeCommandDelegate and a file path base name
+-   (while running) Write to the file (baseName_pid) to trigger the actions
+
+For a good example, see Air Quality:
+
+-   [Delegate](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/AirQualitySensorAppAttrUpdateDelegate.cpp)
+-   [main](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/main.cpp)
+-   [README](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/README.md)
+
+[RVC Clean Mode](https://github.com/project-chip/connectedhomeip/blob/master/src/python_testing/TC_RVCCLEANM_2_1.py)
+gives an example of how to use named pipes in testing.
+
+## Fault Injection
+
+Fault injection is used to inject conditional code paths at runtime, e.g.
+errors. This is very useful for things like client testing, to check error
+handling for paths that are difficult to hit under normal operating conditions.
+
+The fault injection framework we are currently using is nlFaultInjection.The
+framework uses a macro for injecting the code paths, and the macro is set to a
+no-op if the option is turned off at compile time. The build option to turn on
+fault inject is `chip_with_nlfaultinjection`.
+
+Fault injection has been plumbed out through a manufacturer-specific
+[fault injection cluster](#fault-injection-cluster) that is available in the
+SDK. This allows fault injection to be turned on and off using standard cluster
+mechanisms during testing. For certification, operating these using a secondary,
+non-DUT controller is recommended. For a good example of this, please see
+[TC-IDM-1.3](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/interactiondatamodel.adoc#tc-idm-1-3-batched-commands-invoke-request-action-from-dut-to-th-dut_client).
+
+The nlFaultInjection allows the application to define multiple managers. In the
+SDK, we have managers for System, inet and CHIP. CHIP should be used for
+anything above the system layer (basically all new cluster development). The
+CHIP fault manager is available at
+[lib/support/CHIPFaultInjection.h](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/CHIPFaultInjection.h).
+
+To add new fault injection code paths:
+
+-   Add new IDs (aFaultID) to the enum in
+    [CHIPFaultInjection](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/CHIPFaultInjection.h)
+-   add CHIP_FAULT_INJECT(aFaultID, aStatements) at the point where the fault
+    injection should occur
+
+### Fault Injection example
+
+```
+CHIP_ERROR CASEServer::OnMessageReceived(Messaging::ExchangeContext * ec,
+   const PayloadHeader & payloadHeader,
+                                        System::PacketBufferHandle && payload)
+{
+   MATTER_TRACE_SCOPE("OnMessageReceived", "CASEServer");
+
+   bool busy = GetSession().GetState() != CASESession::State::kInitialized;
+   CHIP_FAULT_INJECT(FaultInjection::kFault_CASEServerBusy, busy = true);
+   if (busy)
+   {
+…
+```
+
+### Fault Injection cluster
+
+The Fault injection cluster is a manufacturer-specific cluster, available in the
+SDK (0xFFF1FC06).
+
+-   [server](https://github.com/project-chip/connectedhomeip/blob/master/src/app/clusters/fault-injection-server/fault-injection-server.cpp)
+-   [xml](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap-templates/zcl/data-model/chip/fault-injection-cluster.xml)
+
+Example apps can be compiled with this cluster for client-side certification and
+integration tests.
+
+To use this cluster to turn on a fault, use the FailAtFault command:
+
+-   Type: FaultType - use FaultType::kChipFault (0x03)
+-   Id: int32u - match the ID you set up for your fault
+-   NumCallsToSkip: int32u - number of times to run normally
+-   NumCallsToFail: int32u - number of times to hit the fault injection
+    condition after NumCallsToSkip
+-   TakeMutex: bool - controls access to the fault injection manager for
+    multi-threaded systems. False is fine.

docs/testing/unit_testing.md

@@ -1,3 +1,122 @@
 # Unit testing
 
-This doc is a placeholder for an guide around unit tests.
+## Why?
+
+-   MUCH faster than integration tests.
+-   Runs as part of build process.
+-   Allows testing specific error conditions that are difficult to trigger under
+    normal operating conditions.
+    -   e.g. out of memory errors etc.
+-   Allows testing different device compositions without defining multiple
+    example applications.
+    -   e.g. feature combinations not in example apps.
+
+## Unit testing in the SDK - nlUnitTest
+
+The following example gives a small demonstration of how to use nlUnitTest to
+write a unit test
+
+```
+#include <lib/support/UnitTestContext.h>
+#include <lib/support/UnitTestRegistration.h>
+#include <nlunit-test.h>
+
+class YourTestContext : public Test::AppContext {
+   ...
+};
+
+
+static void TestName(nlTestSuite * apSuite, void * apContext) {
+    // If you register the test suite with a context, cast
+    // apContext as appropriate
+    YourTestContext * ctx = static_cast<YourTestContext *>(aContext);
+
+    // Do some test things here, then check the results using NL_TEST_ASSERT
+    NL_TEST_ASSERT(apSuite, <boolean condition>)
+}
+
+static const nlTest sTests[] =
+{
+    NL_TEST_DEF("TestName", TestName), // Can have multiple of these
+    NL_TEST_SENTINEL() // If you forget this, you’re going to have a bad time
+};
+
+nlTestSuite sSuite =
+{
+    "TheNameOfYourTestSuite", // Test name
+    &sTests[0], // The list of tests to run
+    TestContext::Initialize, // Runs before all the tests (can be nullptr)
+    TestContext::Finalize // Runs after all the tests (can be nullptr)
+};
+
+int YourTestSuiteName()
+{
+    return chip::ExecuteTestsWithContext<YourTestContext>(&sSuite); // or “without”
+}
+
+CHIP_REGISTER_TEST_SUITE(YourTestSuiteName)
+```
+
+Each test gets an nlTestSuite object (apSuite) that is passed into the test
+assertions, and a void\* context (apContext) that is yours to do with as you
+require.
+
+The apContext should be derived from
+[Test::AppContext](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/AppTestContext.h)
+
+See
+[TestSpan.cpp](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/TestSpan.cpp)
+for a great example of a good unit test.
+
+## nlUnitTest - Compiling and running
+
+-   Add to src/some_directory/tests/BUILD.gn
+    -   chip_test_suite_using_nltest("tests")
+        -   See for example
+            [src/lib/support/tests/BUILD.gn](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/BUILD.gn)
+-   [./gn_build.sh](https://github.com/project-chip/connectedhomeip/blob/master/gn_build.sh)
+    will build and run all tests
+    -   CI runs this, so any unit tests that get added will automatically be
+        added to the CI
+-   Test binaries are compiled into:
+    -   out/debug/<host_compiler>/tests
+    -   e.g. out/debug/linux_x64_clang/tests
+-   Tests are run when ./gn_build.sh runs, but you can run them individually in
+    a debugger from their location.
+
+## Debugging unit tests
+
+-   After running ./gn_build.sh, test binaries are compiled into
+    -   out/debug/<host_compiler>/tests
+    -   e.g. out/debug/linux_x64_clang/tests
+-   Individual binaries, can be run through regular tools:
+    -   gdb
+    -   valgrind
+    -   Your favorite tool that you tell everyone about.
+
+## Utilities
+
+We have a small number of unit testing utilities that should be used in unit
+tests.
+
+Consider adding more utilities for general use if you require them for your
+tests.
+
+### Mock clock
+
+The mock clock is located in
+[src/system/SystemClock.h](https://github.com/project-chip/connectedhomeip/blob/master/src/system/SystemClock.h)
+as `System::Clock::Internal::MockClock`.
+
+To use the mock clock, use the `chip::System::SystemClock()` function as normal.
+In the test, instantiate a MockClock and use the `SetSystemClockForTesting` to
+inject the clock. The Set and Advance functions in the MockClock can then be
+used to set exact times for testing. This allows testing specific edge
+conditions in tests, helps reduce test flakiness due to race conditions, and
+reduces the time required for testing as tests no long require real-time waits.
+
+### TestPersistentStorageDelegate
+
+The TestPersistentStorageDelegate is an in-memory version of storage that easily
+allows removal of keys, presence checks, etc. It is available at
+[src/lib/support/TestPersistentStorageDelegate.h](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/TestPersistentStorageDelegate.h)