From 754a938dd639de1939a2929eebbbc64975269687 Mon Sep 17 00:00:00 2001 From: Andrei Litvin Date: Fri, 3 Nov 2023 14:21:34 -0400 Subject: [PATCH] Add some documentation about matter idl tools that we have for validating data (#30198) * Add some documentation about matter idl tooling * Restyle * Add zigbee only note, because that caught me a bit * Restyle * Add quotes on cmdline examples * clearer docs a bit * Add an external data model for the data model md (make sphinx happy) * make build succeed * Some better orphan markers to let sphinx process these files without a TOC entry. I was unsure how to make a proper toc here * fix spelling --------- Co-authored-by: Andrei Litvin --- data_model/README.md | 4 ++ docs/conf.py | 5 +- docs/guides/index.md | 1 + docs/guides/matter_idl_tooling.md | 86 ++++++++++++++++++++++++ scripts/tools/memory/README-GitHub-CI.md | 4 ++ scripts/tools/memory/memdf/README.md | 4 ++ scripts/tools/memory/platform/README.md | 4 ++ 7 files changed, 105 insertions(+), 3 deletions(-) create mode 100644 docs/guides/matter_idl_tooling.md diff --git a/data_model/README.md b/data_model/README.md index ada763cd7f2804..36f38cae0ff1f8 100644 --- a/data_model/README.md +++ b/data_model/README.md @@ -1,3 +1,7 @@ +--- +orphan: true +--- + ## Contents This folder contains a machine-readable representation of matter clusters. diff --git a/docs/conf.py b/docs/conf.py index 1f2f0d96cf74e8..52402d86518b80 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -32,9 +32,6 @@ "examples/thermostat/nxp/linux-se05x/README.md", "examples/common/m5stack-tft/repo", "docs/guides/README.md", - "scripts/tools/memory/memdf/README.md", - "scripts/tools/memory/platform/README.md", - "scripts/tools/memory/README-GitHub-CI.md", ] @@ -67,6 +64,7 @@ external_content_contents = [ (MATTER_BASE / "docs", "[!_R]*"), + (MATTER_BASE, "data_model/**/*.md"), (MATTER_BASE, "README.md"), (MATTER_BASE, "examples/**/*.md"), (MATTER_BASE, "examples/**/*.png"), @@ -82,5 +80,6 @@ "README", # cannot detect README.md "scripts/", "examples/android/", + "data_model/", ] external_content_link_extensions = [".md", ".png", ".jpg", ".svg"] diff --git a/docs/guides/index.md b/docs/guides/index.md index a818ee2b777912..2119b46059a8c0 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -55,6 +55,7 @@ ti/ti_platform_overview - [Access Control](./access-control-guide.md) - [IP Commissioning](./ip_commissioning.md) +- [Matter IDL tooling and validation](./matter_idl_tooling.md) ## Setup Guides diff --git a/docs/guides/matter_idl_tooling.md b/docs/guides/matter_idl_tooling.md new file mode 100644 index 00000000000000..09953e1d5ef0ca --- /dev/null +++ b/docs/guides/matter_idl_tooling.md @@ -0,0 +1,86 @@ +# The `.matter` IDL file format + +The matter IDL file format is designed to be a human-readable representation of +data structures, cluster definitions and endpoint composition. + +Since it is designed to be easy for both machine and humans to read, it is the +basis of some tools to make validating zap-based cluster definitions easier. + +More details on the format in +[matter_idl/README.md](../../scripts/py_matter_idl/matter_idl/README.md). + +## Parsing CSA XML Data definitions + +The SDK contains a copy of CSA XML data definitions in `data_model/clusters`. + +The files there are updated by running a scraper against the official matter +specification and are a good source of truth for actual definitions. Update +information available in [data_model/README.md](../../data_model/README.md). + +NOTE: scraper is a work in progress, XML data may be incomplete or have errors +still. + +The script `./scripts/py_matter_idl/matter_idl/data_model_xml_parser.py` has the +ability to parse one or more CSA data model XML files and output their content +in `.matter` format. For example: + +```sh +./scripts/py_matter_idl/matter_idl/data_model_xml_parser.py data_model/clusters/BooleanState.xml +``` + +The tool supports several options that are useful for development: + +| Argument(s) | Description | +| ----------------------- | ----------------------------------------------------------------------------------- | +| `-o/--output PATH` | Output the matter file into a path instead of STDOUT. | +| `--compare PATH` | Also read another `.matter` file for compare. MUST be used with `--compare-output`. | +| `--compare-output PATH` | Output the subset of `--compare` clusters that matches XML into PATH. | + +Using `--compare` AND `--compare-output` produce output that is easier to +compare as opposed to using existing zap-generated matter files because it +strips out comments and it also alpha-sorts elements so that diffs are +human-readable. + +### Comparing a `.matter` file against the spec + +Combining arguments to the tool allows getting a diff between SDK and +specification: + +- `data_model/clusters/*.xml` are assumed to be the official specification + definitions +- `src/controller/data_model/controller-clusters.matter` contains _all_ + clusters defined in the SDK + +As such one can run compares such as: + +```sh +./scripts/py_matter_idl/matter_idl/data_model_xml_parser.py \ + -o out/spec.matter \ + --compare-output out/sdk.matter \ + --compare src/controller/data_model/controller-clusters.matter \ + data_model/clusters/DoorLock.xml \ + && diff out/{spec,sdk}.matter +``` + +NOTE: due to specification data scraper still being in development, the diff +should be human-validated (e.g. for tool errors or Zigbee-only markers in the +spec). + +## Linting `.matter` files for devices + +For device validation, `./scripts/idl_lint.py` provides the ability to validate +a matter file for some basic conformance logic. These rules are expressed in +`scripts/rules.matterlint`. + +The rules generally are: + +- pre-loaded from silabs XML files (validates that mandatory attributes are + present) +- Hard-coded rules (e.g. mandatory clusters and attributes on specific + endpoints) + +Usage: + +```sh +./scripts/idl_lint.py ./examples/window-app/common/window-app.matter +``` diff --git a/scripts/tools/memory/README-GitHub-CI.md b/scripts/tools/memory/README-GitHub-CI.md index e34af1f9f23566..d9ff7172ebe5a6 100644 --- a/scripts/tools/memory/README-GitHub-CI.md +++ b/scripts/tools/memory/README-GitHub-CI.md @@ -1,3 +1,7 @@ +--- +orphan: true +--- + # Scripts for GitHub CI A set of `gh_*.py` scripts work together to produce size comparisons for PRs. diff --git a/scripts/tools/memory/memdf/README.md b/scripts/tools/memory/memdf/README.md index 83628c7676e25a..2ea3c2f7a4ffe3 100644 --- a/scripts/tools/memory/memdf/README.md +++ b/scripts/tools/memory/memdf/README.md @@ -1,3 +1,7 @@ +--- +orphan: true +--- + This package contains routines to to collect, aggregate, and report memory usage, using Pandas `DataFrame` as the primary representation. diff --git a/scripts/tools/memory/platform/README.md b/scripts/tools/memory/platform/README.md index f90ba441f98b32..b3332d0ff4a34f 100644 --- a/scripts/tools/memory/platform/README.md +++ b/scripts/tools/memory/platform/README.md @@ -1,3 +1,7 @@ +--- +orphan: true +--- + A config file is a Python (nested) dictionary used by `memdf.util.config.Config`.