Add automated JSON schema validation

[ajlende]

Changes proposed in this Pull Request:

Adds an npm script and GitHub action for validating JSON for themes.

1. Gets the schema from the "Requires at least" minimum WP version.
2. Warns when the $schema version doesn't match the minimum WP version.
3. Validates the theme.json and style variations against the minimum WP version schema.

Validation script

I may have had a bit too much fun with building the tables and adding colors, so the code exploded a bit. 😅

$ node theme-utils.mjs help validate-theme                    

validate-theme [--format=FORMAT] [--color=WHEN] [--table-width=COLUMNS] <array of theme slugs>

Validates a theme against the WordPress theme requirements.

--format=FORMAT

        Output format. Possible values: *table*, json, dir.

--color=WHEN

        Colorize the output for table or dir formats. The automatic mode only enables colors if 
        an interactive terminal is detected. Possible values: *auto*, always, never.

--table-width=COLUMNS

        Explicitly set the width of the table format instead of determining it automatically.
        Will default to 120 if omitted and width cannot be determined automatically.

But I think the new result is really nice. ✨

And it works with pagers.

$ # It can take a while to process longer lists of themes
$ node theme-utils.mjs validate-theme --color=always --table-width=$(( $(tput cols) )) atheme,adventurer,grammer,skatepark | less -R

Or without if you only have a theme or two to validate.

$ node theme-utils.mjs validate-theme grammer                                                                  

Progress: [ '■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■', 100 ] 1/1

╔═════════════════════════════════════════════════════════════╤══════════════════════════╗
║ ERROR                                                       │ Message : missing 'Requi ║
║ Theme : Grammer                                             │ res at least' header met ║
║ File  : style.css                                           │ adata                    ║
╟─────────────────────────────────────────────────────────────┼──────────────────────────╢
║ WARNING                                                     │ Actual   : undefined     ║
║ Theme : Grammer                                             │ Expected : 5.9 or greate ║
║ File  : style.css                                           │ r                        ║
║                                                             │ Message  : the 'Requires ║
║                                                             │  at least' version does  ║
║                                                             │ not support theme.json   ║
╟─────────────────────────────────────────────────────────────┼──────────────────────────╢
║ WARNING                                                     │ Instance path : /setting ║
║ Theme  : Grammer                                            │ s/spacing/spacingScale   ║
║ File   : theme.json                                         │ Schema path   : #/defini ║
║ Schema : https://schemas.wp.org/trunk/theme.json            │ tions/settingsSpacingPro ║
║                                                             │ perties/properties/spaci ║
║                                                             │ ng/properties/spacingSca ║
║                                                             │ le/additionalProperties  ║
║                                                             │ Keyword       : addition ║
║                                                             │ alProperties             ║
║                                                             │ Params        : { additi ║
║                                                             │ onalProperty: 'theme' }  ║
║                                                             │ Message       : must NOT ║
║                                                             │ have additional properti ║
║                                                             │ es                       ║
╟─────────────────────────────────────────────────────────────┼──────────────────────────╢
║ WARNING                                                     │ Instance path :          ║
║ Theme  : Grammer                                            │ Schema path   : #/requir ║
║ File   : assets/fonts/font-collection.json                  │ ed                       ║
║ Schema : https://schemas.wp.org/wp/6.5/font-collection.json │ Keyword       : required ║
║                                                             │ Params        : { missin ║
║                                                             │ gProperty: 'slug' }      ║
║                                                             │ Message       : must hav ║
║                                                             │ e required property 'slu ║
║                                                             │ g'                       ║
║                                                             ├──────────────────────────╢
║                                                             │ Instance path :          ║
║                                                             │ Schema path   : #/requir ║
║                                                             │ ed                       ║
║                                                             │ Keyword       : required ║
║                                                             │ Params        : { missin ║
║                                                             │ gProperty: 'name' }      ║
║                                                             │ Message       : must hav ║
║                                                             │ e required property 'nam ║
║                                                             │ e'                       ║
╚═════════════════════════════════════════════════════════════╧══════════════════════════╝

Validation passed with warnings.

And the added --format=json option is also super helpful when combined with jq to drill down into the data.

For example, this is the breakdown of all the current themes and a count of the types of problems that they have:

$ node theme-utils.mjs validate-theme --format=json $(find . -name 'theme.json' | awk -F/ '{print $2}' | uniq | sort | paste -s -d, -) | jq '.[].data[].message' | sort | uniq -c | sort -bgr
5815 "must NOT have additional properties"
 696 "must be object"
 551 "must be string"
 328 "the $schema version does not match style.css 'Requires at least' version"
 153 "must match exactly one schema in oneOf"
  73 "must be equal to constant"
  71 "the 'Requires at least' version does not support theme.json"
  71 "must be equal to one of the allowed values"
  47 "Missing $schema URI: undefined"
   6 "property name must be valid"
   5 "must be number"
   2 "must be >= 1"
   2 "missing 'Requires at least' header metadata"
   1 "must have required property 'version'"
   1 "must have required property 'slug'"
   1 "must have required property 'name'"

GitHub Action

The GitHub action only runs on the modified themes.

It can be tested by committing some modifications to a theme and running like this:

$ 
$ act -s GITHUB_TOKEN="$(gh auth token)" pull_request -j validate-theme
INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock' 
WARN  ⚠ You are using Apple M-series chip and you have not specified container architecture, you might encounter issues while running act. If so, try running it with '--container-architecture linux/amd64'. ⚠  
[Validate Themes/validate-theme] 🚀  Start image=catthehacker/ubuntu:act-latest
[Validate Themes/validate-theme]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform= username= forcePull=true
[Validate Themes/validate-theme]   🐳  docker create image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[Validate Themes/validate-theme]   🐳  docker run image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[Validate Themes/validate-theme]   ☁  git clone 'https://github.com/actions/setup-node' # ref=v4
[Validate Themes/validate-theme] Non-terminating error while running 'git clone': some refs were not updated
[Validate Themes/validate-theme] ⭐ Run Main Checkout
[Validate Themes/validate-theme]   🐳  docker cp src=/Users/ajlende/Documents/a8c-themes/. dst=/Users/ajlende/Documents/a8c-themes
[Validate Themes/validate-theme]   ✅  Success - Main Checkout
[Validate Themes/validate-theme] ⭐ Run Main Setup Node
[Validate Themes/validate-theme]   🐳  docker cp src=/Users/ajlende/.cache/act/actions-setup-node@v4/ dst=/var/run/act/actions/actions-setup-node@v4/
[Validate Themes/validate-theme]   🐳  docker exec cmd=[node /var/run/act/actions/actions-setup-node@v4/dist/setup/index.js] user= workdir=
| Resolved .nvmrc as 20
[Validate Themes/validate-theme]   💬  ::debug::isExplicit: 
[Validate Themes/validate-theme]   💬  ::debug::explicit? false
[Validate Themes/validate-theme]   💬  ::debug::isExplicit: 18.20.4
[Validate Themes/validate-theme]   💬  ::debug::explicit? true
[Validate Themes/validate-theme]   💬  ::debug::isExplicit: 20.10.0
[Validate Themes/validate-theme]   💬  ::debug::explicit? true
[Validate Themes/validate-theme]   💬  ::debug::isExplicit: 20.15.0
[Validate Themes/validate-theme]   💬  ::debug::explicit? true
[Validate Themes/validate-theme]   💬  ::debug::evaluating 3 versions
[Validate Themes/validate-theme]   💬  ::debug::matched: 20.15.0
[Validate Themes/validate-theme]   💬  ::debug::checking cache: /opt/hostedtoolcache/node/20.15.0/arm64
[Validate Themes/validate-theme]   💬  ::debug::Found tool in cache node 20.15.0 arm64
| Found in cache @ /opt/hostedtoolcache/node/20.15.0/arm64
[Validate Themes/validate-theme]   ❓  ::group::Environment details
| node: v20.15.0
| npm: 10.7.0
| yarn: 
[Validate Themes/validate-theme]   ❓  ::endgroup::
| [command]/opt/hostedtoolcache/node/20.15.0/arm64/bin/npm config get cache
| /root/.npm
[Validate Themes/validate-theme]   💬  ::debug::npm's cache folder "/root/.npm" configured for the root directory
[Validate Themes/validate-theme]   💬  ::debug::followSymbolicLinks 'true'
[Validate Themes/validate-theme]   💬  ::debug::followSymbolicLinks 'true'
[Validate Themes/validate-theme]   💬  ::debug::implicitDescendants 'true'
[Validate Themes/validate-theme]   💬  ::debug::matchDirectories 'true'
[Validate Themes/validate-theme]   💬  ::debug::omitBrokenSymbolicLinks 'true'
[Validate Themes/validate-theme]   💬  ::debug::Search path '/Users/ajlende/Documents/a8c-themes/package-lock.json'
[Validate Themes/validate-theme]   💬  ::debug::/Users/ajlende/Documents/a8c-themes/package-lock.json
[Validate Themes/validate-theme]   💬  ::debug::Found 1 files to hash.
[Validate Themes/validate-theme]   💬  ::debug::primary key is node-cache-Linux-npm-9df091025ae38d8512d20d182f34873fc5425eec86d0742d8c8bdbe8dd0f3e69
[Validate Themes/validate-theme]   💬  ::debug::Resolved Keys:
[Validate Themes/validate-theme]   💬  ::debug::["node-cache-Linux-npm-9df091025ae38d8512d20d182f34873fc5425eec86d0742d8c8bdbe8dd0f3e69"]
[Validate Themes/validate-theme]   💬  ::debug::Checking zstd --quiet --version
[Validate Themes/validate-theme]   💬  ::debug::1.4.8
[Validate Themes/validate-theme]   💬  ::debug::zstd version: 1.4.8
[Validate Themes/validate-theme]   💬  ::debug::Resource Url: http://192.168.68.55:64143/_apis/artifactcache/cache?keys=node-cache-Linux-npm-9df091025ae38d8512d20d182f34873fc5425eec86d0742d8c8bdbe8dd0f3e69&version=b3f0cb83629d634645a5146420c017462ebb5229bd60271a7a86e489a6066469
[Validate Themes/validate-theme]   💬  ::debug::Failed to delete archive: Error: ENOENT: no such file or directory, unlink ''
| npm cache is not found
[Validate Themes/validate-theme]   ❓ add-matcher /run/act/actions/actions-setup-node@v4/.github/tsc.json
[Validate Themes/validate-theme]   ❓ add-matcher /run/act/actions/actions-setup-node@v4/.github/eslint-stylish.json
[Validate Themes/validate-theme]   ❓ add-matcher /run/act/actions/actions-setup-node@v4/.github/eslint-compact.json
[Validate Themes/validate-theme]   ✅  Success - Main Setup Node
[Validate Themes/validate-theme]   ⚙  ::set-output:: node-version=v20.15.0
[Validate Themes/validate-theme]   ⚙  ::set-output:: cache-hit=false
[Validate Themes/validate-theme]   ⚙  ::add-path:: /opt/hostedtoolcache/node/20.15.0/arm64/bin
[Validate Themes/validate-theme] ⭐ Run Main Install dependencies
[Validate Themes/validate-theme]   🐳  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2] user= workdir=
| npm warn deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
| npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
| npm warn deprecated [email protected]: This package is no longer supported.
| npm warn deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
| npm warn deprecated [email protected]: this library is no longer supported
| npm warn deprecated [email protected]: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
| npm warn deprecated [email protected]: Glob versions prior to v9 are no longer supported
| npm warn deprecated [email protected]: This package is no longer supported.
| npm warn deprecated [email protected]: Use your platform's native DOMException instead
| npm warn deprecated [email protected]: This package is no longer supported.
| npm warn deprecated [email protected]: Use your platform's native atob() and btoa() methods instead
| npm warn deprecated @npmcli/[email protected]: This functionality has been moved to @npmcli/fs
| npm warn deprecated @humanwhocodes/[email protected]: Use @eslint/object-schema instead
| npm warn deprecated @humanwhocodes/[email protected]: Use @eslint/config-array instead
| npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated [email protected]: This package is no longer supported.
| npm warn deprecated [email protected]: This package is no longer supported.
| npm warn deprecated [email protected]: This package is no longer supported.
| npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated [email protected]: Glob versions prior to v9 are no longer supported
| npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
| npm warn deprecated [email protected]: Node Sass is no longer supported. Please use `sass` or `sass-embedded` instead.
| npm warn deprecated [email protected]: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
| 
| > [email protected] prepare
| > husky
| 
| HUSKY=0 skip install
| added 1909 packages, and audited 1945 packages in 1m
| 
| 288 packages are looking for funding
|   run `npm fund` for details
| 
| 9 vulnerabilities (4 moderate, 5 high)
| 
| To address all issues (including breaking changes), run:
|   npm audit fix --force
| 
| Run `npm audit` for details.
[Validate Themes/validate-theme]   ✅  Success - Main Install dependencies
[Validate Themes/validate-theme] ⭐ Run Main Validate Themes
[Validate Themes/validate-theme]   🐳  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/3] user= workdir=
| 
| Progress: [ '■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■', 100 ] 1/1
| 
| ╔═════════════════════════════════════════════════════════════╤════════════════════════════════════════════════════════╗
| ║ warning                                                     │ actual   : https://schemas.wp.org/trunk/theme.json     ║
| ║ file   : grammer/theme.json                                 │ expected : https://schemas.wp.org/wp/6.5/theme.json    ║
| ║ schema : https://schemas.wp.org/wp/6.5/theme.json           │ message  : mismatched $schema with "Requires at least" ║
| ║                                                             │ metadata in style.css                                  ║
| ╟─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────╢
| ║ error                                                       │ instance path : /version                               ║
| ║ file   : grammer/theme.json                                 │ schema path   : #/properties/version/enum              ║
| ║ schema : https://schemas.wp.org/wp/6.5/theme.json           │ keyword       : enum                                   ║
| ║                                                             │ message       : must be equal to one of the allowed va ║
| ║                                                             │ lues                                                   ║
| ║                                                             ├────────────────────────────────────────────────────────╢
| ║                                                             │ instance path : /settings/color                        ║
| ║                                                             │ schema path   : #/definitions/settingsPropertiesColor/ ║
| ║                                                             │ properties/color/additionalProperties                  ║
| ║                                                             │ keyword       : additionalProperties                   ║
| ║                                                             │ message       : must NOT have additional properties    ║
| ║                                                             ├────────────────────────────────────────────────────────╢
| ║                                                             │ instance path : /settings/spacing/spacingScale         ║
| ║                                                             │ schema path   : #/definitions/settingsPropertiesSpacin ║
| ║                                                             │ g/properties/spacing/properties/spacingScale/additiona ║
| ║                                                             │ lProperties                                            ║
| ║                                                             │ keyword       : additionalProperties                   ║
| ║                                                             │ message       : must NOT have additional properties    ║
| ║                                                             ├────────────────────────────────────────────────────────╢
| ║                                                             │ instance path : /settings/typography                   ║
| ║                                                             │ schema path   : #/properties/typography/additionalProp ║
| ║                                                             │ erties                                                 ║
| ║                                                             │ keyword       : additionalProperties                   ║
| ║                                                             │ message       : must NOT have additional properties    ║
| ╟─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────╢
| ║ warning                                                     │ actual   : https://schemas.wp.org/trunk/theme.json     ║
| ║ file   : grammer/styles/lightmode.json                      │ expected : https://schemas.wp.org/wp/6.5/theme.json    ║
| ║ schema : https://schemas.wp.org/wp/6.5/theme.json           │ message  : mismatched $schema with "Requires at least" ║
| ║                                                             │ metadata in style.css                                  ║
| ╟─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────╢
| ║ error                                                       │ instance path : /version                               ║
| ║ file   : grammer/styles/lightmode.json                      │ schema path   : #/properties/version/enum              ║
| ║ schema : https://schemas.wp.org/wp/6.5/theme.json           │ keyword       : enum                                   ║
| ║                                                             │ message       : must be equal to one of the allowed va ║
| ║                                                             │ lues                                                   ║
| ╟─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────╢
| ║ error                                                       │ instance path :                                        ║
| ║ file   : grammer/assets/fonts/font-collection.json          │ schema path   : #/required                             ║
| ║ schema : https://schemas.wp.org/wp/6.5/font-collection.json │ keyword       : required                               ║
| ║                                                             │ message       : must have required property 'slug'     ║
| ║                                                             ├────────────────────────────────────────────────────────╢
| ║                                                             │ instance path :                                        ║
| ║                                                             │ schema path   : #/required                             ║
| ║                                                             │ keyword       : required                               ║
| ║                                                             │ message       : must have required property 'name'     ║
| ╚═════════════════════════════════════════════════════════════╧════════════════════════════════════════════════════════╝
| 
| Validation failed.
[Validate Themes/validate-theme]   ❌  Failure - Main Validate Themes
[Validate Themes/validate-theme] exitcode '1': failure
[Validate Themes/validate-theme] 🏁  Job failed
Error: Job 'validate-theme' failed

Related issue(s):

• Validate theme.json files against the JSON Schema WordPress/community-themes#221
• Add automated validation for JSON schemas WordPress/community-themes#230

[madhusudhand]

Thanks @ajlende for the changes. It is amazing 🥇
I dropped few comments inline.

[madhusudhand]

Excellent work! 🙇

The new table format looks neat and is very helpful to quickly fix the changes.
Changes looks good to me. 🚀

[madhusudhand]

I've detected changes to the following themes in this PR: Eventual, Pique.

@vcanales Bot posted preview links even though there are no relevant changes for these themes. (or may be one of the commit had those changes).
May be the script should add/remove preview links based on the latest commit?

Edit:

Hmm, I did a quick test of the script, and it seems to be removing links based on the latest push.
But it is failing to update or remove the comment when all theme changes are removed. (when the PR has zero theme changes)

Created an issue here: WordPress/action-wp-playground-pr-preview#19