The checkmetrics tool is used to check the metrics results files in
JSON format. Results files are checked against configs stored in a TOML
file that contains baseline expectations for the results.
checkmetrics checks for a matching results file for each entry in the
TOML file with an appropriate json file extension. Failure to find a matching
file is classified as a failure for that individual TOML entry.
checkmetrics continues to process all entries in the TOML file and prints its
final results in a summary table to stdout.
checkmetrics exits with a failure code if any of the TOML entries did not
complete successfully.
JSON results files only need to be valid JSON, and contain some form of numeric results that can be extracted into a string or list of numeric results using the jq JSON query tool.
The baseline TOML file is composed of one [[metric]] section per result that is processed.
Each section contains a number of parameters, some optional:
|name | type | description |
|----------------------------------------------------------------------------
|`name` | string | Filename containing results (minus .json ext.)|
|`type` | string | json (optional, json is the default) |
|`description` | string | Description of test (optional) |
|`checkvar` | string | jq query string to extract results from JSON |
|`checktype` | string | Property to check ("mean", "max" etc.) |
|`minval` | float | Minimum value the checked property should be |
|`maxval` | float | Maximum value the checked property should be |
|`midval` | float | Middle value used for percentage range check |
|`minpercent` | float | Minimum percentage from midval check boundary |
|`maxpercent` | float | Maximum percentage from midval check boundary |
At this time only JSON formatted results files are supported.
The following checktypes are supported. All are tested to fall within the bounds set by the minval
and maxval. That is:
minval <= Result <= maxval
|check |description |
|-----------------------------------------------------------------|
|mean |the mean of all the results extracted by the jq query |
|min |the minimum (smallest) result |
|max |the maximum (largest) result |
|sd |the standard deviation of the results |
|cov the coefficient of variation (relative standard deviation)|
checkmetrics takes a number of options. Some are mandatory.
--basefile value path to baseline TOML metrics file
--debug enable debug output in the log
--log value set the log file path
--metricsdir value directory containing results files
--percentage present results as percentage differences
--help, -h show help
--version, -v print the version
The checkmetrics tool outputs a summary table after processing all metrics sections, and returns
a non-zero return code if any of the metrics checks fail.
Example output:
Report Summary:
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
| P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS |
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
| F | boot-times | 0.50 | 1.36 | 0.70 | 40.0% | 1.34 | 1.38 | 2.7% | 1.3% | 2 |
| F | memory-footprint | 100000.00 | 284570.56 | 110000.00 | 10.0% | 284570.56 | 284570.56 | 0.0% | 0.0% | 1 |
| P | memory-footprint-ksm | 100000.00 | 101770.22 | 110000.00 | 10.0% | 101770.22 | 101770.22 | 0.0% | 0.0% | 1 |
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
Fails: 2, Passes 1
Example percentage mode output:
Report Summary:
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
| P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS |
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
| *F* | boot-times | 83.3% | 226.8% | 116.7% | 33.3% | 223.8% | 229.8% | 2.7% | 1.3% | 2 |
| *F* | memory-footprint | 95.2% | 271.0% | 104.8% | 9.5% | 271.0% | 271.0% | 0.0% | 0.0% | 1 |
| P | memory-footprint-ksm | 92.7% | 99.3% | 107.3% | 14.6% | 99.3% | 99.3% | 0.0% | 0.0% | 1 |
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
Fails: 2, Passes 1
|name |description |
|-----------------------------------------------------------------------|
|P/F |Pass/Fail |
|NAME |Name of the test/check |
|FLR |Floor - the minval to check against |
|MEAN |The mean of the results |
|CEIL |Ceiling - the maxval to check against |
|GAP |The range (gap) between the minval and maxval, as a % of minval|
|MIN |The minimum result in the data set |
|MAX |The maximum result in the data set |
|RNG |The % range (spread) between the min and max result, WRT min |
|COV |The coefficient of variation of the results |
|ITS |The number of results (iterations) |
For example, to invoke the checkmetrics tool, enter the following:
BASEFILE=`pwd`/../../metrics/baseline/baseline.toml
METRICSDIR=`pwd`/../../metrics/results
$ ./checkmetrics --basefile ${BASEFILE} --metricsdir ${METRICSDIR}