Commits on Oct 23, 2024

1. Improve testing with stats and errors per test section …

   * TestRunner: separated the running of each "section" of a test into its own try-catch so that we can evaluate each section separately even if another section failed (e.g. evaluate all examples even if one test in "facts" throws an exception). I've added tests that exercise this "new feature".
   
   * TestResults: refactored it a bit so that TestResults now has the three sections and the old TestResults becomes TestSectionResults. Also, TestSectionResults now has only *one* error, since that's what we get when there's an exception evaluating the Pkl code.
   
   * SimpleReport:
     * Separated reporting of each test section so that it's easy to see which facts or which examples fail.
   
     * Added a "stats line" to the module level that reports how many tests and assertions pass/fail. Note that examples are equivalent to "one test with an 'all-or-nothing assertion' so they are counted as 1 test with 1 assertion. Similarly errors will also count as "one test with one assertion failed" because, at the moment, individual errors are not being surfaced individually. Instead, only the first error is being surfaced and thus it masks the rest of the tests and assertions in the test section where it happens.
   
     * For the examples converted to tests, when multiple examples share the same name I've also added a counter ("# 0" and so on) to disambiguate them so that it's easier to identify which one is failing.
   
   * JUnitReport: fixed the reporting of failures as it wasn't consistent with "tests": the tests were the number of tests that we run but "failures" were the number of assertions that failed.

   

   jjmaestro committed Oct 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 25fd96e
   • Browse repository at this point

   Copy the full SHA

   25fd96e View commit details

   Browse the repository at this point in the history