Make the test suite less awk-ward (#419)

This simplifies the design of the test suite by:

1. Removing the use of `awk`, as it is not needed, and
2. Remove the use of `.ghcXX.template` files in favor of ordinary
   `.golden` files.

This allows us to replace the current tailor-made (and buggy)
`tasty-golden` setup with a more conventional one and delete lots
of code in the process. I've also made an effort to clean up the
documentation in `tests/README.md` and the Notes in
`SingletonsTestSuiteUtils` to better reflect the current design.

Fixes #417.

.gitignore

@@ -13,7 +13,6 @@ tags
 
 # Other files
 *.actual
-*.golden
 Main
 
 # cabal sandboxes

Makefile

@@ -50,5 +50,4 @@ clean-tests:
 	find tests -name "*.o"      | xargs rm -f
 	find tests -name "*.dyn_o"  | xargs rm -f
 	find tests -name "*.actual" | xargs rm -f
-	find tests -name "*.golden" | xargs rm -f
 	rm -f tests/compile-and-dump/GradingClient/Main

singletons.cabal

@@ -10,16 +10,15 @@ maintainer:     Ryan Scott <[email protected]>
 bug-reports:    https://github.com/goldfirere/singletons/issues
 stability:      experimental
 tested-with:    GHC == 8.8.1
-extra-source-files: README.md, CHANGES.md,
-                    tests/compile-and-dump/buildGoldenFiles.awk,
+extra-source-files: README.md, CHANGES.md, tests/README.md,
                     tests/compile-and-dump/GradingClient/*.hs,
                     tests/compile-and-dump/InsertionSort/*.hs,
                     tests/compile-and-dump/Promote/*.hs,
                     tests/compile-and-dump/Singletons/*.hs
-                    tests/compile-and-dump/GradingClient/*.ghc88.template,
-                    tests/compile-and-dump/InsertionSort/*.ghc88.template,
-                    tests/compile-and-dump/Promote/*.ghc88.template,
-                    tests/compile-and-dump/Singletons/*.ghc88.template
+                    tests/compile-and-dump/GradingClient/*.golden,
+                    tests/compile-and-dump/InsertionSort/*.golden,
+                    tests/compile-and-dump/Promote/*.golden,
+                    tests/compile-and-dump/Singletons/*.golden
 license:        BSD3
 license-file:   LICENSE
 build-type:     Custom

tests/README.md

@@ -1,13 +1,13 @@
-Singletons testsuite notes
-==========================
+`singletons` testsuite notes
+============================
 
 The `singletons` testsuite is built using the
 [`tasty`](http://hackage.haskell.org/package/tasty) testing framework. Aside
 from the standard `HUnit` and `QuickCheck` tests, it contains a number of
 compile-and-dump tests. These tests run GHC to compile a test file that uses
-`singletons` library and compare the generated Template Haskell splices against
-expected output saved in a "golden" file. Below are some details about these
-tests:
+the `singletons` library and compare the generated Template Haskell splices
+against expected output saved in a "golden" file. Below are some details about
+these tests:
 
  * GHC uses the in-tree `singletons` library. This means there is no need to
    install the library in your system before running the tests. You can simply
@@ -17,38 +17,34 @@ tests:
       cabal test
       ```
 
- * Compile-and-dump tests are stored in subdirectories of
-   `tests/compile-and-dump/` directory. Files with `.template` extension store
-   expected output and are used to generate the actual `.golden` files. (see
-   section "Generating golden files from templates" below). Running the
-   testsuite produces files with `.actual` extension which contain actual output
-   produced by GHC. These files are not deleted by the testsuite, which allows
-   to inspect them in case the test fails. To remove the `.actual` and `.golden`
-   files run:
+ * Compile-and-dump tests are stored in the subdirectories of the
+   `tests/compile-and-dump/` directory. Files with a `.golden` extension store
+   the expected output. Running the testsuite produces files with an `.actual`
+   extension which contain the actual output produced by GHC. These files are
+   not deleted by the testsuite, which allows one to inspect them in case a
+   test fails. To remove the `.actual` files, run:
 
       ```bash
       make clean-tests
       ```
 
- * Running the testsuite requires `awk`, `sed` and `diff`. `awk` is used to
-   generate golden files from templates (see below). `sed` is used to normalize
-   output from GHC (see Note [Normalization with sed]).`diff` is used to compare
-   golden and actual files.
+ * Running the testsuite requires `diff`, as `diff` is used to compare golden
+   and actual files.
 
  * Each compile-and-dump test requires a set of GHC options to be used for
-   compilation. Testsuite defines a default set of options that enable on the
-   command line all the extensions required by `singletons`. This makes writing
+   compilation. The testsuite defines a default set of options that enable all
+   of the language extensions required by `singletons`. This makes writing
    tests easier since there is no need to put all the extensions into a source
-   file. The default options also enable `-ddump-splices` (dumps splices
-   generated by Template Haskell), `-dsuppress-uniques` (avoids test failures
-   due to unique identifiers), `-v0` (silences GHC) and `-fforce-recomp` (forces
-   recompilation each time a testsuite is run). There is a convenience function
-   that creates a test with standard GHC options defined by the testsuite.
+   file. The default options also enable `-ddump-splices` (which dumps splices
+   generated by Template Haskell), `-dsuppress-uniques` (which avoids test
+   failures due to unique identifiers), `-v0` (which silences GHC's build
+   output) and `-fforce-recomp` (which forces recompilation each time a
+   testsuite is run). There is a convenience function (`compileAndDumpTest`)
+   that creates a test with the standard GHC options defined by the testsuite.
 
- * Testsuite supports multiple GHC versions because each version produces
-   different Template Haskell output. Every test requires a separate golden
-   file. GHC HEAD is currently not supported. Golden files for GHC 8.4 have
-   `.ghc84` appended before their extension.
+ * Because `singletons` only supports one version of GHC, the `.golden` files
+   should not be assumed to be portable across multiple versions of GHC. Beware
+   of this should you try testing `singletons` against GHC HEAD.
 
  * You can run single tests or groups of tests whose name match a regexp using
    tasty's pattern feature. For example:
@@ -57,8 +53,8 @@ tests:
    cabal test --test-options="--pattern=Testsuite/Singletons/*"
    ```
 
-   runs all tests in the Testsuite/Singletons branch of the test tree.
-   SingletonsTestSuite module defines structure of the test tree.
+   runs all tests in the `Testsuite/Singletons` branch of the test tree.
+   The `SingletonsTestSuite` module defines the structure of the test tree.
 
  * If you modify `singletons`, you may cause the actual output of some tests to
    change. If these changes are what you intended, you can accept the new
@@ -68,33 +64,8 @@ tests:
    cabal test --test-options="--accept"
    ```
 
-## Generating golden files from templates
-
-It is possible to have definitions generated in one tests used in another
-test. A good example of this is `SingletonsNat` test which singletonizes
-definition of `Nat` datatype. `SingletonsNat` is imported from other tests to
-reuse the generated `Nat` definitions. Due to `-fforce-recomp` option
-`SingletonsNat` module will be recompiled for every test and therefore splices
-generated during its compilation will be included in the output of every test
-that imports `SingletonsNat`. One way to deal with the problem would be to
-prepare golden files that include all generated splices, including those from
-imported modules. The problem with this approach is that if we have 100 tests
-that include `SingletonsNat` and change the way `singletons` are generated we
-will have to update 100 files with exected output. This problem is solved with
-templates. Template file contains expected output but it may also include
-`INSERT tests/compile-and-dump/Foo.XXXX.template` pragma, which inserts the
-contents of a specified template file (XXXX represents GHC version).
-
-This feature is recursive but can be a bit tricky. See `Singletons/EqInstances`
-test for example.
-
 ## ByHand files
 
 `tests` directory contains `ByHand.hs` and `ByHandAux.hs` files. They are
 intended as a sandbox where all the definitions generated by `singletons` are
-written by hand. These files don't test correctness of `singletons` library and
-are thus not part of the testsuite. You can compile these files with:
-
-  ```bash
-  make byhand
-  ```
+written by hand.

tests/SingletonsTestSuiteUtils.hs

@@ -13,23 +13,18 @@ module SingletonsTestSuiteUtils (
  ) where
 
 import Build_singletons   ( ghcPath, ghcFlags, rootDir          )
-import Control.DeepSeq    ( NFData(..)                          )
-import Control.Exception  ( Exception, evaluate, throw          )
-import Data.Foldable      ( asum )
-import Data.List          ( intercalate                         )
+import Control.Exception  ( Exception                           )
+import Data.Foldable      ( asum                                )
 import Data.Text          ( Text                                )
 import Data.String        ( IsString(fromString)                )
-import System.Exit        ( ExitCode(..)                        )
 import System.FilePath    ( takeBaseName, pathSeparator         )
-import System.IO          ( IOMode(..), hGetContents, openFile  )
 import System.FilePath    ( (</>)                               )
+import System.IO          ( IOMode(..), openFile                )
 import System.Process     ( CreateProcess(..), StdStream(..)
                           , createProcess, proc, waitForProcess
                           , callCommand                         )
 import Test.Tasty         ( TestTree, testGroup                 )
-import Test.Tasty.Golden.Advanced
-                          ( goldenTest                          )
-import qualified Data.ByteString as BS
+import Test.Tasty.Golden  ( goldenVsFileDiff                    )
 import qualified Turtle
 
 -- Some infractructure for handling external process errors
@@ -41,9 +36,6 @@ newtype ProcessException = ProcessException String
 goldenPath :: FilePath
 goldenPath = rootDir </> "tests/compile-and-dump/"
 
-ghcVersion :: String
-ghcVersion = ".ghc88"
-
 -- GHC options used when running the tests
 ghcOpts :: [String]
 ghcOpts = ghcFlags ++ [
@@ -89,15 +81,14 @@ ghcOpts = ghcFlags ++ [
 -- with no ".hs".
 compileAndDumpTest :: FilePath -> [String] -> TestTree
 compileAndDumpTest testName opts =
-    goldenTest
+    goldenVsFileDiff
       (takeBaseName testName)
-      (return ())
+      (\ref new -> ["diff", "-w", "-B", ref, new]) -- see Note [Diff options]
+      goldenFilePath
+      actualFilePath
       compileWithGHC
-      cmp
-      acceptNewOutput
   where
     testPath         = testName ++ ".hs"
-    templateFilePath = goldenPath ++ testName ++ ghcVersion ++ ".template"
     goldenFilePath   = goldenPath ++ testName ++ ".golden"
     actualFilePath   = goldenPath ++ testName ++ ".actual"
 
@@ -108,37 +99,10 @@ compileAndDumpTest testName opts =
                                               { std_out = UseHandle hActualFile
                                               , std_err = UseHandle hActualFile
                                               , cwd     = Just goldenPath }
-      _ <- waitForProcess pid      -- see Note [Ignore exit code]
+      _ <- waitForProcess pid        -- see Note [Ignore exit code]
       normalizeOutput actualFilePath -- see Note [Output normalization]
-      buildGoldenFile templateFilePath goldenFilePath
       return ()
 
-    -- See Note [Diff options]
-    cmd = ["diff", "-w", "-B", goldenFilePath, actualFilePath]
-
-    -- This is invoked when the test suite is run with the --accept flag.
-    -- In addition to updating the golden file, we should also update the
-    -- template file (which is what is actually checked into version control).
-    acceptNewOutput _ = do
-      actualContents <- BS.readFile actualFilePath
-      BS.writeFile goldenFilePath   actualContents
-      BS.writeFile templateFilePath actualContents
-
-    -- This is largely cargo-culted from the internals of
-    -- tasty-golden's goldenVsFileDiff.cmp function.
-    cmp _ _ | null cmd = error "compileAndDumpTest: empty command line"
-    cmp _ _ = do
-      (_, Just sout, _, pid)
-        <- createProcess (proc (head cmd) (tail cmd)) { std_out = CreatePipe }
-      -- strictly read the whole output, so that the process can terminate
-      out <- hGetContents sout
-      evaluate . rnf $ out
-
-      r <- waitForProcess pid
-      return $ case r of
-        ExitSuccess -> Nothing
-        _ -> Just out
-
 -- Compile-and-dump test using standard GHC options defined by the testsuite.
 -- It takes two parameters: name of a file containing a test (no ".hs"
 -- extension) and directory where the test is located (relative to
@@ -156,40 +120,39 @@ testCompileAndDumpGroup :: FilePath -> [FilePath -> TestTree] -> TestTree
 testCompileAndDumpGroup testDir tests =
     testGroup testDir $ map ($ testDir) tests
 
--- Note [Ignore exit code]
--- ~~~~~~~~~~~~~~~~~~~~~~~
----- It may happen that compilation of a source file fails. We could find out
--- whether that happened by inspecting the exit code of GHC process. But it
--- would be tricky to get a helpful message from the failing test: we would need
--- to display stderr which we just wrote into a file. Luckliy we don't have to
--- do that - we can ignore the problem here and let the test fail when the
--- actual file is compared with the golden file.
-
--- Note [Diff options]
--- ~~~~~~~~~~~~~~~~~~~
---
--- We use following diff options:
---  -w - Ignore all white space.
---  -B - Ignore changes whose lines are all blank.
+{-
+Note [Ignore exit code]
+~~~~~~~~~~~~~~~~~~~~~~~
+It may happen that the compilation of a source file fails. We could find out
+whether that happened by inspecting the exit code of the `ghc` process. But it
+would be tricky to get a helpful message from the failing test; we would need
+to display the stderr that we just wrote into a file. Luckliy, we don't have to
+do that - we can ignore the problem here and let the test fail when the
+actual file is compared with the golden file.
 
--- Note [Output normalization]
--- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
---
--- Output file is normalized inplace. Line numbers generated in splices:
---
---   Foo:(40,3)-(42,4)
---   Foo.hs:7:3:
---   Equals_1235967303
---
--- are turned into:
---
---   Foo:(0,0)-(0,0)
---   Foo.hs:0:0:
---   Equals_0123456789
---
--- This allows to insert comments into test file without the need to modify the
--- golden file to adjust line numbers.
---
+Note [Diff options]
+~~~~~~~~~~~~~~~~~~~
+We use following diff options:
+ -w - Ignore all white space.
+ -B - Ignore changes whose lines are all blank.
+
+Note [Output normalization]
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Output file is normalized inplace. Line numbers generated in splices:
+
+  Foo:(40,3)-(42,4)
+  Foo.hs:7:3:
+  Equals_1235967303
+
+are turned into:
+
+  Foo:(0,0)-(0,0)
+  Foo.hs:0:0:
+  Equals_0123456789
+
+This allows inserting comments into test files without the need to modify the
+golden file to adjust line numbers.
+-}
 
 normalizeOutput :: FilePath -> IO ()
 normalizeOutput file = Turtle.inplace pat (fromString file)
@@ -212,27 +175,5 @@ normalizeOutput file = Turtle.inplace pat (fromString file)
     numPeriod = zipWith const (cycle "0123456789876543210")
     d = Turtle.some Turtle.digit
 
-buildGoldenFile :: FilePath -> FilePath -> IO ()
-buildGoldenFile templateFilePath goldenFilePath = do
-  hGoldenFile <- openFile goldenFilePath WriteMode
-  runProcessWithOpts (UseHandle hGoldenFile) "awk"
-            [ "-f", goldenPath </> "buildGoldenFiles.awk"
-            , templateFilePath
-            ]
-
-runProcessWithOpts :: StdStream -> String -> [String] -> IO ()
-runProcessWithOpts stdout program opts = do
-  (_, _, Just serr, pid) <-
-      createProcess (proc "bash" ["-c", (intercalate " " (program : opts))])
-                    { std_out = stdout
-                    , std_err = CreatePipe }
-  ecode <- waitForProcess pid
-  case ecode of
-    ExitSuccess   -> return ()
-    ExitFailure _ -> do
-       err <- hGetContents serr -- Text would be faster than String, but this is
-                                -- a corner case so probably not worth it.
-       throw $ ProcessException ("Error when running " ++ program ++ ":\n" ++ err)
-
 cleanFiles :: IO ()
 cleanFiles = callCommand $ "rm -f " ++ rootDir </> "tests/compile-and-dump/*/*.{hi,o}"