Expose API for integrating with other test suites

[munro]

Just setup Tasty so I could run HSpec and doctests, except to find out that doctest can't plug into other test suites. [1] 😭

Looking at the Tasty API, what it needs to expose is pretty simple—

• Test name :: String
• If the test passed / failed
• Description :: String (containing pass message, or fail message)

From this example doctest ...

-- | Parse a number
-- >>> parse (parseNumber) "fail" "1231231.12312312313120000"
-- Right (Number "1231231.12312312313120000")
-- >>> parse (parseNumber) "fail" "-1231231.12312312313120000"
-- Right (Number "-1231231.12312312313120000")
-- >>> parse (parseNumber <|> operator) "fail" "-    "
-- Right (Operator "-")

... we could infer the same test API.

• Test name = "Parse a number"
• Pass, empty description
• Or on fail, standard doctest error—

### Failure in YourBadCode.hs:1337: expression `parse (parseNumber) "fail" "1231231.12312312313120000"'
expected: Right (Number "1231231.12312312313120000")
 but got: The right thing

Or we could expose more of the doctest test API, so the runner has more control over how to display everything. I'm actual more partial to doing this!

data TestExpression = TestExpression
    { line :: Integer
    , expression :: String
    , expected :: String
    , returned :: String
    }

data Test = Test
    { file :: String
    , line :: Integer
    , comment :: String
    , expressions :: [TestExpression] }

---

The next step would be to figure out an API to export the tests from, so they can then be translated to another test runner. What do you think about the following—

module Test.DocTest where

doctestTests :: [String] -> IO [Test]

Looking at the code, I can already start to see where I would plug this in, and make the current runner just sit on top of the API.

Let me know what you think!

❤️ doctest

[1] http://www.reddit.com/r/haskell/comments/1s0t1f/24_days_of_hackage_tasty/cdsvu7n

[sol]

@munro I'm excited to hear that you like both doctest and hspec. I'll write a more through reply when I can free up some time.

But one quick question, both tasty and hspec are testing frameworks. If you are already happy with hspec, may I ask what your motivation is to run your hspec tests through tasty?

Have you looked at automatic spec discovery. You loose this functionality if you use hspec in combination with tasty. In addition, both doctest and hspec work with GHC 7.*, while tasty requires GHC 7.4+.

What exact features of tasty are you after?

[munro]

But one quick question, both tasty and hspec are testing frameworks. If you are already happy with hspec, may I ask what your motivation is to run your hspec tests through tasty?

I liked the DSL that hspec provided, since it lets me do IO—which is nice because it feels more like a scratch pad for my ideas.

What exact features of tasty are you after?

I wanted to combine both my doctests and hspec into one test suite, tasty just made that easy for hspec. Tasty has a gross API for writing tests though, so I just stuck with hspec.

Have you looked at automatic spec discovery. You loose this functionality if you use hspec in combination with tasty. In addition, both doctest and hspec work with GHC 7.*, while tasty requires GHC 7.4+.

:O That's pretty cool, I've just globbed my tests together manually. For some reason I'm infatuated with running all the tests together.

[sol]

For me this does not have high priority for the following reasons:

• The current approach is not tied to any particular testing framework and is robust. We had test-framework-doctest before and deprecated it in favor of using standalone test suite sections.
• Due to issues with GHC 7.8.* depending on an old version of transformers (which prevents you from using a newer version of transformers for your library/executable if you depend on doctest in a test-suite section), I even tend to run doctest as a standalone executable on Travis instead of having a test-suite section by now (e.g. https://github.com/sol/interpolate/blob/master/.travis.yml).
• Doctest is slow, I want my unit tests to be fast. For that reason I prefer to have Doctest run outside my unit tests for the projects I work on / contribute to. (I take Doctest's own test suite as evidence for what a pain a slow test suite is).

If it's still important to you, please feel free to give it a try. On a technical side:

• Doctest reuses a single GHCi session to speed up things. This implies that you have to share that session across test cases. In addition, you may want to use some form of synchronization to prevent concurrent access to the GHCi session.

My requirements for getting it on master are:

• Two reference implementations that use the API for two different testing frameworks, so that we have some evidence that the API is not tied to the needs of one particular testing framework (ideally one of the reference implementations would be for Hspec, yes Hspec is a testing framework)
• Cycles on my side have to be minimal. It's not that I don't want to help, it's just that my typing time is restricted. That means separate commits for refactoring and features, optimizing things for reviewability, and possibly breaking up a big PR into smaller ones, etc.

[simonmichael]

Count me as another who'd like to be able to run doctest from one of the higher-level test frameworks. I was surprised that the current popular one, tasty, can't run doctests - it seems to run everything else. The old test-framework-doctest seems too deprecated to be an option these days.

[simonmichael]

I meant to add a use case: the hledger-lib package defines two test suites, one is hunit tests run by test-framework, the other is for doctests. Having each suite report its results separately is not always ideal, eg recently I missed doctest failures because the hunit test output pushed them offscreen. It would be good to be able to summarise all the test results together at the end.