Import statements get counted as separate tests

[nikita-volkov]

I have only a single doctest, which is the following:

-- >>> import qualified Data.Vector as V
-- >>> import qualified Data.Vector.Unboxed as VU
-- >>> streamValueDistributions $ V.fromList $ fmap VU.fromList $ [ [1, 2, 3], [2, 3], [2] ]
-- [(2,3),(3,2),(1,1)]

Executing doctest on it, produces the following summary:

Examples: 3  Tried: 3  Errors: 0  Failures: 0

Version: doctest-0.9.11.1

[orlitzky]

Isn't this a good thing?

The import statements are commands that your users are expected to type into ghci. If they don't work for some reason, I want the test suite to fail.

But suppose we didn't count the import statements as tests. So we ignore them if they fail. What good can come if it? if an import fails, the subsequent tests will probably fail anyway. But if the subsequent tests do pass, then you have a useless import, which is something I'd like to know! If I know the import is failing but the rest of the test suite works, I can simply remove the import.

[nikita-volkov]

It's a single output, hence a single example. Import statements are only an environment for this example. If they fail, the test should fail. I'm really surprised someone can look at it another way.

[orlitzky]

If import failure implies test failure, aren't imports "tests" then? =)

I see what you're saying -- that all you really want to test is output -- but it's hard to differentiate. It might be nice to have another output statistic, "nonempty output lines" or something like that. But consider the following test:

>>> putStr ""

which also tests something that produces no output. You want that to fail if the putStr fails, of course. Which means the criteria is the same as that for the imports. So I think it makes sense to treat the imports as tests. I don't feel strongly about it, though, I just happened to be browsing the issue list.

[nikita-volkov]

Testing import statements - that is definitely something new. Though a documentation with examples, which produce no visible result, could be another hit.

Please pardon my sarcasm, but it is very hard to come up with a reply to this that is any more delicate.

[orlitzky]

Ok, so the putStr thing is stupid. But there are better examples.

-- | This module makes heavy use of the 'GHC.Generics' module, and the
-- associated \"DeriveGeneric\" extension which allows instances of 'Generic'
-- to be, well, derived. First, we'll need to enable a few GHC extensions:
--
-- >>> :set -XDeriveGeneric
-- >>> :set -XNoMonomorphismRestriction
--
-- We'll also need to import the 'Generics' module from GHC:
--
-- >>> import GHC.Generics
--
-- With that out of the way, we can now define a new data type which
-- automatically derives its 'Generic' instance:
--
-- >>> data Foo = Bar Int Int Int deriving (Show, Generic)
--
-- This should give you a feel for how the typeclass and extension are used below.
--
module Main
where

main :: IO ()
main = putStrLn "Actual code goes here."

None of the >>> lines produce output, but they should all be tested.

[sol]

@nikita-volkov Doctest is agnostic about what kind of statements you feed it. So it can not differentiate import statements from expressions that produce output. This has the advantage that

If something works with GHCi, it works with Doctest, too!

What we could change is how we count examples. Currently every line that starts with a >>> is counted as a example. We could e.g. change that to count each Haddock comment as an example (regardless how many >>>s it contains).

I'm not very opinionated about it, but I most likely will not invest any cycles in that myself. But if somebody has a vision and wants to outline a different approach, let's go for it.

[munro]

@orlitzky ah yea, what you said makes sense, but also brings up a lot of complexity.

1. if a >>> statement is immediately followed by another >>>, the assumption is that GHCi returned "" for the former, and counts both as tests.

-- | MyTest
-- >>> 1 + 2
-- >>> 2 + 3
-- 5

-- | MyTest2
-- >>> let foo = 2
-- >>> foo + 4
-- 6

2. if a >>> is an IO value, GHCi runs with real world, any printed output is shown, which then used for test assertion on the next line (which could implicitly be "" if the next line is a >>>)

-- | MyTest3
-- >>> import Control.Concurrent.MVar
-- >>> newEmptyMVar
-- >>> newEmptyMVar
--

-- | MyTest4
-- >>> putStr ""
-- >>> putStr "foo"
-- >>> putStr "bar"
-- bar

3. if a >>> returns a value, GHCi calls show on it, and prints it, output is used for test assertion.

-- | MyTest5
-- >>> data Foo = Foo deriving Show
-- >>> Foo
-- >>> Foo
-- Foo

4. Any other type of statement doesn't return anthing, but I guess GHCi counts it as "", and doctest asserts that.

---

@orlitzky also, your example you provided is kind of weird when you think about it. Because it doesn't look like your asserting anything, it just looks like your documenting the setup of a multiline test. Since you can't write:

-- >>> import GHC.Generics
-- With that out of the way, we can now define a new data type which
-- automatically derives its 'Generic' instance:

Though it doesn't look like it to a human, there's actually an assertion here.

-- >>> import GHC.Generics
-- "" <-- Is this really an explicit assertion by the programmer?  It just looks like whitespacing to me.
-- With that out of the way, we can now define a new data type which

---

@sol I've just thought about it for a bit, and I think that's a great idea to leave all the behavior how it is, and just treat a single Haddock comment as a test.

Since if one assert in the chain fails, the all the rest of the >>> don't run ... sounds line a single test!

Also, it's normal for a programmer to do multiple assertions in a single test.

Finally, it relates to #106 since I had already assumed the entire Haddock comment was treated as one test internally. 😄

[orlitzky]

@munro

-- >>> import GHC.Generics
-- "" <-- Is this really an explicit assertion by the programmer?

If the import statement outputs something other than "", don't you want the test suite to fail?