"not in scope" errors when running with GHC 9 + stack test

[simonmichael]

Some tests that have been working with GHC 8.x are giving a lot of name not in scope errors when run with GHC 9.0.

With doctest-0.18.1, other pkgs, GHC 8.10:

$ stack test hledger-lib:test:doctest  --stack-yaml stack8.10.yaml
hledger-lib> test (suite: doctest)

Loading and searching for doctests in 50 files, plus any files they import:
Examples: 218  Tried: 218  Errors: 0  Failures: 0

hledger-lib> Test suite doctest passed

With doctest-0.18.1, other pkgs, GHC 9.0:

$ stack test hledger-lib:test:doctest
hledger-lib> test (suite: doctest)

Loading and searching for doctests in 50 files, plus any files they import:
Hledger/Read/JournalReader.hs:182: failure in expression `rejp (journalp <* eof) "2015/1/1\n a  0\n"'
expected: Right (Right Journal  with 1 transactions, 1 accounts)
 but got: 
          ^
          <interactive>:31:1: error:
              Variable not in scope: rejp :: f0 a0 -> t0 -> t
          
          <interactive>:31:7: error: Variable not in scope: journalp :: f0 a0
          
          <interactive>:31:19: error: Variable not in scope: eof :: f0 b0
...

I didn't find anything relevant in the issue tracker or https://hackage.haskell.org/package/doctest-0.18.1/changelog or GHC release notes. What could explain this ?

[simonmichael]

It appears not to see the bindings in the current module (the module containing the doctests).

More data points:

• it works if I run tests with the standard doctest runner
• it works if I build and run my custom runner from the command line
• it fails if I build and run my custom runner as a package test suite via stack test (stack test hledger-lib:test:doctest)
• it works with stack and GHC 8.x (stack test hledger-lib:test:doctest --stack-yaml stack8.10.yaml)
• it works with cabal and GHC 9 (cabal test hledger-lib:test:doctest -w $(stack exec -- which ghc)

[simonmichael]

I have maybe narrowed this down to stack test, let's wait and see: commercialhaskell/stack#5591

[simonmichael]

Any idea why doctests would fail to see bindings in scope, when invoked with those GHC_ENVIRONMENT settings ?
commercialhaskell/stack#5591 (comment)

[simonmichael]

Possibly related, "template-haskell" is among the tests failing with GHC 9: #303

[tonyday567]

Same error with cabal test on GHC 9.

[LeventErkok]

@simonmichael I'm running into this issue; and I do not use stack at all. Wondering if you had any further insights.

[simonmichael]

I have not, sorry.

[LeventErkok]

@simonmichael

Thanks.. Unfortunately this renders doctest pretty much unusable with the most recent GHC9.2.1 release. It's hard to maintain open-source projects in the presence of a fast moving target like GHC. The doctest project filled a very important hole in the testing story for Haskell, with no alternatives to it that I am aware of. I wonder if we should seek help at GHC head-quarters, or elsewhere to make this issue better known; should this be a GHC issue to start with.

[simonmichael]

I don't use doctest much, but I'd like to more, so I'm in favour. Yes, worth asking #ghc and also proposing it in the appropriate Haskell Foundation channel indeed.

[simonmichael]

(I suspect it's not a hard problem to fix, just requires deep knowledge of these tools.)

[LeventErkok]

My bad, I thought you were the maintainer! I see it's @sol Perhaps he has some plans to pursue this further?

[ysangkok]

One alternative could be cabal-docspec.

[LeventErkok]

@ysangkok https://hackage.haskell.org/package/doctest-extract is another attempt, which has the right idea of extracting tests and compiling them on the side for speed; but so far as I know it's not ready for prime time.

As far as I can see cabal-docspec isn't really ready to be used either. I wasn't aware of it before, nor I see any indication that it can replace doctest as is. Do you know anyone using it in a real project?

[LeventErkok]

@ysangkok

Thanks for the cabal-docspec pointer. I've been experimenting with it for the past hour, and indeed it seems to be a viable alternative. I'm mostly motivated by the fact that it actually uses the compiled code for test-execution so it goes a lot faster, and doesn't depend on ghc as a library to do the parsing, so much more resilient to GHC version changes.

One down-side is it's not on hackage, which makes it a bit of a hassle to get your hands on. But not a big deal; I expect it'll eventually land on hackage.

This, of course, takes nothing from doctest as is, and efforts of @sol. It's a great piece of software and I'd love to see it flourish. It's just that there's an alternative that can be utilized in the meantime. Much appreciated.

[sol]

@LeventErkok I'm sorry I haven't been involved with maintaining doctest for some time now. Due to a lack of cycles and issues with RSI most of my effort goes towards hspec.

@quasicomputational did a great job and I think @andreasabel tried to keep on the lights for the last year or so. But from what I see we are at the verge of bit rot. However, I'm not sure if I'm ready to let doctest fade away.

Looking at it right now I think we don't support cabal -v2 at all + we certainly do not use it for doctests's test suite. That's probably the first thing I would want to address.

There are two issues here:

1. ghci does not seem to respect -v0 when loading package environments for some versions of GHC (specifically < 8.10).
2. It's non-trivial to determine the location of the Paths_* module.

(1) should be easy to work around. Not sure about (2).

[sol]

I think this is a GHC upstream bug: https://gitlab.haskell.org/ghc/ghc/-/issues/20670

I also assume that this is fixed on main now (by #312).

@LeventErkok @simonmichael can you give it a try and report back if there are still any issues left? Basically, what you want to do is:

cabal exec -- cabal test doctests --test-show-details=direct

with a recent version of cabal-install.

[sol]

doctest + cabal-v2 should now work with GHC >= 8.* as long as you don't need the Paths_* module.

The issues with GHC == 7.* and cabal-v2 as I understand it is that GHC < 8 does not support ghc env files. So cabal exec will not bring any packages into scope (I guess cabal could still support those versions of GHC by constructing a traditional package db and bring it into scope, but apparently it doesn't. @phadej any input?). cabal-v1 still works with GHC < 8.

The issue with Paths_* and cabal-v2 is that I'm not aware of any easy way to get it's location. Ideally we would want to ask cabal, but I don't think cabal provides an interface for that (@phadej are you aware of anything here?). This is not doctest specific though, you would run into the same issue if you would e.g. try to start ghci with cabal exec.

I think one way to make it work is to creatively (mis)use cabal repl --with-ghc=doctest. To make this work, doctest has to pretend to be ghc, transparently forwarding any invocations to ghc, until it is called with --interactive. This will not only solve the Paths_* issue but make it generally easier to use doctest + also solve any issues related to GHC < 8.

[sol]

#314 (4677e38) is a proof-of-concept for cabal repl --with-ghc=doctest. It's not ready for prime time yet. As is, it will break doctest in any other context but cabal repl.

[LeventErkok]

Thanks @sol. I've since moved to cabal-docspec. It's slightly different in its methodology than doctest, but I like the fact that it doesn't depend on the GHC API. Your work on doctest has been phenomenal and very helpful over the years though. Much appreciated.

[sol]

Unless given evidence to the contrary, I assume that this is fixed with doctest-0.19.0.

[Xitian9]

I'm still getting this error with doctest-0.20.0.

[sol]

@Xitian9 can you please open a new issue + also give the exact steps to reproduce.

This will help a lot.

[vaclavsvejcar]

@sol I'm getting this error with doctest-0.20.0 and GHC 9.0.1 as well, when trying to build project with stack and Nightly stackage. Everything works fine with lts-18.18 and GHC 8.10.7. I can provide some more details if that would help.

[echatav]

I'm getting this issue as well. Works with GHC 8 but not GHC 9. Same error with doctest 0.18.2 and 0.20.0. Here's a link to what the error looks like.

[sol]

@vaclavsvejcar @echatav if you can open a new issue with exact steps on how to reproduce (bonus points, if you can provide a minimal example), that would help a lot.

What I assume could happen is that https://gitlab.haskell.org/ghc/ghc/-/issues/20670 is triggered by other conditions besides -XTemplateHaskell.

[echatav]

I opened a new issue with reproduction steps and failure details.