Test failures can be hard to interpret

[sjakobi]

In dhall-lang/dhall-haskell#1957 (comment) I had some difficulty interpreting this test failure:

/home/simon/src/dhall-haskell/dhall/src/Dhall/Tutorial.hs:452: failure in expression `input auto "[]" :: IO (Vector Natural)'
expected: "*** Exception:"
          "...Error...: Invalid input"
          "..."
          "(input):1:3:"
          "  |"
          "1 | []"
          "  |   ^"
          "Empty list literal without annotation"
 but got: "*** Exception: "
          "\ESC[1;31mError\ESC[0m: Invalid input"
          ""
          "(input):1:3:"
          "  |"
          "1 | []"
          "  |   ^"
          "Empty list literal without annotation"
          ""

I thought the problem was with the escape codes on the second line, not realizing that the ... handled these alright.

Gabriel then suggested that the real problem was the missing trailing space on the first line, but that turned out not to be a problem.

The actual problem turned out to be the empty last line ("") which I addressed by adding a ... line.

I think it would be good if doctest could be clearer where exactly the mismatch is.

Ideally doctest would even output a fixed version of the expected output, or have a mode where it updates the expected output directly in the source file.

[quasicomputational]

Ideally doctest would even output a fixed version of the expected output, or have a mode where it updates the expected output directly in the source file.

I don't think that's possible to do in general, because the 'fixed version' is under-determined - replacing the previous expected output with a single wildcard would 'work' in that the tests would pass, but it's quite obviously not what you'd want to happen. It might be possible to do this usefully in some common cases with some heuristics (e.g., if the match failed due to extra lines at the end, append those), but it'd take a bit of thought and design.

There's already some affordance for pointing to the specific divergence where it occurs within a single line, but, yeah, this isn't a great experience where the divergence is due to extra lines at the end. Looking at the code for matching (https://github.com/sol/doctest/blob/master/src/Runner/Example.hs), I think that case might be the only really bad UX, and I think that it shouldn't be too hard to do something smarter like specifically pointing to the extra lines dangling off the end.

[sjakobi]

Thanks for looking into this, @quasicomputational! :)

Looking at the code for matching (https://github.com/sol/doctest/blob/master/src/Runner/Example.hs), I think that case might be the only really bad UX, and I think that it shouldn't be too hard to do something smarter like specifically pointing to the extra lines dangling off the end.

Sounds good to me!