Many unit test failures on local setup

[rockobonaparte]

I had previously put up a few PRs around the time of the PyCon dev sprints without being able to locally run the unit tests, but now I'm trying to run them before advancing because I'm starting to seriously rearrange the furniture in the transpiler and AST visitor code. I had a ton of failures, but they were all still there when I rolled back to my last upstream version.

Here's the executive summary:
Ran 14192 tests in 3613.337s

FAILED (failures=1461, expected failures=971, unexpected successes=3)

I dumped everything out to a text file but it was 8MB, so up on DropBox it went:
https://www.dropbox.com/s/1cu1f9pootfp9kt/testlog.txt?dl=0

It looks like I'm the unique snowflake doing stuff on Windows, so I am going to go over the environment:
Windows 10 Pro
Python 3.6.5 64-bit (from Python.org, not ActiveState, not custom build)

I'm running the Oracle JVM and the version string is:
java version "10.0.1" 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)

My first impression is that it's a lot of Unicode-related errors. During the dev sprint we just floated my changes up to the build server and the CI environment passed the tests. My laptop was falling apart and would have never run those tests in any sane time span anyways. I'd like to note I'm on my development desktop now, so we can rule out shenanigans there.

I thought @freakboy3742 might have brought up that something was up with the unit tests, but I can't remember among the clutter of that week. At this point I want to try to get them going because I'm probably about to extend into them for an MSIL transpiler, and I don't look forward to wading through real failures versus false negatives.

[freakboy3742]

AFIAK, the unit test suite is stable, and currently passing.

My initial impression is that this is an encoding issue with the terminal you're running the tests in. Codepage handling on Windows is a tricky thing; I don't know if the problem here is with VOC, our test harness, or something in between. We've had problems in the past with users on non-English Windows installs. I can only assume something similar is going on here.

[rockobonaparte]

Thanks for this prompt heads up here. I'll just pick one of the simpler tests complaining about something with Unicode and go from there.

Given at PyCon I think I was the only freak running Windows and my edits came about from trying to run on Windows, I'm assuming it's not really a target for any of this work right now. I'm going to expect to bump into some Windows shenanigans in the process of walking through the test, there will be a groan, and then I'll post back.

In my case, I was just running good ol' cmd.exe--without Administrative privileges. I wasn't even trying to plow through the unit tests using PyCharm.

[freakboy3742]

Windows support isn't something I am actively working on - but that doesn't mean it's something we're not interested in. Windows is clearly an important platform, it's just not the platform that is my daily driver. From my perspective, having something that is 100% complete on one platform is more important than having something 20% complete on multiple platforms.

Having someone who is motivated to look into Windows issues is definitely helpful to the project as a whole.

[rockobonaparte]

Oh you don't have to defend that to me. It was pretty obvious at the dev sprints that it wasn't anybody else's daily driver either.

I decided to try to replicate in PyCharm so I could step through the stuff in a handful of failures. Of course the handful I picked passed just fine haha.

I'm going to let PyCharm run a regression and see if there's any intersectionality. I'm going to bet no; they're probably going to pass. If that's the case, then I think your hunch about the terminal is about right. It would have to do with cmd.exe. Then I'll have to use pdb or set up the tests to attach.

[rockobonaparte]

PyCharm had completely different results on Windows:
14192 tests done
971 ignored
11 failed

I was looking at the failures and I think PyCharm is just botching them. I think you mentioned not using PyCharm so that's not a surprise. For example, it failed builtins.test_pow.BuiltinTwoargPowFunctionTests.test_complex_complex. If I'm following test_pow.py, that's in a not_implemented block, so I'm going to assume that means it isn't supposed to actually be run. In the same family is a failing test_int_complex due to a really goofy floating-point precision miscompare, and I don't even see that test case anywhere in a recursive search of the source tree.

Anyways I'm inclined to close the issue even though it would be a nice spot to correspond on some of this. It's just that the problems vary so keeping this issue open means it could be open for a long time.