error message: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

[kloczek]

Can someone help with simplejson/simplejson#281

pytest in this case shows series of errors like below:

================================================================================== ERRORS ==================================================================================
________________________________________________________ ERROR collecting simplejson/tests/test_bigint_as_string.py ________________________________________________________
import file mismatch:
imported module 'simplejson.tests.test_bigint_as_string' has this __file__ attribute:
  /usr/lib64/python3.8/site-packages/simplejson/tests/test_bigint_as_string.py
which is not the same as the test file we want to collect:
  /home/tkloczko/rpmbuild/BUILD/simplejson-3.17.2/simplejson/tests/test_bigint_as_string.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

[nicoddemus]

I can explain the message in more detail:

pytest is trying to import/collect this file:

/home/tkloczko/rpmbuild/BUILD/simplejson-3.17.2/simplejson/tests/test_bigint_as_string.py

However after importing it, it inspects the __file__ attribute and it points to another file:

/usr/lib64/python3.8/site-packages/simplejson/tests/test_bigint_as_string.py

In other words, you are trying to execute test_bigint_as_string.py, but that was already imported from /usr/lib64 somehow.

You already have simplejson installed in the system site-packages, and somehow the test from the system site-packages is already imported. Not sure how that's happening, links might be involved perhaps?

[hroncok]

Anyway, I don't think this has to do anything with rpm. I think this is simply a conflict of "tests files from local checkout" vs. "installed test modules of the same name". I.e. as @RonnyPfannschmidt says, you should run the tests with --pyargs. I don't know why we don't hit this problem with simplejson in Fedora -- https://kojipkgs.fedoraproject.org//packages/python-simplejson/3.17.2/6.fc35/data/logs/x86_64/build.log

[RonnyPfannschmidt]

@hroncok i beleive you dont hit the issue thanks to a PYTHONDONTWRITEBYTECODE - unless assert rewrite is also disabled, its easy to miss as a new pyc will be made

i suspect a number of packages may be affected (all which install tests and need to run installed tests)

[kloczek]

[..]

the correct invocation would use pytest --pyargs simplejson.tests when using that PYTHONPATH

@nicoddemus i believe the import errors for conflicts need to get better when its about pythonpath vs pytest managed additional path

Just tested that and ItWorks™️😀

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-simplejson-3.17.2-7.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-simplejson-3.17.2-7.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra --pyargs simplejson.tests
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/tkloczko/rpmbuild/BUILD/simplejson-3.17.2
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, toolbox-0.5, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, pyfakefs-4.5.0, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, hypothesis-6.14.4, xprocess-0.18.1, black-0.3.12, checkdocs-2.7.1, anyio-3.3.0, Faker-8.11.0, asyncio-0.15.1, trio-0.7.0, httpbin-1.0.0, subtests-0.5.0
collected 140 items

test_bigint_as_string.py ....                                                                                                                                        [  2%]
test_bitsize_int_as_string.py ......                                                                                                                                 [  7%]
test_check_circular.py .....                                                                                                                                         [ 10%]
test_decimal.py ......                                                                                                                                               [ 15%]
test_decode.py ...........                                                                                                                                           [ 22%]
test_default.py .                                                                                                                                                    [ 23%]
test_dump.py ..............                                                                                                                                          [ 33%]
test_encode_basestring_ascii.py ...                                                                                                                                  [ 35%]
test_encode_for_html.py ....                                                                                                                                         [ 38%]
test_errors.py .....                                                                                                                                                 [ 42%]
test_fail.py ...                                                                                                                                                     [ 44%]
test_float.py .....                                                                                                                                                  [ 47%]
test_for_json.py ...........                                                                                                                                         [ 55%]
test_indent.py ...                                                                                                                                                   [ 57%]
test_item_sort_key.py ...                                                                                                                                            [ 60%]
test_iterable.py .                                                                                                                                                   [ 60%]
test_namedtuple.py ......                                                                                                                                            [ 65%]
test_pass1.py .                                                                                                                                                      [ 65%]
test_pass2.py .                                                                                                                                                      [ 66%]
test_pass3.py .                                                                                                                                                      [ 67%]
test_raw_json.py ....                                                                                                                                                [ 70%]
test_recursion.py ...                                                                                                                                                [ 72%]
test_scanstring.py .....                                                                                                                                             [ 75%]
test_separators.py .                                                                                                                                                 [ 76%]
test_speedups.py .......                                                                                                                                             [ 81%]
test_str_subclass.py .                                                                                                                                               [ 82%]
test_subclass.py ..                                                                                                                                                  [ 83%]
test_tool.py ...                                                                                                                                                     [ 85%]
test_tuple.py ..                                                                                                                                                     [ 87%]
test_unicode.py ..................                                                                                                                                   [100%]

=========================================================================== 140 passed in 1.01s ============================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.

Thank you very much😀
I'm going to test that modification on other packages where I see exactly the same flood of errors.

Q: is it possible to embed that pytest switch in pytest.ini? IMO it would be the best way to sort out that kind of cases (cannot find any details about that).

[RonnyPfannschmidt]

Embedding it would be very incorrect and break most selective testing

[kloczek]

@kloczek additionally this may be a detail with the rpm build process - it seems to store bytecode files intended to be compiled to be stored somewhere else into the build root, pytest currently doesn't handle that well

What is typical output of that class of effects?

[RonnyPfannschmidt]

@hroncok i identified a PYTHONDONTWRITEBYTECODE in the default rpm macros, which may or may not hide this issue in general, im under the impression the rpm python infra kind of manage this

however i'm also under the impression that @kloczek is building custom rpms using custom specs/macros and not disclosing those details, a quick look on the issues created seems to indicate such issues with dozens of packages

[kloczek]

@hroncok i identified a PYTHONDONTWRITEBYTECODE in the default rpm macros, which may or may not hide this issue in general, im under the impression the rpm python infra kind of manage this

however i'm also under the impression that @kloczek is building custom rpms using custom specs/macros and not disclosing those details, a quick look on the issues created seems to indicate such issues with dozens of packages

I'm using $PYTHONDONTWRITEBYTECODE as well in my rpm macros however I think that in most of the cases Fedora python modules packages don't know that this variable does not solve the issue.
Reason of that is very simple:

[tkloczko@barrel SPECS.fedora]$ grep ^%pytest python-* | wc -l; ls -1 python-* | wc -l
331
2630

in at least few cases using %__pythonn3 -Bm pytest in fedora spec files is masking that class of issues.
The same stats on my specs:

[tkloczko@barrel SPECS]$ grep ^%pytest python-* | wc -l; ls -1 python-* | wc -l
480
567

[kloczek]

however i'm also under the impression that @kloczek is building custom rpms using custom specs/macros and not disclosing those details, a quick look on the issues created seems to indicate such issues with dozens of packages

Indeed ..

As explanation .. I'm sometimes really tired talk to TheWall™️ of lack of understanding/lack of experience.
Trust me I've been really trying to disclosing that but typical answer that I'm using WrongWay™️ and nothing more really discouraged me to dig bigger hole around many subjects.
I've already been "awarder" ban by few gihub users .. ONLY because I've been c&p long output with errors (it was my fault that I've not added -q or -qq to shorten extra verbose default pytest output .. only then I've learned how to shorten than output to not store to much in build logs I've started asking why some pytest logs are soooo long 😋).
People sometimes are doing really strange things in python modules and they are blaming those who are showing them default output of they've own work.😋

My nose already is really quite flat and bloody by bouncing many times over that wall 😔

[RonnyPfannschmidt]

while i see where you are coming from

my perception of you is a user that posts hundreds of issues with incomplete debug details and/or rather tricky changes in packaging,
burning a lot of unwarranted time in dozens and dozens of upstream projects.

i also lost quite some time until i figured that you are in fact NOT using the upstreams your outputs indicate.

some would call that out your behaviour as downright abusive and i certainly can understand why they reject/ban you

personally I'm also at a point that i will not debug any more issues you send in,
as based on your github history my impression is that incomplete/misleading issues are common/regular occurrences.

I'm not willing to burn my personal time on that.

[kloczek]

personally I'm also at a point that i will not debug any more issues you send in,
as based on your github history my impression is that incomplete/misleading issues are common/regular occurrences

If you are thinking that I'm expecting to debug something for me that is completely wrong impression.

I have +25 tears log history packaging stuff (with rpm I've started at the time when it was written perl) and during that time I've been able learn many thing however even after so long time time to time I'm stumping on completely new areas where I must do something quickly without proper knowledge.
In that kind of situation all what I really need are suggestions in which direction possibly I can find the answer on some questions.
Problem with entering on such new areas is that initially I'm not even able to describe issue using typical set of terms by people familiars wit the subject with which I'm dealing.

[RonnyPfannschmidt]

i completely and utterly fail to understand how one files an issue against a software project without expecting someone to debug it and figure it out

[taraskuzyk]

In my case adding __init__.py to my test folder solved the issue for me, deleting __pycache__ was failing in the venv folder

[Hoozits728]

This did it for me! Thanks.

[Unalian]

Same. It works. Thanks.

[JohnAGS88]

in my case I delete the pycache folder and the error continues. I don't know what to do anymore!

[The-Compiler]

Consider opening a new discussion with the full output. With zero information, there's no way to help you.