PyFlask failures - January 2024

[CodyCBakerPhD]

Test suite applied to built versions of the app on Mac (intel) and ubuntu have been failing for past few days

After analyzing differences between passing to failing environments as of 5 days ago, the only difference is boto3 and botocore (the only package we use that cut a new release on Jan 5)

Pinning those in https://github.com/NeurodataWithoutBorders/nwb-guide/actions/runs/7452039831/job/20274447966?pr=558 did not fix the issue however

Core error has something to do with the mysterious .defs clib missing from https://github.com/h5py/h5py/blob/master/h5py/h5.pyx#L14C2-L14C21 of h5py

Attempts to collect-all on h5py and h5py.defs have failed: #557 & #556

[CodyCBakerPhD]

@garrettmflynn Are you able to reproduce these issues on a local build with your mac?

[CodyCBakerPhD]

The biggest mystery of all is of course 'why now'? since there were no major obvious changes to h5py around that time

[CodyCBakerPhD]

Was confused why the .defs import works if it's not in the h5py lib, turns out it's generated on the fly: https://github.com/h5py/h5py/blob/4c01efa9714db40ffe27a322c4f1ba4635816e44/api_gen.py#L180-L181

[garrettmflynn]

Just reproduced on my M2.

Was confused why the .defs import works if it's not in the h5py lib, turns out it's generated on the fly

Huh, that's weird.

[CodyCBakerPhD]

@garrettmflynn On your M2, if you dig into the site-packages for the nwb-guide conda environment, and find h5py, do you see anything resembling a defs file? (for me, for example, it's defs.cp311-win_amd64.pyd)

[rly]

Some forum posts suggest we need to import all of these:

     datas=[],
     hiddenimports=[
       'h5py.defs',
       'h5py.utils',
       'h5py.h5ac',
       'h5py._proxy',
     ],
     hookspath=[],

https://stackoverflow.com/questions/70205979/modulenotfounderror-no-module-named-h5py-utils
https://stackoverflow.com/a/45842873/20177
pyinstaller/pyinstaller#2540

However, when I do so in #557, I get

ImportError: cannot import name 'Blosc' from 'numcodecs' (/home/runner/work/nwb-guide/nwb-guide/build/flask/nwb-guide/numcodecs/__init__.pyc)

The fact that the ImportError is now in a different package suggests a more fundamental problem. Perhaps something changed in pyinstaller that changes how it finds imports / dependencies from the python code and packages them.

[garrettmflynn]

@CodyCBakerPhD Yep! lib/python3.9/site-packages/h5py/defs.cpython-39-darwin.so

[CodyCBakerPhD]

@rly Looks like you're ahead of us and on the right track then - can you keep hammering away at including hiddenImports for numcodecs and anything beyond that on your PR?

The fact that the ImportError is now in a different package suggests a more fundamental problem. Perhaps something changed in pyinstaller that changes how it finds imports / dependencies from the python code and packages them.

I would agree, but this only compounds the mystery since we hard pin our PyInstaller version throughout all environments

Yep! lib/python3.9/site-packages/h5py/defs.cpython-39-darwin.so

@garrettmflynn OK and then can you check the same exact place in the PyInstaller bundle (in the build)?

[rly]

@CodyCBakerPhD I'll poke around and see what I can figure out

[CodyCBakerPhD]

@rly Figured out the mystery

Culprit was https://pypi.org/project/pyinstaller-hooks-contrib/#history