h5py 2.10.0 is incompatible with numpy 1.24.1

[tueda]

The current image has h5py 2.10.0 (specified in requirements.txt), which is incompatible with numpy 1.24.1 installed in the image:

docker run -it --rm madminertool/madminer-jupyter-env:0.3.5 /usr/bin/python3

Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import madminer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/madminer/__init__.py", line 1, in <module>
    from .analysis import DataAnalyzer
  File "/usr/local/lib/python3.8/dist-packages/madminer/analysis/__init__.py", line 1, in <module>
    from .dataanalyzer import DataAnalyzer
  File "/usr/local/lib/python3.8/dist-packages/madminer/analysis/dataanalyzer.py", line 5, in <module>
    from madminer.utils.interfaces.hdf5 import load_events
  File "/usr/local/lib/python3.8/dist-packages/madminer/utils/interfaces/hdf5.py", line 13, in <module>
    import h5py
  File "/usr/local/lib/python3.8/dist-packages/h5py/__init__.py", line 46, in <module>
    from ._conv import register_converters as _register_converters
  File "h5py/h5t.pxd", line 14, in init h5py._conv
  File "h5py/h5t.pyx", line 293, in init h5py.h5t
  File "/usr/local/lib/python3.8/dist-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'typeDict'

Maybe numpy version could be fixed to an older one, as in https://stackoverflow.com/q/74852225.

[matthewfeickert]

@irinaespejo @Sinclert If I fix this can I bump up the release number to 0.3.6 or will that not get picked up by workflows? I guess another way of saying this is what needs to get updated anytime there's a new release of an image?

[Sinclert]

Hey @tueda, thanks for your report.

@matthewfeickert Feel free to do so. IIRC this Docker image is only used within the madminer-tutorial repository. You would need to update every reference to the new tag, similar to what this commit did.

[yowdz0709]

I am facing the same issue and now I cannot downgrade numpy, some common errors happen when using the function delphes.analyse_delphes_samples(), any suggestions?

Here is my error message:

Traceback (most recent call last): File "/Users/younes/Desktop/madminer/examples/tutorial_particle_physics/delphes_level.py", line 77, in <module> _ = plot_distributions( ^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/plotting/distributions.py", line 134, in plot_distributions sa = SampleAugmenter(filename, include_nuisance_parameters=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/sampling/sampleaugmenter.py", line 74, in __init__ super().__init__(filename, disable_morphing, include_nuisance_parameters) File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/analysis/dataanalyzer.py", line 53, in __init__ ) = load_madminer_settings(filename, include_nuisance_benchmarks=include_nuisance_parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/utils/interfaces/hdf5.py", line 63, in load_madminer_settings analysis_params = _load_analysis_params(file_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/madminer/utils/interfaces/hdf5.py", line 872, in _load_analysis_params with h5py.File(file_name, "r") as file: ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/h5py/_hl/files.py", line 567, in __init__ fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/h5py/_hl/files.py", line 231, in make_fid fid = h5f.open(name, flags, fapl=fapl) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 106, in h5py.h5f.open OSError: Unable to open file (file signature not found)

[Sinclert]

👋🏻 Apologies for the delay on the resolution.

I decided to forward fix the h5py - numpy compatibility issue, by removing the h5py requirement from the madminer-jupyter-env requirements.txt file in commit 2db21db (it must be exclusively installed as a madminer dependency).

Please, @tueda , could you try madminer-jupyter-env v0.3.6 image and let us know if your issue is solved?

[tueda]

Thank you for the fix. I have tried the 0.3.6 image and it worked: at least import madminer does not give any errors.

[tueda]

Recently, my collaborator told me that the Docker image does not work properly with the physics tutorial (part 2A). It turned out that miner.run() fails silently because MG5_aMC_v2.9.4 is not compatible with NumPy 1.24.4.
Log file: mg_processes/signal1/run_01_tag_1_debug.log).

See also this bug report and the corresponding fix commit. Upgrading the MG5_aMC to 2.9.16+ could be a potential fix.

[Sinclert]

Thanks for the bug report. I just pushed image version 0.3.7 containing the described MadGraph version upgrade.

[tueda]

Thanks for uploading 0.3.7. Unfortunately, the image still has some problems.

When you run /madminer/software/MG5_aMC_v2_9_16/bin/mg5_aMC, you see:

    A Pythia8 path is specified via the option 'pythia8_path' but no path for option
    'mg5amc_py8_interface_path' is specified. This means that Pythia8 cannot be used
    leading order simulations with MadEvent.
    Consider installing the MG5_aMC-PY8 interface with the following command:
     MG5_aMC>install mg5amc_py8_interface

This causes the physics tutorial (part 2B) to fail because miner.run() silently fails (log).

Actually, it seems that installing mg5amc_py8_interface failed during the build of the image (RUN echo "install pythia8" | python3 ${MG_BINARY_PATH}). When you look at /madminer/software/MG5_aMC_v2_9_16/HEPTools/MG5aMC_PY8_interface/mg5amc_py8_interface_install.log, you find:

MG5aMC_PY8_interface compilation output log:
b"g++ MG5aMC_PY8_interface.cc -o MG5aMC_PY8_interface -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/include  -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//include -ldl -fPIC -lstdc++ -std=c++11 -O2 -DHEPMC2HACK -DGZIP -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/include -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/lib -lz -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib -lpythia8 -ldl -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/lib -lHepMC\n/usr/bin/ld: /madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib/libpythia8.so: undefined reference to `pthread_create'\ncollect2: error: ld returned 1 exit status\nmake: *** [Makefile_mg5amc_py8_interface_static:8: MG5aMC_PY8_interface] Error 1\n"
------------------------------------------------
Error during the compilation of MG5aMC_PY8_interface:
b"g++ MG5aMC_PY8_interface.cc -o MG5aMC_PY8_interface -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/include  -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//include -ldl -fPIC -lstdc++ -std=c++11 -O2 -DHEPMC2HACK -DGZIP -I/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/include -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/zlib/lib -lz -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib -lpythia8 -ldl -L/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/lib -Wl,-rpath,/madminer/software/MG5_aMC_v2_9_16/HEPTools/hepmc/lib -lHepMC\n/usr/bin/ld: /madminer/software/MG5_aMC_v2_9_16/HEPTools/pythia8//lib/libpythia8.so: undefined reference to `pthread_create'\ncollect2: error: ld returned 1 exit status\nmake: *** [Makefile_mg5amc_py8_interface_static:8: MG5aMC_PY8_interface] Error 1\n"

This appear to be the pthread issue raised in here, here or here.

I'm not sure why this pthread issue occurred in your build; currently, if I build the image locally, then the compilation of MG5aMC_PY8_interface successfully completes. (But I had the same in the past with Ubuntu images; perhaps due to the layer cache??)

If this pthread issue can be fixed, then the next problem is in the MadMiner physics tutorial (part 4A), where limits.expected_limits() eventually calls np.histogramdd() with normed in Histo._fit(). This parameter has been removed since NumPy 1.24.0.

[Sinclert]

Thanks again for the detailed bug report.

At this stage, I would say you know more about what is going wrong with the tutorial Docker image, than myself, or any other person associated to this project (which has been unmaintained for a few years now).

Seems like the easiest path forward would be for you to open a PR, and for us to review it. Feel free to tag me once you do.

[tueda]

Maybe the first step would be adding CI, which checks if changes cause further problems: #10.