import error

[fedecaccia]

Using numpy 1.15.0 and last dragnet version I can't import dragnet from python3:

Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dragnet
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/local/lib/python3.5/dist-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d

neither from python2:

Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from dragnet import features
 File "/usr/local/lib/python2.7/dist-packages/dragnet/features/__init__.py", line 4, in <module>
   from dragnet.features.readability import ReadabilityFeatures
 File "/usr/local/lib/python2.7/dist-packages/dragnet/features/readability.py", line 3, in <module>
   from ._readability import make_readability_features
 File "dragnet/features/_readability.pyx", line 3, in init dragnet.features._readability
 File "__init__.pxd", line 1000, in numpy.import_array
ImportError: numpy.core.multiarray failed to import

[pakelley]

with fresh environments I'm not able to reproduce these errors :/
I do get the sklearn DeprecationWarning, but that's it.
Are you still running into this, @fedecaccia ?

FWIW, here are my pip list outputs:
For Python 3.6:

Package          Version
---------------- ---------
appnope          0.1.0
backcall         0.1.0
certifi          2018.8.24
Cython           0.28.5
decorator        4.3.0
dragnet          2.0.3
ftfy             4.4.3
html5lib         1.0.1
ipython          7.0.1
ipython-genutils 0.2.0
jedi             0.13.1
lxml             4.2.5
numpy            1.15.2
parso            0.3.1
pexpect          4.6.0
pickleshare      0.7.5
pip              18.1
prompt-toolkit   2.0.5
ptyprocess       0.6.0
Pygments         2.2.0
scikit-learn     0.19.2
scipy            1.1.0
setuptools       40.4.3
simplegeneric    0.8.1
six              1.11.0
traitlets        4.3.2
wcwidth          0.1.7
webencodings     0.5.1
wheel            0.32.1

For Python 2.7:

Package                            Version
---------------------------------- ---------
appnope                            0.1.0
backports.shutil-get-terminal-size 1.0.0
certifi                            2018.8.24
Cython                             0.28.5
decorator                          4.3.0
dragnet                            2.0.3
enum34                             1.1.6
ftfy                               4.4.3
html5lib                           1.0.1
ipython                            5.8.0
ipython-genutils                   0.2.0
lxml                               4.2.5
numpy                              1.15.2
pathlib2                           2.3.2
pexpect                            4.6.0
pickleshare                        0.7.5
pip                                18.1
prompt-toolkit                     1.0.15
ptyprocess                         0.6.0
Pygments                           2.2.0
scandir                            1.1
scikit-learn                       0.19.2
scipy                              1.1.0
setuptools                         40.4.3
simplegeneric                      0.8.1
six                                1.11.0
traitlets                          4.3.2
wcwidth                            0.1.7
webencodings                       0.5.1
wheel                              0.32.1

[bdewilde]

Hello! FWIW I also get this error in a Python 2 environment:

Python 2.7.10 (default, Aug 17 2018, 17:41:52)
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from dragnet import features
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-3705de598490> in <module>()
----> 1 from dragnet import features

/Users/burtondewilde/.pants.d/python-setup/chroots/6a8bbfd8688faa57326b41c0ae9cdf9cbd977473/.deps/dragnet-2.0.3-cp27-cp27m-macosx_10_13_x86_64.whl/dragnet/__init__.py in <module>()
      1 from dragnet.blocks import Blockifier, PartialBlock, BlockifyError
----> 2 from dragnet import features
      3 from dragnet.extractor import Extractor
      4 from dragnet.util import load_pickled_model
      5

/Users/burtondewilde/.pants.d/python-setup/chroots/6a8bbfd8688faa57326b41c0ae9cdf9cbd977473/.deps/dragnet-2.0.3-cp27-cp27m-macosx_10_13_x86_64.whl/dragnet/features/__init__.py in <module>()
      2 from dragnet.features.css import CSSFeatures
      3 from dragnet.features.kohlschuetter import KohlschuetterFeatures
----> 4 from dragnet.features.readability import ReadabilityFeatures
      5 from dragnet.features.weninger import WeningerFeatures, ClusteredWeningerFeatures
      6

/Users/burtondewilde/.pants.d/python-setup/chroots/6a8bbfd8688faa57326b41c0ae9cdf9cbd977473/.deps/dragnet-2.0.3-cp27-cp27m-macosx_10_13_x86_64.whl/dragnet/features/readability.py in <module>()
      1 from sklearn.base import BaseEstimator, TransformerMixin
      2
----> 3 from ._readability import make_readability_features
      4
      5

dragnet/features/_readability.pyx in init dragnet.features._readability()

__init__.pxd in numpy.import_array()

ImportError: numpy.core.multiarray failed to import

In [2]: import numpy as np

In [3]: np.__version__
Out[3]: '1.13.3'

Unfortunately, this is a production environment over which I have little control. Locally, I can create a fresh environment for dragnet, which imports with a warning but no error:

Python 2.7.15 (default, Jun 28 2018, 10:19:24)
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from dragnet import features
/Users/burtondewilde/.pyenv/versions/2.7.15/envs/dragnet-test/lib/python2.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d

In [2]: import numpy as np

In [3]: np.__version__
Out[3]: '1.15.4'

If I manually downgrade the version of numpy to that used in the production environment (1.13.3), I provoke that same error locally. Does dragnet need to set a lower bound on numpy version in requirements.txt?

[pakelley]

I can import dragnet and tests pass for me on numpy 1.13.3 as well 😕
Here's what I get in ipython:

Python 2.7.15 | packaged by conda-forge | (default, Jul 27 2018, 10:23:44)
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from dragnet import features

In [2]: import numpy as np

In [3]: np.__version__
Out[3]: '1.13.3'

@bdewilde would you mind sharing a pip list here so I can make sure I'm using the same packages as you? Also, what OS are you running this on?

[bdewilde]

Hi @pakelley , I just went through the process once more in a fresh environment and reproduced the result — you had me doubting myself! 😅

I'm using Python 2.7.15 on macOS. Here are the packages in this virtualenv:

(dragnet-test) ~$ pip list
Package      Version
------------ -------
Cython       0.29
dragnet      2.0.3
ftfy         4.4.3
html5lib     1.0.1
lxml         4.2.5
numpy        1.13.3
pip          18.1
scikit-learn 0.19.2
scipy        1.1.0
setuptools   40.5.0
six          1.11.0
wcwidth      0.1.7
webencodings 0.5.1
wheel        0.32.2

[pakelley]

ugh, I was having the worst time trying to reproduce this until I realized I was working inside the dragnet repo... so it was just importing straight from source 😓 Working in another directory gives me that ImportError.
When I build the package on numpy 1.13.3 the package works fine though. I figure we could either host a pypi package for older versions of numpy or just drop support for those older versions and add a minimum numpy version. WRT hosting another package, I don't have access to the PyPI repo and don't know the workflow for hosting different builds of packages, so I don't know if I can be much help with that if that's the better route.