Issue with running first time index on macOS #122

flinge · 2020-07-02T14:03:09Z

I've installed libmagic as per https://filemagic.readthedocs.io/en/latest/guide.html

denis@MacBook-Pro ~> promnesia index
/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/kython/klogging2.py:27: UserWarning: You might want to install 'logzero' for nice colored logs!
  warnings.warn("You might want to install 'logzero' for nice colored logs!")
[INFO    2020-07-02 14:57:20,143 promnesia common.py:417] extracting via promnesia.sources.auto:index ('/usr/share/doc/python3/html/faq',) {} ... ...
[INFO    2020-07-02 14:57:20,143 promnesia common.py:417] extracting via promnesia.sources.auto:index ('/usr/share/doc/python3/html/faq',) {} ... ...
[ERROR   2020-07-02 14:57:20,149 promnesia common.py:433] indexer emitted exception
Traceback (most recent call last):
  File "/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/sources/auto.py", line 291, in _index
    yield from _index_file(pp, opts=opts)
  File "/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/sources/auto.py", line 317, in _index_file
    pm = mime(pp)
  File "/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/sources/auto.py", line 26, in mime
    return _magic().from_file(str(path))
  File "/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/sources/auto.py", line 22, in _magic
    return magic.Magic(mime=True)
TypeError: __init__() got an unexpected keyword argument 'mime'

[ERROR   2020-07-02 14:57:20,149 promnesia common.py:433] indexer emitted exception
Traceback (most recent call last):
  File "/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/sources/auto.py", line 291, in _index
    yield from _index_file(pp, opts=opts)
  File "/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/sources/auto.py", line 317, in _index_file
    pm = mime(pp)
  File "/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/sources/auto.py", line 26, in mime
    return _magic().from_file(str(path))
  File "/Users/denis/Library/Python/3.7/lib/python/site-packages/promnesia/sources/auto.py", line 22, in _magic
    return magic.Magic(mime=True)
TypeError: __init__() got an unexpected keyword argument 'mime'

[INFO    2020-07-02 14:57:20,149 promnesia common.py:443] extracting via promnesia.sources.auto:index ('/usr/share/doc/python3/html/faq',) {} ...: got 0 visits
[INFO    2020-07-02 14:57:20,149 promnesia common.py:443] extracting via promnesia.sources.auto:index ('/usr/share/doc/python3/html/faq',) {} ...: got 0 visits
[INFO    2020-07-02 14:57:20,167 promnesia dump.py:51] saved database to /Users/denis/Library/Application Support/promnesia/promnesia.sqlite
[INFO    2020-07-02 14:57:20,167 promnesia dump.py:51] saved database to /Users/denis/Library/Application Support/promnesia/promnesia.sqlite

The text was updated successfully, but these errors were encountered:

clintgibler · 2020-07-03T23:36:33Z

I'm observing the same behavior:

macOS Catalina - 10.15.5
Python 3.7.7

$ pip3 show filemagic
Name: filemagic
Version: 1.6
Summary: A Python API for libmagic, the library behind the Unix file command
Home-page: http://filemagic.readthedocs.org
Author: Aaron Iles
Location: /Users/clint/Library/Python/3.7/lib/python/site-packages

@karlicoss which version of filemagic are you using?

clintgibler · 2020-07-03T23:37:14Z

Reviewing the link @flinge shared above, it appears the filemagic API has changed. Or at least, updating to code based on the Usage text in the filemagic README fixed the indexing errors for me, see PR #124.

karlicoss · 2020-07-05T13:38:36Z

Hi everyone! Sorry for a late reply, busy days...

So, promnesia is actually using python-magic.

promnesia/setup.py

Line 67 in 8fbaa66

'python-magic', # for detecting mime types

I didn't know of filemagic, but python-magic seems to be more mature (and kept up to date)? Any reason you tried to use filemagic instead?

Let me know if python-magic works for you (possibly you'd need to install filemagic first to avoid name conflicts)!

wadkar · 2020-07-05T17:27:14Z

I am putting this hear because future users might stumble upon this thread due to filemagic libmagic and python-magic related keywords in this thread and also ran into below error when I executed promnesia index for the first time:

❮ promnesia index
[INFO    2020-07-05 22:37:23,902 promnesia common.py:417] extracting via promnesia.sources.auto:index ('/usr/share/doc/python3/html/faq',) {} ... ...
[ERROR   2020-07-05 22:37:23,909 promnesia common.py:433] indexer emitted exception
    Traceback (most recent call last):
      File "/Users/wadkar/.pyenv/versions/3.8.3/envs/promenesia/lib/python3.8/site-packages/promnesia/sources/auto.py", line 291, in _index
        yield from _index_file(pp, opts=opts)
      File "/Users/wadkar/.pyenv/versions/3.8.3/envs/promenesia/lib/python3.8/site-packages/promnesia/sources/auto.py", line 317, in _index_file
        pm = mime(pp)
      File "/Users/wadkar/.pyenv/versions/3.8.3/envs/promenesia/lib/python3.8/site-packages/promnesia/sources/auto.py", line 26, in mime
        return _magic().from_file(str(path))
      File "/Users/wadkar/.pyenv/versions/3.8.3/envs/promenesia/lib/python3.8/site-packages/promnesia/sources/auto.py", line 21, in _magic
        import magic # type: ignore
      File "/Users/wadkar/.pyenv/versions/3.8.3/envs/promenesia/lib/python3.8/site-packages/magic.py", line 201, in <module>
        raise ImportError('failed to find libmagic.  Check your installation')
    ImportError: failed to find libmagic.  Check your installation

There's line in your scrips/ci/run which says to brew install libmagic and it's also easy to figure out if you search the interweb. Let me know if you want me to send in a PR to clarify in the docs or edit the wiki (you'll have to enable wiki for the repo).

~~(Edit1: spoke too soon; bumped into another error - but it's not related to this one, so creating another issue)~~
(Edit2: Never mind, false alarm: it's the default config.py with a Source('/usr/share/doc/python3/html/faq') line which throws FileNotFoundError on macOS which expected, commenting out that line does the trick)

clintgibler · 2020-07-05T20:24:28Z

@karlicoss no problem at all, thanks for your response!

Uninstalling filemagic, installing python-magic, and running scripts/promnesia index from master (git commit 8fbaa66ea9dc94fa2475530cd74ef52d0305111e) works for me 👍

To help future people coming across this issue, would it be useful to update the README to indicate that if you're on macOS and you're getting errors related to magic, make sure to pip3 install python-magic, not libmagic?

You could also add a requirements.txt or something. setup.py lists it, as you commented above, but that might not be as obvious to people. At least for me, I look for a requirements.txt first, but I'm not an experienced Python dev :)

karlicoss · 2020-07-05T20:45:38Z

@wadkar hm, so as far as I understand (don't have a mac, so was doing it semi-blind :) ), libmagic is the C library, whereas python-magic and filemagic are simply python bindings to it. I guess I could suggest installing libmagic in the docs, good idea.

@clintgibler cool! So, setup.py already contains the dependencies (hence no need for requirements.txt). They can be installed with setup.py install --user -e . ('editable' install). I guess I'll add it to the readme.

wadkar · 2020-07-05T20:54:03Z

@karlicoss
Yes, did a quick brew uninstall libmagic and then installed python-magic in a fresh python3 virtualenv. import magic threw same error. And it looks like the python-magic docs seem to agree with it.

Also, thanks for this - I need to do a proper SOURCES setup, but the demo looks cool 😄

(Edit: Just to add, it seems that the docs say that even Linux(Debian/Ubuntu) users need to do sudo apt-get install libmagic1 for python-magic to work. But then, you might as well install python bindings to libmagic - potato, potato 😆 )

karlicoss · 2020-07-05T21:31:40Z

@wadkar ah, right! I guess file is de-facto default on most linux distributions, so that's why I never even noticed that technically it's a dependency. I'll link to this in the documentation (and as a runtime warning), thanks!

santhosh82 · 2020-07-07T08:39:57Z

Uninstalling and installing a specific version of python-magic worked for me.

pip uninstall python-magic
pip install python-magic-bin==0.4.14

Source/Credits: Yelp/elastalert#1927 (comment)

karlicoss · 2020-07-11T16:12:49Z

@santhosh82 hmm, unlikely it's about the version, it's just a different package. Seems that python-magic-bin also includes the C library too, which makes it work. Generally it's preferable to install it from the standard package manager so you don't have random libraries around in your system. Does brew install libmagic work for you?

santhosh82 · 2020-07-12T00:29:12Z

I think I didn't try installing libmagic using brew.

karlicoss · 2020-07-12T20:19:26Z

Ok! Anyway, I added brew install libmagic to the readme, hopefully that will make it easier :)

clintgibler mentioned this issue Jul 3, 2020

Update filemagic to new API to fix meme crash #124

Closed

karlicoss added the documentation documentation/readme enhancements label Jul 11, 2020

karlicoss closed this as completed Jul 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with running first time index on macOS #122

Issue with running first time index on macOS #122

flinge commented Jul 2, 2020

clintgibler commented Jul 3, 2020

clintgibler commented Jul 3, 2020

karlicoss commented Jul 5, 2020

wadkar commented Jul 5, 2020 •

edited

Loading

clintgibler commented Jul 5, 2020 •

edited

Loading

karlicoss commented Jul 5, 2020

wadkar commented Jul 5, 2020 •

edited

Loading

karlicoss commented Jul 5, 2020

santhosh82 commented Jul 7, 2020

karlicoss commented Jul 11, 2020

santhosh82 commented Jul 12, 2020

karlicoss commented Jul 12, 2020

Issue with running first time index on macOS #122

Issue with running first time index on macOS #122

Comments

flinge commented Jul 2, 2020

clintgibler commented Jul 3, 2020

clintgibler commented Jul 3, 2020

karlicoss commented Jul 5, 2020

wadkar commented Jul 5, 2020 • edited Loading

clintgibler commented Jul 5, 2020 • edited Loading

karlicoss commented Jul 5, 2020

wadkar commented Jul 5, 2020 • edited Loading

karlicoss commented Jul 5, 2020

santhosh82 commented Jul 7, 2020

karlicoss commented Jul 11, 2020

santhosh82 commented Jul 12, 2020

karlicoss commented Jul 12, 2020

wadkar commented Jul 5, 2020 •

edited

Loading

clintgibler commented Jul 5, 2020 •

edited

Loading

wadkar commented Jul 5, 2020 •

edited

Loading