pybloomfiltermmap stack overflow crash on startup - Mac OSX blocker #485
Comments
Just today I asked in out mailing list about the potential replacement of setuptools in mac.py. Do you think it should be applied to w3af's source?
The very ugly crash that should never happen (and it's the first time I see something similar) seems to be because of something in bloomfilter. As you can see in that code, we have different filter implementations and we try to use the most efficient first. The problem seems to be near (kind of guessing): try:
bf = CMmapFilter(1000, 0.01, temp_file)
bf.add(1)
assert 1 in bf
assert 2 not in bf
except:
WrappedBloomFilter = FileSeekFilter
else:
WrappedBloomFilter = CMmapFilterWhere that except doesn't (of course) catch errors at the C level. Try to change that file to force it to use WrappedBloomFilter = FileSeekFilter and let me know how it goes. |
|
Seems that you've found the issue. I tried simply replacing the entire block of code above with I did struggle getting everything installed correctly, so the problem may be in my Python installation. I'll have to try this again, starting with a cleaner machine. I should mention that I'm running this on Mac OSX. Regarding the py27-distribute issue... it seems there are several people with this problem. I found the workaround for that on a mailing list post by someone else running up against this issue a few days ago. (Looks like the workaround was provided by you.. so thanks again!) |
Well, that's good to know. I'll leave this open and try to think about a solution that's not very ugly.
|
|
Hey, thanks a lot for the help! |
|
Had the same problem, so confirmed for my Mac OSX system, workaround seems to work for this issue. |
|
By that I mean... we reported the issue upstream and are waiting for a solution. If you get that crash you could:
|
|
@axiak released a new version of pybloomfiltermmap at pypi. Could you guys see if it works for you on MAC? cd w3af
git pull
git checkout feature/module
git status # make sure you don't have any local changes
pip install --upgrade pybloomfiltermmap
./w3af_console # run a scan |
|
It seems that @axiak's version of pybloomfiltermmap has installation issues 👎 |
|
@axiak closed axiak/pybloomfiltermmap#34 , which was our original bug report for Mac OSX. It was closed without any comment (fixed? ignored?) and can't test without MAC OSX :( |
|
Well, after upgrading to 0.3.14 (which is supposed to fix the MAC issue) the scalable bloom filter break: pablo@eulogia:~/pch/w3af$ nosetests w3af/core/data/bloomfilter/tests/
...........FFF............FFF................................
======================================================================
FAIL: test_bloom_int (w3af.core.data.bloomfilter.tests.test_cmmap_bloom.TestCMmapBloomfilterLarge)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/tests/pylint_plugins/decorator.py", line 38, in test_only_subclass
return meth(self, *args, **kwds)
File "/home/pablo/PycharmProjects/w3af/w3af/core/data/bloomfilter/tests/generic_filter_test.py", line 61, in test_bloom_int
self.assertNotIn(r, self.filter)
AssertionError: 26794 unexpectedly found in <BloomFilter capacity: 20000, error: 0.001, num_hashes: 9>
======================================================================
FAIL: test_bloom_string (w3af.core.data.bloomfilter.tests.test_cmmap_bloom.TestCMmapBloomfilterLarge)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/tests/pylint_plugins/decorator.py", line 38, in test_only_subclass
return meth(self, *args, **kwds)
File "/home/pablo/PycharmProjects/w3af/w3af/core/data/bloomfilter/tests/generic_filter_test.py", line 79, in test_bloom_string
self.assertNotIn(saved_str[::-1], self.filter)
AssertionError: 'ViMcxIVqABeKvjRgYVtYxpbmycISEQtKXACvbmwQ' unexpectedly found in <BloomFilter capacity: 20000, error: 0.001, num_hashes: 9>
======================================================================
FAIL: test_bloom_url_objects (w3af.core.data.bloomfilter.tests.test_cmmap_bloom.TestCMmapBloomfilterLarge)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/tests/pylint_plugins/decorator.py", line 38, in test_only_subclass
return meth(self, *args, **kwds)
File "/home/pablo/PycharmProjects/w3af/w3af/core/data/bloomfilter/tests/generic_filter_test.py", line 95, in test_bloom_url_objects
self.assertNotIn(url_char, self.filter)
AssertionError: <URL for "http://moth/index20076.html"> unexpectedly found in <BloomFilter capacity: 20000, error: 0.001, num_hashes: 9>
======================================================================
FAIL: test_bloom_int (w3af.core.data.bloomfilter.tests.test_scalable_bloom.TestScalableBloomFilterLargeCmmap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/tests/pylint_plugins/decorator.py", line 38, in test_only_subclass
return meth(self, *args, **kwds)
File "/home/pablo/PycharmProjects/w3af/w3af/core/data/bloomfilter/tests/generic_filter_test.py", line 61, in test_bloom_int
self.assertNotIn(r, self.filter)
AssertionError: 22472 unexpectedly found in <w3af.core.data.bloomfilter.scalable_bloom.ScalableBloomFilter object at 0x3258ad0>
======================================================================
FAIL: test_bloom_string (w3af.core.data.bloomfilter.tests.test_scalable_bloom.TestScalableBloomFilterLargeCmmap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/tests/pylint_plugins/decorator.py", line 38, in test_only_subclass
return meth(self, *args, **kwds)
File "/home/pablo/PycharmProjects/w3af/w3af/core/data/bloomfilter/tests/generic_filter_test.py", line 79, in test_bloom_string
self.assertNotIn(saved_str[::-1], self.filter)
AssertionError: 'kcgJhOFSTurBYeSsNenhJsXWKmnUeTfsUnyzXbEw' unexpectedly found in <w3af.core.data.bloomfilter.scalable_bloom.ScalableBloomFilter object at 0x3258bd0>
======================================================================
FAIL: test_bloom_url_objects (w3af.core.data.bloomfilter.tests.test_scalable_bloom.TestScalableBloomFilterLargeCmmap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/tests/pylint_plugins/decorator.py", line 38, in test_only_subclass
return meth(self, *args, **kwds)
File "/home/pablo/PycharmProjects/w3af/w3af/core/data/bloomfilter/tests/generic_filter_test.py", line 95, in test_bloom_url_objects
self.assertNotIn(url_char, self.filter)
AssertionError: <URL for "http://moth/index20167.html"> unexpectedly found in <w3af.core.data.bloomfilter.scalable_bloom.ScalableBloomFilter object at 0x3258cd0>
----------------------------------------------------------------------
Ran 61 tests in 28.329s
FAILED (failures=6)
pablo@eulogia:~/pch/w3af$ find . -name *bloom*^C
pablo@eulogia:~/pch/w3af$ sudo pip install pybloomfiltermmap==0.3.11
[sudo] password for pablo:
Downloading/unpacking pybloomfiltermmap==0.3.11
Downloading pybloomfiltermmap-0.3.11.tar.gz (435kB): 435kB downloaded
Running setup.py egg_info for package pybloomfiltermmap
info: Building from C
Installing collected packages: pybloomfiltermmap
Found existing installation: pybloomfiltermmap 0.3.14
Uninstalling pybloomfiltermmap:
Successfully uninstalled pybloomfiltermmap
Running setup.py install for pybloomfiltermmap
info: Building from C
building 'pybloomfilter' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c src/mmapbitarray.c -o build/temp.linux-x86_64-2.7/src/mmapbitarray.o
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c src/bloomfilter.c -o build/temp.linux-x86_64-2.7/src/bloomfilter.o
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c src/md5.c -o build/temp.linux-x86_64-2.7/src/md5.o
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c src/primetester.c -o build/temp.linux-x86_64-2.7/src/primetester.o
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c src/pybloomfilter.c -o build/temp.linux-x86_64-2.7/src/pybloomfilter.o
In file included from src/pybloomfilter.c:255:0:
/usr/include/python2.7/pythread.h:5:1: warning: ‘always_inline’ attribute ignored [-Wattributes]
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/src/mmapbitarray.o build/temp.linux-x86_64-2.7/src/bloomfilter.o build/temp.linux-x86_64-2.7/src/md5.o build/temp.linux-x86_64-2.7/src/primetester.o build/temp.linux-x86_64-2.7/src/pybloomfilter.o -lcrypto -o build/lib.linux-x86_64-2.7/pybloomfilter.so
Successfully installed pybloomfiltermmap
Cleaning up...
pablo@eulogia:~/pch/w3af$ nosetests w3af/core/data/bloomfilter/tests/
.............................................................
----------------------------------------------------------------------
Ran 61 tests in 33.906s
OK
pablo@eulogia:~/pch/w3af$ |
|
I don't know too much about those tests, but those look to me like it's testing against false positives? bloom filters will have false positives. |
|
Yes, agreed on the FP, but... they always PASS on 0.3.11 and always fail on 0.3.14 . So there is either something wrong with pybloomfiltermmap OR my wrapper to make ScalableBloomfilters. I'll research more later and report a but to you if needed. Thanks! |
|
This test fails and I'm not doing any scalable bloom filter, just testing the 0.3.14: pablo@eulogia:~/pch/w3af$ nosetests w3af/core/data/bloomfilter/tests/test_cmmap_bloom.py:TestCMmapBloomfilterLarge.test_bloom_int
F
======================================================================
FAIL: test_bloom_int (w3af.core.data.bloomfilter.tests.test_cmmap_bloom.TestCMmapBloomfilterLarge)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/tests/pylint_plugins/decorator.py", line 38, in test_only_subclass
return meth(self, *args, **kwds)
File "/home/pablo/PycharmProjects/w3af/w3af/core/data/bloomfilter/tests/generic_filter_test.py", line 61, in test_bloom_int
self.assertNotIn(r, self.filter)
AssertionError: 26794 unexpectedly found in <BloomFilter capacity: 20000, error: 0.001, num_hashes: 9>
----------------------------------------------------------------------
Ran 1 test in 0.031s
FAILED (failures=1)
pablo@eulogia:~/pch/w3af$ This is the test that fails: @only_if_subclass
def test_bloom_int(self):
for i in xrange(0, self.CAPACITY):
self.filter.add(i)
for i in xrange(0, self.CAPACITY):
self.assertIn(i, self.filter)
for i in xrange(0, self.CAPACITY / 2):
r = random.randint(self.CAPACITY, self.CAPACITY * 2)
self.assertNotIn(r, self.filter)Test starts to PASS if in test_cmmap_bloom.py I set All this only occurs with Will read the latest commits for the pybloomfiltermmap project to investigate. |
|
Suspect: axiak/pybloomfiltermmap@a06bb1b |
|
It's true I changed the hash function, but the error bounds are now much more correct (the tests indicate random input is very close to the error rate). Previously there were issues so I completely erred on the side of too exact. |
|
MurmurHash3 is platform dependent, documenting that I'm using 64bit for these tests. |
|
Who uses 32-bit? |
|
Could you ping me on GTalk / IRC? It will be easier to chat for 10min than to exchange 20 messages over here :) |
|
Maybe I can better test all this with my own mac in the cloud 👍 |
|
Failed to install Mac OSX on virtualbox using http://www.macbreaker.com/2012/07/mountain-lion-virtualbox.html , won't be able to debug myself, so the plan is now:
|
|
Workaround: DONE. Changing severity to 3 since now Mac users can run w3af and this issue is just a performance thing. |
andresriancho
modified the milestones:
1.7 - Multiprocessing release,
1.6 - Consumer/producer release
andresriancho
modified the milestones:
OLD .17,
1.7 - Multiprocessing release,
1.6.1 - Bug fixes after 1.6 release
andresriancho
modified the milestones:
1.6.1 - Bug fixing after 1.6,
1.7.0 - Increase WAVSEP Coverage and add long vulnerability descriptions
I have just installed w3af and it crashes as soon as it starts up. Details below... I can provide more information if needed.
Installation steps:
Cloned latest w3af from github
Installed dependencies using MacPorts & pip
(Replaced py27-distribute with py27-setuptools in mac.py as the former is deprecated)
Running either the GUI or the console application produces the following crash:
The text was updated successfully, but these errors were encountered: