Add ElasticSearch support to Kippo #135
Conversation
|
This also solves: #138 |
|
Thanks for your pull request! I've done some quick tests with the help of honeydrive; My concern here is the same as with the hpfeeds logger, the lack of asynchronous operation. In case of connection problems, the entire operation of the application is blocked. Simulated with: sudo iptables -A OUTPUT -p tcp --dport 9200 -j DROP If the underlying library doesn't allow for asynchronous operation, perhaps we can use twisted.internet.threads.deferToThread to work around it. I also roughly tested how it manages a disappearing elasticsearch, and while there are nasty errors in the console, the honeypot seems to continue operation fine. |
|
Hi @desaster, thanks a lot for testing this! I understand your concern, blocking operations can be problematic. Your suggestion seems fine to me. Although I don't have any experience with the Twisted library I can take a look and see what can be done. Unless you're interested in trying yourself. I've also tested what happens if the ES service goes down, and had the same experience. The fact that the honeypot continues working is "good", although masking of the errors is probably needed. |
|
there seems to be some existing work on making elasticsearch work with twisted, such as this https://github.com/jkoelker/txes however this is just after a quick google search, i haven't looked at how they work yet. |
|
There is also this Tornado based library for async communication with ES: https://github.com/globocom/tornado-es |
* master: (39 commits) Make MakingKippoReachable link work quick fixes: call getPeer() from another transport, fixes desaster#146 add virtualenv support to start.sh add .gitignore based on ikoniaris' pull request desaster#135 restructuring and cleanup remove obsolete config entry change exec handling so the command is allowed to run long enough for wget to work we don't need recvline.HistoricRecvLine for execCommand, so branch off the protocol into two protocols code style changes when using execCommand, ls breaks without windowSize cleanup key generation [i] does not generally mean anything in kippo add bash -c (Closes desaster#104) Add php Add simple free command Add simple killall Add perl Add /proc/version add support for -O parameter (fixes desaster#131) ... Conflicts: README.md kippo/core/honeypot.py
Hi desaster.
This pull request adds ElasticSearch support to Kippo.
Configuration:
I have added a new section to the config file where the ES instance/cluster's values should be set.
Logging module:
I have added a new file named "elasticsearch.py" into "kippo/dblog/" in the same fashion as the other logging modules and followed the same function overriding style.
Testing:
I have tested it myself and it works correctly. The login attempts can be visualized with Kibana, like in Kippo2ElasticSearch's case: http://bruteforce.gr/kippo2elasticsearch (see the screenshots). If you don't want to mess with installing ELK, you can use the latest version of HoneyDrive (released today!) to test it yourself: http://bruteforce.gr/honeydrive
Additional requirements:
1 and 2 are easily installable via pip. Regarding number 3 above, I have included a database in the pull request that I use to initialize the geoip object I use for geolocating the remote IP. You can skip it if you like though and add an extra configuration option to kippo.cfg for the user to select whether to enable geolocation or not. If yes, he will have to manually download the GeoIP.dat file and unzip it in the root folder for example. By the way, a requirements.txt file should also be added at some point to track those packages.
Extras:
I've also included a more proper .gitignore file. I should have opened another pull request for this perhaps but I was coding the ElasticSearch part at the same time.
Let me know what you think.