ThreatBus fails to start in FreeBSD 13

[clopmz]

Good morning,

I am trying to configure ThreatBus to work with MISP and Zeek as an app. I have installed threatbus using the following commands:

python3 -m venv /opt/zeek-bindings
/opt/zeek-bindings/bin/pip3 install -U pip btest wheel
/opt/zeek-bindings/bin/pip3 install -U threatbus
/opt/zeek-bindings/bin/pip3 install -U threatbus-misp
/opt/zeek-bindings/bin/pip3 install -U threatbus-zeek

Also, zkg is installed in the same virtual environment. After this, I have created the following config yaml:

logging:
  console: false
  console_verbosity: DEBUG
  file: true
  file_verbosity: DEBUG
  filename: /tmp/threatbus.log

plugins:
  apps:
    zeek:
      host: "127.0.0.1"
      port: 47761
      module_namespace: Tenzir

  misp:
    api:
      host: https://172.22.58.6
      ssl: false
      key: <MISP_API_KEY>
    filter: # filter are optional. you can omit the entire section.
      - orgs: # org IDs must be strings: https://github.com/MISP/PyMISP/blob/main/pymisp/data/schema.json
          - "1"
        tags:
          - "TLP:WHITE"
          - "TLP:AMBER"
          - "TLP:RED"
        types: # MISP attribute types https://github.com/MISP/misp-objects/blob/main/schema_objects.json
          - ip-src
          - ip-dst
          - hostname
          - domain
          - url
          - md5
          - sha256
          - email
          - filename
          - ja3-fingerprint-md5
          - ssdeep
          - uri
    zmq:
      host: 172.22.58.6
      port: 50000

When I have tried to start threatbus with the followin command, the following error is displayed:

root@zeekhost:/tmp # /opt/zeek-bindings/bin/threatbus -c /usr/local/etc/threatbus/config.yaml
Traceback (most recent call last):
  File "/opt/zeek-bindings/bin/threatbus", line 8, in <module>
    sys.exit(main())
  File "/opt/zeek-bindings/lib/python3.7/site-packages/threatbus/threatbus.py", line 209, in main
    start(config)
  File "/opt/zeek-bindings/lib/python3.7/site-packages/threatbus/threatbus.py", line 170, in start
    apps.load_setuptools_entrypoints("threatbus.app")
  File "/opt/zeek-bindings/lib/python3.7/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/opt/zeek-bindings/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 178, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/zeek-bindings/lib/python3.7/site-packages/threatbus_zeek/plugin.py", line 1, in <module>
    import broker
ModuleNotFoundError: No module named 'broker'

Zeek 4.0.3 is installed in the same host as threatbus in /opt/zeek directory and /opt/zeek/bin is configured in root's path.

What am I doing wrong?

[mavam]

It looks like the Python Broker bindings are not installed, based on this piece of the error message:

    import broker
ModuleNotFoundError: No module named 'broker'

Does pip install broker fix it?

[mavam]

According to Zeek's docs, my python bindings are installed.

Mind verifying that this is indeed the case, by typing python followed by import broker? Then you're in business.

Do I need to pass any special switch to "pip installl threatbus-zeek" to use my custom Zeek's install?

For this to work, Zeek and Threat Bus should use the same Python installation. When Zeek installs the Python bindings, Threat Bus needs to be able to pick them up. Threat Bus doesn't install anything Zeek/Broker related via pip.

For testing, you could create a venv and install into it from Both Zeek and Threat Bus.

Also, how can I add ZeroMQ support for MISP under threatbus? I have tried to install using "pip install threatbus-misp.zmq" without luck,

Mind sharing the error that you get here?

[clopmz]

HI @mavam

I have solved the problem: I only need to create a softlink inside threatbus venv to broker's python bindings directory and now it works.

But now appears the following error:

2021-07-09 10:51:55,630 INFO     [threatbus] Starting plugins...
2021-07-09 10:51:55,631 INFO     [threatbus_inmem.plugin] In-memory backbone started.
2021-07-09 10:51:55,632 INFO     [threatbus_inmem.plugin] Adding subscription to: threatbus/snapshotrequest
2021-07-09 10:51:55,632 INFO     [threatbus_inmem.plugin] Adding subscription to: threatbus/snapshotenvelope
2021-07-09 10:51:55,695 INFO     [threatbus_zeek.plugin] Zeek plugin started
2021-07-09 10:51:55,703 CRITICAL [threatbus_misp.plugin] Invalid config for plugin misp: MISP attribute export is configured via ZeroMQ, but the dependency is not installed. Install `threatbus-misp[zmq]` to use this setting.

I have installed threatbus-misp .... Where is the problem?

root@zeek:/tmp # /opt/threatbus/bin/pip3 list
Package Version

---

antlr4-python3-runtime 4.8
appdirs 1.4.4
attrs 21.2.0
black 21.6b0
btest 0.68
certifi 2021.5.30
chardet 4.0.0
click 8.0.1
coloredlogs 15.0.1
confuse 1.5.0
Deprecated 1.2.12
humanfriendly 9.2
idna 2.10
jsonschema 3.2.0
mypy-extensions 0.4.3
pathspec 0.8.1
pip 21.1.3
pluggy 0.13.1
pymisp 2.4.144
pyrsistent 0.18.0
python-dateutil 2.8.1
pytz 2021.1
PyYAML 5.4.1
regex 2021.7.6
requests 2.25.1
setuptools 56.0.0
simplejson 3.17.2
six 1.16.0
sqlite3 0.0.0
stix2 2.1.0
stix2-patterns 1.3.2
threatbus 2021.6.24
threatbus-inmem 2021.6.24
threatbus-misp 2021.6.24
threatbus-zeek 2021.6.24
toml 0.10.2
urllib3 1.26.6
wheel 0.36.2
wrapt 1.12.1

[dominiklohmann]

You might want to use python -m venv --system-site-packages venv to allow access to system packages. See also the documentation at docs.tenzir.com/threatbus on Bare Metal installations

[mavam]

I have installed threatbus-misp .... Where is the problem?

There are basically two ways to install the MISP plugin:

pip install threatbus-misp[zmq]
pip install threatbus-misp[kafka]

You want the first invocation. The brackets represent optional dependencies. See the plugin README for details.

[clopmz]

Ok, done. My issue is solved.

Many thanks @mavam and @dominiklohmann