Querying stored IOCs

[clopmz]

Good afternoon,

After seting up threatbus with Zeek and MISP plugins enabled and working, I have a doubt regarding how IOC's are queried by threatbus.

The IOC's that are loaded through the workers via a scheduled job, I see that they are consulted via ZeroMQ. But what happens with the rest of the IOC's stored in previous days?

I don't see that they are consulted via ThreatBus. Is it possible to query all IOC's stored in a MISP instance?

Thanks.

[lava]

Hi,

the high-level idea for that is to send a snapshot request via threatbus. The app (ie. threatbus_misp) should then translate that into a request for the historic data from MISP, and put the results back onto the bus.

[clopmz]

UHmm ... If I am not wrong, snapshot option is:

option snapshot_intel: interval = 0 sec &redef;

Is this correct?.

I did a simple test: I stopped the MISP instance, stopped the ThreatBus process on the FreeBSD host, restarted the MISP instance and the threatbus process. In ThreatBus log appears the following:

2021-07-15 06:05:14,499 INFO     [threatbus] Starting plugins...
2021-07-15 06:05:14,501 INFO     [threatbus_inmem.plugin] In-memory backbone started.
2021-07-15 06:05:14,501 INFO     [threatbus_inmem.plugin] Adding subscription to: threatbus/snapshotrequest
2021-07-15 06:05:14,501 INFO     [threatbus_inmem.plugin] Adding subscription to: threatbus/snapshotenvelope
2021-07-15 06:05:14,531 INFO     [threatbus_zeek.plugin] Zeek plugin started
2021-07-15 06:05:15,186 INFO     [threatbus_inmem.plugin] Adding subscription to: stix2/sighting
2021-07-15 06:05:15,195 INFO     [threatbus_misp.plugin] MISP plugin started
2021-07-15 06:09:42,948 INFO     [threatbus_zeek.plugin] Received subscription for topic 'stix2/indicator' with snapshot '0:00:00'
2021-07-15 06:09:42,949 INFO     [threatbus_inmem.plugin] Adding subscription to: stix2/indicator
2021-07-15 06:09:43,205 INFO     [threatbus_zeek.plugin] Received subscription for topic 'stix2/indicator' with snapshot '0:00:00'
2021-07-15 06:09:43,207 INFO     [threatbus_inmem.plugin] Adding subscription to: stix2/indicator
2021-07-15 06:09:46,034 INFO     [threatbus_zeek.plugin] Received unsubscription from topic 'stix2/indicatoroekylazdev'
2021-07-15 06:09:46,034 INFO     [threatbus_inmem.plugin] Removing subscription from: stix2/indicator
2021-07-15 06:09:46,036 INFO     [threatbus_zeek.plugin] Received subscription for topic 'stix2/indicator' with snapshot '0:00:00'
2021-07-15 06:09:46,038 INFO     [threatbus_inmem.plugin] Adding subscription to: stix2/indicator
2021-07-15 06:09:46,989 INFO     [threatbus_zeek.plugin] Received unsubscription from topic 'stix2/indicatorahzvsdovga'
2021-07-15 06:09:46,989 INFO     [threatbus_inmem.plugin] Removing subscription from: stix2/indicator
2021-07-15 06:09:46,996 INFO     [threatbus_zeek.plugin] Received subscription for topic 'stix2/indicator' with snapshot '0:00:00'
2021-07-15 06:09:46,997 INFO     [threatbus_inmem.plugin] Adding subscription to: stix2/indicator
2021-07-15 06:09:47,086 INFO     [threatbus_zeek.plugin] Received unsubscription from topic 'stix2/indicatorinmoamjbze'
2021-07-15 06:09:47,086 INFO     [threatbus_inmem.plugin] Removing subscription from: stix2/indicator
2021-07-15 06:09:47,088 INFO     [threatbus_zeek.plugin] Received subscription for topic 'stix2/indicator' with snapshot '0:00:00'
2021-07-15 06:09:47,089 INFO     [threatbus_inmem.plugin] Adding subscription to: stix2/indicator

And on Zeek's reporter.log:

{"ts":"2021-07-15T06:09:42.900111Z","level":"Reporter::INFO","message":"Threat Bus peered","location":"/nsm/zeek/spool/installed-scripts-do-not-touch/site/threatbus/threatbus.zeek, line 134"}
{"ts":"2021-07-15T06:09:43.345350Z","level":"Reporter::INFO","message":"Subscribed to p2p_topic: stix2/indicatoroekylazdev","location":"/nsm/zeek/spool/installed-scripts-do-not-touch/site/threatbus/threatbus.zeek, line 260"}
{"ts":"2021-07-15T06:09:46.032767Z","level":"Reporter::INFO","message":"unsubscribing from p2p_topic stix2/indicatoroekylazdev","location":"/nsm/zeek/spool/installed-scripts-do-not-touch/site/threatbus/threatbus.zeek, line 110"}
{"ts":"2021-07-15T06:09:46.042336Z","level":"Reporter::INFO","message":"Subscribed to p2p_topic: stix2/indicatorahzvsdovga","location":"/nsm/zeek/spool/installed-scripts-do-not-touch/site/threatbus/threatbus.zeek, line 260"}
{"ts":"2021-07-15T06:09:46.987408Z","level":"Reporter::INFO","message":"unsubscribing from p2p_topic stix2/indicatorahzvsdovga","location":"/nsm/zeek/spool/installed-scripts-do-not-touch/site/threatbus/threatbus.zeek, line 110"}
{"ts":"2021-07-15T06:09:47.017457Z","level":"Reporter::INFO","message":"Subscribed to p2p_topic: stix2/indicatorinmoamjbze","location":"/nsm/zeek/spool/installed-scripts-do-not-touch/site/threatbus/threatbus.zeek, line 260"}
{"ts":"2021-07-15T06:09:47.071716Z","level":"Reporter::INFO","message":"unsubscribing from p2p_topic stix2/indicatorinmoamjbze","location":"/nsm/zeek/spool/installed-scripts-do-not-touch/site/threatbus/threatbus.zeek, line 110"}
{"ts":"2021-07-15T06:09:47.100606Z","level":"Reporter::INFO","message":"Subscribed to p2p_topic: stix2/indicatorklbxwixoiv","location":"/nsm/zeek/spool/installed-scripts-do-not-touch/site/threatbus/threatbus.zeek, line 260"}

Which seems to me to be correct. But I have done the simple test of resolving a domain listed in MISP and ThreatBus has not taken any action ..... This IOC, without stopping the MISP instance, was detected by threatbus:

{"ts":"2021-07-14T14:14:54.642342Z","uid":"CixEex4OdDdDwoexJ3","id.orig_h":"172.22.55.5","id.orig_p":52999,"id.resp_h":"172.22.55.1","id.resp_p":53,"seen.indicator":"say.f322.net","seen.indicator_type":"Intel::DOMAIN","seen.where":"DNS::IN_REQUEST","seen.node":"idps-prod","matched":["Intel::DOMAIN"],"sources":["threatbus"]}

Do I need to reconfigure snapshot_option under Zeek to retrieve IOCs from the last 30 days, for example?

Regards.

[clopmz]

Thanks @lava .

There is no log message related to this domain. In fact, the problem comes because ThreatBus only queries the IOC's that are loaded into MISP during a fetch. Previous IOCs are not queried by ThreatBus even if the snapshot option is configured.

[clopmz]

Hi,

Any news regarding this issue?

[mavam]

Sorry for long latency here.

We are going to reproduce this specific issue in our lab setup this week, including a MISP configured with the ThreatFox feed. We have an idea already what may cause, but not yet confirmed the issue.

Stay tuned.

[lava]

So, this took a while since we got caught up rebuilding large parts of our internal test infrastructure internally, and this got pushed back until that was finished. Sorry for that.

However, I finally got around to reproducing the setup, and ultimately couldn't reproduce the issue:

I started threatbus with the threatbus-zeek and threatbus-misp plugins configured, then I started a MISP instance and added the ThreatFox feed, and finally I started zeek with the "Tenzir::snapshot_intel=30 days".

As expected, all iocs that were already in MISP were sent over the threat bus (with some of them generating the Cannot find STIX-2 mapping for MISP attribute type 'sha256' error message you also saw)

[mavam]

@clopmz could you maybe try once more with current master?

[clopmz]

Many thanks @mavam