Adapt to Zeek "files" log losing conn_uids+tx_hosts+rx_hosts and gaining uid+id #2981
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #2980
When #2971 merged, Zui went from using a Zeek artifact based on Zeek v3.2.1 to an artifact based on Zeek v6.0.2. This exposed Zui to a "breaking change" as part of Zeek v5.1.0 that broke the Download Packets feature in some circumstances as shown in #2980. From the Zeek release notes:
Here's the affected code as it stands on the tip of
main
:zui/apps/zui/src/plugins/brimcap/zeek/queries.ts
Lines 21 to 34 in e275ddd
zui/apps/zui/src/plugins/brimcap/zeek/queries.ts
Lines 8 to 10 in e275ddd
If I manually construct that query after loading the
ifconfig.pcapng
test data from #2980 and drop thehead 1
, we can see why it started failing.This shows that the "breaking change" to the
files
record means it has taken on so many characteristics of theconn
log that it ended up slipping through all the filter logic infindConnLog()
only to be found unusable elsewhere when the pcap extraction logic looked for theproto
field that's normally found on a trueconn
record. In this PR I've tightened up the logic sofindConnLog()
only finds a trueconn
record.Once I became hip to the scope of the change, I looked into other possible side effects of the add/remove of those fields. As I recall it having been explained to me, the complexity with
conn_uids
,tx_hosts
, andrx_hosts
all beingset
types was an attempt to capture situations where multiple hosts could simultaneously be involved in send/receive of a given file, such as may happen in peer-to-peer file sharing protocols. And I seem to recall that the "breaking change" to thefiles
log was effectively an acceptance of how that never really worked properly and just created confusion. Therefore the singleuid
would now correlate afiles
event with its correspondingconn
record, and likewise theid.orig_h
andid.resp_h
would show the hosts involved in the transfer of a given file.This all means that reacting to the change actually simplifies and fixes some things. Most of the other Zeek log types are trivially joined by
uid
, and nowfiles
is much the same, which meansconn_uids
can be dropped from the add-ons in ouruid
filtering logic. Likewise, while Zed v1.5.0 didn't have the #2980 problem, withtx_hosts
andrx_hosts
as sets, the Detail pane showed the count summary like this:Whereas the primitive values plus the fixes in this branch means we can actually show meaningful IP addresses again.
Other things to note:
After recognizing the impact this Zeek change had on us, I browsed all their release notes in versions since the Zeek v3.2.1 we used in the past. Nothing stood out.
I fixed the hover text descriptions in this PR, but then I realized they don't actually work anymore and broke when Zui Polish 2 #2895 merged, so they were presumably a casualty of the "New Detail Pane Design" cited there. I don't know if this is a surprise but since we've not done another GA release since that merged we could make a call on if we're ready to let that functionality go (in which case we should probably remove the dead code) or if it can be fixed (I don't know how many users rely on it, but it always seemed to me like a nice proof-of-concept of something we could find a way for users to do with their own custom data sources, perhaps using plugins?)
The fact that the Download Packets button failed so quietly in the Can't open pcap slice if the conn contains "files" activity #2980 repro is somewhat concerning to me, but I doubt I have the skills to dive into that. I could perhaps open another issue to pursue that separately.