feat: Support Temporal Proximity correlation

[fukusuket]

What Changed

• Closed Support Temporal Proximity correlation #1446

Evidence

Integration-Test

• https://github.com/Yamato-Security/hayabusa/actions/runs/12388332964

I would appreciate it if you could check it out when you have time🙏

[fukusuket]

diff check

Resulting files are the same size before and after This PR.
(Sorting of aggregate results does not match exactly due to the following issue #1466)

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -D -n -u -q -o timeline-new.csv -C
% ./hayabusa-3.0.0-mac-aarch64 csv-timeline -d ../hayabusa-sample-evtx -w -D -n -u -q -o timeline-old.csv -C
% ls -la
...
-rw-r--r--@  1 fukusuke  staff  33475930 12 18 15:37 timeline-new.csv
-rw-r--r--   1 fukusuke  staff  33475930 12 18 15:37 timeline-old.csv

[fukusuket]

Temporal rule

I confirmed that temporal rule can be detected in following simple case.

title: Correlation Test Rule
id: 23179f25-6fce-4827-bae1-b219deaf563a
author: fukusuket
correlation:
  type: temporal
  rules:
    - failed_logins
    - successful_login
  group-by:
    - Computer
  timespan: 1m
  generate: true
level: critical
ruletype: Hayabusa
---
id: 23179f25-6fce-4827-bae1-b219deaf563c
title: Failed Logins
name: failed_logins
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4625
    LogonType: 2
  condition: selection
---
id: 23179f25-6fce-4827-bae1-b219deaf563x
title: Successful Login
name: successful_login
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4624
    LogonType: 2
  condition: selection

./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -o timeline.csv -r test.yml -q -C
Start time: 2024/12/18 15:44
Total event log files: 598
Total file size: 139.2 MB

Loading detection rules. Please wait.


Undefined rules: 3 (100.00%)

Correlation rules: 1 (33.33%)
Correlation referenced rules: 2 (66.67%)

Expand rules: 0 (0.00%)
Enabled expand rules: 0 (0.00%)

Hayabusa rules: 1
Other rules: 2
Total detection rules: 3

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 248
Detection rules enabled after channel filter: 5

Output profile: standard

Scanning in progress. Please wait.

[00:00:00] 248 / 248   [========================================] 100%
...
╭──────────────────────────────────────────────╮
│ Top critical alerts:        Top high alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Correlation Test Rule (1)   n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤

following records detected by temporal rule.

"2020-09-09 22:18:23.627 +09:00","Failed Logins","-","MSEDGEWIN10","Sec",4625,137222,"Type: 2 - INTERACTIVE ¦ TgtUser: IEUser ¦ SrcComp: MSEDGEWIN10 ¦ SrcIP: - ¦ AuthPkg: Negotiate ¦ Proc: C:\Program Files (x86)\Google\Chrome\Application\chrome.exe","FailureReason: BAD USER OR PW ¦ KeyLength: 0 ¦ LogonProcessName: Chrome ¦ ProcessId: 4952 ¦ Status: BAD USER OR PW ¦ SubStatus: WRONG PW ¦ SubjectLogonId: 0x79e59 ¦ SubjectUserSid: S-1-5-21-3461203602-4096304019-2269080069-1000 ¦ TargetUserSid: S-1-0-0"
"2020-09-09 22:18:27.714 +09:00","Successful Login","-","MSEDGEWIN10","Sec",4624,137224,"Type: 2 - INTERACTIVE ¦ TgtUser: IEUser ¦ SrcComp: MSEDGEWIN10 ¦ SrcIP: - ¦ LID: 0x1cd8f6","AuthenticationPackageName: Negotiate ¦ ElevatedToken: YES ¦ ImpersonationLevel: IMPERSONATION ¦ KeyLength: 0 ¦ LogonGuid: 00000000-0000-0000-0000-000000000000 ¦ LogonProcessName: Chrome ¦ ProcessId: 4952 ¦ ProcessName: C:\Program Files (x86)\Google\Chrome\Application\chrome.exe ¦ SubjectLogonId: 0x79e59 ¦ SubjectUserSid: S-1-5-21-3461203602-4096304019-2269080069-1000 ¦ TargetLinkedLogonId: 0x1cd964 ¦ TargetUserSid: S-1-5-21-3461203602-4096304019-2269080069-1000 ¦ VirtualAccount: NO"
"2020-09-09 22:18:27.714 +09:00","Successful Login","-","MSEDGEWIN10","Sec",4624,137225,"Type: 2 -

[YamatoSecurity]

@fukusuket Thanks so much! When generate: false is set, things look good but when generate: true is set, it shows:

╭───────╮
│ - (1) │
╰───────╯

under the Rule Authors. I think it should include fukusuket for the correlation rule and - or n/a for the referenced rules when the author is not defined.

Could you check this?

[fukusuket]

@YamatoSecurity
Thank you so much for checking! I fixed Rule Authors output!

./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -r test.yml -q -p super-verbose -o timeline.csv -C
Start time: 2024/12/18 19:59
Total event log files: 598
Total file size: 139.2 MB

Loading detection rules. Please wait.


Undefined rules: 3 (100.00%)

Correlation rules: 1 (33.33%)
Correlation referenced rules: 2 (66.67%)

Expand rules: 0 (0.00%)
Enabled expand rules: 0 (0.00%)

Hayabusa rules: 1
Other rules: 2
Total detection rules: 3

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 248
Detection rules enabled after channel filter: 5

Output profile: super-verbose

Scanning in progress. Please wait.

[00:00:00] 248 / 248   [========================================] 100%

Scanning finished. Please wait while the results are being saved.
                                                                                                                                               Rule Authors:

╭───────────────╮
│ fukusuket (1) │
╰───────────────╯

Results Summary:

Events with hits / Total events: 51 / 26,393 (Data reduction: 26,342 events (99.81%))

Total | Unique detections: 82 | 3
Total | Unique critical detections: 1 (1.22%) | 1 (66.67%)
Total | Unique high detections: 0 (0.00%) | 0 (0.00%)
Total | Unique medium detections: 0 (0.00%) | 0 (0.00%)
Total | Unique low detections: 0 (0.00%) | 0 (0.00%)
Total | Unique informational detections: 0 (0.00%) | 0 (0.00%)

Dates with most total detections:
critical: 2020-09-09 (1), high: n/a, medium: n/a, low: n/a, informational: n/a

Top 5 computers with most unique detections:
critical: n/a
high: n/a
medium: n/a
low: n/a
informational: n/a

╭──────────────────────────────────────────────╮
│ Top critical alerts:        Top high alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Correlation Test Rule (1)   n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:          Top low alerts:  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
╰───────────────────────────╌──────────────────╯

Saved file: timeline.csv (60.5 KB)

[YamatoSecurity]

@fukusuket Thanks! There seems to be a bug with the author name being overwritten.
Here is my test rule:

title: Correlation Test Rule
id: 23179f25-6fce-4827-bae1-b219deaf563a
author: fukusuket
correlation:
  type: temporal
  rules:
    - failed_logins
    - successful_login
  group-by:
    - Computer
  timespan: 1m
  generate: true
level: critical
ruletype: Hayabusa
---
id: 23179f25-6fce-4827-bae1-b219deaf563c
title: Failed Logins
author: author1
name: failed_logins
level: medium
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4625
    LogonType: 2
  condition: selection
---
id: 23179f25-6fce-4827-bae1-b219deaf563x
title: Successful Login
author: author2
name: successful_login
level: low
logsource:
  product: windows
  service: security
detection:
  selection:
    Channel: Security
    EventID: 4624
    LogonType: 2
  condition: selection

there should be fukusuket, author1 and author2 in the Rule Authors but it shows

Rule Authors:

╭─────────────╮
│ author2 (1) │
╰─────────────╯

Could you check this?

[fukusuket]

@YamatoSecurity
Sorry for the bug! I fixed #1532 (comment) ! Could you check this?🙏

./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -r test2.yml -q -p super-verbose -o timeline.csv -C
Start time: 2024/12/19 22:34
Total event log files: 598
Total file size: 139.2 MB

Loading detection rules. Please wait.


Undefined rules: 3 (100.00%)

Correlation rules: 1 (33.33%)
Correlation referenced rules: 2 (66.67%)

Expand rules: 0 (0.00%)
Enabled expand rules: 0 (0.00%)

Hayabusa rules: 1
Other rules: 2
Total detection rules: 3

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 248
Detection rules enabled after channel filter: 5

Output profile: super-verbose

Scanning in progress. Please wait.

[00:00:00] 248 / 248   [========================================] 100%

Scanning finished. Please wait while the results are being saved.
                                                                                                                                               Rule Authors:

╭───────────────────────────────────────────╮
│ author1 (1)   author2 (1)   fukusuket (1) │
╰─────────────╌─────────────╌───────────────╯

Results Summary:

Events with hits / Total events: 51 / 26,393 (Data reduction: 26,342 events (99.81%))

Total | Unique detections: 82 | 3
Total | Unique critical detections: 1 (1.22%) | 1 (0.00%)
Total | Unique high detections: 0 (0.00%) | 0 (0.00%)
Total | Unique medium detections: 17 (20.73%) | 1 (33.33%)
Total | Unique low detections: 64 (78.05%) | 1 (33.33%)
Total | Unique informational detections: 0 (0.00%) | 0 (0.00%)

Dates with most total detections:
critical: 2020-09-09 (1), high: n/a, medium: 2023-09-26 (16), low: 2013-10-24 (26), informational: n/a

Top 5 computers with most unique detections:
critical: n/a
high: n/a
medium: WIN (1), MSEDGEWIN10 (1)
low: IE8Win7 (1), PC02.example.corp (1), FS03.offsec.lan (1), IE9Win7 (1), IE10Win7 (1)
informational: n/a

╭───────────────────────────────────────────────────╮
│ Top critical alerts:        Top high alerts:      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Correlation Test Rule (1)   n/a                   │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:          Top low alerts:       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Failed Logins (17)          Successful Login (64) │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                         n/a                   │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
│ n/a                         n/a                   │
╰───────────────────────────╌───────────────────────╯

Saved file: timeline.csv (61.2 KB)

Elapsed time: 00:00:00.887

[YamatoSecurity]

@fukusuket LGTM! Thanks so much!!!