Support Ordered Temporal Proximity correlation

[fukusuket]

What Changed

• Closed Support Ordered Temporal Proximity correlation #1447
• In addition, I have noticed a bug in the temporal rule's time windows detection logic, which has also been corrected.

Evidence

Integration-Test

https://github.com/Yamato-Security/hayabusa/actions/runs/12445888496

I would appreciate it if you could check it out when you have time🙏

[fukusuket]

diff check

% ./hayabusa-3.0.0-mac-aarch64 csv-timeline -d ../hayabusa-sample-evtx -w -D -n -u -q -o timeline-old.csv -C
% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -D -n -u -q -o timeline-new.csv -C
% ls -la
total 175128
...
-rw-r--r--@  1 fukusuke  staff  33475930 12 21 23:12 timeline-new.csv
-rw-r--r--   1 fukusuke  staff  33475930 12 21 23:12 timeline-old.csv

[fukusuket]

I checked #1447 (comment) sample rule!

case1 detect

title: Successful password spray
id: 23179f25-6fce-4827-bae1-b219deaf563a
author: yamatosecurity
correlation:
  type: temporal_ordered
  rules:
    - many_failed_logins
    - successful_login

 ./hayabusa csv-timeline -d ../tlp-red -w -q -r test.yml -o timeline.csv -C
Start time: 2024/12/21 23:23
Total event log files: 1
Total file size: 1.1 MB

Loading detection rules. Please wait.


Undefined rules: 4 (100.00%)

Correlation rules: 2 (50.00%)
Correlation referenced rules: 3 (75.00%)

Expand rules: 0 (0.00%)
Enabled expand rules: 0 (0.00%)

Hayabusa rules: 1
Other rules: 3
Total detection rules: 4

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 1
Detection rules enabled after channel filter: 4

Output profile: standard

Scanning in progress. Please wait.

[00:00:00] 1 / 1   [========================================] 100%

Scanning finished. Please wait while the results are being saved.
                                                                                                                                               Rule Authors:

╭────────────────────────────────────────────────╮
│ author1 (1)   author2 (1)   yamatosecurity (1) │
╰─────────────╌─────────────╌────────────────────╯

Results Summary:

Events with hits / Total events: 110 / 277 (Data reduction: 167 events (60.29%))

Total | Unique detections: 8 | 3
Total | Unique critical detections: 1 (12.50%) | 1 (0.00%)
Total | Unique high detections: 1 (12.50%) | 1 (33.33%)
Total | Unique medium detections: 0 (0.00%) | 0 (0.00%)
Total | Unique low detections: 0 (0.00%) | 0 (0.00%)
Total | Unique informational detections: 6 (75.00%) | 1 (33.33%)

Dates with most total detections:
critical: 2024-12-21 (1), high: 2024-12-21 (1), medium: n/a, low: n/a, informational: 2024-12-21 (6)

Top 5 computers with most unique detections:
critical: n/a
high: n/a
medium: n/a
low: n/a
informational: Sec504Student (1)

╭─────────────────────────────────────────────────────────╮
│ Top critical alerts:            Top high alerts:        │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Successful password spray (1)   Many Failed Logons! (1) │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:              Top low alerts:         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Successful Login (6)            n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
╰───────────────────────────────╌─────────────────────────╯

Saved file: timeline.csv (5.5 KB)

Elapsed time: 00:00:00.025

case2 (not detect)

title: Successful password spray
id: 23179f25-6fce-4827-bae1-b219deaf563a
author: yamatosecurity
correlation:
  type: temporal_ordered
  rules:
    - successful_login
    - many_failed_logins

% ./hayabusa csv-timeline -d ../tlp-red -w -q -r test.yml -o timeline.csv -C
Start time: 2024/12/21 23:21
Total event log files: 1
Total file size: 1.1 MB

Loading detection rules. Please wait.


Undefined rules: 4 (100.00%)

Correlation rules: 2 (50.00%)
Correlation referenced rules: 3 (75.00%)

Expand rules: 0 (0.00%)
Enabled expand rules: 0 (0.00%)

Hayabusa rules: 1
Other rules: 3
Total detection rules: 4

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 1
Detection rules enabled after channel filter: 4

Output profile: standard

Scanning in progress. Please wait.

[00:00:00] 1 / 1   [========================================] 100%

Scanning finished. Please wait while the results are being saved.
                                                                                                                                               Rule Authors:

╭───────────────────────────╮
│ author2 (1)   author1 (1) │
╰─────────────╌─────────────╯

Results Summary:

Events with hits / Total events: 110 / 277 (Data reduction: 167 events (60.29%))

Total | Unique detections: 7 | 2
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 1 (14.29%) | 1 (50.00%)
Total | Unique medium detections: 0 (0.00%) | 0 (0.00%)
Total | Unique low detections: 0 (0.00%) | 0 (0.00%)
Total | Unique informational detections: 6 (85.71%) | 1 (50.00%)

Dates with most total detections:
critical: n/a, high: 2024-12-21 (1), medium: n/a, low: n/a, informational: 2024-12-21 (6)

Top 5 computers with most unique detections:
critical: n/a
high: n/a
medium: n/a
low: n/a
informational: Sec504Student (1)

╭─────────────────────────────────────────────────────╮
│ Top critical alerts:        Top high alerts:        │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                         Many Failed Logons! (1) │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:          Top low alerts:         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                         n/a                     │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                           │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Successful Login (6)        n/a                     │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
│ n/a                         n/a                     │
╰───────────────────────────╌─────────────────────────╯

Saved file: timeline.csv (5.3 KB)

Elapsed time: 00:00:00.024

[YamatoSecurity]

@fukusuket Thanks!! Great work!

I noticed one bug. When generate: true is set for the correlation rule, the alert shows up:

╭─────────────────────────────────────────────────────────╮
│ Top critical alerts:            Top high alerts:        │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Successful password spray (1)   Many Failed Logons! (1) │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:              Top low alerts:         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Successful Login (6)            n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
│ n/a                             n/a                     │
╰───────────────────────────────╌─────────────────────────╯

But when set to the default false, no alerts show up and does not match:

╭──────────────────────────────────────────────╮
│ Top critical alerts:        Top high alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:          Top low alerts:  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
│ n/a                         n/a              │
╰───────────────────────────╌──────────────────╯

It only seems to work when generate: true is set.

[fukusuket]

@YamatoSecurity
Thank you so much for checking! I fixed it!

./hayabusa csv-timeline -d ../tlp-red -r test.yml -w -o timeline.csv -C -q
Start time: 2024/12/22 11:01
Total event log files: 1
Total file size: 1.1 MB

Loading detection rules. Please wait.


Undefined rules: 4 (100.00%)

Correlation rules: 2 (50.00%)
Correlation referenced rules: 3 (75.00%)

Expand rules: 0 (0.00%)
Enabled expand rules: 0 (0.00%)

Hayabusa rules: 1
Other rules: 3
Total detection rules: 4

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 1
Detection rules enabled after channel filter: 3

Output profile: standard

Scanning in progress. Please wait.

[00:00:00] 1 / 1   [========================================] 100%

Scanning finished. Please wait while the results are being saved.
                                                                                                                                               Rule Authors:

╭────────────────────╮
│ yamatosecurity (1) │
╰────────────────────╯

Results Summary:

Events with hits / Total events: 104 / 277 (Data reduction: 173 events (62.45%))

Total | Unique detections: 1 | 1
Total | Unique critical detections: 1 (100.00%) | 1 (0.00%)
Total | Unique high detections: 0 (0.00%) | 0 (0.00%)
Total | Unique medium detections: 0 (0.00%) | 0 (0.00%)
Total | Unique low detections: 0 (0.00%) | 0 (0.00%)
Total | Unique informational detections: 0 (0.00%) | 0 (0.00%)

Dates with most total detections:
critical: 2024-12-21 (1), high: n/a, medium: n/a, low: n/a, informational: n/a

Top 5 computers with most unique detections:
critical: Sec504Student (1)
high: n/a
medium: n/a
low: n/a
informational: n/a

╭──────────────────────────────────────────────────╮
│ Top critical alerts:            Top high alerts: │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Successful password spray (1)   n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:              Top low alerts:  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                             n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                        │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                             n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
│ n/a                             n/a              │
╰───────────────────────────────╌──────────────────╯

Saved file: timeline.csv (268 B)

Elapsed time: 00:00:00.024

[YamatoSecurity]

@fukusuket Everything looks good now! Thanks again!