feat: exclude rules/evtx when target `Channel` does not exists in files #1334

fukusuket · 2024-04-29T05:23:14Z

What Changed

Test

baseline-evtx v8

rev	Events with hits / Total events	Memory usage stats	Elapsed time
main	5,969,042 / 6,611,184	16.0 GiB	00:11:07.2717
This PR	5,968,974 / 6,462,495	16.0 GiB	00:10:33.2383

fukusuke@fukusukenoAir hayabusa-2.15.0-mac-arm % ./hayabusa csv-timeline -D -n -u -o new-big.csv -w -d ../all-evtx --debug -C

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/04/30 01:09

Total event log files: 2,239
Total file size: 8.8 GB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12

Deprecated rules: 208 (4.76%)
Experimental rules: 903 (20.64%)
Stable rules: 251 (5.74%)
Test rules: 2,967 (67.83%)
Unsupported rules: 45 (1.03%)

Hayabusa rules: 174
Sigma rules: 4,200
Total enabled detection rules: 4,374

Evtx files loaded after channel filter: 139
Detection rules loaded after channel filter: 4,341

Output profile: standard

Scanning in progress. Please wait.

[00:07:58] 139 / 139   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Rule Authors:

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (91)                   Nasreddine Bencherchali (63)      frack113 (59)                     Florian Roth (38)               │
│ oscd.community (38)                Tim Shelton (15)                  Daniil Yugoslavskiy (12)          Roberto Rodriguez (11)          │
│ Roberto Rodriguez @Cyb3r... (11)   Timur Zinniatullin (10)           OTR (9)                           Victor Sergeev (8)              │
│ Jonhnathan Ribeiro (6)             Gleb Sukhodolskiy (6)             KarneadesMarkus Neis (4)          juju4 (4)                       │
│ Jakob Weinzettl (4)                Ján Trenčanský (3)                Sander Wiebing (3)                Wietze Beukema (3)              │
│ SOC Prime (3)                      Markus Neis (3)                   Thomas Patzke (3)                 Michael Haag (3)                │
│ Teymur Kheirkhabarov (3)           Bhabesh Raj (3)                   Alexandr Yampolskyi (3)           Patrick Bareiss (2)             │
│ Christopher Peacock @sec... (2)    Dimitrios Slamaris (2)            Aleksey Potapov (2)               Oddvar Moe (2)                  │
│ James Pemberton@4A616D65... (2)    Sreeman (2)                       Mark Woan (2)                     Anton Kutepov (2)               │
│ Center for Threat Inform... (2)    Endgame (2)                       Mark Russinovich (2)              @gott_cyber (2)                 │
│ JHasenbusch (2)                    Fukusuke Takahashi (2)            Sherif Eldeeb (2)                 Swachchhanda Shrawan Poudel (2) │
│ SCYTHE @scythe_io (2)              Samir Bousseaden (1)              Janantha Marasinghe (1)           Matthew Green @mgreen27 (1)     │
│ Luc Génaux (1)                     Ecco (1)                          Andreas Hunkeler (1)              Harish Segar (1)                │
│ Connor Martin (1)                  D3F7A5105 (1)                     Stephen Lincoln @slincol... (1)   Eric Conrad (1)                 │
│ xorxes (1)                         Thurein Oo (1)                    Tim Rauch (1)                     Zach Stanford @svch0st (1)      │
│ pH-T (1)                           FPT.EagleEye (1)                  @neu5ron (1)                      Open Threat Research (1)        │
│ Cybex (1)                          Tom Kern (1)                      AlertIQ (1)                       X__Junior (1)                   │
│ Elastic (1)                        Beyu Denis (1)                    Anish Bogati (1)                  Cédric Hien (1)                 │
│ James Pemberton @4A616D6573 (1)    Timur Zinniatullin oscd.... (1)   Yusuke Matsui (1)                 @redcanary (1)                  │
│ Joshua Wright (1)                  Maxime Thiebaut (1)               Perez Diego (1)                   xknow (1)                       │
│ @0xrawsec (1)                      Dmitry Uchakin (1)                James Dickenson (1)               Natalia Shornikova (1)          │
│ Max Altgelt (1)                    mdecrevoisier (1)                 Michael R. (1)                                                    │
╰──────────────────────────────────╌─────────────────────────────────╌─────────────────────────────────╌─────────────────────────────────╯

Results Summary:

Events with hits / Total events: 5,968,974 / 6,462,495 (Data reduction: 493,521 events (7.64%))

Total | Unique detections: 6,108,811 | 308
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 17,313 (0.28%) | 35 (22.73%)
Total | Unique medium detections: 41,212 (0.67%) | 115 (28.57%)
Total | Unique low detections: 1,704,534 (27.90%) | 88 (37.34%)
Total | Unique informational detections: 4,345,752 (71.14%) | 70 (11.36%)

Dates with most total detections:
critical: n/a, high: 2023-11-06 (5,568), medium: 2023-11-06 (18,886), low: 2022-09-18 (915,314), informational: 2022-03-02 (1,231,211)

Top 5 computers with most unique detections:
critical: n/a
high: WinDev2310Eval (22), DESKTOP-6D0DBMB (10), DESKTOP-A8CALR3 (9), Agamemnon (9), evtx-PC (7)
medium: WinDev2310Eval (85), Agamemnon (40), DESKTOP-6D0DBMB (23), DESKTOP-A8CALR3 (23), evtx-PC (16)
low: WinDev2310Eval (57), DESKTOP-6D0DBMB (40), Agamemnon (34), DESKTOP-A8CALR3 (33), evtx-PC (22)
informational: WinDev2310Eval (49), DESKTOP-6D0DBMB (48), DESKTOP-A8CALR3 (47), WIN-TKC15D7KHUR (43), Agamemnon (40)

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                               Top high alerts:                                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                                                File Creation Date Changed to Another Year (15,884)            │
│ n/a                                                                Windows Shell/Scripting Application File Write to Sus... (991) │
│ n/a                                                                Mimikatz Detection LSASS Access (131)                          │
│ n/a                                                                Proc Exec (Non-Exe Filetype) (60)                              │
│ n/a                                                                File Download with Headless Browser (60)                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                                 Top low alerts:                                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (12,311)                                           Proc Access (1,613,391)                                        │
│ Process Ran With High Privilege (7,673)                            Possible Timestomping (71,065)                                 │
│ Potential Credential Dumping Activity Via LSASS (6,135)            Scheduled Task Created - Registry (8,185)                      │
│ Autorun Keys Modification (3,606)                                  Shell Context Menu Command Tampering (4,283)                   │
│ LSASS Access From Program In Potentially Suspicious F... (2,396)   Suspicious In-Memory Module Execution (2,878)                  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                                         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Reg Key Create/Delete (Noisy) (1,714,224)                          Pipe Conn (39,460)                                             │
│ Reg Key Value Set (Noisy) (1,151,506)                              Proc Exec (23,695)                                             │
│ DLL Loaded (Noisy) (727,396)                                       Proc Terminated (14,674)                                       │
│ File Created (542,438)                                             Net Conn (14,234)                                              │
│ File Deleted (94,703)                                              Pipe Created (10,602)                                          │
╰──────────────────────────────────────────────────────────────────╌────────────────────────────────────────────────────────────────╯

Saved file: new-big.csv (4.2 GB)

Elapsed time: 00:10:33.2383

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240430_011958.log for details.

Rule Parse Processing Time: 00:00:01.876
Analysis Processing Time: 00:08:00.386
Output Processing Time: 00:01:37.417

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    16.0 GiB    16.0 GiB     0          16.0 GiB
 committed:     1.0 GiB    16.0 GiB   785.1 GiB  -769.1 GiB                          ok
     reset:     0
    purged:    56.4 GiB
   touched:   128.5 KiB    11.1 MiB   126.6 GiB  -126.6 GiB                          ok
  segments:    18         179         168          11                                not all freed!
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0           1.1 Mi     -1.1 Mi                           ok
-abandoned:     5           5           5           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    29.1 Ki
   threads:    17          17           1          16                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   635.394 s
   process: user: 3385.846 s, system: 53.633 s, faults: 60, rss: 2.3 GiB, commit: 1.0 GiB

…ocess

fukusuket · 2024-04-29T16:00:03Z

hayabusa-sample-evtx(--enable-all-rules --scan-all-evtx-files)

I confirmed that there are no differences as shown below.

% ./hayabusa csv-timeline --enable-all-rules --scan-all-evtx-files -D -n -u -o new.csv -w -d ../hayabusa-sample-evtx -C
% ./hayabusa-2.15.0-mac-aarch64 csv-timeline -D -n -u -o old.csv -w -d ../hayabusa-sample-evtx -C
% diff new.csv old.csv
%

I would appreciate it if you could check it out when you have time🙏

fukusuket · 2024-04-30T00:04:59Z

hayabusa-sample-evtx(no option)

I confirmed that only the rules that do not use Channel have differences as follows.

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -D -n -u -o old.csv
% ./hayabusa-2.15.0-mac-aarch64 csv-timeline -d ../hayabusa-sample-evtx -w -D -n -u -o new.csv
% cat new.csv| awk -F, '{print $2}' | sort | uniq > rule-new.txt
% cat old.csv| awk -F, '{print $2}' | sort | uniq > rule-old.txt
% diff rule-new.txt rule-old.txt
221d220
< "Mimikatz Use"
284d282
< "Possible Hidden Shellcode"

fukusuket · 2024-05-01T00:30:33Z

Sorry, there were two rules that used wildcards in Channel, so I will add processing...

fukusuket · 2024-05-01T00:52:47Z

I fixed #1334 (comment) :)

baseline-evtx v8

I confirmed that there are differences only for rules that do not use Channel.

% ./hayabusa-2.15.0-mac-aarch64 csv-timeline -d ../all-evtx -w -D -n -u --debug -o old-big.csv
% ./hayabusa csv-timeline -d ../all-evtx -w -D -n -u --debug -o new-big.csv
% cat new-big.csv| awk -F, '{print $2}' | sort | uniq > rule-new.txt
% cat old-big.csv| awk -F, '{print $2}' | sort | uniq > rule-old.txt
% diff rule-old.txt rule-new.txt
123d122
< "Possible Hidden Shellcode"

and in terms of performance, there is almost no difference compared to the #1334 (comment) result.

% ./hayabusa csv-timeline -d ../all-evtx -w -D -n -u --debug -o new-big.csv

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/05/01 09:26

Total event log files: 2,239
Total file size: 8.8 GB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12

Deprecated rules: 208 (4.76%)
Experimental rules: 903 (20.64%)
Stable rules: 251 (5.74%)
Test rules: 2,967 (67.83%)
Unsupported rules: 45 (1.03%)

Hayabusa rules: 174
Sigma rules: 4,200
Total enabled detection rules: 4,374

Evtx files loaded after channel filter: 145
Detection rules loaded after channel filter: 4,343

Output profile: standard

Scanning in progress. Please wait.

[00:07:48] 145 / 145   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Rule Authors:

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (91)                   Nasreddine Bencherchali (63)      frack113 (59)                     Florian Roth (38)               │
│ oscd.community (38)                Tim Shelton (15)                  Daniil Yugoslavskiy (12)          Roberto Rodriguez (11)          │
│ Roberto Rodriguez @Cyb3r... (11)   Timur Zinniatullin (10)           OTR (9)                           Victor Sergeev (8)              │
│ Jonhnathan Ribeiro (6)             Gleb Sukhodolskiy (6)             KarneadesMarkus Neis (4)          juju4 (4)                       │
│ Jakob Weinzettl (4)                Bhabesh Raj (4)                   Ján Trenčanský (3)                Sander Wiebing (3)              │
│ Wietze Beukema (3)                 SOC Prime (3)                     Markus Neis (3)                   Thomas Patzke (3)               │
│ Michael Haag (3)                   Teymur Kheirkhabarov (3)          Alexandr Yampolskyi (3)           Patrick Bareiss (2)             │
│ Christopher Peacock @sec... (2)    Dimitrios Slamaris (2)            Aleksey Potapov (2)               Oddvar Moe (2)                  │
│ James Pemberton@4A616D65... (2)    Sreeman (2)                       Mark Woan (2)                     Anton Kutepov (2)               │
│ Center for Threat Inform... (2)    Endgame (2)                       Mark Russinovich (2)              @gott_cyber (2)                 │
│ JHasenbusch (2)                    Fukusuke Takahashi (2)            Sherif Eldeeb (2)                 Swachchhanda Shrawan Poudel (2) │
│ SCYTHE @scythe_io (2)              Samir Bousseaden (1)              Janantha Marasinghe (1)           Matthew Green @mgreen27 (1)     │
│ Luc Génaux (1)                     Ecco (1)                          Andreas Hunkeler (1)              Harish Segar (1)                │
│ Connor Martin (1)                  D3F7A5105 (1)                     Stephen Lincoln @slincol... (1)   Eric Conrad (1)                 │
│ xorxes (1)                         Thurein Oo (1)                    Tim Rauch (1)                     Zach Stanford @svch0st (1)      │
│ pH-T (1)                           FPT.EagleEye (1)                  @neu5ron (1)                      Open Threat Research (1)        │
│ Cybex (1)                          Tom Kern (1)                      AlertIQ (1)                       X__Junior (1)                   │
│ Elastic (1)                        Beyu Denis (1)                    Anish Bogati (1)                  Cédric Hien (1)                 │
│ James Pemberton @4A616D6573 (1)    Timur Zinniatullin oscd.... (1)   Yusuke Matsui (1)                 @redcanary (1)                  │
│ Joshua Wright (1)                  Maxime Thiebaut (1)               Perez Diego (1)                   xknow (1)                       │
│ @0xrawsec (1)                      Dmitry Uchakin (1)                James Dickenson (1)               Natalia Shornikova (1)          │
│ Max Altgelt (1)                    mdecrevoisier (1)                 Michael R. (1)                                                    │
╰──────────────────────────────────╌─────────────────────────────────╌─────────────────────────────────╌─────────────────────────────────╯

Results Summary:

Events with hits / Total events: 5,968,978 / 6,463,018 (Data reduction: 494,040 events (7.64%))

Total | Unique detections: 6,108,815 | 309
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 17,317 (0.28%) | 36 (22.65%)
Total | Unique medium detections: 41,212 (0.67%) | 115 (28.48%)
Total | Unique low detections: 1,704,534 (27.90%) | 88 (37.22%)
Total | Unique informational detections: 4,345,752 (71.14%) | 70 (11.65%)

Dates with most total detections:
critical: n/a, high: 2023-11-06 (5,572), medium: 2023-11-06 (18,886), low: 2022-09-18 (915,314), informational: 2022-03-02 (1,231,211)

Top 5 computers with most unique detections:
critical: n/a
high: WinDev2310Eval (23), DESKTOP-6D0DBMB (10), DESKTOP-A8CALR3 (9), Agamemnon (9), evtx-PC (7)
medium: WinDev2310Eval (85), Agamemnon (40), DESKTOP-6D0DBMB (23), DESKTOP-A8CALR3 (23), evtx-PC (16)
low: WinDev2310Eval (57), DESKTOP-6D0DBMB (40), Agamemnon (34), DESKTOP-A8CALR3 (33), evtx-PC (22)
informational: WinDev2310Eval (49), DESKTOP-6D0DBMB (48), DESKTOP-A8CALR3 (47), WIN-TKC15D7KHUR (43), Agamemnon (40)

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                               Top high alerts:                                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                                                File Creation Date Changed to Another Year (15,884)            │
│ n/a                                                                Windows Shell/Scripting Application File Write to Sus... (991) │
│ n/a                                                                Mimikatz Detection LSASS Access (131)                          │
│ n/a                                                                Proc Exec (Non-Exe Filetype) (60)                              │
│ n/a                                                                File Download with Headless Browser (60)                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                                 Top low alerts:                                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (12,311)                                           Proc Access (1,613,391)                                        │
│ Process Ran With High Privilege (7,673)                            Possible Timestomping (71,065)                                 │
│ Potential Credential Dumping Activity Via LSASS (6,135)            Scheduled Task Created - Registry (8,185)                      │
│ Autorun Keys Modification (3,606)                                  Shell Context Menu Command Tampering (4,283)                   │
│ LSASS Access From Program In Potentially Suspicious F... (2,396)   Suspicious In-Memory Module Execution (2,878)                  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                                         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Reg Key Create/Delete (Noisy) (1,714,224)                          Pipe Conn (39,460)                                             │
│ Reg Key Value Set (Noisy) (1,151,506)                              Proc Exec (23,695)                                             │
│ DLL Loaded (Noisy) (727,396)                                       Proc Terminated (14,674)                                       │
│ File Created (542,438)                                             Net Conn (14,234)                                              │
│ File Deleted (94,703)                                              Pipe Created (10,602)                                          │
╰──────────────────────────────────────────────────────────────────╌────────────────────────────────────────────────────────────────╯

Saved file: new-big.csv (4.2 GB)

Elapsed time: 00:10:37.1818

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240501_093655.log for details.

Rule Parse Processing Time: 00:00:01.902
Analysis Processing Time: 00:07:50.036
Output Processing Time: 00:01:50.179

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    16.0 GiB    16.0 GiB     0          16.0 GiB
 committed:     1.0 GiB    16.0 GiB   835.1 GiB  -819.1 GiB                          ok
     reset:     0
    purged:    52.9 GiB
   touched:   128.5 KiB    11.6 MiB   129.5 GiB  -129.5 GiB                          ok
  segments:    18         187         176          11                                not all freed!
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0           1.1 Mi     -1.1 Mi                           ok
-abandoned:     5           5           5           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    24.9 Ki
   threads:    17          17           1          16                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   638.832 s
   process: user: 3316.145 s, system: 57.614 s, faults: 67, rss: 2.2 GiB, commit: 1.0 GiB

fukusuket · 2024-05-01T00:59:06Z

I also confirmed #1334 (comment) again and there is no diff!

YamatoSecurity · 2024-05-01T01:22:32Z

@fukusuket Thanks! I updated the changelog and added a short hand option. Can you translate the Japanese changelog? (Ignore the part about This gives a speed benefit of TODO as I am still taking benchmarks. So far it looks very good!

fukusuket · 2024-05-01T03:20:06Z

@YamatoSecurity
Thank you so much for update changelog! I updated japanese changelog!

YamatoSecurity · 2024-05-01T07:53:19Z

@fukusuket I took benchmarks and updated the changelog. For big scans were most .evtx files are scanned and most rules are still used, there is about a 10% performance increase. When only enabling a single rule or scanning a single file, there is a big performance increase of 60% or more. 😄

Just one thing, when scanning lots of data (130GB), it takes a while to create the rule <=> evtx mapping, so could you add a message asking the user to wait?

Before:

Hayabusa rules: 162
Sigma rules: 3,948
Total enabled detection rules: 4,110

Evtx files loaded after channel filter: 1,220
Detection rules loaded after channel filter: 4,094

After:

Hayabusa rules: 162
Sigma rules: 3,948
Total enabled detection rules: 4,110

Enabling the channel filter. Please wait.

Evtx files loaded after channel filter: 1,220
Detection rules loaded after channel filter: 4,094

fukusuket · 2024-05-01T09:16:31Z

@YamatoSecurity
Thank you so much for benchmarking! I'm glad that the performance seems to have improved. I added console message!

YamatoSecurity

@fukusuket Looks great to me! Thanks so much for this!

feat: exclude rules when target evtx(channel) does not exists.

e780aea

fukusuket added the enhancement New feature or request label Apr 29, 2024

fukusuket added this to the v2.16.0 milestone Apr 29, 2024

fukusuket self-assigned this Apr 29, 2024

fukusuket added 10 commits April 29, 2024 22:37

feat: exclude evtx when rule target channel does not exists.

05ad734

test: update actions/checkout@v4

6fe50ec

chg: change the timing to output the number of items after filtering

df3fa96

feat: add --enable-all-rules/--scan-all-evtx-files options

36d3991

chg: if --enable-all-rules/--scan-all-evtx-files on skip filtering pr…

502983a

…ocess

chg: if --enable-all-rules/--scan-all-evtx-files on skip filtering pr…

0e9cfe8

…ocess

chg: if --enable-all-rules/--scan-all-evtx-files on skip filtering pr…

36bf6cf

…ocess

fix: cargo fmt

072e5d9

fix: cargo fmt

8fdc364

fix: cargo fmt

e370ee7

fukusuket added 2 commits April 30, 2024 01:00

fix: cargo fmt

8a92dc0

fix: cargo fmt

ce70607

fukusuket marked this pull request as ready for review April 29, 2024 16:21

fukusuket requested review from hitenkoku and YamatoSecurity April 29, 2024 16:21

hitenkoku approved these changes Apr 29, 2024

View reviewed changes

fukusuket changed the title ~~feat: exclude rules when target evtx(channel) does not exists.~~ feat: exclude rules/evtx when target Channel does not exists in files Apr 30, 2024

fix: add process for channel wildcard pattern

8cde8f8

YamatoSecurity added 2 commits May 1, 2024 10:01

update changelog

8bc08d2

2.16.0 dev build

94e8e19

doc: update changelog

d984595

YamatoSecurity added 3 commits May 1, 2024 16:42

update changelog

61820ce

add short options

2e7ccd4

update wording

c2e234a

fix: add console message

fc91f2d

Reword

8488fba

YamatoSecurity approved these changes May 1, 2024

View reviewed changes

YamatoSecurity merged commit d578a89 into main May 1, 2024
7 checks passed

YamatoSecurity deleted the 1317-exclude-rule-evtx-not-exists branch May 1, 2024 11:08

fukusuket mentioned this pull request May 8, 2024

[bug] Nothing is detected when using the -J, --JSON-input option with the timeline command because of Channel filter #1343

Closed

fukusuket mentioned this pull request Oct 12, 2024

[bug] High memory usage even in low-memory mode #1438

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: exclude rules/evtx when target `Channel` does not exists in files #1334

feat: exclude rules/evtx when target `Channel` does not exists in files #1334

fukusuket commented Apr 29, 2024 •

edited

Loading

fukusuket commented Apr 29, 2024 •

edited

Loading

fukusuket commented Apr 30, 2024

fukusuket commented May 1, 2024 •

edited

Loading

fukusuket commented May 1, 2024 •

edited

Loading

fukusuket commented May 1, 2024

YamatoSecurity commented May 1, 2024

fukusuket commented May 1, 2024

YamatoSecurity commented May 1, 2024

fukusuket commented May 1, 2024

YamatoSecurity left a comment

feat: exclude rules/evtx when target Channel does not exists in files #1334

feat: exclude rules/evtx when target Channel does not exists in files #1334

Conversation

fukusuket commented Apr 29, 2024 • edited Loading

What Changed

Test

baseline-evtx v8

fukusuket commented Apr 29, 2024 • edited Loading

hayabusa-sample-evtx(--enable-all-rules --scan-all-evtx-files)

fukusuket commented Apr 30, 2024

hayabusa-sample-evtx(no option)

fukusuket commented May 1, 2024 • edited Loading

fukusuket commented May 1, 2024 • edited Loading

baseline-evtx v8

fukusuket commented May 1, 2024

YamatoSecurity commented May 1, 2024

fukusuket commented May 1, 2024

YamatoSecurity commented May 1, 2024

fukusuket commented May 1, 2024

YamatoSecurity left a comment

Choose a reason for hiding this comment

feat: exclude rules/evtx when target `Channel` does not exists in files #1334

feat: exclude rules/evtx when target `Channel` does not exists in files #1334

fukusuket commented Apr 29, 2024 •

edited

Loading

fukusuket commented Apr 29, 2024 •

edited

Loading

fukusuket commented May 1, 2024 •

edited

Loading

fukusuket commented May 1, 2024 •

edited

Loading