fix: Added constraint that -A, --enable-all-rules option is required to -J, --JSON-input option

[fukusuket]

What Changed

• Closed [bug] Nothing is detected when using the -J, --JSON-input option with the timeline command because of Channel filter #1343
  • Added constraint that -A, --enable-all-rules option is required to -J, --JSON-input option
    • (After thinking about it, the -a, --scan-all-evtx-files option was unnecessary)

Evidence

Without --enable-all-rules option:

% ./hayabusa csv-timeline -f ../apt29/apt29_evals_day1_manual_2020-05-01225525.json -J -w -o out.csv -C
error: the following required arguments were not provided:
  --enable-all-rules

Usage: hayabusa csv-timeline --enable-all-rules --output <FILE> --JSON-input --no-wizard --clobber <--directory <DIR>|--file <FILE>|--live-analysis>

With --enable-all-rules option:

% ./hayabusa csv-timeline -f ../apt29/apt29_evals_day1_manual_2020-05-01225525.json -J -w -o out.csv --enable-all-rules -C

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/05/08 23:49

Total event log files: 1
Total file size: 385.3 MB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12 (Disabled)

Deprecated rules: 208 (5.06%) (Disabled)
Experimental rules: 856 (20.83%)
Stable rules: 240 (5.84%)
Test rules: 3,014 (73.33%)
Unsupported rules: 45 (1.09%) (Disabled)

Hayabusa rules: 162
Sigma rules: 3,948
Total detection rules: 4,110

Output profile: standard

Scanning in progress. Please wait.

[00:00:16] 1 / 1   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Rule Authors:

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (53)                  frack113 (34)                     Florian Roth (24)                 oscd.community (23)             │
│ Roberto Rodriguez (19)            OTR (16)                          Nasreddine Bencherchali (13)      Tim Shelton (9)                 │
│ Roberto Rodriguez @Cyb3r... (9)   Natalia Shornikova (5)            Thomas Patzke (4)                 Teymur Kheirkhabarov (4)        │
│ Christian Burkard (3)             Nikita Nazarov (3)                Timur Zinniatullin (3)            Christopher Peacock @sec... (2) │
│ Sean Metcalf (2)                  Markus Neis (2)                   John Lambert (2)                  Relativity (2)                  │
│ Oleg Kolesnikov @securon... (2)   E.M. Anhaus (2)                   Center for Threat Inform... (2)   X__Junior (2)                   │
│ Daniel Bohannon (2)               Bartlomiej Czyz (2)               Kutepov Anton (2)                 @ROxPinTeddy (2)                │
│ Yassine Oukessou (2)              Romaissa Adjailia (2)             Daniil Yugoslavskiy (1)           Samir Bousseaden (1)            │
│ Georg Lauenstein (1)              Ján Trenčanský (1)                Tuan Le (1)                       Patryk Prauze - ING Tech (1)    │
│ Dimitrios Slamaris (1)            Aleksey Potapov (1)               Tom Kern (1)                      Tobias Michalski (1)            │
│ Mustafa Kaan Demir (1)            Bartlomiej Czyz @bczyz1 (1)       Micah Babinski (1)                AlertIQ (1)                     │
│ Mark Russinovich (1)              Michael Haag (1)                  Gleb Sukhodolskiy (1)             Austin Songer (1)               │
│ Joshua Wright (1)                 Timur Zinniatullin oscd.... (1)   Perez Diego (1)                   SCYTHE (1)                      │
│ omkar72 (1)                       Hieu Tran (1)                     Bhabesh Raj (1)                   Tim Burrell (1)                 │
│ Fukusuke Takahashi (1)            Max Altgelt (1)                   Vasiliy Burov (1)                 Swachchhanda Shrawan Poudel (1) │
│ SCYTHE @scythe_io (1)                                                                                                                 │
╰─────────────────────────────────╌─────────────────────────────────╌─────────────────────────────────╌─────────────────────────────────╯

Results Summary:

Events with hits / Total events: 28,049 / 196,081 (Data reduction: 168,032 events (85.70%))

Total | Unique detections: 58,388 | 177
Total | Unique critical detections: 1 (0.00%) | 1 (0.00%)
Total | Unique high detections: 375 (0.64%) | 34 (25.42%)
Total | Unique medium detections: 2,956 (5.06%) | 67 (16.95%)
Total | Unique low detections: 40,232 (68.90%) | 30 (37.85%)
Total | Unique informational detections: 14,824 (25.39%) | 45 (19.21%)

Dates with most total detections:
critical: 2020-05-02 (1), high: 2020-05-02 (375), medium: 2020-05-02 (2,956), low: 2020-05-02 (40,232), informational: 2020-05-02 (14,824)

Top 5 computers with most unique detections:
critical: SCRANTON.dmevals.local (1)
high: SCRANTON.dmevals.local (27), NASHUA.dmevals.local (8), NEWYORK.dmevals.local (1)
medium: SCRANTON.dmevals.local (54), NASHUA.dmevals.local (16), NEWYORK.dmevals.local (9), UTICA.dmevals.local (4)
low: SCRANTON.dmevals.local (26), NASHUA.dmevals.local (18), NEWYORK.dmevals.local (6), UTICA.dmevals.local (3)
informational: SCRANTON.dmevals.local (38), NASHUA.dmevals.local (23), NEWYORK.dmevals.local (22), UTICA.dmevals.local (10)

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                      Top high alerts:                                      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Malicious Service Installations (1)                       Potential Shellcode Injection (122)                   │
│ n/a                                                       File Creation Date Changed to Another Year (108)      │
│ n/a                                                       Suspicious Svchost Process Access (74)                │
│ n/a                                                       HackTool - SysmonEnte Execution (24)                  │
│ n/a                                                       Potential File Overwrite Via Sysinternals SDelete (6) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                        Top low alerts:                                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Alternate PowerShell Hosts - PowerShell Module (954)      Proc Access (39,259)                                  │
│ Raw Access Read (652)                                     Scheduled Task Created - Registry (298)               │
│ Process Ran With High Privilege (440)                     Creation of an Executable by an Executable (256)      │
│ Python Initiated Connection (348)                         Possible Timestomping (209)                           │
│ Potential Binary Or Script Dropper Via PowerShell (293)   Susp CmdLine (Possible LOLBIN) (38)                   │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ PwSh Pipeline Exec (5,080)                                Deleted File Archived (422)                           │
│ Net Conn (4,392)                                          PwSh Scriptblock (414)                                │
│ File Created (1,649)                                      Proc Terminated (401)                                 │
│ Proc Exec (906)                                           Pipe Conn (362)                                       │
│ PowerShell Decompress Commands (620)                      Pipe Created (84)                                     │
╰─────────────────────────────────────────────────────────╌───────────────────────────────────────────────────────╯

Saved file: out.csv (159.8 MB)

Elapsed time: 00:00:17.2127

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls

[YamatoSecurity]

@fukusuket Thanks for this! Very sorry, after thinking about this, it might be better just to automatically turn off the channel filter whenever -J is specified so that the user doesn't have to worry about another option. What do you think? If it requires changing lots of code though, then lets go with this simple fix.
Is -a not required because it is scanning a file? Does -J support directories?

[fukusuket]

@YamatoSecurity
I was concerned that it was a little difficult to understand that the Channel filter ON/OFF was implicitly changed using the J option🤔

However, after thinking about it again... I feel that there are very few users who want the J option and the Channel filter, so I agree👍

I will close this PR!(and reopen with new branch)