added low memory mode option and organize menu

[hitenkoku]

What Changed

• added low-memory-mode option
• added conflicts with remove-duplicate-detections, remove-duplicate-data to low-memory-mode option

[codecov]

Codecov Report

Attention: Patch coverage is 96.22642% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 81.09%. Comparing base (ce04f24) to head (20d2177).

Files	Patch %	Lines
src/detections/configs.rs	86.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1302      +/-   ##
==========================================
+ Coverage   81.06%   81.09%   +0.03%     
==========================================
  Files          27       27              
  Lines       24626    24676      +50     
==========================================
+ Hits        19963    20012      +49     
- Misses       4663     4664       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[YamatoSecurity]

@hitenkoku Thanks so much! The low memory option looks good. Could you move the two options from Input to General Options?

  -J, --JSON-input       Scan JSON formatted logs instead of .evtx (.json or .jsonl)
  -x, --recover-records  Carve evtx records from slack space (default: disabled)

[hitenkoku]

@YamatoSecurity I apologize for the omission in response.
I fixed in c304c12.
Would you check it?

[YamatoSecurity]

@hitenkoku LGTM! Thanks so much!

[fukusuket]

@hitenkoku
I confirmed that the option has been added!
I have one question! The memory usage seems to be not reduced with the -s option, are you planning to implement it in another PR?

% ./hayabusa csv-timeline -d ../all-evtx -s --debug -o new.csv -C
...
Saved file: new.csv (2.3 GB)

Elapsed time: 00:10:15.2493

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240308_100204.log for details.

Rule Parse Processing Time: 00:00:01.713
Analysis Processing Time: 00:09:50.914
Output Processing Time: 00:00:22.297

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    10.0 GiB    10.0 GiB     0          10.0 GiB
 committed:     1.0 GiB    10.0 GiB   748.8 GiB  -738.8 GiB                          ok

[hitenkoku]

@fukusuket Sorry for my late reply.
I'll check it.

[hach1yon]

LGTM

[hitenkoku]

@fukusuket Sorry for my late reply.
I fixed in 20d2177.
Would you check it?

• without -s option

> .\1298.exe csv-timeline -d ../all-evtx --debug -o old.csv -C -D -u -n -w
...
Saved file: old.csv (3.0 GB)

Elapsed time: 00:05:28.2577

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Rule Parse Processing Time: 00:00:01.871
Analysis Processing Time: 00:04:36.890
Output Processing Time: 00:00:48.569

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    12.0 GiB    12.0 GiB     0          12.0 GiB
 committed:    11.2 GiB    48.4 GiB    37.4 GiB    10.9 GiB
     reset:     0
    purged:    10.8 GiB
   touched:   128.5 KiB    26.1 MiB    95.2 GiB   -95.2 GiB                          ok
  segments:    17         418         408          10                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0         905.7 Ki   -905.7 Ki                           ok
-abandoned:     2           2           0           2                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   310.9 Ki
    resets:     0
    purges:    11.6 Ki
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   330.590 s
   process: user: 302.000 s, system: 8.046 s, faults: 7653510, rss: 10.7 GiB, commit: 11.2 GiB

• with -s option

> .\1298.exe csv-timeline -d ../all-evtx --debug -o new.csv -C -D -u -n -w -s
...
Saved file: new.csv (3.0 GB)

Elapsed time: 00:05:34.1893

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Rule Parse Processing Time: 00:00:01.887
Analysis Processing Time: 00:05:32.614
Output Processing Time: 00:00:00.133

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     3.0 GiB     3.0 GiB     0           3.0 GiB
 committed:     2.4 GiB    49.6 GiB    51.6 GiB    -1.9 GiB                          ok
     reset:     0
    purged:    18.0 GiB
   touched:   128.5 KiB    29.6 MiB    89.8 GiB   -89.8 GiB                          ok
  segments:    23         475         462          13                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0         914.4 Ki   -914.4 Ki                           ok
-abandoned:     2           2           0           2                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   278.7 Ki
    resets:     0
    purges:     6.2 Ki
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   335.906 s
   process: user: 379.703 s, system: 4.031 s, faults: 6267949, rss: 1.3 GiB, commit: 1.5 GiB

[fukusuket]

@hitenkoku
Thank you so much for quick fix :) I also confirmed json-timeline works! LGTM!!🚀

% ./hayabusa json-timeline -d ../all-evtx -o out.csv --debug -C -w -s
...
Saved file: out.csv (3.1 GB)

Elapsed time: 00:10:24.919

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240309_022245.log for details.

Rule Parse Processing Time: 00:00:01.787
Analysis Processing Time: 00:10:23.044
Output Processing Time: 00:00:00.067

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:     1.0 GiB     2.0 GiB   609.4 GiB  -607.4 GiB                          ok