Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract Windows event logs messages attributes #2910

Merged
merged 38 commits into from
Nov 2, 2023

Conversation

roshanmaskey
Copy link
Collaborator

@roshanmaskey roshanmaskey commented Sep 27, 2023

Windows event log contains details about the event in the message field (EventData XML attribute). Extracting the information in the EventData would enable analysts to query/filter event logs based on the attributes in EventData.

The figure below shows the high-level schema.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
  </System>
  <EventData>
  </EventData>
</Event>

Plaso extracts the fields and stores the values as a list in the field strings.

"strings": [ "S-1-5-18", "WIN-MDLVGLNGOM0$", "WORKGROUP", "0x00000000000003e7", "S-1-5-18", "SYSTEM", "NT AUTHORITY", "0x00000000000003e7", "5", "Advapi ", "Negotiate", "-", "{00000000-0000-0000-0000-000000000000}", "-", "-", "0", "0x000000000000026c", "C:\\Windows\\System32\\services.exe", "-", "-", "%%1833", "-", "-", "-", "%%1843", "0x0000000000000000", "%%1842" ]

This PR maps the extracted event log strings to the respective attributes using winevt.yaml (data/winevt.yaml`)

Checks

  • All tests succeed.
  • Unit tests added.
  • e2e tests added.
  • Documentation updated.

Closing issues

Put closes #2911 in your comment to auto-close the issue that your PR fixes
(if such).

@jaegeral
Copy link
Collaborator

fwiw, I think once this is in, it could mean we can make a lot of those field mappings better in https://github.com/google/timesketch/blob/master/data/sigma_config.yaml

@jkppr jkppr linked an issue Sep 29, 2023 that may be closed by this pull request
Copy link
Collaborator

@jkppr jkppr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work on this feature. Looking forward to the great possibilities we will get with the mapping.

Please see the comments below. If you have questions regarding the multi analyzer approach, let me know.

data/winevt.yaml Outdated Show resolved Hide resolved
data/winevt.yaml Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature.py Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature_plugins/winevt.py Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature_plugins/winevt.py Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature_plugins/winevt.py Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature_plugins/winevt.py Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature_plugins/winevt.py Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature_plugins/winevt.py Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature_plugins/winevt.py Outdated Show resolved Hide resolved
@roshanmaskey roshanmaskey marked this pull request as ready for review October 9, 2023 03:24
Copy link
Collaborator

@jkppr jkppr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second round:
This works really well already! Main comments are around wording of errors and checking required dict entries.

What are your plans for moving the existing feature_extraction.py into the plugin system?

If you are fine with it, I would like to test both changes together, so either merge it in one PR or in two dependent PRs.

data/winevt_features.yaml Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature.py Outdated Show resolved Hide resolved
timesketch/lib/analyzers/feature.py Outdated Show resolved Hide resolved
@berggren berggren added this to the Release: 20231206 milestone Oct 30, 2023
@jkppr
Copy link
Collaborator

jkppr commented Nov 2, 2023

Notes from offline sync:

  • This PR will merge the new feature extraction plugins currently including the (old) regex extractor and the new winevt mapping extractor.
  • The old feature extractor analyzer will be removed in a next separate PR.
  • Documentation will be updated in a separate PR as well.

@jkppr jkppr merged commit 7a7b575 into google:master Nov 2, 2023
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Windows Event Logs Message Fields Extraction
4 participants