Artifact definitions filter helper #1883

jnettesheim · 2018-05-14T23:45:10Z

One line description of pull request

Issue 1313, plaso engine updates to add forensic artifact filtering capabilities.

Description:

Per discussion in #1732 adding plaso engine changes in separate PR.

Related issue (if applicable): fixes #

Notes:

All contributions to Plaso undergo code
review. This makes sure
that the code has appropriate test coverage and conforms to the Plaso style
guide.

One of the maintainers will examine your code, and may request changes.

Checklist:

Local tests pass
Tests on TravisCI pass
Codacy passes (or flags issues that you think are acceptable)
Coveralls indicates test coverage is sufficient

If new dependencies are required:

l2tdevtools has been updated

…om/jnettesheim/plaso into artifact-definitions-filter-helper

joachimmetz

Please add tests, there are a couple of complex functions added where I would like to see tests to better understand them.

Besides that various style nits.

joachimmetz · 2018-05-15T03:49:25Z

plaso/engine/artifact_filters.py

+from dfwinreg import registry_searcher
+
+from plaso.engine import path_helper
+


joachimmetz · 2018-05-15T03:49:50Z

plaso/engine/artifact_filters.py

+from plaso.engine import path_helper
+
+class ArtifactDefinitionsFilterHelper(object):
+  """Helper to create filters based on artifact defintions.


defintions => definitions

joachimmetz · 2018-05-15T03:50:20Z

plaso/engine/artifact_filters.py

+class ArtifactDefinitionsFilterHelper(object):
+  """Helper to create filters based on artifact defintions.
+
+  Builds extraction and parsing filters from forensic artifact definitions.


what are "parsing filters" ?

joachimmetz · 2018-05-15T03:51:06Z

plaso/engine/artifact_filters.py

+  https://github.com/ForensicArtifacts/artifacts/blob/master/docs/Artifacts%20definition%20format%20and%20style%20guide.asciidoc
+  """
+
+  ARTIFACT_FILTERS = 'ARTIFACT_FILTERS'


If these are only used by this class please prefix with an underscore.

Removed several of these and added underscore to one only used in this class.

joachimmetz · 2018-05-15T03:51:51Z

plaso/engine/artifact_filters.py

+    self._knowledge_base = knowledge_base
+
+  def BuildFindSpecs(self, environment_variables=None):
+    """Build find specification from a forensic artifact definitions.


Build => Builds

joachimmetz · 2018-05-15T04:33:21Z

plaso/engine/path_helper.py

@@ -133,3 +147,26 @@ def GetRelativePathForPathSpec(cls, path_spec, mount_path=None):
      location = location[len(mount_path):]

    return location
+
+  @classmethod
+  def ExpandUserHomeDirPath(cls, path, user_accounts):


Windows calls this profile directory not home directory, also don's use abbreviations such as Dir. This is more clear for an international audience.

This could be for any operating system. Removed abbreviation.

joachimmetz · 2018-05-15T04:35:13Z

tests/engine/artifact_filters.py

+
+  @shared_test_lib.skipUnlessHasTestFile(['artifacts'])
+  @shared_test_lib.skipUnlessHasTestFile(['SYSTEM'])
+  def testBuildRegistryFindSpecs(self):


testBuildRegistryFindSpecs => testBuildFindSpecsWithRegistry or testBuildFindSpecsOnRegistry

joachimmetz · 2018-05-15T04:35:34Z

tests/engine/artifact_filters.py

+  @shared_test_lib.skipUnlessHasTestFile(['artifacts'])
+  @shared_test_lib.skipUnlessHasTestFile(['SYSTEM'])
+  def testBuildRegistryFindSpecs(self):
+    """Tests the BuildFindSpecs function for registry artifacts."""


for => on
registry => Registry

joachimmetz · 2018-05-15T04:36:33Z

tests/engine/artifact_filters.py

+  @shared_test_lib.skipUnlessHasTestFile(['System.evtx'])
+  @shared_test_lib.skipUnlessHasTestFile(['testdir', 'filter_1.txt'])
+  @shared_test_lib.skipUnlessHasTestFile(['testdir', 'filter_3.txt'])
+  def testBuildFileFindSpecs(self):


testBuildFileFindSpecs => testBuildFindSpecsWithFileSystem or testBuildFindSpecsOnFileSystem

joachimmetz · 2018-05-15T04:38:37Z

tests/engine/artifact_filters.py

+
+    # Three key paths found
+    self.assertEqual(len(key_paths), 3)
+


Please add tests for BuildFindSpecsFromFileArtifact, _CheckKeyCompatibility, _ExpandRecursiveGlobs, _BuildRecursivePaths. The last 2 methods I have a hard time understanding from the code directly.

codecov · 2018-05-27T06:58:21Z

Codecov Report

Merging #1883 into master will increase coverage by 0.01%.
The diff coverage is 88%.

@@            Coverage Diff             @@
##           master    #1883      +/-   ##
==========================================
+ Coverage   84.03%   84.05%   +0.01%     
==========================================
  Files         435      436       +1     
  Lines       30998    31174     +176     
==========================================
+ Hits        26049    26202     +153     
- Misses       4949     4972      +23

Impacted Files	Coverage Δ
plaso/engine/path_helper.py	`98.11% <100%> (+1.62%)`	⬆️
plaso/engine/artifact_filters.py	`80% <80%> (ø)`
plaso/storage/interface.py	`77.35% <0%> (-0.42%)`	⬇️
plaso/parsers/trendmicroav.py	`86.52% <0%> (-0.08%)`	⬇️
plaso/formatters/trendmicroav.py	`96.87% <0%> (+1.04%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46d484b...10c448a. Read the comment docs.

jnettesheim · 2018-05-28T06:13:58Z

@joachimmetz - PTAL.

joachimmetz · 2018-06-01T04:56:37Z

@jnettesheim I'm having a look this weekend.

joachimmetz · 2018-06-04T05:00:57Z

plaso/engine/artifact_filters.py

+          dynamically populate environment variables in key.
+      user_accounts (list[str]): identified user accounts stored in the
+          knowledge base.
+      find_specs (dict[artifacts.artifact_types]):  Dictionary containing


find_specs (dict[artifacts.artifact_types]): Dictionary containing ... I had a brief look at this but this arg looks weird to me. Find specs are a dfVFS class, but type information says artifacts.artifact_types ?

returning a list of dfvfs.FindSpect will simplify the interface.

joachimmetz · 2018-06-04T05:03:24Z

@jnettesheim I've started looking at this. I've made some minor changes, mainly nits about docstrings and style formatting. I'll continue as soon as time permits

joachimmetz

more comments and changes

joachimmetz · 2018-06-09T15:51:13Z

plaso/engine/artifact_filters.py

+
+from __future__ import unicode_literals
+
+import logging


Use from plaso.engine import logger instead of logging

joachimmetz · 2018-06-09T16:15:50Z

plaso/engine/artifact_filters.py

+    """
+    find_specs = {}
+
+    artifact_definitions = []


this loop can be merged with the next loop and remove the need for the artifact_definitions list

joachimmetz · 2018-06-09T16:27:59Z

plaso/engine/artifact_filters.py

+          dynamically populate environment variables in key.
+      user_accounts (list[str]): identified user accounts stored in the
+          knowledge base.
+      find_specs (dict[artifacts.artifact_types]):  Dictionary containing


returning a list of dfvfs.FindSpect will simplify the interface.

joachimmetz · 2018-06-09T17:04:11Z

tests/engine/path_helper.py

+        '\\Windows\\Test\\*\\*\\*\\*\\*\\*\\*\\*',
+        '\\Windows\\Test\\*\\*\\*\\*\\*\\*\\*\\*\\*',
+        '\\Windows\\Test\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*']
+    self.assertItemsEqual(paths, check_paths)


not portable to python 3
Instance of 'PathHelperTest' has no 'assertItemsEqual' member (no-member)

joachimmetz · 2018-06-09T17:08:25Z

plaso/engine/path_helper.py

@@ -3,6 +3,9 @@

 from __future__ import unicode_literals

+import logging


user engine.logger instead

joachimmetz · 2018-06-09T17:08:49Z

plaso/engine/path_helper.py

@@ -42,9 +49,19 @@ def ExpandWindowsPath(cls, path, environment_variables):
          not path_segment.endswith('%')):
        continue

-      lookup_key = path_segment.upper()[1:-1]
+      check_for_drive_letter = False
+      if path_segment.upper().startswith('%%ENVIRON_'):


store upper case path in a variable seeing you're reusing it

joachimmetz · 2018-06-09T17:32:49Z

plaso/engine/path_helper.py

+          _, _, new_path = new_path.rpartition(':')
+        user_paths.append(new_path)
+    else:
+      user_paths = [path]


drive letter is not stripped here

joachimmetz · 2018-06-09T17:39:29Z

tests/engine/artifact_filters.py

+    path_segments = ['Users', 'testuser2', 'Documents', 'WindowsPowerShell',
+                     'profile\\.ps1']
+    self.assertEqual(
+        find_specs[artifact_types.TYPE_INDICATOR_FILE][2]._location_segments,


seeing file_specs is a dict this test will be flaky

joachimmetz · 2018-06-09T18:39:18Z

Yes pending completion of CI tests, then I'll merge

jnettesheim added 7 commits May 4, 2018 00:24

Initial commit artifact artifact definitions filter helper

99d22df

Minor code changes and adding path_helper changes

a220538

Docstring updates

f40837a

Initial commit artifact artifact definitions filter helper

0b47f08

Minor code changes and adding path_helper changes

9f75e53

Docstring updates

15ebd78

Merge branch 'artifact-definitions-filter-helper' of https://github.c…

ae02c6e

…om/jnettesheim/plaso into artifact-definitions-filter-helper

joachimmetz self-requested a review May 15, 2018 03:34

joachimmetz requested changes May 15, 2018

View reviewed changes

Code Review Changes

fc73658

jnettesheim added 5 commits May 27, 2018 00:00

Merge branch 'master' into artifact-definitions-filter-helper

e432f5c

Codefactor and other whitespace changes

4652ba9

PyLine and codefactor changes

ba8a693

Further pylint changes and a few nits

4e16ceb

Whitepace

d79edc3

joachimmetz added 4 commits June 4, 2018 06:19

Update artifact_filters.py

8c01638

Update artifact_filters.py

927cb28

Update artifact_filters.py

f8004d7

Update artifact_filters.py

6fe99d1

joachimmetz reviewed Jun 4, 2018

View reviewed changes

Changes for review

116b9fc

joachimmetz reviewed Jun 9, 2018

View reviewed changes

joachimmetz approved these changes Jun 9, 2018

View reviewed changes

Changes for review

10c448a

joachimmetz merged commit 12454e4 into log2timeline:master Jun 9, 2018

jnettesheim deleted the artifact-definitions-filter-helper branch June 9, 2018 20:08

jnettesheim mentioned this pull request Jun 16, 2018

Issue #1313 added artifacts filter support for files and Windows regi… #1732

Merged

1 task

		from dfwinreg import registry_searcher

		from plaso.engine import path_helper

		@@ -3,6 +3,9 @@

		from __future__ import unicode_literals

		import logging

Artifact definitions filter helper #1883

Artifact definitions filter helper #1883

Conversation

jnettesheim commented May 14, 2018

One line description of pull request

Description:

Notes:

Checklist:

joachimmetz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 27, 2018 • edited Loading

Codecov Report

jnettesheim commented May 28, 2018 • edited Loading

joachimmetz commented Jun 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joachimmetz commented Jun 4, 2018

joachimmetz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joachimmetz commented Jun 9, 2018

codecov bot commented May 27, 2018 •

edited

Loading

jnettesheim commented May 28, 2018 •

edited

Loading