Skip to content

misp-stix v2.4.193 - Finalised the Observed Data and Observable objects Converter
Compare
Choose a tag to compare
@chrisr3d chrisr3d released this 21 Jun 13:18
· 191 commits to main since this release
v2.4.193
ac98494
This commit was signed with the committer’s verified signature.
chrisr3d Christian Studer
GPG key ID: 6BBED1B63A6D639F
Learn about vigilant mode.

v2.4.193 - 2024-06-06

Included in this release

  • The Observed Data & Observable objects Converter is now finalised as a separate converter and the branch containing the changes is now merged
    • Including some major improvement on the Observable objects conversion to MISP, such as:
      • standalone Observable objects are now correctly parsed
      • long lists of Observables referenced by - or containing in - a single Observed Data object, with no specific meaning are now correctly handled and parsed as separate objects even though they are now strictly respecting the Observable objects format specification
    • Observable objects mapping improved

Add

  • [tests] Tests for Email Message objects - and references - import from STIX 2.x
  • [stix2 import] Updated the STIX 2.x Email objects mappings
  • [stix2 import] Added organisation_uuid argument to use to generate the custom clusters UUID
  • [tests] Tests for Autonomous System observable objects with observed data import from STIX 2.x
  • [stix2 import] Parsing Observed Data with Autonomous System observable objects from converters

Chg

  • [poetry] Bumped latest version in lock file
  • [poetry] Updated version
  • [tests] Updated tests for domain-ip objects import from STIX 2.1 to cover specific cases with UUIDs handling
  • [stix2 import] Adding source information to the custom Galaxy Clusters imported from STIX 2.x objects
  • [stix2 import] Using the file observable references parsing method to convert v2.0 observable objects
  • [stix2 import] Making the network-traffic objects parsing more generic
  • [stix2 import] Simplify loading JSON files
  • [stix2 import] Added generic conversion methods for observable objects associated to observed data objects imported as MISP objects
  • [tests] Deduplicating existing tests for external directory observable objects

Fix

  • [stix2 import] Making Python 3.8 & 3.9 happy with the typing
  • [stix2 import] Post Observed Data Converter merge clean up and reassembling
  • [stix2 import] Merged missing conflicts
  • [stix2 import] Fixed UUID handling for email object attributes parsed from email-message references
  • [stix2 import] Fixed domain-ip objects UUID handling
  • [stix2 import] Handling domains resolving other domains with object references
  • [stix2 import] Removed unnecessary intermediary method
  • [stix2 import] Avoiding domain-name observable objects to be skipped because they're referenced by another domain-name object
  • [stix2 import] Fixed domain-ip attributes UUIDs handling
  • [stix2 import] Fixed domain-ip object attributes handling as _sanitise_attribute_uuid already returns a dict with the uuid key included
  • [stix2 import] Fixed _observable variable name
  • [stix2 import] Protocols error message made clearer
  • [tests] Better UUID tests for objects imported from STIX 2.x Network Traffic Observable objects
  • [stix2 import] Better internal http-request objects import from Observable objects
  • [stix2 import] Better handling of attributes uuid for values converted from internal Network Traffic Observable objects
  • [stix2 import] Fixing the internal STIX2 Network Traffic Observable objects and references IDs handling
  • [stix2 import] Fixed Network Traffic Observable objects from internal STIX 2.x content parsing
  • [stix2 import] Fixed STIX 2.0 Network Traffic Observable objects parsing
  • [stix2 import] Added missing protocol_attribute property in STIX2Mapping parent class
  • [stix2 import] Better handling of internal Galaxy & Cluster description
  • [stix2 import] Updated Network Traffic observables objects mapping to MISP objects
  • [stix2 import] Importing Network Traffic observable objects referenced by external Observed Data objects with the network-traffic generic MISP object template
  • [stix2 import] Fixed email message objects parsing
  • [stix2 import] Invalid typehint
  • [stix2 import] Avoid running git process
  • [stix2 import] No longer require to exclude patterns with 'AND' and 'OR'
  • [stix2 import] Avoiding issues introduced since we updated the observables fetching method
  • [stix2 import] Avoiding issues with the internal STIX 2.1 Autonomous System observable objects fetching method
  • [stix2 import] Making the multiple observables fetching method available to both internal and external STIX 2 Observed Data object converters
  • [stix2 import] Avoiding issues with ssdeep hash type in STIX 2.0 external content
  • [stix2 import] Updated pe object mapping with the compilation-timestamp attribute
  • [stix2 import] Better STIX 2.0 windows-pebinary-ext within File observable object handling
  • [stix2 import] MISP object references handling method name
  • [stix2 import] Error exceptions handling method name
  • [stix2 import] Fixed the MISP object reference duplicates checking
  • [stix2 import] Deduplication of MISP object references
  • [stix2 import] Fixed File PE extension parsing method name to avoid confusion with the generic method used then from the observable objects converter class
  • [stix2 import] Avoiding issues with observables references, by keeping track of each reference within a single STIX 2.0 observed data objects list
  • [stix2 import] Returning MISPAttributes in some generic observable objects conversion methods
  • [stix2 import] Fixed wrong variable name for a MISP object meta fields check
  • [tests] Fixed tests for external STIX 2.x SDOs imported as Galaxy Clusters following the recent add of the organisation_uuid argument
  • [stix2 import] Setting single_event when parsing a bundle with a single report/grouping, to avoid issues raised with multiple reports/groupings handling methods
  • [stix2 import] Fixed the case with multiple events as result
  • [stix2 import] In the end we have to parse the Sighting & Opinion objects and convert them as MISP Sighting when they are used
  • [stix2 import] Fixed relationships handling between sighting & opinion objects, and their references
  • [stix2 import] Fixed MISP Sightings handling
  • [stix2 import] Removed unused import
  • [stix2 import] Avoiding issues with STIX 2.x content coming from a TAXII collection or embedded into a single list instead of a Bundle
  • [stix2 import] Removed unsued import & added missing blank lines to make pep8 happy
  • [stix2 import] Added the missing sorting statement for observable objects types passed to match mapping
  • [stix2 import] Clearer observable objects mapping handling in the observed data conversion methods
  • [stix2 import] Reusing the STIX 2.1 observable objects fetching method
  • [stix2 import] Setting MISP objects timestamp with the datetime value instead of an int
  • [stix2 import] Fixed AttributeError with method from parent conversion class
  • [tests] Passing observable ids instead of objects themselves for some tests that only need to know about ids
  • [tests] Testing MISP Object comment when its uuid is v5
  • [stix2 import] Added observed data id as comment for misp objects converted from STIX 2.0 when it has a v5 uuid
  • [stix2 import] Some typings fixed
  • [stix2 import] Quick reordering to allow more reusability
  • [stix2 import] Avoiding issues with marking definitions referenced but not present in a file
  • [stix2 import] Better tags from indicators parsing & simplified the tags handling method
  • [stix2 import] Some methods deduplication between main parser & converters
  • [stix2 import] Yield syntax
  • [stix2 import] Copy-paste typo
  • [tests] Quick fix on the created or created_time field from a process observable object
  • [stix2 import] Avoid future potential issues with object names in generic conversion methods
  • [stix2 import] Quick fix in the Process observable objects associated with Observed Data objects conversion method
  • [stix2 import] Utilising the newly added environment-variables attribute to properly import the environment variables & arguments of a STIX 2.x process object
  • [stix2 import] Updated typings
  • [stix2 import] Typo on the generic observable object parsing method to call
  • [stix2 import] Deduplication in the STIX 2.1 Directory objects parsing
  • [stix2 import] Removed duplicated MISP Attribute dict creation methods
  • [stix2 import] Better handling of generic observable object parsers
  • [stix2 import] Quick clean-up on some observed data method arguments
  • [stix2 import] Fixed Observable objects types mapping
  • [stix2 import] Better overall UUID sanitation & comments handling for MISP attributes creation
  • [tests] Removed spec_version fields in STIX 2.0 samples
  • [stix2 import] Properly calling the UUID sanitation method
  • [stix2 import] Removing unused variable in marking definitions parsing
  • [stix2 import] Fixed directory observable objects parsing method header
  • [tests] Added missing tests for directory path attribute types
  • [stix2 import] Reuse of the method parsing Directory observable objects with an id field
  • [stix2 import] Using the AS value parsing method for an AS value that was missing it
  • [stix2 import] Fixed directory mapping
  • [stix2 import] Quick pep8 clean-up
  • [stix2 import] Fixed the converters composition
  • [tests] A tiny clarification change
  • [stix2 import] Observable objects fetcher moved to the parent class as it will be reused for internal & external conversion
  • [stix2 import] Quick syntax fix

Wip

  • [tests] Tests for domain-ip objects import from external STIX 2.x
  • [tests] Tests for Network Traffic Observable objects imported from external STIX 2 bundles as network-traffic objects
  • [stix2 import] Better conversion of Network Traffic references observable objects
  • [stix2 import] Parsing Network Traffic Observable objects referenced in Observed Data from the Observed Data Converter
  • [stix2 import] Parsing EmailMessage observable objects from Observed Data converter
  • [stix2 import] Reusing EmailMessage observable parsing method
  • [stix2 import] Parsing DomainName and IP observable objects resolving each others
  • [stix2 import] Parsing archive-ext from standalone file observable objects
  • [tests] Added tests for file objects with extensions
  • [stix2 import] Parsing File objects extensions
  • [stix2 import] Parsing STIX 2.0 Observed Data objects with multiple embedded observable objects with no specific mapping
  • [stix2 import] Better observable objects fetching methods
  • [stix2 import] Parsing Observable objects referenced together by a single Observed Data object with no specific mapping
  • [tests] Tests for File objects and their Directory & Artifact references import from STIX 2.x
  • [stix2 import] Converting File observable objects and their Directory & Artifact references
  • [stix2 import] Better observable objects parsing
  • [stix2 import] Better embedded directory observable object references parsing
  • [stix2 import] Parsing the observable objects referenced with contains_refs references in a generic method that will be reused later
  • [tests] Tests for some objects referenced by Opinions
  • [tests] Tests for user account observable objects referenced by registry keys as creators
  • [stix2 import] Handling cases where some STIX 2.1 observable objects are referenced by multiple observed data objects
  • [stix2 import] Parsing User Account observables referenced by registry keys to be the creator reference
  • [tests] Tests for STIX 2.x Windows Registry Key objects conversion
  • [stix2 import] Converting STIX 2.x Windows Registry Key objects
  • [tests] Tests for External STIX 2.x User Account observable objects import as MISP objects
  • [stix2 import] Parsing external STIX 2.x User Account observable objects from converters
  • [tests] Tests for external STIX 2.x Process observable objects associated with Observed Data object import as MISP process objects
  • [stix2 import] Parsing Process observable objects from converters
  • [tests] Tests for X509 Certificate objects import from STIX 2.x
  • [stix2 import] Reusing the generic observed data parsing methods to support X509 observable objects conversion from the converters
  • [tests] Tests for external Software Observable objects - within or referenced by Observed data objects - import to MISP objects
  • [stix2 import] Reusing the generic observed data parsing methods to support Software observable objects conversion from the converters
  • [tests] Tests for external STIX 2.x Observed Data with artifact observable objects import to MISP
  • [stix2 import] Parsing external STIX 2.x Observed data with artifact observable objects, from converters
  • [stix2 import] Handling the observable relationships after the observed data objects are all parsed
  • [tests] Tests for Observable objects converted in a generic way to MISP attributes
  • [stix2 import] Parsing some Observable objects - converted to MISP attributes - in a generic way, from Observed Data converter
  • [tests] Tests for email address observable objects in observed data import from external STIX 2.x content
  • [stix2 import] Parsing email address observable objects in observed data from external STIX 2.x content, in converters
  • [tests] Tests for directory observable objects import from STIX 2.x
  • [stix2 import] Porting Observed Data objects conversion ability to converters, starting with Directory objects

Pull Requests

  • Merge pull request #65 from JakubOnderka/fix-git
Assets 2
Loading