-
-
Notifications
You must be signed in to change notification settings - Fork 437
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DataModel Plugin #2494
Open
cristinaascari
wants to merge
63
commits into
develop
Choose a base branch
from
datamodel_plugin
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
DataModel Plugin #2494
Changes from 57 commits
Commits
Show all changes
63 commits
Select commit
Hold shift + click to select a range
fa34c52
added data models
cristinaascari 604738b
updated data models
cristinaascari 1cf2e0a
updated data models
cristinaascari 45b7f10
updated data models
cristinaascari a234319
updated data models
cristinaascari 38c0c29
fix
cristinaascari 3bc44c5
fix
cristinaascari 29f4313
update data models
cristinaascari 1913000
fix
cristinaascari 90187fb
update file data model
cristinaascari b4462a0
update file data model
cristinaascari 4140f2f
updates file data model
cristinaascari 5ab5614
updates file data model
cristinaascari e82f98e
updates file data model
cristinaascari f12776d
updates
cristinaascari 845406c
updates data models
cristinaascari 6f91d84
fix
cristinaascari 0a6529c
fix
cristinaascari 210f008
fix CharFiled max_length
cristinaascari 88376fc
fixes
cristinaascari 858b200
fixes
cristinaascari 7ebe465
Added BaseDataModel
cristinaascari 3717b95
updated BaseDataModel
cristinaascari 8e78524
updates data models
cristinaascari bdddb86
Merge branch 'develop' into datamodel_plugin
cristinaascari e6b289d
updates data models
cristinaascari beb3ff8
added admin data models
cristinaascari 6261ec0
field names fixes
cristinaascari de6b938
fix ip data model
cristinaascari dca7a50
fix ip data model admin
cristinaascari db3f59a
fixes FileDataModel fields
cristinaascari b0119d6
Update external_references field
cristinaascari e0e64da
fix linters
cristinaascari a7beacb
fixes
cristinaascari 0b934ff
updates signature field
cristinaascari dbbd637
updates tags field
cristinaascari f57abe3
fix
cristinaascari 1018903
updates data model admin
cristinaascari 3dab81c
fix
cristinaascari 8c96d36
moved data_model into api_app
cristinaascari 7c81328
moved data_model into api_app
cristinaascari 48bfb83
Added unique_together constraint in IETFReport
cristinaascari c03102a
Merge branch 'develop' into datamodel_plugin
0ssigeno 1b84446
More stuff
0ssigeno a1cba27
Stuff
0ssigeno f853d59
More fixes
0ssigeno c7fa248
More test and logs
0ssigeno 0d85592
Blake
0ssigeno 5dacc79
More
0ssigeno 8e3a3e4
Mini rework
0ssigeno 6914ecd
Blake
0ssigeno 9aeebe3
Blake
0ssigeno f2384be
Fixes
0ssigeno 84421b4
Merge branch 'develop' into datamodel_plugin
0ssigeno dba0c22
Fixes
0ssigeno 89875f5
Fixes
0ssigeno 4ad19ab
Technically we can have some mapping with post processing
0ssigeno bf17b04
bgp_ranking mapping
cristinaascari c2ef65f
more analyzer mappings
cristinaascari b175d72
fixes analyzer mappings
cristinaascari 5cd8f3a
more mappings
cristinaascari fa5a40b
Tor mapping
cristinaascari a084c17
Fix retrieval
0ssigeno File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
20 changes: 20 additions & 0 deletions
20
api_app/analyzers_manager/migrations/0123_analyzerconfig_mapping_data_model.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# Generated by Django 4.2.15 on 2024-10-14 07:24 | ||
|
||
from django.db import migrations, models | ||
|
||
|
||
class Migration(migrations.Migration): | ||
|
||
dependencies = [ | ||
("analyzers_manager", "0122_alter_soft_time_limit"), | ||
] | ||
|
||
operations = [ | ||
migrations.AddField( | ||
model_name="analyzerconfig", | ||
name="mapping_data_model", | ||
field=models.JSONField( | ||
default=dict, help_text="Mapping data_model_key: analyzer_report_key. " | ||
), | ||
), | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,58 @@ | ||
# Generated by Django 4.2.15 on 2024-10-14 07:24 | ||
|
||
from django.db import migrations | ||
|
||
|
||
def migrate_urlhaus(apps, schema_editor): | ||
AnalyzerConfig = apps.get_model("analyzers_manager", "AnalyzerConfig") | ||
ac = AnalyzerConfig.objects.filter(name="URLhaus").first() | ||
if not ac: | ||
return | ||
ac.mapping_data_model = { | ||
"urlhaus_reference": "external_references", | ||
"$Malicious": "evaluation", | ||
"urls.url": "related_threats", | ||
} | ||
ac.save() | ||
|
||
|
||
def migrate_maxmind(apps, schema_editor): | ||
AnalyzerConfig = apps.get_model("analyzers_manager", "AnalyzerConfig") | ||
ac = AnalyzerConfig.objects.filter(name="MaxMindGeoIP").first() | ||
if not ac: | ||
return | ||
ac.mapping_data_model = { | ||
"country_code": "country.iso_code", | ||
"registered_country_code": "registered_country_code.iso_code", | ||
"asn": "autonomous_system_number", | ||
"isp": "autonomous_system_organization", | ||
} | ||
ac.save() | ||
|
||
|
||
def migrate_abuse_ipdb(apps, schema_editor): | ||
AnalyzerConfig = apps.get_model("analyzers_manager", "AnalyzerConfig") | ||
ac = AnalyzerConfig.objects.filter(name="AbuseIPDB").first() | ||
if not ac: | ||
return | ||
ac.mapping_data_model = { | ||
"country_code": "data.countryCode", | ||
"external_references": "permalink", | ||
"resolutions": "data.hostnames", | ||
"isp": "data.isp", | ||
"tags": "categories_found", | ||
} | ||
ac.save() | ||
|
||
|
||
class Migration(migrations.Migration): | ||
|
||
dependencies = [ | ||
("analyzers_manager", "0123_analyzerconfig_mapping_data_model"), | ||
] | ||
|
||
operations = [ | ||
migrations.RunPython(migrate_maxmind, migrations.RunPython.noop), | ||
migrations.RunPython(migrate_abuse_ipdb, migrations.RunPython.noop), | ||
migrations.RunPython(migrate_urlhaus, migrations.RunPython.noop), | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -1,12 +1,14 @@ | ||||||||||||||||||
# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl | ||||||||||||||||||
# See the file 'LICENSE' for copying permission. | ||||||||||||||||||
|
||||||||||||||||||
import json | ||||||||||||||||||
from logging import getLogger | ||||||||||||||||||
from typing import Optional | ||||||||||||||||||
from typing import Dict, Optional, Type | ||||||||||||||||||
|
||||||||||||||||||
from django.contrib.contenttypes.fields import GenericRelation | ||||||||||||||||||
from django.contrib.postgres.fields import ArrayField | ||||||||||||||||||
from django.core.exceptions import ValidationError | ||||||||||||||||||
from django.db import models | ||||||||||||||||||
from django.db.models import ForeignKey | ||||||||||||||||||
|
||||||||||||||||||
from api_app.analyzers_manager.constants import ( | ||||||||||||||||||
HashChoices, | ||||||||||||||||||
|
@@ -16,6 +18,12 @@ | |||||||||||||||||
from api_app.analyzers_manager.exceptions import AnalyzerConfigurationException | ||||||||||||||||||
from api_app.analyzers_manager.queryset import AnalyzerReportQuerySet | ||||||||||||||||||
from api_app.choices import TLP, PythonModuleBasePaths | ||||||||||||||||||
from api_app.data_model_manager.models import ( | ||||||||||||||||||
BaseDataModel, | ||||||||||||||||||
DomainDataModel, | ||||||||||||||||||
FileDataModel, | ||||||||||||||||||
IPDataModel, | ||||||||||||||||||
) | ||||||||||||||||||
from api_app.fields import ChoiceArrayField | ||||||||||||||||||
from api_app.models import AbstractReport, PythonConfig, PythonModule | ||||||||||||||||||
|
||||||||||||||||||
|
@@ -32,6 +40,83 @@ class Meta: | |||||||||||||||||
unique_together = [("config", "job")] | ||||||||||||||||||
indexes = AbstractReport.Meta.indexes | ||||||||||||||||||
|
||||||||||||||||||
@property | ||||||||||||||||||
def data_model_class(self) -> Type[BaseDataModel]: | ||||||||||||||||||
if self.job.is_sample: | ||||||||||||||||||
return FileDataModel | ||||||||||||||||||
if self.job.observable_classification == ObservableTypes.IP.value: | ||||||||||||||||||
return IPDataModel | ||||||||||||||||||
if self.job.observable_classification == ObservableTypes.DOMAIN.value: | ||||||||||||||||||
return DomainDataModel | ||||||||||||||||||
raise NotImplementedError( | ||||||||||||||||||
f"Unable to find data model for {self.job.observable_classification}" | ||||||||||||||||||
) | ||||||||||||||||||
|
||||||||||||||||||
def _validation_before_data_model(self) -> bool: | ||||||||||||||||||
if not self.status == self.STATUSES.SUCCESS.value: | ||||||||||||||||||
logger.info( | ||||||||||||||||||
f"Skipping data model of {self.config.name} for job {self.config.pk} because status is " | ||||||||||||||||||
f"{self.status}" | ||||||||||||||||||
) | ||||||||||||||||||
Comment on lines
+57
to
+60
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
return False | ||||||||||||||||||
data_model_keys = self.data_model_class.get_fields().keys() | ||||||||||||||||||
for data_model_key in self.config.mapping_data_model.values(): | ||||||||||||||||||
if data_model_key not in data_model_keys: | ||||||||||||||||||
self.errors.append( | ||||||||||||||||||
f"Field {data_model_key} not present in {self.data_model_class.__name__}" | ||||||||||||||||||
) | ||||||||||||||||||
return True | ||||||||||||||||||
|
||||||||||||||||||
def _create_data_model_dictionary(self) -> Dict: | ||||||||||||||||||
result = {} | ||||||||||||||||||
data_model_fields = self.data_model_class.get_fields() | ||||||||||||||||||
logger.info(f"Mapping is {json.dumps(self.config.mapping_data_model)}") | ||||||||||||||||||
for report_key, data_model_key in self.config.mapping_data_model.items(): | ||||||||||||||||||
# this is a constant | ||||||||||||||||||
if report_key.startswith("$"): | ||||||||||||||||||
value = report_key | ||||||||||||||||||
# this is a field of the report | ||||||||||||||||||
else: | ||||||||||||||||||
try: | ||||||||||||||||||
value = self.get_value(self.report, report_key.split(".")) | ||||||||||||||||||
logger.info(f"Retrieved {value} from key {report_key}") | ||||||||||||||||||
except Exception: | ||||||||||||||||||
# validation | ||||||||||||||||||
self.errors.append(f"Field {report_key} not present in report") | ||||||||||||||||||
continue | ||||||||||||||||||
# create the related object if necessary | ||||||||||||||||||
if isinstance(data_model_fields[data_model_key], ForeignKey): | ||||||||||||||||||
# to create an object we need at least | ||||||||||||||||||
if not isinstance(value, dict): | ||||||||||||||||||
self.errors.append( | ||||||||||||||||||
f"Field {report_key} has type {type(report_key)} while a dictionary is expected" | ||||||||||||||||||
) | ||||||||||||||||||
continue | ||||||||||||||||||
value, _ = data_model_fields[ | ||||||||||||||||||
data_model_key | ||||||||||||||||||
].related_model.objects.get_or_create(**value) | ||||||||||||||||||
elif isinstance(data_model_fields[data_model_key], ArrayField): | ||||||||||||||||||
if data_model_key not in result: | ||||||||||||||||||
result[data_model_key] = [] | ||||||||||||||||||
if isinstance(value, list): | ||||||||||||||||||
result[data_model_key].extend(value) | ||||||||||||||||||
elif isinstance(value, dict): | ||||||||||||||||||
result[data_model_key].extend(list(value.keys())) | ||||||||||||||||||
else: | ||||||||||||||||||
result[data_model_key].append(value) | ||||||||||||||||||
result[data_model_key] = value | ||||||||||||||||||
return result | ||||||||||||||||||
|
||||||||||||||||||
def create_data_model(self) -> Optional[BaseDataModel]: | ||||||||||||||||||
if not self._validation_before_data_model(): | ||||||||||||||||||
return None | ||||||||||||||||||
dictionary = self._create_data_model_dictionary() | ||||||||||||||||||
data_model = self.data_model_class.objects.create( | ||||||||||||||||||
**dictionary, analyzer_report=self | ||||||||||||||||||
) | ||||||||||||||||||
|
||||||||||||||||||
return data_model | ||||||||||||||||||
|
||||||||||||||||||
|
||||||||||||||||||
class MimeTypes(models.TextChoices): | ||||||||||||||||||
# IMPORTANT! in case you update this Enum remember to update also the frontend | ||||||||||||||||||
|
@@ -188,6 +273,10 @@ class AnalyzerConfig(PythonConfig): | |||||||||||||||||
orgs_configuration = GenericRelation( | ||||||||||||||||||
"api_app.OrganizationPluginConfiguration", related_name="%(class)s" | ||||||||||||||||||
) | ||||||||||||||||||
mapping_data_model = models.JSONField( | ||||||||||||||||||
default=dict, | ||||||||||||||||||
help_text="Mapping analyzer_report_key: data_model_key. Keys preceded by the symbol $ will be considered as constants.", | ||||||||||||||||||
) | ||||||||||||||||||
|
||||||||||||||||||
@classmethod | ||||||||||||||||||
@property | ||||||||||||||||||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, consider
self.config_id