Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataModel Plugin #2494

Open
wants to merge 63 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
fa34c52
added data models
cristinaascari Jul 30, 2024
604738b
updated data models
cristinaascari Jul 31, 2024
1cf2e0a
updated data models
cristinaascari Jul 31, 2024
45b7f10
updated data models
cristinaascari Jul 31, 2024
a234319
updated data models
cristinaascari Aug 1, 2024
38c0c29
fix
cristinaascari Aug 1, 2024
3bc44c5
fix
cristinaascari Aug 2, 2024
29f4313
update data models
cristinaascari Aug 5, 2024
1913000
fix
cristinaascari Aug 5, 2024
90187fb
update file data model
cristinaascari Aug 5, 2024
b4462a0
update file data model
cristinaascari Aug 5, 2024
4140f2f
updates file data model
cristinaascari Aug 6, 2024
5ab5614
updates file data model
cristinaascari Aug 6, 2024
e82f98e
updates file data model
cristinaascari Aug 6, 2024
f12776d
updates
cristinaascari Aug 9, 2024
845406c
updates data models
cristinaascari Aug 9, 2024
6f91d84
fix
cristinaascari Aug 9, 2024
0a6529c
fix
cristinaascari Aug 9, 2024
210f008
fix CharFiled max_length
cristinaascari Aug 26, 2024
88376fc
fixes
cristinaascari Aug 26, 2024
858b200
fixes
cristinaascari Aug 26, 2024
7ebe465
Added BaseDataModel
cristinaascari Aug 26, 2024
3717b95
updated BaseDataModel
cristinaascari Aug 26, 2024
8e78524
updates data models
cristinaascari Aug 26, 2024
bdddb86
Merge branch 'develop' into datamodel_plugin
cristinaascari Aug 27, 2024
e6b289d
updates data models
cristinaascari Aug 27, 2024
beb3ff8
added admin data models
cristinaascari Aug 29, 2024
6261ec0
field names fixes
cristinaascari Aug 29, 2024
de6b938
fix ip data model
cristinaascari Aug 29, 2024
dca7a50
fix ip data model admin
cristinaascari Aug 29, 2024
db3f59a
fixes FileDataModel fields
cristinaascari Aug 30, 2024
b0119d6
Update external_references field
cristinaascari Aug 30, 2024
e0e64da
fix linters
cristinaascari Aug 30, 2024
a7beacb
fixes
cristinaascari Aug 30, 2024
0b934ff
updates signature field
cristinaascari Aug 30, 2024
dbbd637
updates tags field
cristinaascari Aug 30, 2024
f57abe3
fix
cristinaascari Aug 30, 2024
1018903
updates data model admin
cristinaascari Aug 30, 2024
3dab81c
fix
cristinaascari Aug 30, 2024
8c96d36
moved data_model into api_app
cristinaascari Sep 2, 2024
7c81328
moved data_model into api_app
cristinaascari Sep 2, 2024
48bfb83
Added unique_together constraint in IETFReport
cristinaascari Sep 2, 2024
c03102a
Merge branch 'develop' into datamodel_plugin
0ssigeno Sep 18, 2024
1b84446
More stuff
0ssigeno Sep 24, 2024
a1cba27
Stuff
0ssigeno Oct 7, 2024
f853d59
More fixes
0ssigeno Oct 14, 2024
c7fa248
More test and logs
0ssigeno Oct 14, 2024
0d85592
Blake
0ssigeno Oct 14, 2024
5dacc79
More
0ssigeno Oct 16, 2024
8e3a3e4
Mini rework
0ssigeno Oct 16, 2024
6914ecd
Blake
0ssigeno Oct 16, 2024
9aeebe3
Blake
0ssigeno Oct 16, 2024
f2384be
Fixes
0ssigeno Oct 16, 2024
84421b4
Merge branch 'develop' into datamodel_plugin
0ssigeno Oct 16, 2024
dba0c22
Fixes
0ssigeno Oct 16, 2024
89875f5
Fixes
0ssigeno Oct 16, 2024
4ad19ab
Technically we can have some mapping with post processing
0ssigeno Oct 16, 2024
bf17b04
bgp_ranking mapping
cristinaascari Oct 21, 2024
c2ef65f
more analyzer mappings
cristinaascari Oct 22, 2024
b175d72
fixes analyzer mappings
cristinaascari Oct 23, 2024
5cd8f3a
more mappings
cristinaascari Oct 23, 2024
fa5a40b
Tor mapping
cristinaascari Oct 23, 2024
a084c17
Fix retrieval
0ssigeno Oct 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added data_model/__init__.py
Empty file.
53 changes: 53 additions & 0 deletions data_model/admin.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
from django.contrib import admin

from data_model.models import BaseDataModel, DomainDataModel, FileDataModel, IPDataModel


@admin.register(BaseDataModel)
class BaseDataModelAdminView(admin.ModelAdmin):
list_display = (
"evaluation",
"external_references",
"related_threats",
"malware_family",
"additional_info",
)


@admin.register(DomainDataModel)
class DomainDataModelAdminView(BaseDataModelAdminView):
list_display = BaseDataModelAdminView.list_display + (
"ietf_report",
"rank",
)


@admin.register(IPDataModel)
class IPDataModelAdminView(BaseDataModelAdminView):
list_display = BaseDataModelAdminView.list_display + (
"ietf_report",
"asn",
"asn_rank",
"certificates",
"org_name",
"country",
"country_code",
"registered_country",
"registered_country_code",
"isp",
"is_anonymizer",
"is_tor_exit_node",
)


@admin.register(FileDataModel)
class FileDataModelAdminView(BaseDataModelAdminView):
list_display = BaseDataModelAdminView.list_display + (
"tags",
"compromised_hosts",
"signatures",
"yara_rules",
"comments",
"file_information",
"stats",
)
6 changes: 6 additions & 0 deletions data_model/apps.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
from django.apps import AppConfig


class DataModelConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "data_model"
Empty file.
109 changes: 109 additions & 0 deletions data_model/models.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
from django.contrib.postgres import fields as pg_fields
from django.db import models


class IETFReport(models.Model):
rrname = models.CharField(max_length=100)
rrtype = models.CharField(max_length=100)
rdata = pg_fields.ArrayField(models.CharField(max_length=100))
time_first = models.DateTimeField()
time_last = models.DateTimeField()


class BaseDataModel(models.Model):
evaluation = models.CharField(
max_length=100, null=True
) # classification/verdict/found/score/malscore
# HybridAnalysisObservable (verdict), BasicMaliciousDetector (malicious),
# GoogleSafeBrowsing (malicious), Crowdsec (classifications),
# GreyNoise (classification), Cymru (found), Cuckoo (malscore),
# Intezer (verdict/sub_verdict), Triage (analysis.score),
# HybridAnalysisFileAnalyzer (classification_tags)
external_references = pg_fields.ArrayField(
models.URLField(), null=True
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved
) # link/external_references/permalink/domains
# Crowdsec (link), UrlHaus (external_references), BoxJs,
# Cuckoo (result_url/permalink), Intezer (link/analysis_url),
# MalwareBazaarFileAnalyzer (permalink/file_information.value), MwDB (permalink),
# StringsInfo (data), Triage (permalink), UnpacMe (permalink), XlmMacroDeobfuscator,
# Yara (report.list_el.url/rule_url), Yaraify (link),
# HybridAnalysisFileAnalyzer (domains),
# VirusTotalV3FileAnalyzer (data.relationships.contacted_urls/contacted_domains)
related_threats = pg_fields.ArrayField(
models.CharField(max_length=100), null=True
) # threats/related_threats
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved
# GoogleSafeBrowsing, QuarkEngineAPK (crimes.crime)
malware_family = models.CharField(
max_length=100, null=True
) # family/family_name/malware_family
# HybridAnalysisObservable, Intezer (family_name), Cuckoo, MwDB,
# Triage (analysis.family), UnpacMe (results.malware_id.malware_family),
# VirusTotalV3FileAnalyzer
# (attributes.last_analysis_results.list_el.results/attributes.names)
additional_info = (
models.JSONField()
) # field for additional information related to a specific analyzer


class DomainDataModel(BaseDataModel):
ietf_report = models.ForeignKey(
IETFReport, on_delete=models.CASCADE, null=True
) # pdns
rank = models.IntegerField(null=True) # Tranco
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved


class IPDataModel(BaseDataModel):
ietf_report = models.ForeignKey(
IETFReport, on_delete=models.CASCADE, null=True
) # pdns
asn = models.IntegerField(null=True) # BGPRanking, MaxMind
asn_rank = models.DecimalField(null=True) # BGPRanking
certificates = models.JSONField(null=True) # CIRCL_PSSL
org_name = models.CharField(max_length=100, null=True) # GreyNoise
country = models.CharField(max_length=100, null=True) # MaxMind, AbuseIPDB
country_code = models.CharField(max_length=100, null=True) # MaxMind, AbuseIPDB
registered_country = models.CharField(
max_length=100, null=True
) # MaxMind, AbuseIPDB
registered_country_code = models.CharField(
max_length=100, null=True
) # MaxMind, AbuseIPDB
isp = models.CharField(max_length=100, null=True) # AbuseIPDB
is_anonymizer = models.BooleanField(null=True) # TorProject, Crowdsec
is_tor_exit_node = models.BooleanField(null=True) # TorProject, Crowdsec
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved
# additional_info
# behavior = models.CharField(max_length=100, null=True) # Crowdsec
# noise = models.BooleanField(null=True) # GreyNoise
# riot = models.BooleanField(null=True) # GreyNoise


class FileDataModel(BaseDataModel):
tags = pg_fields.ArrayField(
models.CharField(max_length=100), null=True
) # HybridAnalysisFileAnalyzer, MalwareBazaarFileAnalyzer, MwDB,
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved
# VirusTotalV3FileAnalyzer (report.data.tags)
compromised_hosts = pg_fields.ArrayField(
models.CharField(max_length=100), null=True
) # HybridAnalysisFileAnalyzer
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved
signatures = pg_fields.ArrayField(
models.CharField(max_length=100), null=True
) # ClamAvFileAnalyzer, MalwareBazaarFileAnalyzer, Yara (report.list_el.match)
yara_rules = pg_fields.ArrayField(
models.JSONField(), null=True
) # MalwareBazaarFileAnalyzer, Yaraify (report.data.tasks.static_result)
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved
comments = pg_fields.ArrayField(
models.CharField(max_length=100), null=True
) # MalwareBazaarFileAnalyzer,
# VirusTotalV3FileAnalyzer (data.relationships.comments)
file_information = pg_fields.ArrayField(
models.JSONField(), null=True
) # MalwareBazaarFileAnalyzer, OneNoteInfo
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved
# (files), QuarkEngineAPK (crimes.confidence, threat_level, total_score)
# RtfInfo (exploit_equation_editor, exploit_ole2link_vuln)
stats = pg_fields.ArrayField(
models.JSONField(), null=True
) # PdfInfo (peepdf_stats)
cristinaascari marked this conversation as resolved.
Show resolved Hide resolved
# additional_info
# pdfid_reports = pg_fields.ArrayField(models.JSONField(), null=True) # PdfInfo
# imphash = models.CharField(max_length=100, null=True) # PeInfo
# type = models.CharField(max_length=100, null=True) # PeInfo
1 change: 1 addition & 0 deletions data_model/views.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Create your views here.
1 change: 1 addition & 0 deletions intel_owl/settings/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
"api_app.pivots_manager",
"api_app.ingestors_manager",
"api_app.investigations_manager",
"data_model",
# auth
"rest_email_auth",
# performance debugging
Expand Down
Loading