-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Datamodel docs #7
base: main
Are you sure you want to change the base?
Conversation
```python3 | ||
{"query_status": "no_results"} | ||
``` | ||
meaning that we can provide use the following code to consider only _real_ results: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
meaning that we can provide use the following code to consider only _real_ results: | |
meaning that we can use the following code to consider only _real_ results: |
If you specify a path that is not present in the `DataModel`, an error will be added to the job. | ||
If you specify a path that is not present in the `AnalyzerConfig`, a warning will be added to the job. | ||
### Analyzer._do_create_data_model | ||
This is a function that every `Analyzer` can override: this functions returns a boolean and, if `False`, the datamodel will not be created. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a function that every `Analyzer` can override: this functions returns a boolean and, if `False`, the datamodel will not be created. | |
This is a function that every `Analyzer` can override: this function returns a boolean and, if `False`, the datamodel will not be created. |
### Analyzer._do_create_data_model | ||
This is a function that every `Analyzer` can override: this functions returns a boolean and, if `False`, the datamodel will not be created. | ||
This can be used if the `Analyzers` can succeed without retrieving useful results. | ||
Let's use as an example `UrlHaus`: if the domain analyzed is not present in its database, the result will be |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use as an example `UrlHaus`: if the domain analyzed is not present in its database, the result will be | |
Let's use `UrlHaus` as an example : if the domain analyzed is not present in its database, the result will be |
If you specify a path that is not present in the `AnalyzerConfig`, a warning will be added to the job. | ||
### Analyzer._do_create_data_model | ||
This is a function that every `Analyzer` can override: this functions returns a boolean and, if `False`, the datamodel will not be created. | ||
This can be used if the `Analyzers` can succeed without retrieving useful results. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be used if the `Analyzers` can succeed without retrieving useful results. | |
This can be useful when a specific `Analyzer` succeeds without retrieving useful results. |
``` | ||
|
||
### Analyzer._create_data_model_mtm | ||
This is a function that every `Analyzer` can override: this functions returns a dictionary where the values are the objects that will be added in a many to many relationship in the datamodel, and the keys the names of the fields. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a function that every `Analyzer` can override: this functions returns a dictionary where the values are the objects that will be added in a many to many relationship in the datamodel, and the keys the names of the fields. | |
This is a function that every `Analyzer` can override: this function returns a dictionary where the values are the objects that will be added in a many to many relationship in the datamodel, and the keys the names of the fields. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this function returns a dictionary where the values are the objects that will be added in a many to many relationship in the datamodel, and the keys the names of the fields.
This is the technical explanation and is understandable. I would like to see another sentence with a more generic use-case explanation, less technical. In practice, you did it for all the cases but this one.
Example This mean that you can use it for more articulate data transformation to parse the `AnalyzerReport` into a `DataModel`.
or This can be used if the `Analyzers` can succeed without retrieving useful results.
Here we are creating many `Signature` objects (using the signatures that matched the sample analyzed) and adding them to the `signatures` field. | ||
|
||
### Analyzer._update_data_model | ||
This is the last function that you can override in the `Analyzer` class: this functions returns nothing, and is called after every other check. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the last function that you can override in the `Analyzer` class: this functions returns nothing, and is called after every other check. | |
This is the last function that you can override in the `Analyzer` class: this function returns nothing, and is called after every other check. |
self.report: AnalyzerReport | ||
if self.report.report["isWhitelisted"]: | ||
evaluation = ( | ||
self.report.data_model_class.EVALUATIONS.FALSE_POSITIVE.value |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.report.data_model_class.EVALUATIONS.FALSE_POSITIVE.value | |
self.report.data_model_class.EVALUATIONS.TRUSTED.value |
data_model.evaluation = evaluation | ||
``` | ||
We are setting the field `evaluation` depending on some logic that we constructed, using the data inside the report. | ||
If the IP has some report but is whitelisted then we set the `evaluation` to `false positive`, otherwise to `malicious`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the IP has some report but is whitelisted then we set the `evaluation` to `false positive`, otherwise to `malicious`. | |
If the IP address has been reported by some AbuseIPDB users but, at the same time, is whitelisted by AbuseIPDB, then we set its `evaluation` to `trusted`. On the contrary, if it's not whitelisted, we set it as `malicious`. |
In version XXX of IntelOwl, a new plugin has been added: the `DataModel`. | ||
Its main functionality is to model an `Analyzer` result to a set of prearranged keys, allowing users to easily search, evaluate and use the analyzer result. | ||
The author of an `AnalyzerConfig` is able to decide mapping between each field of the `AnalyzerReport` and the corresponding in the `DataModel`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The basic description of each Plugin should be put in the "Usage section" so I expect these 3 rows there while I would name this overall section "How to create/customize the Data Model" or something similar to align it with the rest of the existing documentation
No description provided.