Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataModel Plugin #2494

Open
wants to merge 63 commits into
base: develop
Choose a base branch
from
Open

DataModel Plugin #2494

wants to merge 63 commits into from

Conversation

cristinaascari
Copy link
Contributor

(Please add to the PR name the issue/s that this PR would close if merged by using a Github keyword. Example: <feature name>. Closes #999. If your PR is made by a single commit, please add that clause in the commit too. This is all required to automate the closure of related issues.)

Description

Please include a summary of the change and link to the related issue.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue).
  • New feature (non-breaking change which adds functionality).
  • Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

  • I have read and understood the rules about how to Contribute to this project
  • The pull request is for the branch develop
  • A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
    • I strictly followed the documentation "How to create a Plugin"
    • Usage file was updated.
    • Advanced-Usage was updated (in case the plugin provides additional optional configuration).
    • I have dumped the configuration from Django Admin using the dumpplugin command and added it in the project as a data migration. ("How to share a plugin with the community")
    • If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive test_files.zip and you added the default tests for that mimetype in test_classes.py.
    • If you created a new analyzer and it is free (does not require any API key), please add it in the FREE_TO_USE_ANALYZERS playbook by following this guide.
    • Check if it could make sense to add that analyzer/connector to other freely available playbooks.
    • I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
    • If the plugin interacts with an external service, I have created an attribute called precisely url that contains this information. This is required for Health Checks.
    • If the plugin requires mocked testing, _monkeypatch() was used in its class to apply the necessary decorators.
    • I have added that raw JSON sample to the MockUpResponse of the _monkeypatch() method. This serves us to provide a valid sample for testing.
  • If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
  • Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
  • I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
  • If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).
  • If the GUI has been modified:
    • I have a provided a screenshot of the result in the PR.
    • I have created new frontend tests for the new component or updated existing ones.
  • After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

Important Rules

  • If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
  • Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

data_model/models.py Outdated Show resolved Hide resolved
data_model/models.py Outdated Show resolved Hide resolved
data_model/models.py Outdated Show resolved Hide resolved
data_model/models.py Outdated Show resolved Hide resolved
@0ssigeno 0ssigeno requested a review from mlodic October 16, 2024 14:49
@0ssigeno
Copy link
Contributor

The errors in the test are not related to the pr, you can review it

@0ssigeno 0ssigeno marked this pull request as ready for review October 16, 2024 14:50
Copy link
Contributor

@code-review-doctor code-review-doctor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some things to consider. View full project report here.

Comment on lines 57 to 59
logger.info(
f"Skipping data model of {self.config.name} for job {self.config.pk} because no data model"
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
logger.info(
f"Skipping data model of {self.config.name} for job {self.config.pk} because no data model"
)
logger.info(
f"Skipping data model of {self.config.name} for job {self.config_id} because no"
" data model"
)

def _validation_before_data_model(self) -> bool:
if not self.config.mapping_data_model:
logger.info(
f"Skipping data model of {self.config.name} for job {self.config.pk} because no data model"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.config.pk performs a database read when pk is evaluated. You could take advantage of Django's caching of related fields by using self.config_id, which does not do a database read. More info.

Comment on lines +62 to +65
logger.info(
f"Skipping data model of {self.config.name} for job {self.config.pk} because status is "
f"{self.status}"
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
logger.info(
f"Skipping data model of {self.config.name} for job {self.config.pk} because status is "
f"{self.status}"
)
logger.info(
f"Skipping data model of {self.config.name} for job {self.config_id} because status"
f" is {self.status}"
)

return False
if not self.status == self.STATUSES.SUCCESS.value:
logger.info(
f"Skipping data model of {self.config.name} for job {self.config.pk} because status is "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, consider self.config_id


class Signature(models.Model):
provider = LowercaseCharField(max_length=100)
url = models.URLField(default=None, null=True, blank=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
url = models.URLField(default=None, null=True, blank=True)
url = models.URLField(default="", blank=True)

null=True on a string field causes inconsistent data types because the value can be either str or None. This adds complexity and maybe bugs, but can be solved by replacing null=True with default="". Explained here.



class DomainDataModel(BaseDataModel):
ietf_report = models.ForeignKey(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Django automatically creates a related_name if it's not set. If it were set then a more readable and explicit relationship is set up. More details.



class IPDataModel(BaseDataModel):
ietf_report = models.ForeignKey(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: with an explicit related_name would be better.



class DataModelEvaluations(Choices):
FALSE_POSITIVE = "false_positive"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
FALSE_POSITIVE = "false_positive"
TRUSTED = "trusted"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants