Skip to content

Commit

Permalink
add documentation of use cmdb and update unit test
Browse files Browse the repository at this point in the history
  • Loading branch information
MikeHV14 committed Jan 8, 2025
1 parent d74f000 commit 4fa2dca
Show file tree
Hide file tree
Showing 17 changed files with 510 additions and 93 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ devsecops-engine-tools --platform_devops ["local","azure","github"] --remote_con
┃ ┗ 📜ConfigTool.json
┃ ┗ 📜Exclusions.json
```

for more information visit [here](https://github.com/bancolombia/devsecops-engine-tools/blob/trunk/example_remote_config_local/README.md)
#### Tools available for the modules (Configuration engine_core/ConfigTool.json)


Expand Down
118 changes: 118 additions & 0 deletions example_remote_config_local/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Example use remote config for vulnerability management

Initially, we need to know that the DevSecOps engine tools include a module for connecting to a vulnerability centralizer (DefectDojo). As a primary requirement for using this module, it must be considered whether a CMDB will be used or not. This is configurable through the remote config located in the [engine_core](https://github.com/bancolombia/devsecops-engine-tools/blob/trunk/example_remote_config_local/engine_core/ConfigTool.json). Below, examples of the two possible cases will be provided:

### Using CMDB:
Let's suppose the CMDB response is as follows:
```json
{
"count": 2,
"value": [
{
"ApplicationCode": "code1-app",
"ApplicationName": "Example Application Name 1",
"ApplicationType": "Example Application type 1",
"ApplicationTag": "Example Application tag 1",
"ApplicationDescription": "Example Application description 1",
"Env": "PDN"
},
{
"ApplicationCode": "code1-app",
"ApplicationName": "Example Application Name 1",
"ApplicationType": "Example Application type 1",
"ApplicationTag": "Example Application tag 1",
"ApplicationDescription": "Example Application description 1",
"Env": "DEV"
}
]
}
```

Then, the remote config settings should look similar to this:
```json
"DEFECT_DOJO": {
"HOST_DEFECT_DOJO": "http://localhost:8080",
"LIMITS_QUERY": 100,
"MAX_RETRIES_QUERY": 5,
"CMDB": {
"USE_CMDB": true,
"HOST_CMDB": "http://host_cmdb_example",
"REGEX_EXPRESSION_CMDB": "^([^-]+)",
"CMDB_MAPPING_PATH": "/path/mapping_cmdb.json",
"CMDB_MAPPING": {
"PRODUCT_TYPE_NAME": "ApplicationType",
"PRODUCT_NAME": "ApplicationName",
"TAG_PRODUCT": "ApplicationTag",
"PRODUCT_DESCRIPTION": "ApplicationDescription",
"CODIGO_APP": "ApplicationCode"
},
"CMDB_REQUEST_RESPONSE": {
"HEADERS": {
"Content-Type": "application/json",
"Api-Key": "tokenvalue"
},
"METHOD": "GET",
"PARAMS": {
"appCode": "codappvalue"
},
"RESPONSE": ["value", 0]
}
}
}
```

**Let’s detail the description for the elements under the CMDB key:**

*USE_CMDB:* The value is a boolean, indicating whether or not CMDB will be used.

*USE_CMDB:* The value is a boolean, indicating whether or not CMDB will be used.

*HOST_CMDB:* The URL of the API for querying the CMDB.

*REGEX_EXPRESSION_CMDB:* Regular expression.

*CMDB_MAPPING_PATH:* Location of the mapping for possible product types.

*CMDB_MAPPING:* Element containing the equivalent mappings between DefectDojo and the CMDB.

*MDB_REQUEST_RESPONSE:* Contains the necessary elements to make a request to the CMDB.

*HEADERS:* Headers required to make the request. Note that the authentication type must be via Api-Key. The value "tokenvalue" should remain as is, as it is necessary for replacing the token.

*METHOD:* Can be either POST or GET.

*PARAMS:* Used if the selected METHOD is GET. The value "codappvalue" should remain as is, as it is necessary for replacement.

*BODY:* Used if the selected METHOD is POST. The value "codappvalue" should remain as is, as it is necessary for replacement.

### Without Using CMDB:
For this case, three environment variables must be created according to the DevOps platform.
```bash
## Platform local
DET_VM_PRODUCT_TYPE_NAME="Example product type name"
DET_VM_PRODUCT_NAME="Example product type name"
DET_VM_PRODUCT_DESCRIPTION="Example product descrition"

## Platform azure
VM_PRODUCT_TYPE_NAME="Example product type name"
VM_PRODUCT_NAME="Example product type name"
VM_PRODUCT_DESCRIPTION="Example product descrition"

## Platform github
VM_PRODUCT_TYPE_NAME="Example product type name"
VM_PRODUCT_NAME="Example product type name"
VM_PRODUCT_DESCRIPTION="Example product descrition"
```

The remote config settings should look similar to this:
```json
"DEFECT_DOJO": {
"HOST_DEFECT_DOJO": "http://localhost:8080",
"LIMITS_QUERY": 100,
"MAX_RETRIES_QUERY": 5,
"CMDB": {
"USE_CMDB": false,
"REGEX_EXPRESSION_CMDB": "^([^-]+)",
}
}
```
44 changes: 25 additions & 19 deletions example_remote_config_local/engine_core/ConfigTool.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,29 +14,35 @@
"main"
],
"DEFECT_DOJO": {
"CMDB_MAPPING_PATH": "",
"HOST_CMDB": "",
"HOST_DEFECT_DOJO": "",
"REGEX_EXPRESSION_CMDB": "",
"LIMITS_QUERY": 100,
"MAX_RETRIES_QUERY": 5,
"CMDB_MAPPING": {
"PRODUCT_TYPE_NAME": "",
"PRODUCT_NAME": "",
"TAG_PRODUCT": "",
"PRODUCT_DESCRIPTION": "",
"CODIGO_APP": ""
},
"CMDB_REQUEST_RESPONSE": {
"HEADERS": {
"Content-Type": "application/json",
"apiKey": "tokenvalue"
"CMDB": {
"USE_CMDB": false,
"HOST_CMDB": "",
"REGEX_EXPRESSION_CMDB": "",
"CMDB_MAPPING_PATH": "",
"CMDB_MAPPING": {
"PRODUCT_TYPE_NAME": "",
"PRODUCT_NAME": "",
"TAG_PRODUCT": "",
"PRODUCT_DESCRIPTION": "",
"CODIGO_APP": ""
},
"METHOD": "GET|POST",
"PARAMS": {
"appCode": "codappvalue"
},
"RESPONSE": [0]
"CMDB_REQUEST_RESPONSE": {
"HEADERS": {
"Content-Type": "application/json",
"Api-Key": "tokenvalue"
},
"METHOD": "GET|POST",
"PARAMS": {
"appCode": "codappvalue"
},
"BODY": {
"appCode": "codappvalue"
},
"RESPONSE": [0]
}
}
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,6 @@ class VulnerabilityManagement:
branch_tag: str
commit_hash: str
environment: str
vm_product_type_name: str
vm_product_name: str
vm_product_description: str
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,9 @@ def _use_vulnerability_management(
self.devops_platform_gateway.get_variable("branch_tag"),
self.devops_platform_gateway.get_variable("commit_hash"),
env,
self.devops_platform_gateway.get_variable("vm_product_type_name"),
self.devops_platform_gateway.get_variable("vm_product_name"),
self.devops_platform_gateway.get_variable("vm_product_description"),
)
)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
SystemVariables,
ReleaseVariables,
AgentVariables,
VMVariables
)
from devsecops_engine_tools.engine_utilities.azuredevops.infrastructure.azure_devops_api import (
AzureDevopsApi,
Expand Down Expand Up @@ -95,6 +96,9 @@ def get_variable(self, variable):
"target_branch": SystemVariables.System_TargetBranchName,
"source_branch": SystemVariables.System_SourceBranch,
"repository_provider": BuildVariables.Build_Repository_Provider,
"vm_product_type_name": VMVariables.Vm_Product_Type_Name,
"vm_product_name": VMVariables.Vm_Product_Name,
"vm_product_description": VMVariables.Vm_Product_Description,
}
try:
return variable_map.get(variable).value()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@

from devsecops_engine_tools.engine_utilities.utils.logger_info import MyLogger
from devsecops_engine_tools.engine_utilities import settings
from devsecops_engine_tools.engine_utilities.defect_dojo.domain.serializers.import_scan import ImportScanSerializer
import time
import concurrent.futures

Expand Down Expand Up @@ -87,45 +88,17 @@ def send_vulnerability_management(
tags = vulnerability_management.dict_args["tool"]
if vulnerability_management.dict_args["tool"] == "engine_iac":
tags = f"{vulnerability_management.dict_args['tool']}_{'_'.join(vulnerability_management.dict_args['platform'])}"
request: ImportScanRequest = Connect.cmdb(
cmdb_mapping={
"product_type_name": vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB_MAPPING"]["PRODUCT_TYPE_NAME"],
"product_name": vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB_MAPPING"]["PRODUCT_NAME"],
"tag_product": vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB_MAPPING"]["TAG_PRODUCT"],
"product_description": vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB_MAPPING"]["PRODUCT_DESCRIPTION"],
"codigo_app": vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB_MAPPING"]["CODIGO_APP"],
},
compact_remote_config_url=f'{vulnerability_management.base_compact_remote_config_url}{vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB_MAPPING_PATH"]}',
personal_access_token=vulnerability_management.access_token,
token_cmdb=token_cmdb,
host_cmdb=vulnerability_management.config_tool[
"VULNERABILITY_MANAGER"
]["DEFECT_DOJO"]["HOST_CMDB"],
cmdb_request_response=vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB_REQUEST_RESPONSE"],
expression=vulnerability_management.config_tool[
"VULNERABILITY_MANAGER"
]["DEFECT_DOJO"]["REGEX_EXPRESSION_CMDB"],
token_defect_dojo=token_dd,
host_defect_dojo=vulnerability_management.config_tool[
"VULNERABILITY_MANAGER"
]["DEFECT_DOJO"]["HOST_DEFECT_DOJO"],
scan_type=scan_type_mapping[vulnerability_management.scan_type],
engagement_name=vulnerability_management.input_core.scope_pipeline,
service=vulnerability_management.input_core.scope_pipeline,
file=vulnerability_management.input_core.path_file_results,
version=vulnerability_management.version,
build_id=vulnerability_management.build_id,
source_code_management_uri=vulnerability_management.source_code_management_uri,
branch_tag=vulnerability_management.branch_tag,
commit_hash=vulnerability_management.commit_hash,
environment=(
enviroment_mapping[vulnerability_management.environment.lower()]
if vulnerability_management.environment is not None
and vulnerability_management.environment.lower()
in enviroment_mapping
else enviroment_mapping["default"]
),
tags=tags,

use_cmdb = vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB"]["USE_CMDB"]

request = self.build_request(
vulnerability_management,
token_cmdb,
token_dd,
scan_type_mapping,
enviroment_mapping,
tags,
use_cmdb
)

def request_func():
Expand Down Expand Up @@ -155,6 +128,74 @@ def request_func():
)
)

def build_request(
self,
vulnerability_management: VulnerabilityManagement,
token_cmdb,
token_dd,
scan_type_mapping,
enviroment_mapping,
tags,
use_cmdb: bool
):
common_fields = {
"scan_type": scan_type_mapping[vulnerability_management.scan_type],
"file": vulnerability_management.input_core.path_file_results,
"engagement_name": vulnerability_management.input_core.scope_pipeline,
"source_code_management_uri": vulnerability_management.source_code_management_uri,
"tags": tags,
"version": vulnerability_management.version,
"build_id": vulnerability_management.build_id,
"branch_tag": vulnerability_management.branch_tag,
"commit_hash": vulnerability_management.commit_hash,
"service": vulnerability_management.input_core.scope_pipeline,
"environment": (
enviroment_mapping[vulnerability_management.environment.lower()]
if vulnerability_management.environment is not None
and vulnerability_management.environment.lower()
in enviroment_mapping
else enviroment_mapping["default"]
),
"token_defect_dojo": token_dd,
"host_defect_dojo": vulnerability_management.config_tool[
"VULNERABILITY_MANAGER"
]["DEFECT_DOJO"]["HOST_DEFECT_DOJO"],
"expression": vulnerability_management.config_tool[
"VULNERABILITY_MANAGER"
]["DEFECT_DOJO"]["CMDB"]["REGEX_EXPRESSION_CMDB"],
}

if use_cmdb:
cmdb_mapping = vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB"]["CMDB_MAPPING"]
return Connect.cmdb(
cmdb_mapping={
"product_type_name": cmdb_mapping["PRODUCT_TYPE_NAME"],
"product_name": cmdb_mapping["PRODUCT_NAME"],
"tag_product": cmdb_mapping["TAG_PRODUCT"],
"product_description": cmdb_mapping["PRODUCT_DESCRIPTION"],
"codigo_app": cmdb_mapping["CODIGO_APP"],
},
compact_remote_config_url=f'{vulnerability_management.base_compact_remote_config_url}{vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB"]["CMDB_MAPPING_PATH"]}',
personal_access_token=vulnerability_management.access_token,
token_cmdb=token_cmdb,
host_cmdb=vulnerability_management.config_tool[
"VULNERABILITY_MANAGER"
]["DEFECT_DOJO"]["CMDB"]["HOST_CMDB"],
cmdb_request_response=vulnerability_management.config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB"]["CMDB_REQUEST_RESPONSE"],
**common_fields,
)
else:
request: ImportScanRequest = ImportScanSerializer().load(
{
"product_type_name":vulnerability_management.vm_product_type_name,
"product_name": vulnerability_management.vm_product_name,
"product_description":vulnerability_management.vm_product_description,
"code_app":vulnerability_management.vm_product_name,
**common_fields,
}
)
return request

def get_product_type_service(self, service, dict_args, secret_tool, config_tool):
try:
session_manager = self._get_session_manager(
Expand All @@ -171,7 +212,7 @@ def request_func():
request={
"name": Connect.get_code_app(
service,
config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"][
config_tool["VULNERABILITY_MANAGER"]["DEFECT_DOJO"]["CMDB"][
"REGEX_EXPRESSION_CMDB"
],
),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
BuildVariables,
SystemVariables,
ReleaseVariables,
AgentVariables
AgentVariables,
VMVariables
)
from devsecops_engine_tools.engine_utilities.github.infrastructure.github_api import (
GithubApi,
Expand Down Expand Up @@ -85,6 +86,9 @@ def get_variable(self, variable):
"target_branch": SystemVariables.github_event_base_ref,
"source_branch": SystemVariables.github_ref,
"repository_provider": BuildVariables.GitHub,
"vm_product_type_name": VMVariables.Vm_Product_Type_Name,
"vm_product_name": VMVariables.Vm_Product_Name,
"vm_product_description": VMVariables.Vm_Product_Description,
}
try:
return variable_map.get(variable).value()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ def get_variable(self, variable):
"temp_directory" : "DET_TEMP_DIRECTORY",
"target_branch" : "DET_TARGET_BRANCH",
"source_branch" : "DET_SOURCE_BRANCH",
"repository_provider" : "DET_REPOSITORY_PROVIDER"
"repository_provider" : "DET_REPOSITORY_PROVIDER",
"vm_product_type_name" : "DET_VM_PRODUCT_TYPE_NAME",
"vm_product_name" : "DET_VM_PRODUCT_NAME",
"vm_product_description" : "DET_VM_PRODUCT_DESCRIPTION",
}
return os.environ.get(env_variables[variable], None)
Loading

0 comments on commit 4fa2dca

Please sign in to comment.