misp-playbooks/pb_query_hash_information.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca1fbfb5",
   "metadata": {
    "tags": [
     "playbook-introduction"
    ]
   },
   "source": [
    "# Query hash information\n",
    "\n",
    "## Introduction\n",
    "\n",
    "- UUID: **9b9f655d-100d-490e-b3b3-ec24cf5526bd**\n",
    "- Started from [issue 15](https://github.com/MISP/misp-playbooks/issues/15)\n",
    "- State: **Published**\n",
    "- Purpose: This playbook is complementary to the playbooks for [static malware analysis](https://github.com/MISP/misp-playbooks/blob/main/misp-playbooks/pb_malware_triage.ipynb) and [dynamic malware analysis](https://github.com/MISP/misp-playbooks/blob/main/misp-playbooks/pb_malware_triage_upload_sample.ipynb) and investigates **file hashes**.\n",
    "    - It checks if the hashes are found on **MISP warninglists**, in MISP events or in MISP feeds.\n",
    "    - Uses the information from **VirusTotal**, **Hashlookup** and **MalwareBazaar** to provide context information on hashes.\n",
    "    - It creates a **MISP report** for each hash.\n",
    "    - Run this playbook if you do not have the sample and have to work with only the hash.\n",
    "- Tags: [ \"hash\", \"reputation\", \"abuse\", \"sample\", \"file\", \"malware\" ]\n",
    "- External resources: **Mattermost**, **VirusTotal**, **Hashlookup**, **MalwareBazaar**\n",
    "- Target audience: **SOC**, **CSIRT**, **CTI**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a512a2c1",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "# Playbook\n",
    "\n",
    "- **Query hash information**\n",
    "    - Introduction\n",
    "- **Preparation**\n",
    "    - PR:1 Initialise environment\n",
    "    - PR:2 Verify MISP modules\n",
    "    - PR:3 Load helper functions\n",
    "    - PR:4 Set helper variables\n",
    "    - PR:5 Provide the list of hashes\n",
    "    - PR:6 MISP event details\n",
    "    - PR:7 Setup MISP event link\n",
    "- **Investigate**\n",
    "    - IN:1 Context details for the hashes\n",
    "    - IN:2 Add the hashes to the event\n",
    "    - IN:3 Matches with MISP warninglists\n",
    "    - IN:4 Download a sample    \n",
    "- **Correlation**\n",
    "    - CR:1 Correlation with MISP events\n",
    "    - CR:2 Correlation with MISP feeds\n",
    "- **Enrichment**\n",
    "    - ER:1 Enrich with information from VirusTotal\n",
    "    - ER:2 Enrich with information from Hashlookup\n",
    "    - ER:3 Enrich with information from MalwareBazaar    \n",
    "    - ER:4 Investigation and enrichment report\n",
    "    - ER:5 PEinfo report\n",
    "- **Summary**\n",
    "    - EN:1 MISP indicators\n",
    "    - EN:2 Create the summary of the playbook\n",
    "    - EN:3 Print the summary\n",
    "    - EN:4 Send a summary to Mattermost\n",
    "    - EN:5 End of the playbook\n",
    "- External references\n",
    "- Technical details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc24913",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Preparation\n",
    "\n",
    "## PR:1 Initialise environment\n",
    "\n",
    "This section **initialises the playbook environment** and loads the required Python libraries. \n",
    "\n",
    "The credentials for MISP (**API key**) and other services are loaded from the file `keys.py` in the directory **vault**. A [PyMISP](https://github.com/MISP/PyMISP) object is created to interact with MISP and the active MISP server is displayed. By printing out the server name you know that it's possible to connect to MISP. In case of a problem PyMISP will indicate the error with `PyMISPError: Unable to connect to MISP`.\n",
    "\n",
    "The contents of the `keys.py` file should contain at least :\n",
    "\n",
    "```\n",
    "misp_url=\"<MISP URL>\"                  # The URL to our MISP server\n",
    "misp_key=\"<MISP API KEY>\"              # The MISP API key\n",
    "misp_verifycert=<True or False>        # Indicate if PyMISP should attempt to verify the certificate or ignore errors\n",
    "mattermost_playbook_user=\"<MATTERMOST USER>\"\n",
    "mattermost_hook=\"<MATTERMOST WEBHOOK>\"\n",
    "virustotal_apikey=\"<VIRUSTOTAL_APIKEY>\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f94c336",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialise Python environment\n",
    "import urllib3\n",
    "import sys\n",
    "import json\n",
    "import base64\n",
    "import uuid\n",
    "import time\n",
    "import re\n",
    "from prettytable import PrettyTable, MARKDOWN\n",
    "from IPython.display import Image, display, display_markdown, HTML\n",
    "from datetime import date\n",
    "import requests\n",
    "from pymisp import *\n",
    "from pymisp.tools import GenericObjectGenerator\n",
    "from abuse_finder import domain_abuse, ip_abuse, email_abuse, url_abuse\n",
    "import textwrap\n",
    "from datetime import datetime\n",
    "\n",
    "# Used for -optionally- drawing the maps.\n",
    "import folium\n",
    "\n",
    "# Load the credentials\n",
    "sys.path.insert(0, \"../vault/\")\n",
    "from keys import *\n",
    "if misp_verifycert is False:\n",
    "    import urllib3\n",
    "    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n",
    "print(\"The \\033[92mPython libraries\\033[90m are loaded and the \\033[92mcredentials\\033[90m are read from the keys file.\")\n",
    "\n",
    "# Create the PyMISP object\n",
    "misp = PyMISP(misp_url, misp_key, misp_verifycert)\n",
    "misp_headers = {\"Authorization\": misp_key,  \"Content-Type\": \"application/json\", \"Accept\": \"application/json\"}\n",
    "print(\"I will use the MISP server \\033[92m{}\\033[90m for this playbook.\\n\".format(misp_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e4c6424-0056-416a-812c-04a7559cbe56",
   "metadata": {},
   "source": [
    "## PR:2 Verify MISP modules\n",
    "\n",
    "This playbook uses the MISP modules to obtain additional correlation or enrichment information. [MISP modules](https://github.com/MISP/misp-modules) are autonomous modules that can be used to extend MISP for new services such as expansion, import and export. The modules are written in Python 3 following a simple API interface. The objective is to ease the extensions of MISP functionalities without modifying core components. The API is available via a simple REST API which is independent from MISP installation or configuration.\n",
    "\n",
    "In the next cell we check if we have access to the **MISP module** server and if the required modules are enabled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a797506-c5d7-45e8-ae58-cbf9a833162c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Where can we find the local MISP Module server? You can leave this to the default setting in most cases.\n",
    "misp_modules_url = \"http://127.0.0.1:6666\"\n",
    "\n",
    "# How long do we wait between queries when using the MISP modules (API rate limiting of external service such as VirusTotal)\n",
    "misp_modules_wait = 3\n",
    "\n",
    "# Initiliasation\n",
    "misp_modules = {}\n",
    "misp_modules_headers = {\n",
    "    \"Content-Type\": \"application/json\",\n",
    "    \"Accept\": \"application/json\"\n",
    "}\n",
    "misp_modules_in_use = [\"hashlookup\", \"virustotal_public\", \"malwarebazaar\"]\n",
    "# Code block to query the MISP module server and check if our modules are enabled\n",
    "res = requests.get(\"{}/modules\".format(misp_modules_url), headers=misp_modules_headers)\n",
    "for module in res.json():\n",
    "    for module_requested in misp_modules_in_use:\n",
    "        if module.get(\"name\", False) == module_requested:\n",
    "            misp_modules[module_requested] = {\"enabled\": True, \"input\": module.get(\"mispattributes\").get(\"input\")}\n",
    "            print(\"Found the \\033[92m{}\\033[90m MISP module (Accepted input: {}).\".format(module_requested, misp_modules[module_requested][\"input\"]))\n",
    "print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e92317-49fd-4746-b61e-abd84468114e",
   "metadata": {},
   "source": [
    "## PR:3 Load helper functions\n",
    "\n",
    "The next cell contains **helper functions** that are used in this playbook. \n",
    "\n",
    "Instead of distributing helper functions as separate Python files this playbook includes all the required code as one code cell. This makes portability of playbooks between instances easier. The downside is that functions defined in this playbook need to be defined again in other playbooks, which is not optimal for code re-use. For this iteration of playbooks it is chosen to include the code in the playbook (more portability), but you can easily create one \"helper\" file that contains all the helper code and then import that file in each playbook (for example by adding to the previous cell `from helpers import *`). Note that the graphical workflow image is included as an external image. A missing image would not influence the further progress of the playbook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfab4370-b5be-483d-9344-0b414a06cd61",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def pb_get_misp_tags(tags=[], local_tags=[]):\n",
    "    '''\n",
    "    Get a list of MISP tags based on a Python list\n",
    "\n",
    "    :param misp: MISP object\n",
    "    :param object_template: which object template to return\n",
    "    '''\n",
    "    misp_tags = []\n",
    "    for el in tags:\n",
    "        t = MISPTag()\n",
    "        t.name = el\n",
    "        t.local = False\n",
    "        misp_tags.append(t)\n",
    "\n",
    "    for el in local_tags:\n",
    "        t = MISPTag()\n",
    "        t.name = el\n",
    "        t.local = True\n",
    "        misp_tags.append(t)\n",
    "    return misp_tags\n",
    "\n",
    "\n",
    "\n",
    "def get_playbook_result(value, source, category, res_type):\n",
    "    '''\n",
    "    Lookup the playbook results for a list of entries\n",
    "    \n",
    "    : param value: The IP address under investigation\n",
    "    : param source: Which source provided the enrichment?\n",
    "    : param category: Enrichment category\n",
    "    : param res_type: Enrichment type\n",
    "    '''\n",
    "    return_value = []\n",
    "    for el in playbook_results[value]:\n",
    "        if el[\"source\"] == source and el[\"category\"] == category and el[\"type\"] == res_type:\n",
    "            if el[\"enriched\"] not in return_value:\n",
    "                if el[\"enriched\"] is list:\n",
    "                    for list_item in el[\"enriched\"]:\n",
    "                        return_value.append(str(list_item))\n",
    "                else:\n",
    "                    return_value.append(str(el[\"enriched\"]))\n",
    "    return_value = ' '.join(return_value).strip()\n",
    "    if len(return_value) < 1:\n",
    "        return_value = \"-\"\n",
    "    return return_value\n",
    "\n",
    "\n",
    "\n",
    "def pb_add_enrichment(playbook_results, field, entry, key, value):\n",
    "    '''\n",
    "    Add an enrichment (or correlation) entry but first check that the value is not already there\n",
    "    \n",
    "    : param playbook_results: all the enrichment results\n",
    "    : param field\n",
    "    : param entry\n",
    "    : param key\n",
    "    : param value\n",
    "    '''\n",
    "    skip_field = False\n",
    "    for existing_entry in playbook_results.get(field, []):\n",
    "        if existing_entry.get(key, False) == value:\n",
    "            skip_field = True\n",
    "            print(\" Not adding to playbook results because of duplicate. Already added via {}\".format(existing_entry.get(\"source\", False)))\n",
    "    if not skip_field:\n",
    "        if field in playbook_results:\n",
    "            playbook_results[field].append(entry)\n",
    "        else:\n",
    "            playbook_results[field] = [entry]\n",
    "    return playbook_results\n",
    "\n",
    "\n",
    "\n",
    "def calculate_risk_score(ratio):\n",
    "    '''\n",
    "    Calculate risk score and return the score and a tag\n",
    "    '''\n",
    "    detection_ratio_perc = 0\n",
    "    score = 0\n",
    "    base = 0\n",
    "    tag = \"\"\n",
    "    if len(ratio) > 0:\n",
    "        for detection_ratio in ratio:\n",
    "            score += int(detection_ratio.split(\"/\")[0])\n",
    "            base += int(detection_ratio.split(\"/\")[1])\n",
    "        if base > 0:\n",
    "            detection_ratio_perc = round((score / base) * 100, 2)\n",
    "            if detection_ratio_perc > 0:\n",
    "                if detection_ratio_perc >= 75:\n",
    "                    tag = \"misp:threat-level=\\\"high-risk\\\"\"\n",
    "                elif detection_ratio_perc >= 50:\n",
    "                    tag = \"misp:threat-level=\\\"medium-risk\\\"\"\n",
    "                #elif detection_ratio_perc >= 25:\n",
    "                else:\n",
    "                    tag = \"misp:threat-level=\\\"low-risk\\\"\"\n",
    "            else:\n",
    "                tag = \"misp:threat-level=\\\"no-risk\\\"\"\n",
    "    return detection_ratio_perc, tag\n",
    "\n",
    "\n",
    "\n",
    "def resolve_hash_type(hash_type):\n",
    "    '''\n",
    "    Do an educated guess for the hash type\n",
    "    '''\n",
    "    if len(hash_type) in file_hash_type_length:\n",
    "        return file_hash_type_length[len(hash_type)]\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "print(\"\\033[92mHelper functions loaded\\033[90m.\\n\".format(misp_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b0fe9f4-23b1-4eec-bc61-8bc965ffc619",
   "metadata": {},
   "source": [
    "## PR:4 Set helper variables\n",
    "\n",
    "This cell contains **helper variables** that are used in this playbook. Their usage is explained in the next steps of the playbook.\n",
    "\n",
    "- `samples` : a dictionary of the objects that are created when the playbook progresses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "645d8ee5-e07f-4bab-953c-8d71f9c1b171",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Dictionary to playbook results and some of the core objects that are created\n",
    "samples = {}\n",
    "\n",
    "# Used as input for educated guesses on the hash type\n",
    "file_hash_type_length = {32:\"md5\", 40:\"sha1\", 64:\"sha256\", 128:\"sha512\"}\n",
    "\n",
    "# Hash types returned by external VT\n",
    "file_hash_types = [\"md5\", \"sha1\", \"sha256\", \"ssdeep\"]\n",
    "\n",
    "# A set of regular expressions that we use to determine the attribute type in a warninglist\n",
    "regular_expressions = {\"sha256\": \"^[a-fA-F0-9]{64}$\",\n",
    "                       \"md5\": \"^[a-fA-F0-9]{32}$\",\n",
    "                       \"hostname\": \"^[a-zA-Z0-9.\\-_]+\\.[a-zA-Z]{2,}$\",\n",
    "                       \"sha1\": \"^[a-fA-F0-9]{40}$\",\n",
    "                       \"url\": \"^(http|https):\\/\\/[-a-zA-Z0-9-]{2,256}\\.[-a-zA-Z0-9-]{2,256}\",\n",
    "                       \"ip-src\": \"[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\",\n",
    "                       \"ip-dst\": \"[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\"\n",
    "                      }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62f98308-a0c2-486d-9675-44f51f9a538e",
   "metadata": {},
   "source": [
    "## PR:5 Provide the list of hashes\n",
    "\n",
    "Add the **hashes** that you want to investigate with this playbook to the variable `query`. You can provide one hash or a list of hashes. You can supply any type of hash, for example MD5, SHA1 or SHA256."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "01fd3a79-9ab2-4fde-a9d3-2a242a621a8f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Provide one or more hashes\n",
    "query = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b67481c1-3e9f-4f18-82b8-284c132057cf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Simple function to make a list. Makes it more consistent to work with the user input in the next cells.\n",
    "if type(query) == str:\n",
    "    query =[query]\n",
    "print(\"The playbook will do the hash information query for \\033[92m{}\\033[90m\\n\".format(query))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bfd0045-13fe-4f83-a6a7-2d70621b7423",
   "metadata": {},
   "source": [
    "## PR:6 MISP event details\n",
    "\n",
    "### Event title\n",
    "\n",
    "In this playbook we create a **new** MISP event with title **IP reputation _IPlist_**. You get the chance to override this default title but remember that it is good practice to choose a self-explanatory **event title**. This event title is shown in the MISP event index and should provide you the necessary information what the event is about. You should avoid using generic event titles. Read the [Best Practices in Threat Intelligence](https://www.misp-project.org/best-practices-in-threat-intelligence.html) for further guidance.\n",
    "\n",
    "### Contexualisation\n",
    "\n",
    "This playbook adds event contexualisation via the **tags** that are defined in `event_additional_global_tags` (for *global* tags) and `event_additional_local_tags` (for *local* tags). As a reminder, whereas *global* tags remain attached to the events that you share with your community, the *local* tags are not shared outside your organisation. It's also a good idea to primarily use tags that are part of a [taxonomy](https://github.com/MISP/misp-taxonomies), this allows you to make the contexualisation more portable accross multiple MISP instances.\n",
    "\n",
    "In this playbook the list of tags is build via one of the helper functions `pb_get_misp_tags`. This function takes two arguments, first a list of tags to convert as *global* tags, and secondly a list of tags to convert as *local* tags. It then returns a Python list of MISPTag objects.\n",
    "\n",
    "### Traffic Light Protocol\n",
    "\n",
    "The default **TLP** for this event is **<span style='color:#FFBF00'>tlp:amber</span>**. The Traffic Light Protocol (TLP) facilitates sharing of potentially sensitive information and allows for more effective collaboration. TLP is a set of four standard labels to indicate the sharing boundaries to be applied by the recipients. TLP is always set by the creator of information. You can find more information at [FIRST](https://www.first.org/tlp/). You can specify the TLP via `event_tlp`.\n",
    "\n",
    "### MISP Galaxies\n",
    "\n",
    "This playbook can also add MISP galaxies to the event with the variable `event_galaxies`. You can also leave the list empty if you do not want to add galaxies in this stage of the investigation.\n",
    "\n",
    "### MISP distribution, threat level and analysis level\n",
    "\n",
    "Optionally you can specifiy a MISP **distribution** (with `event_distribution`), **threat level** (with `event_threat_level_id`) or **analysis state** (with `event_analysis`). The event **date** is set to today via `event_date`.\n",
    "\n",
    "If you cannot remember the options for distribution, threat level or the analysis state then use the next cell to guide you. This cell is set as **raw**. If you **change its type to code** and execute the cell you get an overview of the options available for creating a MISP event."
   ]
  },
  {
   "cell_type": "raw",
   "id": "0f74da01-e0db-4140-9386-09be040d7fbd",
   "metadata": {
    "tags": []
   },
   "source": [
    "# This cell is 'raw' by default. Change it to 'code' to execute it.\n",
    "\n",
    "print(\"Distribution value\")\n",
    "for e in Distribution:\n",
    "    print(\" \", e.value, \"=\", e)\n",
    "print(\"\\n\")\n",
    "print(\"Threat level\")\n",
    "for e in ThreatLevel:\n",
    "    print(\" \", e.value, \"=\", e)\n",
    "print(\"\\n\")\n",
    "print(\"Analysis level\")\n",
    "for e in Analysis:\n",
    "    print(\" \", e.value, \"=\", e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f46eaf74-a42e-41c5-8f10-e8882d5dd31d",
   "metadata": {},
   "source": [
    "## PR:7 Setup MISP event link\n",
    "\n",
    "By default the playbook will generate a **title** with a prefix and the IPs you want to investigate. You can override this event title with the variable `event_title`. If you leave `event_title` empty the playbook will generate the MISP event title for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f2f2568-9ee6-4f82-a267-075e98f4e477",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Provide the event title for a new event. Leave blank for the playbook to auto generate one\n",
    "event_title = \"\"\n",
    "\n",
    "# Prefix for auto generate event title\n",
    "event_title_default_prefix = \"Hash information\"\n",
    "\n",
    "# Optionally, you can change TLP, add additional event (local and global) tags, threatlevel, analysis state or distribution level\n",
    "event_tlp = \"tlp:amber\"\n",
    "\n",
    "# Event context\n",
    "event_additional_global_tags = []                                 # This needs to be a Python list\n",
    "event_additional_local_tags = [\"workflow:state=\\\"incomplete\\\"\"]   # This needs to be a Python list\n",
    "\n",
    "# Event galaxies\n",
    "event_galaxies = [\"misp-galaxy:mitre-attack-pattern=\\\"Malware - T1587.001\\\"\", \n",
    "                  \"misp-galaxy:mitre-attack-pattern=\\\"Malware - T1588.001\\\"\"]\n",
    "\n",
    "# Additional MISP event settings\n",
    "event_threat_level_id = ThreatLevel.low\n",
    "event_analysis = Analysis.ongoing\n",
    "event_distribution = Distribution.your_organisation_only\n",
    "event_date = date.today()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a6b266f-577f-4a4c-882b-10542107987e",
   "metadata": {},
   "source": [
    "### Create MISP event\n",
    "\n",
    "The next cell **creates the MISP event** and stores the event object in `misp_event`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "864d7c62-a923-472a-937a-ba196c148b77",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Code block to create the event or add data to an existing event\n",
    "event_title = event_title.strip()\n",
    "\n",
    "if not(len(event_title) > 0):\n",
    "    query_sample = \"\"\n",
    "    for key in query:\n",
    "        query_sample = \"{} {}\".format(key, query_sample)\n",
    "    event_title = \"{} for {}\".format(event_title_default_prefix, query_sample)\n",
    "\n",
    "# Construct the event tags\n",
    "event_additional_global_tags.append(event_tlp)\n",
    "if len(event_galaxies) > 0:\n",
    "    event_additional_global_tags.append(event_galaxies)\n",
    "event_tags = pb_get_misp_tags(event_additional_global_tags, event_additional_local_tags)\n",
    "\n",
    "# Create the PyMISP object for an event\n",
    "event = MISPEvent()\n",
    "event.info = event_title\n",
    "event.distribution = event_distribution\n",
    "event.threat_level_id = event_threat_level_id\n",
    "event.analysis = event_analysis\n",
    "event.set_date(event_date)\n",
    "\n",
    "# Create the MISP event on the server side\n",
    "misp_event = misp.add_event(event, pythonify=True)\n",
    "print(\"Continue the playbook with the new \\033[92mcreated\\033[90m MISP event ID {} with title \\033[92m{}\\033[90m and UUID {}\".format(misp_event.id, misp_event.info, misp_event.uuid))\n",
    "for tag in event_tags:\n",
    "    if len(tag) > 0:\n",
    "        misp.tag(misp_event.uuid, tag, local=tag.local)\n",
    "        print(\" \\033[92mAdded\\033[90m event tag {}\".format(tag))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2243ae8b-cb77-4835-a0cd-2738ee8c9bec",
   "metadata": {},
   "source": [
    "# Investigate\n",
    "\n",
    "This section starts the investigation for the hash information.\n",
    "\n",
    "## IN:1 Context details for the hashes\n",
    "\n",
    "The first step sets the contextualisation (tags) for the hashes with `attribute_tags`. These tags are added to the MISP attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "171aee2b-bf8b-4668-9992-3c64f04e7ec3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "attribute_tags = pb_get_misp_tags([\"PAP:GREEN\", \"course-of-action:passive=\\\"discover\\\"\",\n",
    "                                                  \"course-of-action:passive=\\\"detect\\\"\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4be95a1-67ee-4c5a-85e3-a9607f335f40",
   "metadata": {},
   "source": [
    "## IN:2 Add the hashes to the event\n",
    "\n",
    "Next we add the hashes to the MISP event. The hashes are added as part of MISP `file` **objects**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d17bcaf4-07d9-4329-bfb4-ec04a1530294",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Add hashes to MISP event.\")\n",
    "for hash_value in query:\n",
    "    hash_type = resolve_hash_type(hash_value)\n",
    "    if hash_type:\n",
    "        file_object = MISPObject(\"file\")\n",
    "        file_object.add_attribute(hash_type, hash_value, tags=attribute_tags, comment=\"Added by playbook\")\n",
    "        response = misp.add_object(misp_event.uuid, file_object, pythonify=True)\n",
    "        if not \"errors\" in response:\n",
    "            samples[hash_value] = {}\n",
    "            samples[hash_value][\"hash_type\"] = hash_type\n",
    "            samples[hash_value][\"fileobject\"] = response\n",
    "            samples[hash_value][\"fileobject_uuid\"] = response.uuid\n",
    "    \n",
    "            samples[hash_value][hash_type] = hash_value\n",
    "            for attribute in response.attributes:\n",
    "                if attribute.object_relation == hash_type:\n",
    "                    samples[hash_value][\"hash_attribute_uuid\"] = attribute.uuid\n",
    "                    break\n",
    "                    \n",
    "            samples[hash_value][\"MISP\"] = []\n",
    "            samples[hash_value][\"Feeds\"] = []\n",
    "            samples[hash_value][\"detection_ratio\"] = []\n",
    "            samples[hash_value][\"detection_ratio_indirect\"] = []\n",
    "            samples[hash_value][\"vt\"] = {}\n",
    "            samples[hash_value][\"warninglist\"] = \"\"\n",
    "            samples[hash_value][\"malwarebazaar\"] = {}\n",
    "            samples[hash_value][\"hashlookup\"] = {}\n",
    "            print(\" Added file object for \\033[92m{}\\033[90m as {} with UUID {}\".format(hash_value, hash_type, response.uuid))\n",
    "        else:\n",
    "            print(result)        \n",
    "    else:\n",
    "        print(\" Unknown hash type \\033[91m{}\\033[90m\".format(hash_value))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e65ab9d-219a-4e96-b540-51fe953ecd90",
   "metadata": {},
   "source": [
    "## IN:3 Matches with MISP warninglists\n",
    "\n",
    "The [misp-warninglists](https://github.com/MISP/misp-warninglists) are lists of **well-known** indicators that can be associated to potential **false positives**, errors or mistakes. The warning lists are integrated in MISP to display an info/warning box at the event and attribute level if such indicators are available in one of the list. The lists are also used to filter potential false-positive at API level. MISP warninglists also serve a dual use. You cannot only use them to identify false positives, you can also use the lists to **track** if specific attributes (for example IPs or domains) are in the threat events you receive from your community.\n",
    "\n",
    "Have a look at the other playbooks on warninglists, such as : [Create a custom MISP warninglist](https://github.com/MISP/misp-playbooks/blob/main/misp-playbooks/pb_create_custom_MISP_warninglist.ipynb) or [Retroscan with a MISP warninglist](https://github.com/MISP/misp-playbooks/blob/main/misp-playbooks/pb_retroscan_with_MISP_warninglist.ipynb), for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "830ed4ca-d253-4e27-9f8f-907caa4971dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Search for matches with MISP warninglists\")\n",
    "for key, sample in samples.items():\n",
    "    result = misp.values_in_warninglist(key)\n",
    "    if not \"errors\" in result and len(result) > 0:\n",
    "        print(\" Match with warninglist for \\033[92m{}\\033[90m\".format(key))\n",
    "        sample[\"warninglist\"] = \"\\n### MISP Warninglists\\n\\n\"\n",
    "        for match in result[key]:\n",
    "            print(\"  Found in \\033[92m{}\\033[90m\".format(match[\"name\"]))\n",
    "            sample[\"warninglist\"] = \"{}- Found in **{}**\\n\".format(sample[\"warninglist\"], match[\"name\"])\n",
    "    else:\n",
    "        print(\" No match found for \\033[91m{}\\033[90m\".format(key))\n",
    "print(\"Finished searching\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e0cc9d1-0f55-4ddb-a036-a4b6331ab90c",
   "metadata": {},
   "source": [
    "## IN:4 Download a sample\n",
    "\n",
    "This playbook does not include automatic downloading of malware samples from external sources. \n",
    "\n",
    "You can search for the samples in [VirusShare](https://virusshare.com/), or if your license permits it, query [VirusTotal](https://www.virustotal.com). If you download the sample you use the playbooks for [static malware analysis](https://github.com/MISP/misp-playbooks/blob/main/misp-playbooks/pb_malware_triage.ipynb) and [dynamic malware analysis](https://github.com/MISP/misp-playbooks/blob/main/misp-playbooks/pb_malware_triage_upload_sample.ipynb) to investigate it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aefa562-90fb-449d-be2e-cccf76060e74",
   "metadata": {},
   "source": [
    "# Correlation\n",
    "\n",
    "## CR:1 Correlation with MISP events\n",
    "\n",
    "Search the **MISP server** for events that match with one of the hashes you specified in `query`. A summary is also shown at the end of the playbook.\n",
    "\n",
    "Only **published** events (`correlation_published`) and attributes that have the **to_ids** flag (`correlation_to_ids`) are considered. You can limit the number of results (`correlation_limit`) and limit the search with tags (`correlation_match_tags`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69ad9cc1-ab8a-4bc1-a485-11565b849785",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Only query for published MISP events\n",
    "correlation_published = False\n",
    "\n",
    "# Only consider those values that have the to_ids field set to True\n",
    "correlation_to_ids = True\n",
    "\n",
    "# Limit the returned results to 1000 attributes\n",
    "correlation_limit = 1000\n",
    "\n",
    "# Only return results corresponding with these tags\n",
    "correlation_match_tags = [\"tlp:amber\", \"tlp:white\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba6ac8f8-cd37-4db6-be8a-12e39275d63b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Search for correlating MISP events\")\n",
    "# Code block to query MISP and find the correlations\n",
    "for key, sample in samples.items():\n",
    "    value = key\n",
    "    search_match = misp.search(\"attributes\", to_ids=correlation_to_ids, value=value, tags=correlation_match_tags,\n",
    "                                   published=correlation_published, limit=correlation_limit, pythonify=True)\n",
    "    if len(search_match) > 0:        \n",
    "        for attribute in search_match:\n",
    "            if attribute.Event.id != misp_event.id:   # Skip the event we just created for this playbook\n",
    "                print(\" Found \\033[92m{}\\033[90m in \\033[92m{}\\033[90m ({})\".format(attribute.value, attribute.Event.info, attribute.Event.id ))\n",
    "                entry = {\"source\": \"MISP\", \"category\": attribute.category, \"type\": attribute.type, \"event_id\": attribute.Event.id, \"event_info\": attribute.Event.info}\n",
    "                samples[key][\"MISP\"].append(entry)\n",
    "print(\"Finished searching for correlations\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c10d315-123f-4622-b764-13c1a036ebb5",
   "metadata": {},
   "source": [
    "### MISP events correlation table\n",
    "\n",
    "The correlation results are now stored in `playbook_results`. Execute the next cell to display them in a table format. The table is also included in the summary for Mattermost and TheHive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d098b83-1733-4804-939e-c9d370d8e596",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Put the correlations in a pretty table. We can use this table later also for the summary\n",
    "table = PrettyTable()\n",
    "table.field_names = [\"Source\", \"Value\", \"Category\", \"Type\", \"Event\", \"Event ID\"]\n",
    "table.align[\"Value\"] = \"l\"\n",
    "table.align[\"Category\"] = \"l\"\n",
    "table.align[\"Type\"] = \"l\"\n",
    "table.align[\"Event\"] = \"l\"\n",
    "table.align[\"Event ID\"] = \"l\"\n",
    "table._max_width = {\"Event\": 50}\n",
    "for key, sample in samples.items():\n",
    "    for match in sample[\"MISP\"]:\n",
    "        table.add_row([match[\"source\"], key, match[\"category\"], match[\"type\"], match[\"event_info\"], match[\"event_id\"]])\n",
    "print(table.get_string(sortby=\"Value\"))\n",
    "table_mispevents = table"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0469a986-b85e-4f99-ac16-8957447ec76e",
   "metadata": {},
   "source": [
    "## CR:2 Correlation with MISP feeds\n",
    "\n",
    "Search the **MISP feeds** for events that match with one of the hashes you specified in `query`. The results highly depend on the feeds you have enabled. For good results, it's advised you enable at least\n",
    "- **Threatfox**\n",
    "- **MalwareBazaar**\n",
    "\n",
    "Be sure to check if other [MISP feeds](https://www.misp-project.org/feeds/) provide valuable results for your environment as well. Note that the correlation lookup in the MISP feeds does not return the name of the MISP event, it returns the UUID of the event as title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21a234ec-0ecd-493c-bfac-c1c6aaa6ce4d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Search in MISP feeds\")\n",
    "misp_cache_url = \"{}/feeds/searchCaches/\".format(misp_url)\n",
    "match = False\n",
    "for key, sample in samples.items():\n",
    "    value = key\n",
    "    # Instead of GET, use POST (https://github.com/MISP/MISP/issues/7478)\n",
    "    cache_results = requests.post(misp_cache_url, headers=misp_headers, verify=misp_verifycert, json={\"value\": value})\n",
    "    for result in cache_results.json():\n",
    "        if \"Feed\" in result:\n",
    "            match = True\n",
    "            print(\" Found \\033[92m{}\\033[90m in \\033[92m{}\\033[90m\".format(value, result[\"Feed\"][\"name\"]))\n",
    "            for match in result[\"Feed\"][\"direct_urls\"]:\n",
    "                entry = {\"source\": \"Feeds\", \"feed_name\": result[\"Feed\"][\"name\"], \"match_url\": match[\"url\"]}\n",
    "                samples[key][\"Feeds\"].append(entry)\n",
    "\n",
    "print(\"Finished searching in MISP feeds\")\n",
    "if not match:\n",
    "    print(\"\\033[93mNo correlating information found in MISP feeds.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "597185dd-bfd2-4dbb-9a93-46dff4f68945",
   "metadata": {},
   "source": [
    "### MISP feed correlations table\n",
    "\n",
    "The correlation results are now stored in `playbook_results`. Execute the next cell to display them in a table format. The table is also included in the summary for Mattermost and TheHive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e1734e3-b012-44ab-b6fa-d7090758bdaf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Put the correlations in a pretty table. We can use this table later also for the summary\n",
    "table = PrettyTable()\n",
    "table.field_names = [\"Source\", \"Value\", \"Feed\", \"URL\"]\n",
    "table.align[\"Value\"] = \"l\"\n",
    "table.align[\"Feed\"] = \"l\"\n",
    "table.align[\"Feed URL\"] = \"l\"\n",
    "table._max_width = {\"Feed\": 50}\n",
    "for key, sample in samples.items():\n",
    "    for match in sample[\"Feeds\"]:\n",
    "        table.add_row([match[\"source\"], key, match[\"feed_name\"], match[\"match_url\"]])\n",
    "print(table.get_string(sortby=\"Value\"))\n",
    "table_mispfeeds = table"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af1b0e9c-1b06-4ac5-b620-c349e5cc893f",
   "metadata": {},
   "source": [
    "# Enrichment\n",
    "\n",
    "Next we enrich the hash information with results from external sources.\n",
    "\n",
    "## ER:1 Enrich with information from VirusTotal\n",
    "\n",
    "Next we use [VirusTotal](https://www.virustotal.com/gui/). First we use the MISP modules to query VirusTotal. The advantage of using the MISP modules is that it returns MISP attributes and objects that can then be easily added to the event.\n",
    "\n",
    "After using the modules, we query VirusTotal directly (without using the module) to get the **threat classification**, **sandbox** results and the **Sigma** rules that triggered detection for the file hashes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46e75637-060a-491d-91d4-e5b5fd7c0c6f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Loop through the hashes\n",
    "\n",
    "# Use the modules to create MISP objects\n",
    "# Then do a second query to get the hash details\n",
    "\n",
    "scan_count = 0\n",
    "module_name = \"virustotal_public\"\n",
    "headers = {\n",
    "    \"accept\": \"application/json\",\n",
    "    \"x-apikey\": virustotal_apikey\n",
    "}\n",
    "\n",
    "print(\"Verifying samples with {}\".format(module_name))\n",
    "for key, sample in samples.items():\n",
    "    attribute_type = sample[\"hash_type\"]\n",
    "    value = key\n",
    "    module_comment = \"From {} for {}\".format(module_name, key)\n",
    "    data = {\n",
    "        \"attribute\": {\n",
    "            \"type\": f\"{attribute_type}\",\n",
    "            \"uuid\": str(uuid.uuid4()),\n",
    "            \"value\": f\"{value}\",\n",
    "        },\n",
    "        \"module\": module_name,\n",
    "        \"config\": {\"apikey\": virustotal_apikey}\n",
    "    }\n",
    "    print(\"Query \\033[92m{}\\033[90m as \\033[92m{}\\033[90m\".format(value, attribute_type))\n",
    "    result = requests.post(\"{}/query\".format(misp_modules_url), headers=misp_modules_headers, json=data)\n",
    "    if \"results\" in result.json() and len(result.json()[\"results\"]) > 0:\n",
    "        result_json = result.json()[\"results\"]\n",
    "\n",
    "        for misp_attribute in result_json.get(\"Attribute\", []):\n",
    "            misp_attribute[\"comment\"] = \"{}{}\".format(module_comment, misp_attribute.get(\"comment\", \"\"))\n",
    "            created_attribute = misp.add_attribute(misp_event.uuid, misp_attribute, pythonify=True)\n",
    "            if not \"errors\" in created_attribute:\n",
    "                print(\" Got {} \\033[92m{}\\033[90m\".format(misp_attribute[\"type\"], misp_attribute[\"value\"]))\n",
    "                misp.add_object_reference(sample.get(\"fileobject\").add_reference(created_attribute.uuid, \"related-to\"))\n",
    "            else:\n",
    "                print(\" Unable to add {} \\033[92m{}\\033[90m to MISP event\".format(misp_attribute[\"type\"], misp_attribute[\"value\"]))\n",
    "        for misp_object in result_json.get(\"Object\", []):\n",
    "            misp_object[\"comment\"] = \"{}{}\".format(module_comment, misp_object.get(\"comment\", \"\"))\n",
    "            if len(misp_object[\"Attribute\"]) > 0:\n",
    "                created_object = misp.add_object(misp_event.uuid, misp_object, pythonify=True)\n",
    "                if not \"errors\" in created_object:\n",
    "                    print(\" Got \\033[92m{}\\033[90m \".format(misp_object[\"name\"]))\n",
    "                    misp.add_object_reference(sample.get(\"fileobject\").add_reference(created_object.uuid, \"related-to\"))\n",
    "                    if misp_object[\"name\"] == \"virustotal-report\":\n",
    "                        direct_vt_report = False\n",
    "                        detection_ratio = False\n",
    "                        for attribute in misp_object.get(\"Attribute\", []):\n",
    "                            # We cannot check for a permalink with the value\n",
    "                            # Permalinks point to the SHA256, and the value can be MD5, SHA1 or others\n",
    "                            #if attribute.get(\"object_relation\") == \"permalink\" and value in attribute.get(\"value\"):\n",
    "                            if attribute.get(\"object_relation\") == \"permalink\" and \"virustotal.com/gui/file/\" in attribute.get(\"value\"):\n",
    "                                direct_vt_report = True\n",
    "                            if attribute.get(\"object_relation\") == \"detection-ratio\":\n",
    "                                detection_ratio = attribute.get(\"value\")\n",
    "                        if detection_ratio:\n",
    "                            if direct_vt_report:\n",
    "                                samples[key][\"detection_ratio\"].append(detection_ratio)\n",
    "                                print(\" Got detection ratio \\033[92m{}\\033[90m \".format(detection_ratio))\n",
    "                            else:\n",
    "                                samples[key][\"detection_ratio_indirect\"].append(detection_ratio)\n",
    "                                print(\" Got indirect detection ratio \\033[92m{}\\033[90m \".format(detection_ratio))\n",
    "                else:\n",
    "                    print(\" Unable to add \\033[92m{}\\033[90m to MISP event\".format(misp_object[\"name\"]))\n",
    "\n",
    "        risk1, tag1 = calculate_risk_score(sample.get(\"detection_ratio\"))\n",
    "        risk2, tag2 = calculate_risk_score(sample.get(\"detection_ratio_indirect\"))\n",
    "        if len(tag1) > 0:\n",
    "            misp.tag(sample.get(\"hash_attribute_uuid\"), tag1)\n",
    "\n",
    "        risk_rating = \"vt-detection-ratio={}\\nvt-detection-ratio-indirect={}\".format(risk1, risk2)\n",
    "        samples[key][\"detection_ratio_perc\"] = risk1\n",
    "        samples[key][\"detection_ratio_indirect_perc\"] = risk2\n",
    "        samples[key][\"risk_rating\"] = risk_rating\n",
    "        sample.get(\"fileobject\").add_attribute(\"text\", risk_rating)\n",
    "        misp.update_object(sample.get(\"fileobject\"))\n",
    "\n",
    "        #################################################################\n",
    "\n",
    "        print(\" Query VirusTotal directly to get file details\")\n",
    "        vt_url = \"https://www.virustotal.com/api/v3/files/{}\".format(value)\n",
    "        result = requests.get(vt_url, headers=headers)\n",
    "        if \"data\" in result.json() and len(result.json()[\"data\"]) > 0:\n",
    "            result_json = result.json()[\"data\"][\"attributes\"]\n",
    "            samples[key][\"vt\"][\"data\"] = result_json\n",
    "            samples[key][\"vt\"][\"popular_threat_classification\"] = result_json.get(\"popular_threat_classification\", [])\n",
    "            samples[key][\"vt\"][\"sigma_analysis_results\"] = result_json.get(\"sigma_analysis_results\", [])\n",
    "            samples[key][\"vt\"][\"sandbox_verdicts\"] = result_json.get(\"sandbox_verdicts\", [])\n",
    "            samples[key][\"vt\"][\"sandboxes\"] = []\n",
    "            samples[key][\"vt\"][\"markdown\"] = \"\"\n",
    "\n",
    "            # Get the tags of our \"core\" hash\n",
    "            attribute_tags = misp.get_attribute(sample[\"hash_attribute_uuid\"], pythonify=True).tags\n",
    "            #sample_hashes = { hash_type: key}\n",
    "            for vt_hash_type in file_hash_types:\n",
    "                if not sample.get(vt_hash_type, False) and result_json.get(vt_hash_type, False):\n",
    "                    #if sample[hash_type] != vt_hash_type and result_json.get(vt_hash_type, False):\n",
    "                    sample.get(\"fileobject\").add_attribute(vt_hash_type, result_json.get(vt_hash_type), comment=module_comment, tags=attribute_tags)\n",
    "                    #sample_hashes[vt_hash_type] = result_json.get(vt_hash_type)\n",
    "                    sample[vt_hash_type] = result_json.get(vt_hash_type)\n",
    "                    misp.update_object(sample.get(\"fileobject\"))\n",
    "\n",
    "            if result_json.get(\"first_submission_date\", False):\n",
    "                print(\"  Got \\033[92mfirst submission date\\033[90m\")\n",
    "                datetime_obj = datetime.utcfromtimestamp(result_json.get(\"first_submission_date\"))\n",
    "                formatted_time = datetime_obj.strftime('%Y-%m-%d %H:%M:%S UTC')\n",
    "                samples[key][\"vt\"][\"markdown\"] = \"{}- First submission date: **{}**\\n\".format(samples[key][\"vt\"][\"markdown\"], formatted_time)\n",
    "                sample.get(\"fileobject\").add_attribute(\"text\", \"First submission date: {}\".format(formatted_time), comment=module_comment)\n",
    "            if result_json.get(\"reputation\", False):\n",
    "                print(\"  Got \\033[92mreputation\\033[90m\")\n",
    "                samples[key][\"vt\"][\"markdown\"] = \"{}- Reputation: **{}**\\n\".format(samples[key][\"vt\"][\"markdown\"], result_json.get(\"reputation\"))\n",
    "                sample.get(\"fileobject\").add_attribute(\"text\", \"Reputation: {}\".format(result_json.get(\"reputation\")), comment=module_comment)\n",
    "            if result_json.get(\"tags\", False):\n",
    "                print(\"  Got \\033[92mtags\\033[90m\")\n",
    "                samples[key][\"vt\"][\"markdown\"] = \"{}- Tags: **{}**\\n\".format(samples[key][\"vt\"][\"markdown\"], result_json.get(\"tags\"))\n",
    "            if result_json.get(\"magic\", False):\n",
    "                print(\"  Got \\033[92mfile magic\\033[90m\")\n",
    "                samples[key][\"vt\"][\"markdown\"] = \"{}- File magic: **{}**\\n\".format(samples[key][\"vt\"][\"markdown\"], result_json.get(\"magic\"))\n",
    "                sample.get(\"fileobject\").add_attribute(\"text\", \"File magic: {}\".format(result_json.get(\"magic\")), comment=module_comment)                \n",
    "            if result_json.get(\"type_extension\", False):\n",
    "                print(\"  Got \\033[92mtype extension\\033[90m\")\n",
    "                samples[key][\"vt\"][\"markdown\"] = \"{}- Type extension: **{}**\\n\".format(samples[key][\"vt\"][\"markdown\"], result_json.get(\"type_extension\"))\n",
    "                sample.get(\"fileobject\").add_attribute(\"text\", \"Type extension: {}\".format(result_json.get(\"type_extension\")), comment=module_comment)\n",
    "            misp.update_object(sample.get(\"fileobject\"))\n",
    "                \n",
    "            if len(samples[key][\"vt\"][\"popular_threat_classification\"]) > 0:\n",
    "                samples[key][\"vt\"][\"markdown\"] = \"{}\\n\\n#### Popular threat classifications\\n\".format(samples[key][\"vt\"][\"markdown\"])\n",
    "                print(\"  Got \\033[92mthreat classifications\\033[90m\")\n",
    "                for el in samples[key][\"vt\"][\"popular_threat_classification\"]:\n",
    "                    if el == \"suggested_threat_label\":\n",
    "                        samples[key][\"vt\"][\"suggested_threat_label\"] = samples[key][\"vt\"][\"popular_threat_classification\"][el]\n",
    "                    samples[key][\"vt\"][\"markdown\"] = \"{}\\n- {}: {}\".format(samples[key][\"vt\"][\"markdown\"], el, samples[key][\"vt\"][\"popular_threat_classification\"][el])\n",
    "\n",
    "            if len(samples[key][\"vt\"][\"sigma_analysis_results\"]) > 0:\n",
    "                print(\"  Got \\033[92msigma analysis results\\033[90m\")\n",
    "                samples[key][\"vt\"][\"markdown\"] = \"{}\\n\\n#### Sigma analysis results\\n\".format(samples[key][\"vt\"][\"markdown\"])\n",
    "                for el in samples[key][\"vt\"][\"sigma_analysis_results\"]:\n",
    "                    samples[key][\"vt\"][\"markdown\"] = \"{}\\n - {}\".format(samples[key][\"vt\"][\"markdown\"], el[\"rule_title\"])\n",
    "\n",
    "            if len(samples[key][\"vt\"][\"sandbox_verdicts\"]) > 0:\n",
    "                print(\"  Got \\033[92msandbox verdicts\\033[90m\")\n",
    "                samples[key][\"vt\"][\"markdown\"] = \"{}\\n\\n#### Sandbox verdicts\\n\".format(samples[key][\"vt\"][\"markdown\"])\n",
    "                for el in samples[key][\"vt\"][\"sandbox_verdicts\"]:\n",
    "                    samples[key][\"vt\"][\"sandboxes\"].append(el)\n",
    "                    samples[key][\"vt\"][\"markdown\"] = \"{}\\n - **{}**\".format(samples[key][\"vt\"][\"markdown\"], el)\n",
    "                    for el_sandbox in samples[key][\"vt\"][\"sandbox_verdicts\"][el]:\n",
    "                        samples[key][\"vt\"][\"markdown\"] = \"{}\\n   - {}: {}\".format(samples[key][\"vt\"][\"markdown\"], el_sandbox, samples[key][\"vt\"][\"sandbox_verdicts\"][el][el_sandbox])\n",
    "\n",
    "            if result_json.get(\"pe_info\", False):\n",
    "                summary_pe = \"## PEFile summary\\n\\n\"\n",
    "                print(\"  Using info from \\033[92mpeinfo\\033[90m\")\n",
    "                summary_pe += \"## {}\\n\\n\".format(key)\n",
    "\n",
    "                pe_info = result_json.get(\"pe_info\")\n",
    "                if pe_info.get(\"imphash\", False):\n",
    "                    sample[\"imphash\"] = pe_info.get(\"imphash\", False)\n",
    "                resource_language = []\n",
    "                if pe_info.get(\"resource_details\", False):\n",
    "                    for resource in pe_info.get(\"resource_details\"):\n",
    "                        if resource[\"lang\"] not in resource_language:\n",
    "                            resource_language.append(resource[\"lang\"])\n",
    "                sample[\"resource_language\"] = resource_language\n",
    "                \n",
    "                summary_pe += \" - Imphash: {}\\n\".format(pe_info.get(\"imphash\",\"\"))\n",
    "                summary_pe += \" - Resource language: {}\\n\".format(resource_language)\n",
    "                summary_pe += \"\\n\\n\"\n",
    "                if pe_info.get(\"import_list\", False):\n",
    "                    summary_pe += \"### Imports\\n\\n\"\n",
    "                    import_list = pe_info.get(\"import_list\")\n",
    "                    for import_library in import_list:\n",
    "                        summary_pe += \"**{}**\\n\".format(import_library[\"library_name\"])\n",
    "                        for import_function in import_library[\"imported_functions\"]:\n",
    "                            summary_pe += \"- {}\\n\".format(import_function)\n",
    "                        summary_pe += \"\\n\\n\"\n",
    "                samples[key][\"vt\"][\"summary_pe\"] = summary_pe\n",
    "                samples[key][\"vt\"][\"summary_pe_raw\"] = pe_info\n",
    "        print(\" Finished query VirusTotal directly\")\n",
    "    else:\n",
    "        print(\"No results for \\033[91m{}\\033[90m.\".format(value))\n",
    "    scan_count += 1\n",
    "    if scan_count > 5:\n",
    "        print(\"Sleeping for {} seconds\".format(misp_modules_wait))\n",
    "        time.sleep(misp_modules_wait)\n",
    "        scan_count = 0\n",
    "print(\"Finished {}\".format(module_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08eb4adf-491e-45b6-bc2f-d4dbd9494050",
   "metadata": {},
   "source": [
    "## ER:2 Enrich with information from Hashlookup\n",
    "\n",
    "We use [CIRCL Hashlookup](https://www.circl.lu/services/hashlookup/) to identify if any of the hashes correspond with **known hashes**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "159cf025-482f-4afb-8e73-0f3b26f684d4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Loop through the hashes\n",
    "\n",
    "module_name = \"hashlookup\"\n",
    "print(\"Lookup the file in {}\".format(module_name))\n",
    "for key, sample in samples.items():\n",
    "    attribute_type = sample[\"hash_type\"]\n",
    "    value = key\n",
    "    module_comment = \"From {} for {}\".format(module_name, key)\n",
    "    data = {\n",
    "        \"attribute\": {\n",
    "            \"type\": f\"{attribute_type}\",\n",
    "            \"uuid\": str(uuid.uuid4()),\n",
    "            \"value\": f\"{value}\",\n",
    "        },\n",
    "        \"module\": module_name,\n",
    "        \"config\": {\"custom_API\": False}\n",
    "    }\n",
    "    print(\"Query \\033[92m{}\\033[90m as \\033[92m{}\\033[90m\".format(value, attribute_type))\n",
    "    result = requests.post(\"{}/query\".format(misp_modules_url), headers=misp_modules_headers, json=data)\n",
    "    if \"results\" in result.json() and len(result.json()[\"results\"]) > 0:\n",
    "        result_json = result.json()[\"results\"]\n",
    "        hashlookup_hit = False\n",
    "        hashlookuptext = \"\"\n",
    "        for misp_attribute in result_json.get(\"Attribute\", []):\n",
    "            misp_attribute[\"comment\"] = \"{}{}\".format(module_comment, misp_attribute.get(\"comment\", \"\"))\n",
    "            created_attribute = misp.add_attribute(misp_event.uuid, misp_attribute, pythonify=True)\n",
    "            if not \"errors\" in created_attribute:\n",
    "                print(\" Got {} \\033[92m{}\\033[90m\".format(misp_attribute[\"type\"], misp_attribute[\"value\"]))\n",
    "                misp.add_object_reference(sample.get(\"fileobject\").add_reference(created_attribute.uuid, \"related-to\"))\n",
    "            else:\n",
    "                print(\" Unable to add {} \\033[92m{}\\033[90m to MISP event\".format(misp_attribute[\"type\"], misp_attribute[\"value\"]))\n",
    "        for misp_object in result_json.get(\"Object\", []):\n",
    "            misp_object[\"comment\"] = \"{}{}\".format(module_comment, misp_object.get(\"comment\", \"\"))\n",
    "            if len(misp_object[\"Attribute\"]) > 0:\n",
    "                created_object = misp.add_object(misp_event.uuid, misp_object, pythonify=True)\n",
    "                if not \"errors\" in created_object:\n",
    "\n",
    "                    print(\" Got \\033[92m{}\\033[90m \".format(misp_object[\"name\"]))\n",
    "                    misp.add_object_reference(sample.get(\"fileobject\").add_reference(created_object.uuid, \"related-to\"))\n",
    "                    for attribute in misp_object.get(\"Attribute\", []):\n",
    "                        if attribute[\"object_relation\"] not in [\"MD5\", \"SHA-1\", \"SHA-256\", \"SSDEEP\"]:\n",
    "                            hashlookuptext = \"{}: {}\\n{}\".format(attribute[\"object_relation\"], attribute[\"value\"], hashlookuptext)\n",
    "                        if attribute.get(\"object_relation\") == \"KnownMalicious\":\n",
    "                            hashlookup_hit = True\n",
    "                else:\n",
    "                    print(\" Unable to add \\033[92m{}\\033[90m to MISP event\".format(misp_object[\"name\"]))\n",
    "\n",
    "        if hashlookup_hit:\n",
    "            misp.tag(sample.get(\"hash_attribute_uuid\"), \"misp-workflow:analysis=\\\"known-file-hash\\\"\")\n",
    "\n",
    "        sample.get(\"fileobject\").add_attribute(\"text\", hashlookuptext)\n",
    "        samples[key][\"hashlookup\"][\"data\"] = result_json\n",
    "        samples[key][\"hashlookup\"][\"markdown\"] = hashlookuptext\n",
    "        misp.update_object(sample.get(\"fileobject\"))\n",
    "    else:\n",
    "        print(\"No results for \\033[91m{}\\033[90m.\".format(value))\n",
    "print(\"Finished {}\".format(module_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee02857f-42f1-4fe9-8990-2f7b0b406955",
   "metadata": {},
   "source": [
    "## ER:3 Enrich with information from MalwareBazaar\n",
    "\n",
    "Now we query [MalwareBazaar](https://bazaar.abuse.ch/). Similar as with VirusTotal, we use the MISP module to get useful attributes and objects. Afterwards we query MalwareBazaar directly to get more details such as vendor intel and tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9034f46-3340-4d2d-9a9e-415e25f69f87",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Loop through the hashes\n",
    "\n",
    "module_name = \"malwarebazaar\"\n",
    "print(\"Lookup the file in {}\".format(module_name))\n",
    "for key, sample in samples.items():\n",
    "    attribute_type = sample[\"hash_type\"]\n",
    "    value = key\n",
    "    module_comment = \"From {} for {}\".format(module_name, key)\n",
    "    data = {\n",
    "        \"attribute\": {\n",
    "            \"type\": f\"{attribute_type}\",\n",
    "            \"uuid\": str(uuid.uuid4()),\n",
    "            \"value\": f\"{value}\",\n",
    "        },\n",
    "        \"module\": module_name,\n",
    "    }\n",
    "    print(\"Query \\033[92m{}\\033[90m as \\033[92m{}\\033[90m\".format(value, attribute_type))\n",
    "    result = requests.post(\"{}/query\".format(misp_modules_url), headers=misp_modules_headers, json=data)\n",
    "    if \"results\" in result.json() and len(result.json()[\"results\"]) > 0:\n",
    "        result_json = result.json()[\"results\"]\n",
    "        for misp_attribute in result_json.get(\"Attribute\", []):\n",
    "            misp_attribute[\"comment\"] = \"{}{}\".format(module_comment, misp_attribute.get(\"comment\", \"\"))\n",
    "            created_attribute = misp.add_attribute(misp_event.uuid, misp_attribute, pythonify=True)\n",
    "            if not \"errors\" in created_attribute:\n",
    "                print(\" Got {} \\033[92m{}\\033[90m\".format(misp_attribute[\"type\"], misp_attribute[\"value\"]))\n",
    "                misp.add_object_reference(sample.get(\"fileobject\").add_reference(created_attribute.uuid, \"related-to\"))\n",
    "            else:\n",
    "                print(\" Unable to add {} \\033[92m{}\\033[90m to MISP event\".format(misp_attribute[\"type\"], misp_attribute[\"value\"]))\n",
    "        for misp_object in result_json.get(\"Object\", []):\n",
    "            misp_object[\"comment\"] = \"{}{}\".format(module_comment, misp_object.get(\"comment\", \"\"))\n",
    "            if len(misp_object[\"Attribute\"]) > 0:\n",
    "                created_object = misp.add_object(misp_event.uuid, misp_object, pythonify=True)\n",
    "                if not \"errors\" in created_object:\n",
    "                    print(\" Got \\033[92m{}\\033[90m \".format(misp_object[\"name\"]))\n",
    "                    misp.add_object_reference(sample.get(\"fileobject\").add_reference(created_object.uuid, \"related-to\"))\n",
    "                else:\n",
    "                    print(\" Unable to add \\033[92m{}\\033[90m to MISP event\".format(misp_object[\"name\"]))\n",
    "\n",
    "        #################################################################\n",
    "\n",
    "        print(\" Query MalwareBazaar directly to get file details\")\n",
    "        url = \"https://mb-api.abuse.ch/api/v1/\"\n",
    "        result = requests.post(url, data={\"query\": \"get_info\", \"hash\": value})\n",
    "        if \"data\" in result.json() and len(result.json()[\"data\"]) > 0:\n",
    "            result_json = result.json()[\"data\"][0]\n",
    "            samples[key][\"malwarebazaar\"][\"markdown\"] = \"\"\n",
    "            samples[key][\"malwarebazaar\"][\"data\"] = result_json\n",
    "            samples[key][\"malwarebazaar\"][\"tags\"] = result_json.get(\"tags\", \"\")\n",
    "            samples[key][\"malwarebazaar\"][\"signature\"] = result_json.get(\"signature\", \"\")\n",
    "\n",
    "            if len(samples[key][\"malwarebazaar\"][\"tags\"]) > 0:\n",
    "                samples[key][\"malwarebazaar\"][\"markdown\"] = \"{}\\n**Tags**\\n\".format(samples[key][\"malwarebazaar\"][\"markdown\"])\n",
    "                for tag in samples[key][\"malwarebazaar\"][\"tags\"]:\n",
    "                    samples[key][\"malwarebazaar\"][\"markdown\"] = \"{}\\n - {}\\n\".format(samples[key][\"malwarebazaar\"][\"markdown\"], tag)\n",
    "\n",
    "            if len(result_json.get(\"vendor_intel\", [])) > 0:\n",
    "                if len(result_json.get(\"vendor_intel\", []).get(\"Triage\", [])) > 0:\n",
    "                    samples[key][\"malwarebazaar\"][\"markdown\"] = \"{}\\n**Triage tags and signatures**\\n\".format(samples[key][\"malwarebazaar\"][\"markdown\"])\n",
    "                    for tag in result_json.get(\"vendor_intel\", []).get(\"Triage\", []).get(\"tags\", []):\n",
    "                        samples[key][\"malwarebazaar\"][\"markdown\"] = \"{}\\n - {}\\n\".format(samples[key][\"malwarebazaar\"][\"markdown\"], tag)\n",
    "                    for signature in result_json.get(\"vendor_intel\", []).get(\"Triage\", []).get(\"signatures\", []):\n",
    "                        samples[key][\"malwarebazaar\"][\"markdown\"] = \"{}\\n - {}\\n\".format(samples[key][\"malwarebazaar\"][\"markdown\"], signature[\"signature\"])\n",
    "        print(\" Finished query MalwareBazaar directly\")\n",
    "    else:\n",
    "        print(\"No results for \\033[91m{}\\033[90m.\".format(value))\n",
    "print(\"Finished {}\".format(module_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb658e6d-7279-4959-94a1-48b2e8b2dfc0",
   "metadata": {},
   "source": [
    "## ER:4 Investigation and enrichment report\n",
    "\n",
    "The next cell summarises the investigation and enrichment details into a **MISP report**. The report is attached to the MISP event, and also used at the end of the playbook to create the summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbc293f5-51e1-458e-8115-75eb5d6e3780",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "current_date = datetime.now()\n",
    "formatted_date = current_date.strftime(\"%Y-%m-%d\")\n",
    "summary_iv = \"## Investigation report\\n\\n\"\n",
    "for key, sample in samples.items():\n",
    "    summary_iv_key = \"\"\n",
    "    summary_iv_key += \"## Analysis for {}\\n\".format(key)\n",
    "    summary_iv_key += \"- Date: **{}**\\n\".format(formatted_date)\n",
    "    summary_iv_key += \"- MISP event: **{}** ({})\\n\".format(misp_event.info, misp_event.id)\n",
    "    summary_iv_key += \"- Hash type: **{}**\\n\".format(sample[\"hash_type\"])\n",
    "    for hash_type in file_hash_types:\n",
    "        if sample.get(hash_type, False) and hash_type != sample[\"hash_type\"]:\n",
    "            summary_iv_key += \"- Associated {}  **{}**\\n\".format(hash_type, sample.get(hash_type))\n",
    "    if sample.get(\"imphash\", False):\n",
    "        summary_iv_key += \"- Associated {}  **{}**\\n\".format(\"imphash\", sample.get(\"imphash\"))\n",
    "    if sample.get(\"resource_language\", False):\n",
    "        summary_iv_key += \"- Resource languages: **{}**\\n\".format(sample.get(\"resource_language\"))\n",
    "    summary_iv_key += \"\\n\\n\"\n",
    "    \n",
    "    if sample.get(\"detection_ratio\", False):\n",
    "        summary_iv_key += \"### Risk\\n\"        \n",
    "        summary_iv_key += \" - Detection ratio: **{}**\\n\".format(sample[\"detection_ratio\"])\n",
    "        summary_iv_key += \" - Detection ratio percentage: **{}**\\n\".format(sample[\"detection_ratio_perc\"])\n",
    "        summary_iv_key += \" - Detection ratio indirect: {}\\n\".format(sample[\"detection_ratio_indirect\"])\n",
    "        summary_iv_key += \" - Detection ratio indirect percentage: {}\\n\".format(sample[\"detection_ratio_indirect_perc\"])\n",
    "        if sample.get(\"risk_rating\", False):\n",
    "            summary_iv_key += \" - Risk rating: **{}**\\n\".format(sample[\"risk_rating\"])\n",
    "        summary_iv_key += \"\\n\\n\"\n",
    "\n",
    "    if len(sample[\"warninglist\"]) > 0:\n",
    "        summary_iv_key += sample[\"warninglist\"]\n",
    "        summary_iv_key += \"\\n\\n\"\n",
    "        \n",
    "    if sample[\"vt\"].get(\"markdown\"):        \n",
    "        summary_iv_key += \"### VirusTotal\\n\"\n",
    "        summary_iv_key += sample[\"vt\"][\"markdown\"]\n",
    "        summary_iv_key += \"\\n\\n\"\n",
    "    if sample[\"hashlookup\"].get(\"markdown\"):    \n",
    "        summary_iv_key += \"### Hashlookup\\n\"\n",
    "        summary_iv_key += sample[\"hashlookup\"][\"markdown\"]\n",
    "        summary_iv_key += \"\\n\\n\"\n",
    "    if sample[\"malwarebazaar\"].get(\"markdown\"):    \n",
    "        summary_iv_key += \"### MalwareBazaar\\n\"\n",
    "        summary_iv_key += sample[\"malwarebazaar\"][\"markdown\"]\n",
    "        summary_iv_key += \"\\n\\n\"\n",
    "    \n",
    "    event_title = \"Hash analysis - {}\".format(key)\n",
    "    print(\"Creating MISP report \\033[92m{}\\033[90m\".format(event_title))\n",
    "    chunk_size = 61500\n",
    "    for i in range(0, len(summary_iv_key), chunk_size):\n",
    "        chunk = summary_iv_key[i:i + chunk_size]\n",
    "        event_report = MISPEventReport()\n",
    "        event_title_edit = event_title\n",
    "        if i > 0:\n",
    "            event_title_edit = \"{} ({} > {})\".format(event_title, i, i + chunk_size)\n",
    "        event_report.name = event_title_edit\n",
    "        event_report.content = chunk\n",
    "        result = misp.add_event_report(misp_event.id, event_report)\n",
    "        if \"EventReport\" in result:\n",
    "            print(\" Report ID: \\033[92m{}\\033[90m\".format(result.get(\"EventReport\", []).get(\"id\", 0)))\n",
    "        else:\n",
    "            print(\"Failed to create report for \\033[91m{}\\033[90m.\".format(event_title))\n",
    "            \n",
    "    summary_iv += summary_iv_key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0f17298-3285-496e-b93b-e87535eaefcd",
   "metadata": {},
   "source": [
    "## ER:5 PEinfo report\n",
    "\n",
    "The previous enrichment with VirusTotal can also return information obtained via peinfo. During the enrichment the results are added to a Markdown blob which you can now add to a MISP event."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7d0b901-6f6a-4483-b184-1b223f0d890f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for key, sample in samples.items():\n",
    "    print(\"Is there peinfo for {}?\".format(key))\n",
    "    if sample[\"vt\"].get(\"summary_pe\") and len(sample[\"vt\"][\"summary_pe\"]) > 0:\n",
    "        print(\" Found!\")\n",
    "        event_title = \"PEFile analysis - {}\".format(key)\n",
    "        print(\" Creating MISP report \\033[92m{}\\033[90m\".format(event_title))\n",
    "        chunk_size = 61500\n",
    "        for i in range(0, len(sample[\"vt\"][\"summary_pe\"]), chunk_size):\n",
    "            chunk = sample[\"vt\"][\"summary_pe\"][i:i + chunk_size]\n",
    "            event_report = MISPEventReport()\n",
    "            event_title_edit = event_title\n",
    "            if i > 0:\n",
    "                event_title_edit = \"{} ({} > {})\".format(event_title, i, i + chunk_size)\n",
    "            event_report.name = event_title_edit\n",
    "            event_report.content = chunk\n",
    "            result = misp.add_event_report(misp_event.id, event_report)\n",
    "            if \"EventReport\" in result:\n",
    "                print(\" Report ID: \\033[92m{}\\033[90m\".format(result.get(\"EventReport\", []).get(\"id\", 0)))\n",
    "            else:\n",
    "                print(\"Failed to create report for \\033[91m{}\\033[90m.\".format(event_title))\n",
    "    else:\n",
    "        print(\" No \\033[91mpeinfo\\033[90m found\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70dea5ca",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Closure\n",
    "\n",
    "In this **closure** or end step we create a **summary** of the actions that were performed by the playbook. The summary is printed in the playbook and can also be send to a chat channel. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf86d67a-de23-4121-a26b-b8fb1c67c44e",
   "metadata": {},
   "source": [
    "## EN:1 MISP indicators\n",
    "\n",
    "The next section first **queries MISP for the indicators added to the MISP event** that is linked to the execution of this playbook.\n",
    "\n",
    "The indicators are stored in the variable `indicator_table` (table format) and `indicator_raw_list` (in raw format) which is used in a later section to create the playbook summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16d50d2c-078c-476f-acfc-bb536934f049",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get all the indicators for our event and store this is in a table. We can also use this for the summary.\n",
    "indicator_search = misp.search(\"attributes\", uuid=misp_event.uuid, to_ids=True, pythonify=True)\n",
    "indicator_raw_list = []\n",
    "indicator_table = PrettyTable()\n",
    "if len(indicator_search) > 0:\n",
    "    indicator_table.field_names = [\"Type\", \"Category\", \"Indicator\", \"Comment\"]\n",
    "    indicator_table.align[\"Type\"] = \"l\"\n",
    "    indicator_table.align[\"Category\"] = \"l\"\n",
    "    indicator_table.align[\"Indicator\"] = \"l\"\n",
    "    indicator_table.align[\"Comment\"] = \"l\"\n",
    "    indicator_table.border = True\n",
    "    for indicator in indicator_search:\n",
    "        if indicator.value not in indicator_raw_list:\n",
    "            comment = indicator.comment\n",
    "            if hasattr(indicator, 'object_relation'):\n",
    "                object_ind = misp.get_object(indicator.object_id, pythonify=True)\n",
    "                comment = \"From '{}' object {} {}\".format(object_ind.name, object_ind.comment.lower(), comment.lower())\n",
    "            indicator_table.add_row([indicator.type, indicator.category, indicator.value, comment])\n",
    "            indicator_raw_list.append(indicator.value)\n",
    "    print(\"Got \\033[92m{}\\033[90m indicator(s) from the event \\033[92m{}\\033[90m ({}).\\n\".format(len(indicator_raw_list), misp_event.info, misp_event.id))\n",
    "else:\n",
    "    print(\"\\033[93mNo indicators found in the event \\033[92m{}\\033[90m ({})\".format(misp_event.info, misp_event.id))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d26d1c3b-dec4-4191-92fe-4c130e3f9686",
   "metadata": {},
   "source": [
    "### Raw list of MISP indicators\n",
    "\n",
    "The indicators are now stored in `indicator_search` (as Python objects) and `indicator_raw_list` (in raw format, only the indicators). Execute the next cell to display them in a table format. The table is also included in the summary sent to Mattermost and TheHive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72cd3e2a-afad-4b9c-9f3a-9dbe9e8e135d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if len(indicator_raw_list) > 0:\n",
    "    print(indicator_table.get_string(sortby=\"Type\"))\n",
    "    print(\"\\n\\nIndicator list in raw format:\")\n",
    "    print(\"---------------------------------------------------\")\n",
    "    for el in indicator_raw_list:\n",
    "        print(\"{}\".format(el))\n",
    "    print(\"---------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b82e2327-1e11-4efb-8869-ccbcc896bfcc",
   "metadata": {},
   "source": [
    "## EN:2 Create the summary of the playbook\n",
    "\n",
    "The next section creates a summary and stores the output in the variable `summary` in Markdown format. It also stores an intro text in the variable `intro`. These variables are later used when sending information to Mattermost or TheHive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3edd5623-4d54-4c36-aace-7da1d8adbc68",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "summary = \"# MISP Playbook summary\\nMalware triage with MISP event: **{}** ({}/events/view/{}). \".format(misp_event.info, misp_url, misp_event.id)\n",
    "\n",
    "summary += \"\\n\"\n",
    "summary += summary_iv\n",
    "summary += \"\\n\"\n",
    "intro = summary\n",
    "\n",
    "summary += \"## Indicators\\n\\n\"\n",
    "summary += \"### Indicators table\\n\\n\"\n",
    "if len(indicator_raw_list) > 0:\n",
    "    indicator_table.set_style(MARKDOWN)\n",
    "    summary += indicator_table.get_string(sortby=\"Type\")\n",
    "    summary += \"\\n\\n\\n\"\n",
    "    summary += \"### Indicators in **raw format**\\n\\n\"\n",
    "    for indicator in indicator_raw_list:\n",
    "        summary += \"{}\\n\\n\".format(indicator)\n",
    "    summary += \"\\n\"\n",
    "else:\n",
    "    summary += \"There are no indicators\"\n",
    "summary += \"\\n\\n\"\n",
    "\n",
    "summary += \"## Correlations\\n\\n\"\n",
    "summary += \"### MISP event matches\\n\\n\"\n",
    "table_mispevents.set_style(MARKDOWN)\n",
    "summary += table_mispevents.get_string()\n",
    "summary += \"\\n\\n\"\n",
    "\n",
    "summary += \"### MISP feed matches\\n\\n\"\n",
    "table_mispfeeds.set_style(MARKDOWN)\n",
    "summary += table_mispfeeds.get_string()\n",
    "summary += \"\\n\\n\"\n",
    "\n",
    "summary += \"\\n\\n\"\n",
    "\n",
    "print(\"The \\033[92msummary\\033[90m of the playbook is available.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d603257f-0cbe-47c8-876f-9bbe52216c0d",
   "metadata": {},
   "source": [
    "## EN:3 Print the summary\n",
    "\n",
    "Apart from the full summary of the investigation, the malware triage summary provides the necessary input for the analyst. This summary was previously also added as a MISP report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12691048-4122-4ba3-aab0-203226137e3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Comment the printing of the summary to avoid cluttering the playbook by accident\n",
    "#display_markdown(summary, raw=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba96062-deaa-4fb9-8302-7b3ef829201e",
   "metadata": {},
   "source": [
    "## EN:4 Send a summary to Mattermost\n",
    "\n",
    "Now you can send the summary to Mattermost. You can send the summary in two ways by selecting one of the options for the variable `send_to_mattermost_option` in the next cell.\n",
    "\n",
    "- The default option where the entire summary is in the **chat**, or\n",
    "- a short intro and the summary in a **card**\n",
    "\n",
    "For this playbook we rely on a webhook in Mattermost. You can add a webhook by choosing the gear icon in Mattermost, then choose Integrations and then **Incoming Webhooks**. Set a channel for the webhook and lock the webhook to this channel with *\"Lock to this channel\"*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c455bc3-35cc-436d-8237-95c81b57208f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "send_to_mattermost_option = \"via a chat message\"\n",
    "#send_to_mattermost_option = \"via a chat message with card\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed2f2cdd-4eea-4125-8e52-8addc47bba5b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "message = False\n",
    "if send_to_mattermost_option == \"via a chat message\":\n",
    "    message = {\"username\": mattermost_playbook_user, \"text\": summary}\n",
    "elif send_to_mattermost_option == \"via a chat message with card\":\n",
    "    message = {\"username\": mattermost_playbook_user, \"text\": intro, \"props\": {\"card\": summary}}\n",
    "\n",
    "if message:\n",
    "    r = requests.post(mattermost_hook, data=json.dumps(message))\n",
    "    r.raise_for_status()\n",
    "if message and r.status_code == 200:\n",
    "    print(\"Summary is \\033[92msent to Mattermost.\\n\")\n",
    "else:\n",
    "    print(\"\\033[91mFailed to sent summary\\033[90m to Mattermost.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "498b8eff-7d1b-47b7-90c4-e13cc738b36c",
   "metadata": {},
   "source": [
    "## EN:5 End of the playbook "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c03bb30",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\033[92m End of the playbook\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b17af9f2",
   "metadata": {},
   "source": [
    "## External references <a name=\"extreferences\"></a>\n",
    "\n",
    "- [The MISP Project](https://www.misp-project.org/)\n",
    "- [Mattermost](https://mattermost.com/)\n",
    "- [MalwareBazaar](https://bazaar.abuse.ch/)\n",
    "- [Hashlookup](https://www.circl.lu/services/hashlookup/)\n",
    "- [VirusTotal](https://www.virustotal.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1c813f6",
   "metadata": {},
   "source": [
    "## Technical details \n",
    "\n",
    "### Documentation\n",
    "\n",
    "This playbook requires these Python **libraries** to exist in the environment where the playbook is executed. You can install them with `pip install <library>`.\n",
    "\n",
    "```\n",
    "pyfaup\n",
    "chardet\n",
    "PrettyTable\n",
    "ipywidgets\n",
    "mattermostdriver\n",
    "folium\n",
    "\n",
    "```\n",
    "\n",
    "### Colour codes\n",
    "\n",
    "The output from Python displays some text in different colours. These are the colour codes\n",
    "\n",
    "```\n",
    "Red = '\\033[91m'\n",
    "Green = '\\033[92m'\n",
    "Blue = '\\033[94m'\n",
    "Cyan = '\\033[96m'\n",
    "White = '\\033[97m'\n",
    "Yellow = '\\033[93m'\n",
    "Magenta = '\\033[95m'\n",
    "Grey = '\\033[90m'\n",
    "Black = '\\033[90m'\n",
    "Default = '\\033[99m'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14dce812-e432-4c18-8358-f8b66e5b94a5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Attachments",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}