jupyterthon2024.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison of collaboration methods between MSTICpy and Splunk SIEM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"<iframe width=\"950\" height=\"550\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/TitlePicture.jpg\"></iframe>\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('\"<iframe width=\"950\" height=\"550\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/TitlePicture.jpg\"></iframe>\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WHOAMI\n",
    "\n",
    "### Tatsuya Hasegawa\n",
    "\n",
    "- **Threat Hunter**, **Application Developer**\n",
    "- **Self-employed** \n",
    "- - mainly, contracting with *\"GoAhead Inc.\"* which is a Japanese small company. (Log analysis and security consulting company using Splunk)\n",
    "- **MSTICpy lover & contributor**\n",
    "- - especially, regarding to Splunk Provider. (driver & uploader)\n",
    "- Infosec Carrer\n",
    "- - University master degree major in AntiVirus\n",
    "- - - MSSP in NTT                                    (3y)\n",
    "- - - - CSIRT in domestic recruit company            (3y)\n",
    "- - - - - Security Researcher in Cylance/BlackBerry  (3y)\n",
    "- - - - - - Security Consultant in GoAhead           (3y)\n",
    "- SNS\n",
    "- - X: @T_8ase\n",
    "- - LinkedIn: tatsuya-hasegawa-aa3279142\n",
    "- **Speaker of APAC SANS DFIR Summit 2023**\n",
    "- - [Practical msticpy use ~ rainbow bridge to SIEM for advanced threat hunting ~](https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/Tatsuya_Hasegawa_msticpy_SANS_APAC_DFIR_SUMMIT_2023.pdf)\n",
    "\n",
    "### MSTICpy is very cool tool for data analysis in terms of threat hunting!\n",
    "### I introduced the reason I use MSTICpy and Jupyter for advanced threat hunting on my DFIR Summit slides."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "1. Splunk Query Provider & Uploader (4 min)\n",
    "    \n",
    "    How MSTICpy's Splunk Query Provider work?\n",
    "\n",
    "2. Splunk App for DSDL (5 min)\n",
    "    \n",
    "    How Splunk DSDL work?\n",
    "    \n",
    "    How can we inject MSTICpy to DSDL frame work?\n",
    "\n",
    "3. Let's compare the both about advantages and disadvantages with Matrix! (5 min)\n",
    "\n",
    "    In fact, case by case.\n",
    "\n",
    "4. Conclusion (1min)\n",
    "\n",
    "    \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Splunk Query Provider & Uploader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.10.0\n"
     ]
    }
   ],
   "source": [
    "import msticpy as mp\n",
    "print(mp.__version__)\n",
    "mp.init_notebook()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connected.\n",
      "Connected.\n"
     ]
    }
   ],
   "source": [
    "# Splunk Query Provider\n",
    "\n",
    "splunk_prov = mp.QueryProvider(\"Splunk\")\n",
    "splunk_prov.connect() # from msticpyconfig.yaml\n",
    "## splunk_prov.connect(host=\"HOST\",port=\"PORT\",username=\"USERNAME\",password=\"PASSWORD\")\n",
    "\n",
    "# Auth Token is better for security to access to Splunk\n",
    "splunk_token_prov = mp.QueryProvider(\"Splunk\")\n",
    "splunk_token_prov.connect() # from msticpyconfig.yaml\n",
    "## splunk_token_prov.connect(host=\"HOST\",port=\"PORT\",bearer_token=\"JWT\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inside Splunk Driver\n",
    "\n",
    "`Splunk Driver manipulates Splunk Python SDK to connect to Splunk by REST API.`\n",
    "\n",
    "- /msticpy/msticpy/data/drivers/splunk_drivers.py\n",
    "\n",
    "```python\n",
    "try:\n",
    "    import splunklib.client as sp_client\n",
    "    import splunklib.results as sp_results\n",
    "    from splunklib.client import AuthenticationError, HTTPError\n",
    "except ImportError as imp_err:\n",
    "    raise MsticpyImportExtraError(\n",
    "        \"Cannot use this feature without splunk-sdk installed\",\n",
    "        title=\"Error importing splunk-sdk\",\n",
    "        extra=\"splunk\",\n",
    "    ) from imp_err\n",
    "    \n",
    "~~\n",
    "        # Different required parameters for the REST API authentication method\n",
    "        # between user/pass and authorization bearer token\n",
    "        if \"username\" in cs_dict:\n",
    "            self._required_params = [\"host\", \"username\", \"password\"]\n",
    "        else:\n",
    "            self._required_params = [\"host\", \"bearer_token\"]\n",
    "\n",
    "~~\n",
    "        # Replace to Splunk python sdk's parameter name of sp_client.connect()\n",
    "        if arg_dict.get(\"bearer_token\"):\n",
    "            arg_dict[\"splunkToken\"] = arg_dict.pop(\"bearer_token\")\n",
    "\n",
    "        try:\n",
    "            self.service = sp_client.connect(**arg_dict)\n",
    "        except AuthenticationError as err:            \n",
    "```\n",
    "\n",
    ">Note. \n",
    "It means user authentification has the priority than token auth if we set the both username and bearer_token in the argument options or in the msticpyconfig.yaml.\n",
    "\n",
    "\n",
    "### Basically, SplunkUploader is using the same trick!\n",
    "\n",
    "It has an dedicated Class outside Query Provider like below, but SplunkUploader also utilizes \"splunk_driver\" for connection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connected.\n"
     ]
    }
   ],
   "source": [
    "# Splunk Uploader\n",
    "\n",
    "from msticpy.data.uploaders.splunk_uploader import SplunkUploader\n",
    "splunk_uploader = SplunkUploader() # from msticpyconfig.yaml\n",
    "## splunk_uploader = SplunkUploader(host=\"HOST\", port=\"PORT\", username=\"USERNAME\",  password=\"PASSWORD\")\n",
    "## splunk_uploader = SplunkUploader(host=\"localhost\", bearer_token=\"JWT\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The expire check of JWT auth token has not implemented in msticpy yet.\n",
    "\n",
    "It just emit query failure getting from splunk server after the auth token expired."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Splunk App for DSDL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Splunk App for Data Science and Deep Learning](https://splunkbase.splunk.com/app/4607) (DSDL) is a Splunk official extension for Deep Leaning. \n",
    "\n",
    "- It usually requires external machine resources such as Docker containers outside of Splunk application.\n",
    "- Data exchange is fundamentally between the machine containers and Splunk by using endpoint URLs.\n",
    "- Of course, Jupyther Notebook is supported on the machine containers.\n",
    "- Thus we can install msticpy in the container side, but manually.\n",
    "- msticpy's Query Provider and Uploader are **not** necessary on DSDL platform, the data transfer has done by DSDL architecture.\n",
    "\n",
    "\n",
    "\n",
    "At first, choose the container deployment that meets your needs:\n",
    "- `single-instance` deployment with Docker or Kubernetes and the Splunk platform running on the same instance.\n",
    "- `side-by-side` deployment where the Splunk platform instance communicates with another instance that serves as the Docker or Kubernetes host.\n",
    "    \n",
    "\n",
    "In fact, there are also two method for data transfer in DSDL.\n",
    "1. pull & push in container: `Almost same of msticpy's Query Provider and Uploader`\n",
    "    - Downloader utilizes Splunk REST API with a valid Splunk auth token which is defined in DSDL app.\n",
    "    - Uploader utilizes Splunk's HTTP Event Collector (HEC).\n",
    "2. listen & return container: `Different from msticpy!`\n",
    "    - fit & apply model\n",
    "    - **I will explain more about this method as followings.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"<iframe width=\"950\" height=\"550\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/SplunkDSDL.jpg\"></iframe>\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML('\"<iframe width=\"950\" height=\"550\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/SplunkDSDL.jpg\"></iframe>\"')\n",
    "## DSDL Architechture https://docs.splunk.com/Documentation/DSDL/5.1.1/User/Architecture\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inside DSDL's MLTKContainer\n",
    "\n",
    "`MLTKContainer utilizes endpoint URLs with api auth token for communication.`\n",
    "- /mltk-container/bin/mltkc/MLTKContainer.py \n",
    "\n",
    "```python\n",
    "    # ---------------------------------------------------------------------------\n",
    "    # helper function to call container endpoint\n",
    "    # communicate over basic urllib GET or POST with container endpoint\n",
    "    # TODO #1: gzip compression of payload\n",
    "    # Arguments : (self for convenience)\n",
    "    # - url     : request endpoint url, usually self.endpoint_url = 'http://localhost:5000'\n",
    "    # - data    : JSON structure send via POST to the container\n",
    "    # - content_type (for future use in case of compression)\n",
    "    def endpoint(self, url, data=None, content_type='application/json'):\n",
    "        # we assume a GET call\n",
    "        method = \"GET\"\n",
    "        # if we have data let's switch to POST call\n",
    "        if data:\n",
    "            method = \"POST\"\n",
    "        # check for logging enabled and log stuff\n",
    "        if _MLTKContainer_logging:\n",
    "            debug_message = method+' endpoint ['+url+'] called '\n",
    "            if data:\n",
    "                debug_message += 'with payload ('+str(len(data))+' bytes)'\n",
    "            _MLTKContainer_logger.info(debug_message)\n",
    "        # let's start pessimistic and override in case of success\n",
    "        returns = 'ERROR on ' + method + ' request to endpoint ['+url+']'\n",
    "        try:\n",
    "            # get endpoint cert and create ssl context with option for incluster traffic\n",
    "            settings = self._get_config(os.path.join(os.path.dirname(__file__),\"..\", \"..\", \"local\", \"docker.conf\"))['connection']\n",
    "            api_token = settings.get('api_token')\n",
    "            header = {\n",
    "                'Authorization': api_token,\n",
    "            }\n",
    "            # do the POST\n",
    "            if data:                \n",
    "                #data_encoded = urllib_parse.urlencode(data).encode('utf-8')\n",
    "                data_encoded = str.encode(data)\n",
    "                header['Content-Type'] = content_type\n",
    "                request = urllib_request.Request(\n",
    "                    url, data_encoded, header)\n",
    "            # or the GET\n",
    "            else:\n",
    "                request = urllib_request.Request(url, None, header)\n",
    "~~~\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Case example and sample code \n",
    "\n",
    "This is a sample code for powershell command line analysis which is developed on the Jupyter Notebook in DSDL's machine container.\n",
    "https://github.com/Tatsuya-hasegawa/MSTICPy_utils/blob/main/splunk_dsdl/msticpy_powershell_ioc.ipynb\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"<iframe width=\"950\" height=\"550\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/DSDLex1.jpg\"></iframe>\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML('\"<iframe width=\"950\" height=\"550\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/DSDLex1.jpg\"></iframe>\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In other words, I injected \"msticpy library\" to DSDL \"apply\" mechanism.\n",
    "\n",
    "`Splunk's fit command is needed only for first once, after that, we can use apply the msticpy functions over Splunk without any touch to Jupyter.`\n",
    "\n",
    "Of course, we can change the codes in the apply function anytime, however we need fit command again to overwrite the .py file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"<iframe width=\"950\" height=\"550\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/MLContainerPy.jpg\"></iframe>\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML('\"<iframe width=\"950\" height=\"550\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/MLContainerPy.jpg\"></iframe>\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"<iframe width=\"1390\" height=\"780\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/DSDLex2.jpg\"></iframe>\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML('\"<iframe width=\"1390\" height=\"780\" src=\"https://github.com/Tatsuya-hasegawa/InfosecJupyterthon2024/blob/main/images/DSDLex2.jpg\"></iframe>\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Let's compare the both about advantages and disadvantages with Matrix!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| | msticpy's QueryProvider＆Uploader | Splunk DSDL fit&apply |\n",
    "|:---- |:----:|:----:|\n",
    "| Direction |  Jupyter -> Splunk | Splunk -> Jupyter |\n",
    "| Action Trigger | MSTICpy code snippet| Splunk SPL |\n",
    "| Secure Channel | By yourself | By DSDL |\n",
    "| Credential Management |  Manage Splunk credential| Manage Jupyter credential |\n",
    "| Visualization |  On Jupyter (On Splunk only when Uploader is used)| On Splunk |\n",
    "| Operation | Individual | Team |\n",
    "| Restriction | No, except for Jupyter machine resource | Yes, Splunk could be a shared resouce |\n",
    "| Advantage |  Support long running process and easy for code modification & debug | Easy to set automatic&repetitive analysis by Splunk scheduled search |\n",
    "| Disadvantage| Data security concerns on Jupyter | Should write more exceptional codes for error handling on Splunk|\n",
    "| Suitable for data scientist | <font color=green>Y | <font color=red>N |\n",
    "| Suitable for security analyst/operator | <font color=red>N | <font color=green>Y |\n",
    "| Suitable for threat hunter | <font color=green>Y | <font color=green>Y | "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Conclusion\n",
    "\n",
    "My conclusion is very simple!!!\n",
    "\n",
    "```\n",
    "For operational purpose, DSDL is easy to build msticpy's practical use platform.\n",
    "\n",
    "For manual deep or rapid analysis, msticpy's QueryProvider & Uploader can be more useful.\n",
    "```\n",
    "\n",
    "On the other hand, smart threat hunters utilize manual batch and continuous auto analysis.\n",
    "\n",
    "That is why msticpy users who connect to Splunk may had better know both of these methods.\n",
    "\n",
    "I'm happy if this matrix is to a help for them.\n",
    "\n",
    "---\n",
    "> Comment for Microsoft Sentinel\n",
    "\n",
    "Probably, we'll have no worry about the choice for Microsoft Sentinel, because it has the highly compatible method by using \"Microsoft Sentinel ML Notebooks\". \n",
    "\n",
    "That uses msticpy's QueryProvider & Uploader of the left side of above matrix, however it mitigates the disadvantage because all the components are inside of secure Azure architecture. Awesome!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thank you for giving this talk opportunity!!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}