examples/Vulture/01-single_node_parallel_advanced.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "062cb680",
   "metadata": {},
   "source": [
    "# Vulture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62be1d40",
   "metadata": {},
   "source": [
    "## Advanced Use Cases of Vulture Text Pre-Processing "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0303ae62",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "import pathlib\n",
    "import pandas as pd\n",
    "\n",
    "from TELF.pre_processing import Vulture\n",
    "\n",
    "from TELF.pre_processing.Vulture.modules import SimpleCleaner\n",
    "from TELF.pre_processing.Vulture.modules import LemmatizeCleaner\n",
    "from TELF.pre_processing.Vulture.modules import SubstitutionCleaner\n",
    "from TELF.pre_processing.Vulture.modules import RemoveNonEnglishCleaner\n",
    "\n",
    "from TELF.pre_processing.Vulture.default_stop_words import STOP_WORDS\n",
    "from TELF.pre_processing.Vulture.default_stop_phrases import STOP_PHRASES"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c49317a5",
   "metadata": {},
   "source": [
    "## 0. Load Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f85d3b47",
   "metadata": {},
   "source": [
    "### Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b472c74a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DATA_DIR = os.path.join('..', '..', 'data')\n",
    "DATA_DIR = pathlib.Path(DATA_DIR).resolve()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1bfebc48",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DATA_FILE = 'documents.p'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c50bc36a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents = pickle.load(open(os.path.join(DATA_DIR, DATA_FILE), 'rb'))\n",
    "len(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45d99b26",
   "metadata": {},
   "source": [
    "### Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f4797ce0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "RESULTS_DIR = 'results'\n",
    "RESULTS_DIR = pathlib.Path(RESULTS_DIR).resolve()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dd163b48",
   "metadata": {},
   "outputs": [],
   "source": [
    "RESULTS_FILE = 'clean_documents.p'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d383cd0b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    os.mkdir(RESULTS_DIR)\n",
    "except FileExistsError:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "016d2c18",
   "metadata": {},
   "source": [
    "### Examine Data Format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "20abd4d9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ad68055e-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad680626-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad680658-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad680680-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad6806a8-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad6806d0-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad6806f8-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad680716-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad68073e-677f-11ee-95d4-4ab2673ea3f0']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# key serve as document unique ids\n",
    "list(documents.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "240fa3db",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# values are the text that needs to be cleaned\n",
    "documents[next(iter(documents))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac15a68b",
   "metadata": {},
   "source": [
    "## 1. Creating a Custom Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cabd4e75",
   "metadata": {},
   "source": [
    "The cleaning modules can be arranged in a custom order specified by the user. They will be executed in the order that they appear. The settings of each component can be modified by changing the settings of the associated object in the list. The current supported modules are:\n",
    "\n",
    "    - SimpleCleaner\n",
    "    - RemoveNonEnglishCleaner\n",
    "    - SubstitutionCleaner\n",
    "    - LemmatizeCleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0e86bfa",
   "metadata": {},
   "source": [
    "Here is an example of a custom pipeline. This pipeline is used to first remove non english text, apply simple cleaning, and then apply lemmatization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "83c199f6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "vulture = Vulture(n_jobs  = 1, \n",
    "                  verbose = 10,  # Disable == 0, Verbose >= 1\n",
    "                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4569721f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "steps = [\n",
    "    RemoveNonEnglishCleaner(ascii_ratio=0.9, stopwords_ratio=0.25),\n",
    "    SimpleCleaner(stop_words = STOP_WORDS,\n",
    "                  stop_phrases = STOP_PHRASES,\n",
    "                  order = [\n",
    "                      'standardize_hyphens',\n",
    "                      'isolate_frozen',\n",
    "                      'remove_copyright_statement',\n",
    "                      'remove_stop_phrases',\n",
    "                      'make_lower_case',\n",
    "                      'remove_formulas',\n",
    "                      'normalize',\n",
    "                      'remove_next_line',\n",
    "                      'remove_email',\n",
    "                      'remove_()',\n",
    "                      'remove_[]',\n",
    "                      'remove_special_characters',\n",
    "                      'remove_nonASCII_boundary',\n",
    "                      'remove_nonASCII',\n",
    "                      'remove_tags',\n",
    "                      'remove_stop_words',\n",
    "                      'remove_standalone_numbers',\n",
    "                      'remove_extra_whitespace',\n",
    "                      'min_characters',\n",
    "                  ]\n",
    "                 ),\n",
    "    LemmatizeCleaner('spacy'),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0cb1caec",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/3 [00:00<?, ?it/s][Vulture]: Running RemoveNonEnglishCleaner module\n",
      "\n",
      "100%|███████████████████████████████████████████| 9/9 [00:00<00:00, 1226.21it/s]\u001b[A\n",
      "[Vulture]: Running SimpleCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 155.22it/s]\u001b[A\n",
      "[Vulture]: Running LemmatizeCleaner module\n",
      "\n",
      "  0%|                                                     | 0/9 [00:00<?, ?it/s]\u001b[A\n",
      " 11%|█████                                        | 1/9 [00:02<00:20,  2.50s/it]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 9/9 [00:02<00:00,  3.40it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 3/3 [00:02<00:00,  1.10it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.84 s, sys: 724 ms, total: 2.56 s\n",
      "Wall time: 2.75 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fbb2279d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'supervisory control acquisition scada serve nervous substation power grid facilitate monitor acquisition control equipment ensure smooth efficient operation substation connect device dependence scada grow risk potential malicious intrusion outage permanent damage grid dimensionality approach principal analysis pca accurate identification anomaly scada scada matrix factorization nmf strong detect anomaly wireless sensor network unsupervise approach normal expect behavior detect unseen type attack anomaly identify event deviate expected behavior approach complex multi-dimensional interaction naturally scada differently tensor decomposition powerful unsupervised machine learn complex multi-faceted activity detail scada event novelly tensor decomposition canonical polyadic alternate poisson regression probabilistic framework anomaly cyber network identify anomaly scada showcase statistical behavior analysis scada communication tensor decomposition improve specificity accuracy identify anomaly electrical grid experiment scada electrical grid operate alamos national laboratory lanl transmission distribution service partnership alamos county detect synthetically generate anomaly')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6caf4c29",
   "metadata": {},
   "source": [
    "## 2. Customizing Components"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bb98321",
   "metadata": {
    "tags": []
   },
   "source": [
    "### SimpleCleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a4fd07c",
   "metadata": {},
   "source": [
    "#### Setting Order of Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f00640c",
   "metadata": {},
   "source": [
    "For the default patterns, the ```order``` parameter can be used to specify the order of operations within the SimpleCleaner component. The following examples shows how 5 simple operations can be applied in sequential order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c3bf7bad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "steps = [\n",
    "    SimpleCleaner(stop_words = STOP_WORDS,\n",
    "                  stop_phrases = STOP_PHRASES,\n",
    "                  order = [\n",
    "                      'make_lower_case',\n",
    "                      'remove_()',\n",
    "                      'remove_stop_words',\n",
    "                      'remove_extra_whitespace',\n",
    "                      'min_characters',\n",
    "                  ]\n",
    "                 ),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "948b3663",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/1 [00:00<?, ?it/s][Vulture]: Running SimpleCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 231.92it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 23.03it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 45.3 ms, sys: 3.52 ms, total: 48.9 ms\n",
      "Wall time: 46.5 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a990f628",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'supervisory control acquisition scada serve nervous substations power grids facilitate monitoring acquisition control equipment ensure smooth efficient operation substation connected devices dependence scada grows risk potential malicious intrusions outages permanent damage grid dimensionality approaches principal analysis pca accurate identification anomalies scada scada matrix factorization nmf strong detecting anomalies wireless sensor networks unsupervised approaches normal expected behavior detect unseen types attacks anomalies identifying events deviate expected behavior approaches complex multi-dimensional interactions naturally scada differently tensor decomposition powerful unsupervised machine learning complex multi-faceted activity details scada events novelly tensor decomposition canonical polyadic alternating poisson regression probabilistic framework anomaly cyber network identify anomalies scada showcase statistical behavior analysis scada communication tensor decomposition improves specificity accuracy identifying anomalies electrical grid experiments scada electrical grid operated alamos national laboratory lanl transmission distribution service partnership alamos county detect synthetically generated anomalies')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18a44f89",
   "metadata": {},
   "source": [
    "#### Setting Min Number of Characters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c27f38bf",
   "metadata": {},
   "source": [
    "The number of minimum characters to keep per token is a class attribute. This example shows how it can be changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "749ea3bb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "steps = [\n",
    "    SimpleCleaner(min_characters = 12,\n",
    "                  order = ['min_characters']\n",
    "                 ),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "35f565c1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/1 [00:00<?, ?it/s][Vulture]: Running SimpleCleaner module\n",
      "\n",
      "100%|███████████████████████████████████████████| 9/9 [00:00<00:00, 9220.50it/s]\u001b[A\n",
      "100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 222.85it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.58 ms, sys: 3.44 ms, total: 10 ms\n",
      "Wall time: 7.56 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1123aea6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'acquisition, dimensionality reduction-based identification specifically non-negative factorization unsupervised multi-dimensional interactions Differently, non-negative decomposition unsupervised multi-faceted decomposition probabilistic state-of-the-art communication decomposition experiments, transmission distribution synthetically')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f833958",
   "metadata": {},
   "source": [
    "#### Custom RegEx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "587c907b",
   "metadata": {},
   "source": [
    "The SimpleCleaner component allows the user to specify custom regular expressions patterns. This enables Vulture to be applied to specific datasets while maintaining the parallel and distributed capabilities. In the following example, we will define a simple regular expression to detect and remove acronyms in the text. An acronym will be assumed to be a series of one or more letters enclosed in parentheses. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "faf0c6d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "custom_patterns = {\n",
    "    'remove_acronyms': (\n",
    "        r\"\\s*\\([A-Z]+\\)\\s*\",  # pattern to detect uppercase letters enclosed by parentheses\n",
    "        ' *** '               # the string to replace matches with. \n",
    "                              # the stars are used for the example to make the pattern stand out. \n",
    "                              # to fully remove the acronyms, an empty string can be used\n",
    "    )\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fc64e654",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "steps = [\n",
    "    SimpleCleaner(order = ['remove_acronyms'],  # only perform the custom operation\n",
    "                  custom_patterns = custom_patterns\n",
    "                 ),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ff664f87",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/1 [00:00<?, ?it/s][Vulture]: Running SimpleCleaner module\n",
      "\n",
      "100%|██████████████████████████████████████████| 9/9 [00:00<00:00, 13116.31it/s]\u001b[A\n",
      "100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 234.74it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.17 ms, sys: 3.13 ms, total: 9.3 ms\n",
      "Wall time: 7.25 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c7e16064",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'Supervisory Control and Data Acquisition *** systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis *** , can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization *** has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning *** method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory *** which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed0a1833",
   "metadata": {
    "tags": []
   },
   "source": [
    "### RemoveNonEnglishCleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "620364e3",
   "metadata": {},
   "source": [
    "This component is used to detect and remove non-english text from the dataset. The detection works through a simple heuristics. This heuristic uses two components: the ratio of ASCII characters to all characters and the ratio of stopwords to all words. While not flawless, this heuristic (with appropriate hyperparameters) provides a quick way to filter out foreign language documents.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee1f2877",
   "metadata": {},
   "source": [
    "##### Define a placeholder Spanish document for Non-English detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b11b3679",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This placeholder text tells a short story about a young girl named Ana who lives in a small \n",
    "# town surrounded by mountains and finds an old book filled with stories of adventure and magic.\n",
    "foreign_text = 'En un pequeño pueblo rodeado de montañas, vivía una joven llamada Ana. Todos los días, ' \\\n",
    "               'ella caminaba por los senderos del bosque, disfrutando de la tranquilidad y la belleza de ' \\\n",
    "               'la naturaleza. Un día, mientras exploraba una parte del bosque que no conocía, Ana encontró ' \\\n",
    "               'un antiguo libro cubierto de polvo y hojas. Curiosa, lo abrió y comenzó a leer. Las páginas ' \\\n",
    "               'contenían historias de aventuras, magia y lugares lejanos. Ana se sintió inspirada por estas ' \\\n",
    "               'historias y decidió que un día ella también viviría sus propias aventuras.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e413db1b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "foreign_documents = {0: foreign_text}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0ffe0fc2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "steps = [\n",
    "    RemoveNonEnglishCleaner(ascii_ratio=0.9, stopwords_ratio=0.25),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9bb1ceb9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 1 documents\n",
      "  0%|                                                     | 0/1 [00:00<?, ?it/s][Vulture]: Running RemoveNonEnglishCleaner module\n",
      "\n",
      "100%|███████████████████████████████████████████| 1/1 [00:00<00:00, 3819.95it/s]\u001b[A\n",
      "100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 232.72it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.14 ms, sys: 3.33 ms, total: 9.47 ms\n",
      "Wall time: 7.31 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(foreign_documents, steps=steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "60ee2573",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('En un pequeño pueblo rodeado de montañas, vivía una joven llamada Ana. Todos los días, ella caminaba por los senderos del bosque, disfrutando de la tranquilidad y la belleza de la naturaleza. Un día, mientras exploraba una parte del bosque que no conocía, Ana encontró un antiguo libro cubierto de polvo y hojas. Curiosa, lo abrió y comenzó a leer. Las páginas contenían historias de aventuras, magia y lugares lejanos. Ana se sintió inspirada por estas historias y decidió que un día ella también viviría sus propias aventuras.',\n",
       " '')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(foreign_documents))\n",
    "foreign_documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a5126b6",
   "metadata": {},
   "source": [
    "### SubstitutitionCleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3b5e912",
   "metadata": {},
   "source": [
    "Substitutions allow Vulture to find specific substrings in the text and replaces them with some other substring. The intent of this component is to provide a Subject Matter Expert (SME) the ability to  normalize important terms in the vocabulary. Multiple terms can be used to represent the same concept. Alternatively, multiple tokens can represent a single term, the meaning of which will be destroyed through tokenization. Substitution can normalize these terms and also signal to Vulture that these terms do not require cleaning. This means that this component can also be used to \"freeze\" terms through a reflective substitution, leaving them unchanged throughout the cleaning process."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13df072b",
   "metadata": {},
   "source": [
    "Substitution can be added as argument to the clean function. This will perform substitution before and after cleaning the text. However, the SubstitutionCleaner can be used like any of the other components as part of the pipeline. This finer-grained control allows the user to specify when substitution occurs and how the substitution map is handled. The SubstitutionCleaner has three attributes that control how the substitution map is processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "6ad74634",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "substitution_map = {\n",
    "    'Foo Bar Bat': 'foo_bar',\n",
    "    'testing': 'test'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecc20124",
   "metadata": {},
   "source": [
    "#### Permute"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7b1334",
   "metadata": {},
   "source": [
    "This attribute permutes the substitution map such that whitespace is replaced with hyphens and vice versa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "a7761407",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Foo Bar Bat': 'foo_bar',\n",
       " 'Foo-Bar Bat': 'foo_bar',\n",
       " 'Foo Bar-Bat': 'foo_bar',\n",
       " 'testing': 'test',\n",
       " 'Foo-Bar-Bat': 'foo_bar'}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sub = SubstitutionCleaner(substitution_map, permute=True)\n",
    "sub.substitution_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82da2c18",
   "metadata": {},
   "source": [
    "#### Lower "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e27fa680",
   "metadata": {},
   "source": [
    "This attribute converts the keys of the substitution map to lowercase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "1bb07950",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Foo Bar Bat': 'foo_bar',\n",
       " 'foo bar bat': 'foo_bar',\n",
       " 'Foo-Bar Bat': 'foo_bar',\n",
       " 'Foo Bar-Bat': 'foo_bar',\n",
       " 'foo-bar bat': 'foo_bar',\n",
       " 'foo bar-bat': 'foo_bar',\n",
       " 'testing': 'test',\n",
       " 'Foo-Bar-Bat': 'foo_bar',\n",
       " 'foo-bar-bat': 'foo_bar'}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sub = SubstitutionCleaner(substitution_map, lower=True)\n",
    "sub.substitution_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccd98d67",
   "metadata": {},
   "source": [
    "#### Lemmatize "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72d132c5",
   "metadata": {},
   "source": [
    "This attribute lemmatizes the keys of the substitution map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6a0382c1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Foo Bar Bat': 'foo_bar',\n",
       " 'foo bar bat': 'foo_bar',\n",
       " 'Foo-Bar Bat': 'foo_bar',\n",
       " 'Foo Bar-Bat': 'foo_bar',\n",
       " 'foo-bar bat': 'foo_bar',\n",
       " 'foo bar-bat': 'foo_bar',\n",
       " 'testing': 'test',\n",
       " 'Foo-Bar-Bat': 'foo_bar',\n",
       " 'foo-bar-bat': 'foo_bar',\n",
       " 'test': 'test',\n",
       " 'Foo-Bar-bat': 'foo_bar'}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sub = SubstitutionCleaner(substitution_map, lemmatize=True)\n",
    "sub.substitution_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8749c9c",
   "metadata": {},
   "source": [
    "#### All Together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "72f023d5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Foo Bar Bat': 'foo_bar',\n",
       " 'foo bar bat': 'foo_bar',\n",
       " 'Foo Bar bat': 'foo_bar',\n",
       " 'Foo-Bar Bat': 'foo_bar',\n",
       " 'Foo Bar-Bat': 'foo_bar',\n",
       " 'foo-bar bat': 'foo_bar',\n",
       " 'foo bar-bat': 'foo_bar',\n",
       " 'Foo-Bar bat': 'foo_bar',\n",
       " 'Foo Bar-bat': 'foo_bar',\n",
       " 'testing': 'test',\n",
       " 'Foo-Bar-Bat': 'foo_bar',\n",
       " 'foo-bar-bat': 'foo_bar',\n",
       " 'test': 'test',\n",
       " 'Foo-Bar-bat': 'foo_bar'}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sub = SubstitutionCleaner(substitution_map, permute=True, lower=True, lemmatize=True)\n",
    "sub.substitution_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e77cfd1",
   "metadata": {},
   "source": [
    "### LemmatizeCleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68ea2a0d",
   "metadata": {},
   "source": [
    "#### Apply NLTK Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "71700daf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "steps = [\n",
    "    LemmatizeCleaner('nltk')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "d9cb3096",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/1 [00:00<?, ?it/s][Vulture]: Running LemmatizeCleaner module\n",
      "\n",
      "  0%|                                                     | 0/9 [00:00<?, ?it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 9/9 [00:01<00:00,  5.66it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 1/1 [00:01<00:00,  1.59s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.36 s, sys: 227 ms, total: 1.59 s\n",
      "Wall time: 1.6 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "cbc2f689",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'Supervisory Control and Data Acquisition (SCADA) system often serve a the nervous system for substation within power grids. These system facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and it connected devices. As the dependence on these SCADA system grows, so doe the risk of potential malicious intrusion that could lead to significant outage or even permanent damage to the grid. Previous work ha shown that dimensionality reduction-based approaches, such a Principal Component Analysis (PCA), can be used for accurate identification of anomaly in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) ha shown strong result at detecting anomaly in wireless sensor networks. These unsupervised approach model the normal or expected behavior and detect the unseen type of attack or anomaly by identifying the event that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interaction that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity detail of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which ha previously shown state-of-the-art anomaly detection result on cyber network data, to identify anomaly in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomaly in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.')"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "044ffa5d",
   "metadata": {},
   "source": [
    "#### Apply Spacy Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "9f195897",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "steps = [\n",
    "    LemmatizeCleaner('spacy')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c828e07d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/1 [00:00<?, ?it/s][Vulture]: Running LemmatizeCleaner module\n",
      "\n",
      "  0%|                                                     | 0/9 [00:00<?, ?it/s]\u001b[A\n",
      " 11%|█████                                        | 1/9 [00:01<00:11,  1.44s/it]\u001b[A\n",
      " 56%|█████████████████████████                    | 5/9 [00:01<00:00,  4.21it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 9/9 [00:01<00:00,  5.30it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 1/1 [00:01<00:00,  1.70s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.3 s, sys: 404 ms, total: 1.71 s\n",
      "Wall time: 1.71 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "c267037c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'Supervisory Control and Data Acquisition ( SCADA ) system often serve as the nervous system for substation within power grid . these system facilitate real-time monitoring , data acquisition , control of equipment , and ensure smooth and efficient operation of the substation and its connected device . as the dependence on these SCADA system grow , so do the risk of potential malicious intrusion that could lead to significant outage or even permanent damage to the grid . previous work have show that dimensionality reduction-base approach , such as Principal Component Analysis ( PCA ) , can be use for accurate identification of anomaly in SCADA system . while not specifically apply to SCADA , non-negative matrix factorization ( NMF ) have show strong result at detect anomaly in wireless sensor network . these unsupervised approach model the normal or expect behavior and detect the unseen type of attack or anomaly by identify the event that deviate from the expected behavior . these approach ; however , do not model the complex and multi-dimensional interaction that be naturally present in SCADA system . differently , non-negative tensor decomposition be a powerful unsupervised machine learn ( ML ) method that can model the complex and multi-faceted activity detail of SCADA event . in this work , we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson regression ( CP-APR ) with a probabilistic framework , which have previously show state-of-the-art anomaly detection result on cyber network datum , to identify anomaly in SCADA system . we showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improve the specificity and accuracy of identify anomaly in electrical grid system . in our experiment , we model real-world SCADA system datum collect from the electrical grid operate by Los Alamos National Laboratory ( LANL ) which provide transmission and distribution service through a partnership with Los Alamos County , and detect synthetically generate anomaly .')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "208318ca",
   "metadata": {},
   "source": [
    "## 3. Creating a New Component"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94ec3784",
   "metadata": {},
   "source": [
    "For more complex text cleaning or specific datasets, it may be necessary to create a new Vulture cleaning component. This is possible by creating a new child class derived from ```VultureModuleBase```. The class needs to implement a ```__call__``` method. The following examples shows an implementation of a Vulture component class for stemming text.\n",
    "\n",
    "### **WARNING:**\n",
    "Each component class contains a ```frozen``` attribute, inherited from ```VultureModuleBase```. This attribute is a set of tokens that have been substituted and need to be preserved throughout the cleaning. A new component implementation must have measures in place to leave tokens seen in ````frozen``` unchanged.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35063565",
   "metadata": {},
   "source": [
    "### Define StemCleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9cbd790",
   "metadata": {
    "tags": []
   },
   "source": [
    "This example implements a new component that will take the stems of the tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "d1d8c22f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "\n",
    "from TELF.pre_processing.Vulture.modules import VultureModuleBase\n",
    "\n",
    "\n",
    "class StemCleaner(VultureModuleBase):\n",
    "    \"\"\"\n",
    "    A text cleaner that normalizes tokens through stemming.\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, frozen=None):\n",
    "        super().__init__(frozen)\n",
    "        self.backend = SnowballStemmer(\"english\")\n",
    "        \n",
    "    def __call__(self, document):\n",
    "        return self.run(document)\n",
    "        \n",
    "        \n",
    "    def run(self, document):\n",
    "        \"\"\"\n",
    "        Run the stemming\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        document: tuple\n",
    "            A document id, document text pair for which to perform stemming\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        tuple\n",
    "            Tuple of document id and cleaned text\n",
    "        \"\"\"\n",
    "        doc_id, doc_text = document\n",
    "        doc_text = self._stem(doc_text)\n",
    "        return (doc_id, doc_text)\n",
    "    \n",
    "    \n",
    "    def _stem(self, text):\n",
    "        \"\"\"\n",
    "        Stem a given string\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        text: str\n",
    "            A string to be stemmed\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        str\n",
    "            The stemmed string\n",
    "        \"\"\"\n",
    "        stemmed_tokens = []\n",
    "        for t in text.split():\n",
    "            \n",
    "            # only stem tokens that are not in frozen\n",
    "            stemmed_tokens.append(self.backend.stem(t) if t not in self.frozen else t)\n",
    "        \n",
    "        # return joined string\n",
    "        return ' '.join(stemmed_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea5dc421",
   "metadata": {},
   "source": [
    "### Apply StemCleaner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b08ad6ae",
   "metadata": {},
   "source": [
    "#### Only Use Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "29039765",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "steps = [\n",
    "    StemCleaner(),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "23e7a182",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/1 [00:00<?, ?it/s][Vulture]: Running StemCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 318.47it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 31.23it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 33.6 ms, sys: 3.48 ms, total: 37.1 ms\n",
      "Wall time: 34.9 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "a9fb8ae3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'supervisori control and data acquisit (scada) system often serv as the nervous system for substat within power grids. these system facilit real-tim monitoring, data acquisition, control of equipment, and ensur smooth and effici oper of the substat and it connect devices. as the depend on these scada system grows, so doe the risk of potenti malici intrus that could lead to signific outag or even perman damag to the grid. previous work has shown that dimension reduction-bas approaches, such as princip compon analysi (pca), can be use for accur identif of anomali in scada systems. while not specif appli to scada, non-neg matrix factor (nmf) has shown strong result at detect anomali in wireless sensor networks. these unsupervis approach model the normal or expect behavior and detect the unseen type of attack or anomali by identifi the event that deviat from the expect behavior. these approaches; however, do not model the complex and multi-dimension interact that are natur present in scada systems. differently, non-neg tensor decomposit is a power unsupervis machin learn (ml) method that can model the complex and multi-facet activ detail of scada events. in this work, we novelli appli the tensor decomposit method canon polyad altern poisson regress (cp-apr) with a probabilist framework, which has previous shown state-of-the-art anomali detect result on cyber network data, to identifi anomali in scada systems. we showcas that the use of statist behavior analysi of scada communic with tensor decomposit improv the specif and accuraci of identifi anomali in electr grid systems. in our experiments, we model real-world scada system data collect from the electr grid oper by los alamo nation laboratori (lanl) which provid transmiss and distribut servic through a partnership with los alamo county, and detect synthet generat anomalies.')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00c8158d",
   "metadata": {},
   "source": [
    "#### Stemming & Substitution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "e216d128",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "substitutions = {\n",
    "    'los alamos national laboratory': 'LANL',\n",
    "    'lanl': 'LANL',\n",
    "    'los alamos': 'Los_Alamos',\n",
    "    'supervisory control and data acquisition': 'SCADA',\n",
    "    'scada': 'SCADA',\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "f12d4640",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/3 [00:00<?, ?it/s][Vulture]: Running SubstitutionCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 514.44it/s]\u001b[A\n",
      "[Vulture]: Running StemCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 322.70it/s]\u001b[A\n",
      "[Vulture]: Running SubstitutionCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 727.15it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 44.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.75 s, sys: 946 ms, total: 3.7 s\n",
      "Wall time: 3.72 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps, substitutions=substitutions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "aa58707a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'SCADA (SCADA) system often serv as the nervous system for substat within power grids. these system facilit real-tim monitoring, data acquisition, control of equipment, and ensur smooth and effici oper of the substat and it connect devices. as the depend on these SCADA system grows, so doe the risk of potenti malici intrus that could lead to signific outag or even perman damag to the grid. previous work has shown that dimension reduction-bas approaches, such as princip compon analysi (pca), can be use for accur identif of anomali in SCADA systems. while not specif appli to SCADA, non-neg matrix factor (nmf) has shown strong result at detect anomali in wireless sensor networks. these unsupervis approach model the normal or expect behavior and detect the unseen type of attack or anomali by identifi the event that deviat from the expect behavior. these approaches; however, do not model the complex and multi-dimension interact that are natur present in SCADA systems. differently, non-neg tensor decomposit is a power unsupervis machin learn (ml) method that can model the complex and multi-facet activ detail of SCADA events. in this work, we novelli appli the tensor decomposit method canon polyad altern poisson regress (cp-apr) with a probabilist framework, which has previous shown state-of-the-art anomali detect result on cyber network data, to identifi anomali in SCADA systems. we showcas that the use of statist behavior analysi of SCADA communic with tensor decomposit improv the specif and accuraci of identifi anomali in electr grid systems. in our experiments, we model real-world SCADA system data collect from the electr grid oper by LANL (LANL) which provid transmiss and distribut servic through a partnership with Los_Alamos county, and detect synthet generat anomalies.')"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db6f0984",
   "metadata": {},
   "source": [
    "#### Stemming & Substitution & Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "f2b46765",
   "metadata": {},
   "outputs": [],
   "source": [
    "steps = [\n",
    "    SimpleCleaner(stop_words = STOP_WORDS,\n",
    "                  stop_phrases = STOP_PHRASES,\n",
    "                  order = [\n",
    "                      'standardize_hyphens',\n",
    "                      'isolate_frozen',\n",
    "                      'remove_copyright_statement',\n",
    "                      'remove_stop_phrases',\n",
    "                      'make_lower_case',\n",
    "                      'remove_formulas',\n",
    "                      'normalize',\n",
    "                      'remove_next_line',\n",
    "                      'remove_email',\n",
    "                      'remove_()',\n",
    "                      'remove_[]',\n",
    "                      'remove_special_characters',\n",
    "                      'remove_nonASCII_boundary',\n",
    "                      'remove_nonASCII',\n",
    "                      'remove_tags',\n",
    "                      'remove_stop_words',\n",
    "                      'remove_standalone_numbers',\n",
    "                      'remove_extra_whitespace',\n",
    "                      'min_characters',\n",
    "                  ]\n",
    "                 ),\n",
    "    StemCleaner(),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "aa806db4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|                                                     | 0/4 [00:00<?, ?it/s][Vulture]: Running SubstitutionCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 774.51it/s]\u001b[A\n",
      "[Vulture]: Running SimpleCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 136.76it/s]\u001b[A\n",
      "[Vulture]: Running StemCleaner module\n",
      "\n",
      "100%|████████████████████████████████████████████| 9/9 [00:00<00:00, 635.37it/s]\u001b[A\n",
      " 75%|█████████████████████████████████▊           | 3/4 [00:00<00:00, 29.64it/s][Vulture]: Running SubstitutionCleaner module\n",
      "\n",
      "100%|███████████████████████████████████████████| 9/9 [00:00<00:00, 1282.40it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 4/4 [00:00<00:00, 35.63it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.57 s, sys: 873 ms, total: 3.45 s\n",
      "Wall time: 3.54 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time cleaned_documents = vulture.clean(documents, steps=steps, substitutions=substitutions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "858fdf6c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'SCADA SCADA serv nervous substat power grid facilit monitor acquisit control equip ensur smooth effici oper substat connect devic depend SCADA grow risk potenti malici intrus outag perman damag grid dimension approach princip analysi pca accur identif anomali SCADA SCADA matrix factor nmf strong detect anomali wireless sensor network unsupervis approach normal expect behavior detect unseen type attack anomali identifi event deviat expect behavior approach complex multi-dimension interact natur SCADA differ tensor decomposit power unsupervis machin learn complex multi-facet activ detail SCADA event novelli tensor decomposit canon polyad altern poisson regress probabilist framework anomali cyber network identifi anomali SCADA showcas statist behavior analysi SCADA communic tensor decomposit improv specif accuraci identifi anomali electr grid experi SCADA electr grid oper LANL LANL transmiss distribut servic partnership Los_Alamos counti detect synthet generat anomali')"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key = next(iter(documents))\n",
    "documents[key], cleaned_documents[key]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "TELF",
   "language": "python",
   "name": "telf"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}