examples/Vulture/02-vulture_operators_Doc_subs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "254cd3b1-e36b-42b5-9297-9b9d174a1011",
   "metadata": {},
   "source": [
    "# Vulture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f74105-f696-41c1-ad8e-1414f1e874b4",
   "metadata": {},
   "source": [
    "## Introduction to Text Operations with Vulture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1c0a0ad2-f847-4cdb-a35e-7ad4f298505b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "import pathlib\n",
    "import pandas as pd\n",
    "\n",
    "from TELF.pre_processing import Vulture\n",
    "from TELF.pre_processing.Vulture.modules import SubstitutionOperator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a1dc990-7de8-40e6-baca-0ecef35c6477",
   "metadata": {},
   "source": [
    "## 0. Load Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "521bce7b-4e83-4639-b784-a066c22478c6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'ad68055e-677f-11ee-95d4-4ab2673ea3f0': 'Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " 'ad680626-677f-11ee-95d4-4ab2673ea3f0': 'Highly specific datasets of scientific literature are important for both research and education. However, it is difficult to build such datasets at scale. A common approach is to build these datasets reductively by applying topic modeling on an established corpus and selecting specific topics. A more robust but time-consuming approach is to build the dataset constructively in which a subject matter expert (SME) handpicks documents. This method does not scale and is prone to error as the dataset grows. Here we showcase a new tool, based on machine learning, for constructively generating targeted datasets of scientific literature. Given a small initial “core” corpus of papers, we build a citation network of documents. At each step of the citation network, we generate text embeddings and visualize the embeddings through dimensionality reduction. Papers are kept in the dataset if they are “similar” to the core or are otherwise pruned through human-in-the-loop selection. Additional insight into the papers is gained through sub-topic modeling using SeNMFk. We demonstrate our new tool for literature review by applying it to two different fields in machine learning.',\n",
       " 'ad680658-677f-11ee-95d4-4ab2673ea3f0': 'We propose an efficient distributed out-of-memory implementation of the non-negative matrix factorization (NMF) algorithm for heterogeneous high-performance-computing systems. The proposed implementation is based on prior work on NMFk, which can perform automatic model selection and extract latent variables and patterns from data. In this work, we extend NMFk by adding support for dense and sparse matrix operation on multi-node, multi-GPU systems. The resulting algorithm is optimized for out-of-memory problems where the memory required to factorize a given matrix is greater than the available GPU memory. Memory complexity is reduced by batching/tiling strategies, and sparse and dense matrix operations are significantly accelerated with GPU cores (or tensor cores when available). Input/output latency associated with batch copies between host and device is hidden using CUDA streams to overlap data transfers and compute asynchronously, and latency associated with collective communications (both intra-node and inter-node) is reduced using optimized NVIDIA Collective Communication Library (NCCL) based communicators. Benchmark results show significant improvement, from 32X to 76x speedup, with the new implementation using GPUs over the CPU-based NMFk. Good weak scaling was demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000 GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size sparse matrix of density 10−6.',\n",
       " 'ad680680-677f-11ee-95d4-4ab2673ea3f0': 'Identification of the family to which a malware specimen belongs is essential in understanding the behavior of the malware and developing mitigation strategies. Solutions proposed by prior work, however, are often not practicable due to the lack of realistic evaluation factors. These factors include learning under class imbalance, the ability to identify new malware, and the cost of production-quality labeled data. In practice, deployed models face prominent, rare, and new malware families. At the same time, obtaining a large quantity of up-to-date labeled malware for training a model can be expensive. In this paper, we address these problems and propose a novel hierarchical semi-supervised algorithm, which we call the HNMFk Classifier, that can be used in the early stages of the malware family labeling process. Our method is based on non-negative matrix factorization with automatic model selection, that is, with an estimation of the number of clusters. With HNMFk Classifier, we exploit the hierarchical structure of the malware data together with a semi-supervised setup, which enables us to classify malware families under conditions of extreme class imbalance. Our solution can perform abstaining predictions, or rejection option, which yields promising results in the identification of novel malware families and helps with maintaining the performance of the model when a low quantity of labeled data is used. We perform bulk classification of nearly 2,900 both rare and prominent malware families, through static analysis, using nearly 388,000 samples from the EMBER-2018 corpus. In our experiments, we surpass both supervised and semi-supervised baseline models with an F1 score of 0.80.',\n",
       " 'ad6806a8-677f-11ee-95d4-4ab2673ea3f0': 'Malware is one of the most dangerous and costly cyber threats to national security and a crucial factor in modern cyber-space. However, the adoption of machine learning (ML) based solutions against malware threats has been relatively slow. Shortcomings in the existing ML approaches are likely contributing to this problem. The majority of current ML approaches ignore real-world challenges such as the detection of novel malware. In addition, proposed ML approaches are often designed either for malware/benign-ware classification or malware family classification. Here we introduce and showcase preliminary capabilities of a new method that can perform precise identification of novel malware families, while also unifying the capability for malware/benign-ware classification and malware family classification into a single framework.',\n",
       " 'ad6806d0-677f-11ee-95d4-4ab2673ea3f0': 'Malware is one of the most dangerous and costly cyber threats to organizations, the public, and national security, and a crucial factor in modern warfare. The adoption of ML-based solutions against malware threats has been relatively slow despite the potential cost savings. The majority of prior malware defense solutions based on ML do not sufficiently address the following real-world challenges; considering the cost associated with labeled malware, use of supervised solutions that poorly generalize to new malware, training and testing models under class imbalance where both rare and prominent malware are included, and incorporating the ability to identify new/novel malware families. Cybersecurity analysts regularly go through large quantities of malware samples to understand if a new specimen belongs to a previously known malware family. Classifying a new malware sample into a known family reduces the number of files analysts need to examine, and aids in understanding the behavior of the malware, which is helpful for estimating the severity of the threat, developing mitigation strategies, and reducing cost and time spend on malware analysis. We have developed a new ML method, named Malware-DNA, for malware family classification, characterization, and identification that achieves state-of-the-art results and addresses major shortcomings in the field. Malware-DNA considers malware analogous to the genomic DNA, while exploring the hidden hierarchical structure of malware data without prior knowledge, using the ideas of our SmartTensors AI Platform, 2021 R&D100 winner in the IT category and recognized with an R&D100 2021 Market Disruptor Bronze Medal, enabling the discovery of multi-structure composition of malware, and separating mixed latent features. This hierarchical exploration is done based on semi-supervised and unsupervised ML techniques, yielding better generalization to new malware data. Under the DNA analogy, our innovation takes an approach that follows ideas of our recent ML methods in human cancer, the mutations to the genome can cause various inherited diseases such as certain cancers. Similarly, this project treats malware as malicious mutations (e.g., cancer) in the software genome (i.e. the computer code), and targets extraction and recognition of these new mutational malware signatures. The hierarchical approach is a tensor factorization technique that incorporates automatic model determination, combined with the ability to perform abstaining predictions (selective classification, predict “we do not know what this is”), allow Malware-DNA to identify and classify both rare and prominent malwares, work under label imbalance, maintain accuracy even when a low-quantity of labeled data is used, and detect novel malware (i.e. malware without labels or the software that we did not see before). Our method first builds an archive of latent multi-modal signatures (ALMAS) whose combinations describe and characterize complex data. This archive is then used for rapid/real-time characterization and classification of new data and detect unknown or novel phenomena. In our preliminary studies, we created a catalog of malware from multiple families and benign specimens with static analysis features from the EMBER-2018 dataset6. We select one malware class to be the novel family. We showcase the performance of our approach by classifying the malware families and benign-ware, detecting the specimens belonging to the novel malware family, and report our results with Area Under the Curve of Risk-Coverage (AURC) score.',\n",
       " 'ad6806f8-677f-11ee-95d4-4ab2673ea3f0': 'Topic modeling is one of the key analytic techniques for organizing and analysis of large text corpora. One approach to topic modeling is the recently introduced SeNMFk, a method based on semantic non-negative matrix factorization (NMF) with automatic model determination (NMFk), where the text-document matrix and word-context (co-occurrence) matrix are jointly factorized. The text-document matrix is the term frequency-inverse document frequency (TF-IDF) matrix, and the word-context matrix represents the number of times two words co-occur in a pre-determined window of text. Incorporating the semantic structure of the text with the ability to estimate the number of topics enables a coherent separation of the latent topics and accurate document clustering. This approach, however, only identifies the highest level of topics or the main topics/themes. Many text corpora often include a very complex structure of sub-topics beyond the main themes. For example, a set of documents can be separated into main topics, such as, sports, politics, science, etc. Each of these main topics can be further separated into sub-topics. For example, sport theme can be separated to the subtopics tennis, soccer, football, etc. This process can be repeated by expanding the separation until finding all the sub-topics in the corpus. Here, we introduce a hierarchical SeNMFk approach, that can extract fine-grained sub-topics and their semantic sub-structures. By hierarchically applying SeNMFk, we break down the main topics and extract previously unknown sub-topics as well as the corresponding sub-semantic structures that can serve as narrow vocabularies – scientific-jargon seeds for local Name Entities Recognition (NER). We demonstrate our hierarchical SeNMFk method by performing topic modeling on all papers posted in arXiv, which is ~2 million+ papers. To enhance the semantic clustering in each topic, we also jointly factorize the category-text matrix, values of which represents the TF-IDF of tokens per document category. Here the categories are pre-determined/reported by the authors of the document based on its field of research in arXiv. Our results show the ability and practicality of our hierarchical SeNMFk to extract meaningful topics and find their semantic sub-structures from large datasets.',\n",
       " 'ad680716-677f-11ee-95d4-4ab2673ea3f0': 'Non-negative matrix factorization (NMF) with missing-value completion is a well-known effective Collaborative Filtering (CF) method used to provide personalized user recommendations. However, traditional CF relies on a privacy-invasive collection of user data to build a central recommender model. One-shot federated learning has recently emerged as a method to mitigate the privacy problem while addressing the traditional communication bottleneck of federated learning. In this paper, we present the first one-shot federated CF implementation, named One-FedCF, for groups of users or collaborating organizations. In our solution, the clients first apply local CF in-parallel to build distinct, client-specific recommenders. Then, the privacy-preserving local item patterns and biases from each client are shared with the processor to perform joint factorization in order to extract the global item patterns. Extracted patterns are then aggregated to each client to build the local models via information retrieval transfer. In our experiments, we demonstrate our approach with two MovieLens datasets and show results competitive with the state-of-the-art federated recommender systems at a substantial decrease in the number of communications.',\n",
       " 'ad68073e-677f-11ee-95d4-4ab2673ea3f0': 'We propose an efficient, distributed, out-of-memory implementation of the truncated singular value decomposition (t-SVD) for heterogeneous (CPU+GPU) high performance computing (HPC) systems. Various implementations of SVD have been proposed, but most only estimate the singular values as an estimation of the singular vectors which can significantly increase the time and memory complexity of the algorithm. In this work, we propose an implementation of SVD based on the power method, which is a truncated singular values and singular vectors estimation method. Memory utilization bottlenecks seen in the power method are typically associated with the computation of the Gram matrix , which can be significant when  is large and dense, or when  is super-large and sparse. The proposed implementation is optimized for out-of-memory problems where the memory required to factorize a given matrix is greater than the available GPU memory. We reduce the memory complexity of  by using a batching strategy where the intermediate factors are computed block by block. We also suppress I/O latency associated with both host-to-device (H2D) and device-to-host (D2H) batch copies by overlapping each batch copy with compute using CUDA streams. Furthermore, we use optimized  based communicators to reduce the latency associated with collective communications (both intra-node and inter-node). In addition, sparse and dense matrix multiplications are significantly accelerated with GPU cores (or tensors cores when available), resulting in an implementation with good scaling. We demonstrate the scalability of our distributed out of core SVD algorithm to successfully decompose dense matrix of size 1TB and sparse matrix of size 128PB with 1e-6 sparsity.'}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DATA_DIR = os.path.join('..', '..', 'data')\n",
    "DATA_DIR = pathlib.Path(DATA_DIR).resolve()\n",
    "DATA_FILE = 'documents.p'\n",
    "\n",
    "documents = pickle.load(open(os.path.join(DATA_DIR, DATA_FILE), 'rb'))\n",
    "print(len(documents))\n",
    "documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "387ef96c-492f-4eab-a72e-bcdb3417efa8",
   "metadata": {},
   "source": [
    "### Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "21aaaf2c-8c07-4126-a68e-aeebb4d01b59",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "RESULTS_DIR = 'results'\n",
    "RESULTS_DIR = pathlib.Path(RESULTS_DIR).resolve()\n",
    "RESULTS_FILE = 'operated_documents'\n",
    "\n",
    "try:\n",
    "    os.mkdir(RESULTS_DIR)\n",
    "except FileExistsError:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "302365f7-5aa8-4e8f-befa-a2e94eaf0f70",
   "metadata": {},
   "source": [
    "### Setup Vulture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ea8b8813-e8c7-42bc-9aa0-538187487841",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "vulture = Vulture(n_jobs  = 1, \n",
    "                  verbose = 10,  # Disable == 0, Verbose >= 1\n",
    "                 )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f936d14-4fd1-4f43-bac7-0fc95348c72e",
   "metadata": {},
   "source": [
    "### Apply Substitutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we do not pass the ```save_path```, it will return a list of results where each entry in the list is for the given operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cf331b3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "document_substitutions = {\n",
    "    \"ad68055e-677f-11ee-95d4-4ab2673ea3f0\":{\"Supervisory Control and Data Acquisition (SCADA)\": \"supervisory_control_and_data_acquisition\", \"SCADA\":\"supervisory_control_and_data_acquisition\"},\n",
    "    \"ad680626-677f-11ee-95d4-4ab2673ea3f0\":{'Highly specific datasets of scientific literature': \"dense_domain_specific_datasets\"},\n",
    "    \"ad680658-677f-11ee-95d4-4ab2673ea3f0\":{},\n",
    "    \"ad680680-677f-11ee-95d4-4ab2673ea3f0\":{\"malware\":'software_designed_to_harm'},\n",
    "    \"ad6806a8-677f-11ee-95d4-4ab2673ea3f0\":{\"Malware\":'malicious_software'},\n",
    "    \"ad6806d0-677f-11ee-95d4-4ab2673ea3f0\":{},\n",
    "    \"ad6806f8-677f-11ee-95d4-4ab2673ea3f0\":{\"Topic modeling\": 'matrix_decomposition'},\n",
    "    \"ad680716-677f-11ee-95d4-4ab2673ea3f0\":{\"NMF\":'matrix_factorization_where_no_values_are_negative'},\n",
    "    \"ad68073e-677f-11ee-95d4-4ab2673ea3f0\":{\"We propose\": \"we show\"},\n",
    "}\n",
    "\n",
    "corpus_substitutions = {\n",
    "    'malware': 'malicious_software',\n",
    "    'NMFk': 'nonnegative_matrix_factorization_k',\n",
    "    'NMF': 'nonnegative_matrix_factorization'\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|          | 0/1 [00:00<?, ?it/s][Vulture]: Running SubstitutionOperator module\n",
      "100%|██████████| 9/9 [00:00<00:00, 7332.70it/s]\n",
      "100%|██████████| 1/1 [00:00<00:00, 344.93it/s]\n"
     ]
    }
   ],
   "source": [
    "vulture.operate(    documents, \n",
    "                    steps=[ SubstitutionOperator(   document_substitutions = document_substitutions,\n",
    "                                                    corpus_substitutions = corpus_substitutions,\n",
    "                                                    document_priority = True )], \n",
    "                    save_path=RESULTS_DIR, file_name=RESULTS_FILE)                                   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each entry is a tuple where index 0 is the name of the operation and index 1 is the results of the operation in dictionary format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c8f39bf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "operation_results = pickle.load(open(os.path.join(RESULTS_DIR, 'operated_documents_SubstitutionOperator.p'), 'rb'))\n",
    "operation_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operation for each document is given in dictionary format where key is the document ID and its value is the operation results, \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "69859fe2-48e5-4a8e-8b8d-ccc64abd51b7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def to_df(documents, operated_documents):\n",
    "    data = {\n",
    "        'id': [],\n",
    "        'text': [],\n",
    "        'replaced_text': []\n",
    "    }\n",
    "\n",
    "    for i, text in documents.items():\n",
    "        data['id'].append(i)\n",
    "        data['text'].append(text)\n",
    "        operation_current_doc =  operated_documents.get(i)\n",
    "        data['replaced_text'].append(operation_current_doc['replaced_text'])\n",
    "\n",
    "    return pd.DataFrame.from_dict(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "087aa962-1a60-40c3-8de2-be90df8a975c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>replaced_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ad68055e-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Supervisory Control and Data Acquisition (SCAD...</td>\n",
       "      <td>Supervisory Control and Data Acquisition (supe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ad680626-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Highly specific datasets of scientific literat...</td>\n",
       "      <td>dense_domain_specific_datasets are important f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ad680658-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>We propose an efficient distributed out-of-mem...</td>\n",
       "      <td>We propose an efficient distributed out-of-mem...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ad680680-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Identification of the family to which a malwar...</td>\n",
       "      <td>Identification of the family to which a softwa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ad6806a8-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Malware is one of the most dangerous and costl...</td>\n",
       "      <td>malicious_software is one of the most dangerou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ad6806d0-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Malware is one of the most dangerous and costl...</td>\n",
       "      <td>Malware is one of the most dangerous and costl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ad6806f8-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Topic modeling is one of the key analytic tech...</td>\n",
       "      <td>matrix_decomposition is one of the key analyti...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ad680716-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Non-negative matrix factorization (NMF) with m...</td>\n",
       "      <td>Non-negative matrix factorization (matrix_fact...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ad68073e-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>We propose an efficient, distributed, out-of-m...</td>\n",
       "      <td>we show an efficient, distributed, out-of-memo...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  \\\n",
       "0  ad68055e-677f-11ee-95d4-4ab2673ea3f0   \n",
       "1  ad680626-677f-11ee-95d4-4ab2673ea3f0   \n",
       "2  ad680658-677f-11ee-95d4-4ab2673ea3f0   \n",
       "3  ad680680-677f-11ee-95d4-4ab2673ea3f0   \n",
       "4  ad6806a8-677f-11ee-95d4-4ab2673ea3f0   \n",
       "5  ad6806d0-677f-11ee-95d4-4ab2673ea3f0   \n",
       "6  ad6806f8-677f-11ee-95d4-4ab2673ea3f0   \n",
       "7  ad680716-677f-11ee-95d4-4ab2673ea3f0   \n",
       "8  ad68073e-677f-11ee-95d4-4ab2673ea3f0   \n",
       "\n",
       "                                                text  \\\n",
       "0  Supervisory Control and Data Acquisition (SCAD...   \n",
       "1  Highly specific datasets of scientific literat...   \n",
       "2  We propose an efficient distributed out-of-mem...   \n",
       "3  Identification of the family to which a malwar...   \n",
       "4  Malware is one of the most dangerous and costl...   \n",
       "5  Malware is one of the most dangerous and costl...   \n",
       "6  Topic modeling is one of the key analytic tech...   \n",
       "7  Non-negative matrix factorization (NMF) with m...   \n",
       "8  We propose an efficient, distributed, out-of-m...   \n",
       "\n",
       "                                       replaced_text  \n",
       "0  Supervisory Control and Data Acquisition (supe...  \n",
       "1  dense_domain_specific_datasets are important f...  \n",
       "2  We propose an efficient distributed out-of-mem...  \n",
       "3  Identification of the family to which a softwa...  \n",
       "4  malicious_software is one of the most dangerou...  \n",
       "5  Malware is one of the most dangerous and costl...  \n",
       "6  matrix_decomposition is one of the key analyti...  \n",
       "7  Non-negative matrix factorization (matrix_fact...  \n",
       "8  we show an efficient, distributed, out-of-memo...  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = to_df(documents, operation_results)\n",
    "df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "TELF_public",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}