examples/different_llm_models_scientific_articles.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"..\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, iText2KG is compatible with all language models supported by LangChain. \n",
    "\n",
    "To use iText2KG, you will need both a chat model and an embeddings model. \n",
    "\n",
    "For available chat models, refer to the options listed at: https://python.langchain.com/v0.2/docs/integrations/chat/. \n",
    "For embedding models, explore the choices at: https://python.langchain.com/v0.2/docs/integrations/text_embedding/. \n",
    "\n",
    "This notebook will show you how to run iText2KG using Mistral, Ollama, and OpenAI models. \n",
    "\n",
    "**Please ensure that you install the necessary package for each chat model before use.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mistral"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Mistral, please set up your model using the tutorial here: https://python.langchain.com/v0.2/docs/integrations/chat/mistralai/. Similarly, for the embedding model, follow the setup guide here: https://python.langchain.com/v0.2/docs/integrations/text_embedding/mistralai/ ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\yassir.lairgi\\Documents\\Projects\\iText2KG\\venv-test\\Lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "c:\\Users\\yassir.lairgi\\Documents\\Projects\\iText2KG\\venv-test\\Lib\\site-packages\\langchain_mistralai\\embeddings.py:169: UserWarning: Could not download mistral tokenizer from Huggingface for calculating batch sizes. Set a Huggingface token via the HF_TOKEN environment variable to download the real tokenizer. Falling back to a dummy tokenizer that uses `len()`.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from langchain_mistralai import ChatMistralAI\n",
    "from langchain_mistralai import MistralAIEmbeddings\n",
    "\n",
    "mistral_api_key = \"##\"\n",
    "mistral_llm_model = ChatMistralAI(\n",
    "    api_key = mistral_api_key,\n",
    "    model=\"mistral-large-latest\",\n",
    "    temperature=0,\n",
    "    max_retries=2,\n",
    ")\n",
    "\n",
    "\n",
    "mistral_embeddings_model = MistralAIEmbeddings(\n",
    "    model=\"mistral-embed\",\n",
    "    api_key = mistral_api_key\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same applies for OpenAI. \n",
    "\n",
    "please setup your model using the tutorial : https://python.langchain.com/v0.2/docs/integrations/chat/openai/\n",
    "The same for embedding model : https://python.langchain.com/v0.2/docs/integrations/text_embedding/openai/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
    "\n",
    "openai_api_key = \"##\"\n",
    "\n",
    "openai_llm_model = llm = ChatOpenAI(\n",
    "    api_key = openai_api_key,\n",
    "    model=\"gpt-4o\",\n",
    "    temperature=0,\n",
    "    max_tokens=None,\n",
    "    timeout=None,\n",
    "    max_retries=2,\n",
    ")\n",
    "\n",
    "openai_embeddings_model = OpenAIEmbeddings(\n",
    "    api_key = openai_api_key ,\n",
    "    model=\"text-embedding-3-large\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ollama"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same applies for Ollama. \n",
    "\n",
    "please setup your model using the tutorial : https://python.langchain.com/v0.2/docs/integrations/chat/ollama/\n",
    "The same for embedding model : https://python.langchain.com/v0.2/docs/integrations/text_embedding/openai/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_ollama import ChatOllama, OllamaEmbeddings\n",
    "\n",
    "llm = ChatOllama(\n",
    "    model=\"llama3\",\n",
    "    temperature=0,\n",
    ")\n",
    "\n",
    "embeddings = OllamaEmbeddings(\n",
    "    model=\"llama3\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# iText2KG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Use case: we aim to connect two scientific papers. \n",
    "\n",
    "* The objective is to detect common key concepts between the two papers and allowing for the identification of central themes, keywords, and topics that dominate each paper. These themes could be linked to show overlaps or gaps in coverage, helping researchers identify areas where more study might be needed or where novel connections could be made."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document Distiller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scientific articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import PyPDFLoader\n",
    "from itext2kg.documents_distiller import DocumentsDistiller, Article\n",
    "from pydantic import BaseModel, Field\n",
    "from typing import List, Tuple\n",
    "\n",
    "\n",
    "class ArticleResults(BaseModel):\n",
    "    abstract:str = Field(description=\"Brief summary of the article's abstract\")\n",
    "    key_findings:str = Field(description=\"The key findings of the article\")\n",
    "    limitation_of_sota : str=Field(description=\"limitation of the existing work\")\n",
    "    proposed_solution : str = Field(description=\"the proposed solution in details\")\n",
    "    paper_limitations : str=Field(description=\"The limitations of the proposed solution of the paper\")\n",
    "\n",
    "# Sample input data as a list of triplets\n",
    "# It is structured in this manner : (document's path, page_numbers_to_exclude, blueprint, document_type)\n",
    "documents_information = [\n",
    "    (\"../datasets/scientific_articles/llm-tikg.pdf\", [11,10], ArticleResults, 'scientific article'),\n",
    "    (\"../datasets/scientific_articles/actionable-cyber-threat.pdf\", [12,11,10], ArticleResults, 'scientific article')\n",
    "]\n",
    "\n",
    "def upload_and_distill(documents_information: List[Tuple[str, List[int], BaseModel]]):\n",
    "    distilled_docs = []\n",
    "    \n",
    "    for path_, exclude_pages, blueprint, document_type in documents_information:\n",
    "        \n",
    "        loader = PyPDFLoader(path_)\n",
    "        pages = loader.load_and_split()\n",
    "        pages = [page for page in pages if page.metadata[\"page\"]+1 not in exclude_pages] # Exclude some pages (unecessary pages, for example, the references)\n",
    "        document_distiller = DocumentsDistiller(llm_model=openai_llm_model)\n",
    "        \n",
    "        IE_query = f'''\n",
    "        # DIRECTIVES : \n",
    "        - Act like an experienced information extractor.\n",
    "        - You have a chunk of a {document_type}\n",
    "        - If you do not find the right information, keep its place empty.\n",
    "        '''\n",
    "        \n",
    "        # Distill document content with query\n",
    "        distilled_doc = document_distiller.distill(\n",
    "            documents=[page.page_content.replace(\"{\", '[').replace(\"}\", \"]\") for page in pages],\n",
    "            IE_query=IE_query,\n",
    "            output_data_structure=blueprint\n",
    "        )\n",
    "        \n",
    "        # Filter and format distilled document results\n",
    "        distilled_docs.append([\n",
    "            f\"{document_type}'s {key} - {value}\".replace(\"{\", \"[\").replace(\"}\", \"]\") \n",
    "            for key, value in distilled_doc.items() \n",
    "            if value and value != []\n",
    "        ])\n",
    "    \n",
    "    return distilled_docs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilled_docs = upload_and_distill(documents_information=documents_information)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## iText2KG for graph construction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itext2kg import iText2KG\n",
    "\n",
    "\n",
    "itext2kg = iText2KG(llm_model = openai_llm_model, embeddings_model = openai_embeddings_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We construct the first knowledge graph of the first distilled documents (for the first article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] ------- Extracting Entities from the Document 1\n",
      "{'entities': [{'name': 'Open-source threat intelligence', 'label': 'Data Source'}, {'name': 'Knowledge graph', 'label': 'Data Structure'}, {'name': 'Intrusion detection', 'label': 'Application'}, {'name': 'LLM-TIKG', 'label': 'Methodology'}, {'name': 'Large language model', 'label': 'Technology'}, {'name': 'Few-shot learning', 'label': 'Technique'}, {'name': 'Data annotation', 'label': 'Process'}, {'name': 'Data augmentation', 'label': 'Process'}, {'name': 'Topic classification', 'label': 'Task'}, {'name': 'Entity extraction', 'label': 'Task'}, {'name': 'Relationship extraction', 'label': 'Task'}, {'name': 'TTP extraction', 'label': 'Task'}, {'name': 'GPT-3.5', 'label': 'Model'}, {'name': 'Llama2-7B', 'label': 'Model'}, {'name': 'Instruction-based Information Extraction', 'label': 'Methodology'}, {'name': 'Threat hunting', 'label': 'Application'}, {'name': 'Attack attribution', 'label': 'Application'}]}\n",
      "[Entity(name=instruction based information extraction, label=Methodology, properties=embeddings=array([ 0.00578922,  0.01256758, -0.01632442, ..., -0.01279191,\n",
      "       -0.0065698 ,  0.00066387])), Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.02329106,  0.01237501, -0.01368894, ..., -0.00108116,\n",
      "       -0.01063947,  0.0007295 ])), Entity(name=threat hunting, label=Application, properties=embeddings=array([-0.02765322, -0.03706892, -0.01849421, ..., -0.01293665,\n",
      "        0.00242633,  0.02040382])), Entity(name=data annotation, label=Process, properties=embeddings=array([-0.01023736, -0.02583196, -0.01102544, ..., -0.00943764,\n",
      "       -0.00671153,  0.00328806])), Entity(name=ttp extraction, label=Task, properties=embeddings=array([-0.04061868, -0.00652669, -0.01061077, ...,  0.00112746,\n",
      "       -0.01573218,  0.02327023])), Entity(name=attack attribution, label=Application, properties=embeddings=array([-0.01950506, -0.0041517 , -0.01346924, ..., -0.00290357,\n",
      "       -0.00359692,  0.00190606])), Entity(name=topic classification, label=Task, properties=embeddings=array([-0.00586902,  0.01346419, -0.01709163, ..., -0.00839443,\n",
      "       -0.00442814,  0.02397172])), Entity(name=llm tikg, label=Methodology, properties=embeddings=array([ 0.00613677, -0.01793396, -0.0202904 , ...,  0.00590273,\n",
      "       -0.00363614,  0.00752743])), Entity(name=relationship extraction, label=Task, properties=embeddings=array([ 0.00177076,  0.00329305, -0.00977207, ...,  0.00350275,\n",
      "       -0.02029999,  0.00142598])), Entity(name=entity extraction, label=Task, properties=embeddings=array([-0.00535627, -0.00183553, -0.00696843, ...,  0.00140995,\n",
      "       -0.01277258,  0.00705865])), Entity(name=intrusion detection, label=Application, properties=embeddings=array([-0.02220008, -0.00092426, -0.01036203, ..., -0.00459398,\n",
      "       -0.00572819,  0.0047737 ])), Entity(name=few shot learning, label=Technique, properties=embeddings=array([ 0.00658833,  0.02395446, -0.01717963, ..., -0.0001922 ,\n",
      "        0.00659942,  0.00829406])), Entity(name=large language model, label=Technology, properties=embeddings=array([-0.0175759 ,  0.01570321, -0.01447532, ...,  0.00558762,\n",
      "       -0.0219313 ,  0.0079291 ])), Entity(name=knowledge graph, label=Data_Structure, properties=embeddings=array([ 0.00967516, -0.00251331, -0.02177497, ..., -0.00496721,\n",
      "       -0.01682499,  0.01573904])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=open source threat intelligence, label=Data_Source, properties=embeddings=array([-0.02941218, -0.01031956, -0.00760298, ..., -0.00892985,\n",
      "       -0.00860804,  0.00475828])), Entity(name=data augmentation, label=Process, properties=embeddings=array([-2.30737679e-02,  4.70109321e-03, -7.46384645e-03, ...,\n",
      "       -9.41984504e-03, -3.26512381e-05,  1.36782876e-02]))]\n",
      "[INFO] ------- Extracting Relations from the Document 1\n",
      "{'relationships': [{'startNode': {'name': 'open source threat intelligence', 'label': 'Data_Source'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'constructed from'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'constructs'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'large language model', 'label': 'Technology'}, 'name': 'applies'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'data annotation', 'label': 'Process'}, 'name': 'leverages'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'data augmentation', 'label': 'Process'}, 'name': 'leverages'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'topic classification', 'label': 'Task'}, 'name': 'fine-tunes for'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'entity extraction', 'label': 'Task'}, 'name': 'fine-tunes for'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'relationship extraction', 'label': 'Task'}, 'name': 'fine-tunes for'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'ttp extraction', 'label': 'Task'}, 'name': 'fine-tunes for'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'data annotation', 'label': 'Process'}, 'name': 'used for'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'data augmentation', 'label': 'Process'}, 'name': 'used for'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'topic classification', 'label': 'Task'}, 'name': 'fine-tuned for'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'entity extraction', 'label': 'Task'}, 'name': 'fine-tuned for'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'relationship extraction', 'label': 'Task'}, 'name': 'fine-tuned for'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'ttp extraction', 'label': 'Task'}, 'name': 'fine-tuned for'}, {'startNode': {'name': 'instruction based information extraction', 'label': 'Methodology'}, 'endNode': {'name': 'large language model', 'label': 'Technology'}, 'name': 'requires'}, {'startNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'endNode': {'name': 'intrusion detection', 'label': 'Application'}, 'name': 'applied to'}, {'startNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'endNode': {'name': 'threat hunting', 'label': 'Application'}, 'name': 'enhances'}, {'startNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'endNode': {'name': 'attack attribution', 'label': 'Application'}, 'name': 'enhances'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-1] Aie; there are some isolated entities without relations [Entity(name=few shot learning, label=Technique, properties=embeddings=array([ 0.00658833,  0.02395446, -0.01717963, ..., -0.0001922 ,\n",
      "        0.00659942,  0.00829406]))]. Solving them ...\n",
      "{'relationships': []}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-2] Aie; there are some isolated entities without relations [Entity(name=instruction based information extraction, label=Methodology, properties=embeddings=array([ 0.00578922,  0.01256758, -0.01632442, ..., -0.01279191,\n",
      "       -0.0065698 ,  0.00066387])), Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.02329106,  0.01237501, -0.01368894, ..., -0.00108116,\n",
      "       -0.01063947,  0.0007295 ])), Entity(name=threat hunting, label=Application, properties=embeddings=array([-0.02765322, -0.03706892, -0.01849421, ..., -0.01293665,\n",
      "        0.00242633,  0.02040382])), Entity(name=data annotation, label=Process, properties=embeddings=array([-0.01023736, -0.02583196, -0.01102544, ..., -0.00943764,\n",
      "       -0.00671153,  0.00328806])), Entity(name=ttp extraction, label=Task, properties=embeddings=array([-0.04061868, -0.00652669, -0.01061077, ...,  0.00112746,\n",
      "       -0.01573218,  0.02327023])), Entity(name=attack attribution, label=Application, properties=embeddings=array([-0.01950506, -0.0041517 , -0.01346924, ..., -0.00290357,\n",
      "       -0.00359692,  0.00190606])), Entity(name=topic classification, label=Task, properties=embeddings=array([-0.00586902,  0.01346419, -0.01709163, ..., -0.00839443,\n",
      "       -0.00442814,  0.02397172])), Entity(name=llm tikg, label=Methodology, properties=embeddings=array([ 0.00613677, -0.01793396, -0.0202904 , ...,  0.00590273,\n",
      "       -0.00363614,  0.00752743])), Entity(name=relationship extraction, label=Task, properties=embeddings=array([ 0.00177076,  0.00329305, -0.00977207, ...,  0.00350275,\n",
      "       -0.02029999,  0.00142598])), Entity(name=entity extraction, label=Task, properties=embeddings=array([-0.00535627, -0.00183553, -0.00696843, ...,  0.00140995,\n",
      "       -0.01277258,  0.00705865])), Entity(name=intrusion detection, label=Application, properties=embeddings=array([-0.02220008, -0.00092426, -0.01036203, ..., -0.00459398,\n",
      "       -0.00572819,  0.0047737 ])), Entity(name=few shot learning, label=Technique, properties=embeddings=array([ 0.00658833,  0.02395446, -0.01717963, ..., -0.0001922 ,\n",
      "        0.00659942,  0.00829406])), Entity(name=large language model, label=Technology, properties=embeddings=array([-0.0175759 ,  0.01570321, -0.01447532, ...,  0.00558762,\n",
      "       -0.0219313 ,  0.0079291 ])), Entity(name=knowledge graph, label=Data_Structure, properties=embeddings=array([ 0.00967516, -0.00251331, -0.02177497, ..., -0.00496721,\n",
      "       -0.01682499,  0.01573904])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=open source threat intelligence, label=Data_Source, properties=embeddings=array([-0.02941218, -0.01031956, -0.00760298, ..., -0.00892985,\n",
      "       -0.00860804,  0.00475828])), Entity(name=data augmentation, label=Process, properties=embeddings=array([-2.30737679e-02,  4.70109321e-03, -7.46384645e-03, ...,\n",
      "       -9.41984504e-03, -3.26512381e-05,  1.36782876e-02]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'instruction based information extraction', 'label': 'Methodology'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'utilizes'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'fine-tuned for'}, {'startNode': {'name': 'threat hunting', 'label': 'Application'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'enhanced by'}, {'startNode': {'name': 'data annotation', 'label': 'Process'}, 'endNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'name': 'performed by'}, {'startNode': {'name': 'ttp extraction', 'label': 'Task'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'includes'}, {'startNode': {'name': 'attack attribution', 'label': 'Application'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'supported by'}, {'startNode': {'name': 'topic classification', 'label': 'Task'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'includes'}, {'startNode': {'name': 'relationship extraction', 'label': 'Task'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'includes'}, {'startNode': {'name': 'entity extraction', 'label': 'Task'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'includes'}, {'startNode': {'name': 'intrusion detection', 'label': 'Application'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'enhanced by'}, {'startNode': {'name': 'few shot learning', 'label': 'Technique'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'leverages'}, {'startNode': {'name': 'large language model', 'label': 'Technology'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'utilizes'}, {'startNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'endNode': {'name': 'open source threat intelligence', 'label': 'Data_Source'}, 'name': 'constructed from'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'utilizes'}, {'startNode': {'name': 'data augmentation', 'label': 'Process'}, 'endNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'name': 'performed by'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] ------- Extracting Entities from the Document 2\n",
      "{'entities': [{'name': 'LLM-TIKG', 'label': 'Methodology'}, {'name': 'GPT-3.5', 'label': 'Model'}, {'name': 'GPT-3.5-turbo', 'label': 'Model'}, {'name': 'Llama2-7B', 'label': 'Model'}, {'name': 'BERT-CRF', 'label': 'Model'}, {'name': 'GPT4', 'label': 'Model'}, {'name': 'LLaMA-Tec', 'label': 'Model'}, {'name': 'Named Entity Recognition', 'label': 'Technique'}, {'name': 'TTP classification', 'label': 'Technique'}, {'name': 'Relationship Extraction', 'label': 'Technique'}, {'name': 'Natural Language Processing', 'label': 'Technique'}, {'name': 'PowerShell', 'label': 'Tool'}, {'name': 'Cobalt Strike', 'label': 'Tool'}, {'name': 'C2 server', 'label': 'Tool'}, {'name': 'Mimikatz', 'label': 'Tool'}, {'name': 'BlackSuit', 'label': 'Malware'}, {'name': 'Royal Ransomware', 'label': 'Malware'}, {'name': 'Conti', 'label': 'Malware'}]}\n",
      "[Entity(name=gpt4, label=Model, properties=embeddings=array([-0.02514089,  0.00620026, -0.01696183, ...,  0.00178123,\n",
      "       -0.02046133,  0.01022713])), Entity(name=mimikatz, label=Tool, properties=embeddings=array([-0.00639734, -0.0301451 , -0.0071878 , ..., -0.00065657,\n",
      "       -0.01847601,  0.00268305])), Entity(name=conti, label=Malware, properties=embeddings=array([-2.29399728e-02, -9.65058944e-03, -1.11327454e-02, ...,\n",
      "        1.51461512e-03, -7.58143142e-05, -2.14438450e-02])), Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.02329609,  0.01235883, -0.01369327, ..., -0.00109288,\n",
      "       -0.01063987,  0.00071635])), Entity(name=llama tec, label=Model, properties=embeddings=array([-0.02831803,  0.01843116, -0.0209661 , ...,  0.00039858,\n",
      "       -0.01783572,  0.00052316])), Entity(name=named entity recognition, label=Technique, properties=embeddings=array([ 0.0103626 , -0.00104141, -0.01232173, ..., -0.0005043 ,\n",
      "       -0.0064373 , -0.00348934])), Entity(name=llm tikg, label=Methodology, properties=embeddings=array([ 0.00613084, -0.01793901, -0.02029365, ...,  0.00590774,\n",
      "       -0.00362842,  0.00754116])), Entity(name=royal ransomware, label=Malware, properties=embeddings=array([-0.04176803, -0.01734229, -0.00392667, ..., -0.01100384,\n",
      "       -0.00981439, -0.00648911])), Entity(name=powershell, label=Tool, properties=embeddings=array([-0.01994595, -0.00196154, -0.00636507, ..., -0.00706335,\n",
      "       -0.00590577,  0.00108121])), Entity(name=c2 server, label=Tool, properties=embeddings=array([-0.01649597, -0.01916149, -0.01717776, ...,  0.00531502,\n",
      "       -0.01209817,  0.01713052])), Entity(name=blacksuit, label=Malware, properties=embeddings=array([-0.0204986 , -0.02075756, -0.0089238 , ..., -0.00750815,\n",
      "       -0.01555299, -0.00529391])), Entity(name=bert crf, label=Model, properties=embeddings=array([-0.00411062, -0.0209413 , -0.02574343, ...,  0.00220911,\n",
      "       -0.0039284 ,  0.01736144])), Entity(name=relationship extraction, label=Technique, properties=embeddings=array([ 2.46262595e-02,  9.46211535e-04, -1.42284963e-02, ...,\n",
      "       -1.61979347e-05, -1.35766134e-02, -4.88389041e-03])), Entity(name=ttp classification, label=Technique, properties=embeddings=array([ 0.00051597,  0.00198268, -0.01813652, ...,  0.00068254,\n",
      "       -0.00575744,  0.02207634])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=gpt 3.5 turbo, label=Model, properties=embeddings=array([-0.03320267, -0.00318877, -0.02027043, ..., -0.00012884,\n",
      "       -0.00777877,  0.01345234])), Entity(name=cobalt strike, label=Tool, properties=embeddings=array([-0.01231122, -0.02793761, -0.01360303, ..., -0.00848108,\n",
      "       -0.01583731,  0.00442502])), Entity(name=natural language processing, label=Technique, properties=embeddings=array([-0.00332394,  0.00871266, -0.01762499, ...,  0.00476473,\n",
      "       -0.00332152,  0.00186552]))]\n",
      "[INFO] Wohoo! Entity was matched --- [gpt4:Model] --merged--> [gpt 3.5:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [llama tec:Model] --merged--> [llama2 7b:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [relationship extraction:Technique] --merged--> [relationship extraction:Task]\n",
      "[INFO] Wohoo! Entity was matched --- [ttp classification:Technique] --merged--> [ttp extraction:Task]\n",
      "[INFO] Wohoo! Entity was matched --- [gpt 3.5 turbo:Model] --merged--> [gpt 3.5:Model]\n",
      "[INFO] ------- Extracting Relations from the Document 2\n",
      "{'relationships': [{'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'named entity recognition', 'label': 'Technique'}, 'name': 'enhances'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'relationship extraction', 'label': 'Task'}, 'name': 'enhances'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'named entity recognition', 'label': 'Technique'}, 'name': 'performs better than'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'ttp extraction', 'label': 'Task'}, 'name': 'performs'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'named entity recognition', 'label': 'Technique'}, 'name': 'improves'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'ttp extraction', 'label': 'Task'}, 'name': 'improves'}, {'startNode': {'name': 'powershell', 'label': 'Tool'}, 'endNode': {'name': 'mimikatz', 'label': 'Tool'}, 'name': 'connects with'}, {'startNode': {'name': 'cobalt strike', 'label': 'Tool'}, 'endNode': {'name': 'c2 server', 'label': 'Tool'}, 'name': 'connects with'}, {'startNode': {'name': 'blacksuit', 'label': 'Malware'}, 'endNode': {'name': 'royal ransomware', 'label': 'Malware'}, 'name': 'has similarities with'}, {'startNode': {'name': 'royal ransomware', 'label': 'Malware'}, 'endNode': {'name': 'conti', 'label': 'Malware'}, 'name': 'is connected to'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-1] Aie; there are some isolated entities without relations [Entity(name=bert crf, label=Model, properties=embeddings=array([-0.00411062, -0.0209413 , -0.02574343, ...,  0.00220911,\n",
      "       -0.0039284 ,  0.01736144])), Entity(name=natural language processing, label=Technique, properties=embeddings=array([-0.00332394,  0.00871266, -0.01762499, ...,  0.00476473,\n",
      "       -0.00332152,  0.00186552]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'BERT-CRF', 'label': 'Model'}, 'endNode': {'name': 'Natural Language Processing', 'label': 'Technique'}, 'name': 'utilizes'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-2] Aie; there are some isolated entities without relations [Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=mimikatz, label=Tool, properties=embeddings=array([-0.00639734, -0.0301451 , -0.0071878 , ..., -0.00065657,\n",
      "       -0.01847601,  0.00268305])), Entity(name=conti, label=Malware, properties=embeddings=array([-2.29399728e-02, -9.65058944e-03, -1.11327454e-02, ...,\n",
      "        1.51461512e-03, -7.58143142e-05, -2.14438450e-02])), Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.02329609,  0.01235883, -0.01369327, ..., -0.00109288,\n",
      "       -0.01063987,  0.00071635])), Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.02329106,  0.01237501, -0.01368894, ..., -0.00108116,\n",
      "       -0.01063947,  0.0007295 ])), Entity(name=named entity recognition, label=Technique, properties=embeddings=array([ 0.0103626 , -0.00104141, -0.01232173, ..., -0.0005043 ,\n",
      "       -0.0064373 , -0.00348934])), Entity(name=llm tikg, label=Methodology, properties=embeddings=array([ 0.00613084, -0.01793901, -0.02029365, ...,  0.00590774,\n",
      "       -0.00362842,  0.00754116])), Entity(name=royal ransomware, label=Malware, properties=embeddings=array([-0.04176803, -0.01734229, -0.00392667, ..., -0.01100384,\n",
      "       -0.00981439, -0.00648911])), Entity(name=powershell, label=Tool, properties=embeddings=array([-0.01994595, -0.00196154, -0.00636507, ..., -0.00706335,\n",
      "       -0.00590577,  0.00108121])), Entity(name=c2 server, label=Tool, properties=embeddings=array([-0.01649597, -0.01916149, -0.01717776, ...,  0.00531502,\n",
      "       -0.01209817,  0.01713052])), Entity(name=blacksuit, label=Malware, properties=embeddings=array([-0.0204986 , -0.02075756, -0.0089238 , ..., -0.00750815,\n",
      "       -0.01555299, -0.00529391])), Entity(name=relationship extraction, label=Task, properties=embeddings=array([ 0.00177076,  0.00329305, -0.00977207, ...,  0.00350275,\n",
      "       -0.02029999,  0.00142598])), Entity(name=ttp extraction, label=Task, properties=embeddings=array([-0.04061868, -0.00652669, -0.01061077, ...,  0.00112746,\n",
      "       -0.01573218,  0.02327023])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=cobalt strike, label=Tool, properties=embeddings=array([-0.01231122, -0.02793761, -0.01360303, ..., -0.00848108,\n",
      "       -0.01583731,  0.00442502]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'enhances'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'named entity recognition', 'label': 'Technique'}, 'name': 'improves'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'relationship extraction', 'label': 'Task'}, 'name': 'supports'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'named entity recognition', 'label': 'Technique'}, 'name': 'performs better than'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'ttp extraction', 'label': 'Task'}, 'name': 'performs'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'named entity recognition', 'label': 'Technique'}, 'name': 'enhances'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'ttp extraction', 'label': 'Task'}, 'name': 'enhances'}, {'startNode': {'name': 'powershell', 'label': 'Tool'}, 'endNode': {'name': 'mimikatz', 'label': 'Tool'}, 'name': 'used with'}, {'startNode': {'name': 'cobalt strike', 'label': 'Tool'}, 'endNode': {'name': 'c2 server', 'label': 'Tool'}, 'name': 'communicates with'}, {'startNode': {'name': 'blacksuit', 'label': 'Malware'}, 'endNode': {'name': 'royal ransomware', 'label': 'Malware'}, 'name': 'associated with'}, {'startNode': {'name': 'royal ransomware', 'label': 'Malware'}, 'endNode': {'name': 'conti', 'label': 'Malware'}, 'name': 'connected to'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [communicates_with] --merged --> [connects_with] \n",
      "[INFO] Wohoo! Relation was matched --- [connected_to] --merged --> [is_connected_to] \n",
      "[INFO][ISOLATED ENTITIES][TRY-3] Aie; there are some isolated entities without relations [Entity(name=bert crf, label=Model, properties=embeddings=array([-0.00411062, -0.0209413 , -0.02574343, ...,  0.00220911,\n",
      "       -0.0039284 ,  0.01736144])), Entity(name=natural language processing, label=Technique, properties=embeddings=array([-0.00332394,  0.00871266, -0.01762499, ...,  0.00476473,\n",
      "       -0.00332152,  0.00186552]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'BERT-CRF', 'label': 'Model'}, 'endNode': {'name': 'Natural Language Processing', 'label': 'Technique'}, 'name': 'utilizes'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [used_with] --merged --> [used_for] \n",
      "[INFO] ------- Extracting Entities from the Document 3\n",
      "{'entities': [{'label': 'Methodology', 'name': 'Knowledge Graph Construction'}, {'label': 'Challenge', 'name': 'Domain-specific attributes of entities'}, {'label': 'Challenge', 'name': 'Analysis of lengthy texts'}, {'label': 'Challenge', 'name': 'Requirement for large amounts of labeled data'}, {'label': 'Challenge', 'name': 'Lack of authoritative open-source annotated threat intelligence datasets'}, {'label': 'Challenge', 'name': 'Low-level, fragmented threat intelligence'}, {'label': 'Challenge', 'name': 'Lack of high-level information and relationships between intelligence entities'}, {'label': 'Challenge', 'name': 'Limitations in accuracy of information extraction'}, {'label': 'Challenge', 'name': 'Absence of attack behaviors in textual information'}, {'label': 'Challenge', 'name': 'Isolation of threat intelligence entities'}, {'label': 'Challenge', 'name': 'Neglect to extract TTPs from attack descriptions'}, {'label': 'Challenge', 'name': 'Misses deeper insights in threat intelligence analysis'}, {'label': 'Challenge', 'name': 'Significant manual effort for data annotation'}, {'label': 'Challenge', 'name': 'Data leakage when models are not run locally'}, {'label': 'Challenge', 'name': 'Lack of comprehensive approach to integrate low-level and high-level threat intelligence'}, {'label': 'Methodology', 'name': 'Named Entity Recognition'}, {'label': 'Methodology', 'name': 'Rule-based or syntactic analysis'}, {'label': 'Model', 'name': 'GPT-3.5'}, {'label': 'Model', 'name': 'BERT-CRF'}, {'label': 'Tool', 'name': 'TTPDrill'}]}\n",
      "[Entity(name=significant manual effort for data annotation, label=Challenge, properties=embeddings=array([-0.03582621,  0.00355693, -0.01749353, ..., -0.01501765,\n",
      "       -0.00291695,  0.00987281])), Entity(name=bert crf, label=Model, properties=embeddings=array([-0.00412153, -0.02094083, -0.0257207 , ...,  0.00221283,\n",
      "       -0.00393244,  0.0173748 ])), Entity(name=ttpdrill, label=Tool, properties=embeddings=array([-0.03246584, -0.01824431, -0.01449919, ...,  0.01126529,\n",
      "       -0.02885598,  0.01902914])), Entity(name=isolation of threat intelligence entities, label=Challenge, properties=embeddings=array([-0.02887981, -0.0094818 , -0.01594832, ..., -0.01256053,\n",
      "       -0.00124256, -0.00060795])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03676913, -0.00358182, -0.01604153, ...,  0.00153045,\n",
      "       -0.01379468,  0.01271043])), Entity(name=lack of high level information and relationships between intelligence entities, label=Challenge, properties=embeddings=array([-0.02238257,  0.00850664, -0.01256885, ...,  0.00772615,\n",
      "       -0.0061358 , -0.00344511])), Entity(name=misses deeper insights in threat intelligence analysis, label=Challenge, properties=embeddings=array([-0.02538196,  0.00469716, -0.01788239, ..., -0.00236396,\n",
      "        0.000703  , -0.0001705 ])), Entity(name=limitations in accuracy of information extraction, label=Challenge, properties=embeddings=array([-0.02443838,  0.01966265, -0.0163872 , ..., -0.00306695,\n",
      "       -0.00418684, -0.0012597 ])), Entity(name=domain specific attributes of entities, label=Challenge, properties=embeddings=array([-0.02195742,  0.00671081, -0.0239307 , ...,  0.00169437,\n",
      "        0.00124357, -0.00296183])), Entity(name=neglect to extract ttps from attack descriptions, label=Challenge, properties=embeddings=array([-0.01812784,  0.01436305, -0.01453913, ...,  0.00552481,\n",
      "       -0.0147643 ,  0.01400769])), Entity(name=requirement for large amounts of labeled data, label=Challenge, properties=embeddings=array([-0.02875953,  0.00890965, -0.01521363, ..., -0.01510029,\n",
      "       -0.00618855,  0.0013291 ])), Entity(name=named entity recognition, label=Methodology, properties=embeddings=array([ 0.00046615, -0.00137518, -0.01471573, ..., -0.00510386,\n",
      "       -0.00767246, -0.00947627])), Entity(name=knowledge graph construction, label=Methodology, properties=embeddings=array([ 0.02683518, -0.00105201, -0.02343734, ..., -0.01786854,\n",
      "       -0.00783603,  0.00416973])), Entity(name=rule based or syntactic analysis, label=Methodology, properties=embeddings=array([-0.02387179, -0.00034877, -0.02141809, ..., -0.00548915,\n",
      "       -0.00420197,  0.00583894])), Entity(name=analysis of lengthy texts, label=Challenge, properties=embeddings=array([-0.03061586, -0.0076548 , -0.03352475, ..., -0.00964745,\n",
      "        0.00439513, -0.00436172])), Entity(name=lack of comprehensive approach to integrate low level and high level threat intelligence, label=Challenge, properties=embeddings=array([-0.03010275, -0.00846667, -0.01371372, ..., -0.00516958,\n",
      "        0.00050238,  0.00738735])), Entity(name=lack of authoritative open source annotated threat intelligence datasets, label=Challenge, properties=embeddings=array([-0.03128357,  0.00012772, -0.01422375, ..., -0.01722111,\n",
      "       -0.00885412,  0.0062309 ])), Entity(name=absence of attack behaviors in textual information, label=Challenge, properties=embeddings=array([-0.0339059 ,  0.00772797, -0.016163  , ..., -0.0027726 ,\n",
      "       -0.01660998, -0.00915352])), Entity(name=data leakage when models are not run locally, label=Challenge, properties=embeddings=array([-0.04599591, -0.00727702, -0.01918665, ...,  0.00975389,\n",
      "        0.00198066,  0.00590902])), Entity(name=low level, fragmented threat intelligence, label=Challenge, properties=embeddings=array([-0.03412367, -0.01527198, -0.01254321, ..., -0.0026963 ,\n",
      "       -0.0036029 ,  0.00058851]))]\n",
      "[INFO] Wohoo! Entity was matched --- [named entity recognition:Methodology] --merged--> [named entity recognition:Technique]\n",
      "[INFO] Wohoo! Entity was matched --- [knowledge graph construction:Methodology] --merged--> [knowledge graph:Data_Structure]\n",
      "[INFO] ------- Extracting Relations from the Document 3\n",
      "{'relationships': [{'startNode': {'name': 'significant manual effort for data annotation', 'label': 'Challenge'}, 'endNode': {'name': 'requirement for large amounts of labeled data', 'label': 'Challenge'}, 'name': 'exacerbates'}, {'startNode': {'name': 'bert crf', 'label': 'Model'}, 'endNode': {'name': 'limitations in accuracy of information extraction', 'label': 'Challenge'}, 'name': 'suffers from'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'biases and inaccuracies in entity extraction', 'label': 'Challenge'}, 'name': 'leads to'}, {'startNode': {'name': 'ttpdrill', 'label': 'Tool'}, 'endNode': {'name': 'fails when new categories and expressions appear', 'label': 'Challenge'}, 'name': 'is ineffective'}, {'startNode': {'name': 'isolation of threat intelligence entities', 'label': 'Challenge'}, 'endNode': {'name': 'lack of high level information and relationships between intelligence entities', 'label': 'Challenge'}, 'name': 'contributes to'}, {'startNode': {'name': 'misses deeper insights in threat intelligence analysis', 'label': 'Challenge'}, 'endNode': {'name': 'low level, fragmented threat intelligence', 'label': 'Challenge'}, 'name': 'results in'}, {'startNode': {'name': 'domain specific attributes of entities', 'label': 'Challenge'}, 'endNode': {'name': 'limitations in accuracy of information extraction', 'label': 'Challenge'}, 'name': 'causes'}, {'startNode': {'name': 'neglect to extract ttps from attack descriptions', 'label': 'Challenge'}, 'endNode': {'name': 'absence of attack behaviors in textual information', 'label': 'Challenge'}, 'name': 'leads to'}, {'startNode': {'name': 'named entity recognition', 'label': 'Technique'}, 'endNode': {'name': 'rule based or syntactic analysis', 'label': 'Methodology'}, 'name': 'relies on'}, {'startNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'endNode': {'name': 'lack of comprehensive approach to integrate low level and high level threat intelligence', 'label': 'Challenge'}, 'name': 'fails to address'}, {'startNode': {'name': 'lack of authoritative open source annotated threat intelligence datasets', 'label': 'Challenge'}, 'endNode': {'name': 'requirement for large amounts of labeled data', 'label': 'Challenge'}, 'name': 'aggravates'}, {'startNode': {'name': 'data leakage when models are not run locally', 'label': 'Challenge'}, 'endNode': {'name': 'significant manual effort for data annotation', 'label': 'Challenge'}, 'name': 'increases risk of'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Challenge' name='biases and inaccuracies in entity extraction' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [biases and inaccuracies in entity extraction:Challenge] --merged--> [limitations in accuracy of information extraction:Challenge]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Challenge' name='fails when new categories and expressions appear' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [fails when new categories and expressions appear:Challenge] --merged--> [data leakage when models are not run locally:Challenge]\n",
      "[INFO][ISOLATED ENTITIES][TRY-1] Aie; there are some isolated entities without relations [Entity(name=analysis of lengthy texts, label=Challenge, properties=embeddings=array([-0.03061586, -0.0076548 , -0.03352475, ..., -0.00964745,\n",
      "        0.00439513, -0.00436172]))]. Solving them ...\n",
      "{'relationships': []}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-2] Aie; there are some isolated entities without relations [Entity(name=significant manual effort for data annotation, label=Challenge, properties=embeddings=array([-0.03582621,  0.00355693, -0.01749353, ..., -0.01501765,\n",
      "       -0.00291695,  0.00987281])), Entity(name=bert crf, label=Model, properties=embeddings=array([-0.00412153, -0.02094083, -0.0257207 , ...,  0.00221283,\n",
      "       -0.00393244,  0.0173748 ])), Entity(name=ttpdrill, label=Tool, properties=embeddings=array([-0.03246584, -0.01824431, -0.01449919, ...,  0.01126529,\n",
      "       -0.02885598,  0.01902914])), Entity(name=isolation of threat intelligence entities, label=Challenge, properties=embeddings=array([-0.02887981, -0.0094818 , -0.01594832, ..., -0.01256053,\n",
      "       -0.00124256, -0.00060795])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03676913, -0.00358182, -0.01604153, ...,  0.00153045,\n",
      "       -0.01379468,  0.01271043])), Entity(name=lack of high level information and relationships between intelligence entities, label=Challenge, properties=embeddings=array([-0.02238257,  0.00850664, -0.01256885, ...,  0.00772615,\n",
      "       -0.0061358 , -0.00344511])), Entity(name=misses deeper insights in threat intelligence analysis, label=Challenge, properties=embeddings=array([-0.02538196,  0.00469716, -0.01788239, ..., -0.00236396,\n",
      "        0.000703  , -0.0001705 ])), Entity(name=limitations in accuracy of information extraction, label=Challenge, properties=embeddings=array([-0.02443838,  0.01966265, -0.0163872 , ..., -0.00306695,\n",
      "       -0.00418684, -0.0012597 ])), Entity(name=domain specific attributes of entities, label=Challenge, properties=embeddings=array([-0.02195742,  0.00671081, -0.0239307 , ...,  0.00169437,\n",
      "        0.00124357, -0.00296183])), Entity(name=neglect to extract ttps from attack descriptions, label=Challenge, properties=embeddings=array([-0.01812784,  0.01436305, -0.01453913, ...,  0.00552481,\n",
      "       -0.0147643 ,  0.01400769])), Entity(name=requirement for large amounts of labeled data, label=Challenge, properties=embeddings=array([-0.02875953,  0.00890965, -0.01521363, ..., -0.01510029,\n",
      "       -0.00618855,  0.0013291 ])), Entity(name=named entity recognition, label=Technique, properties=embeddings=array([ 0.0103626 , -0.00104141, -0.01232173, ..., -0.0005043 ,\n",
      "       -0.0064373 , -0.00348934])), Entity(name=knowledge graph, label=Data_Structure, properties=embeddings=array([ 0.00967516, -0.00251331, -0.02177497, ..., -0.00496721,\n",
      "       -0.01682499,  0.01573904])), Entity(name=rule based or syntactic analysis, label=Methodology, properties=embeddings=array([-0.02387179, -0.00034877, -0.02141809, ..., -0.00548915,\n",
      "       -0.00420197,  0.00583894])), Entity(name=analysis of lengthy texts, label=Challenge, properties=embeddings=array([-0.03061586, -0.0076548 , -0.03352475, ..., -0.00964745,\n",
      "        0.00439513, -0.00436172])), Entity(name=lack of comprehensive approach to integrate low level and high level threat intelligence, label=Challenge, properties=embeddings=array([-0.03010275, -0.00846667, -0.01371372, ..., -0.00516958,\n",
      "        0.00050238,  0.00738735])), Entity(name=lack of authoritative open source annotated threat intelligence datasets, label=Challenge, properties=embeddings=array([-0.03128357,  0.00012772, -0.01422375, ..., -0.01722111,\n",
      "       -0.00885412,  0.0062309 ])), Entity(name=absence of attack behaviors in textual information, label=Challenge, properties=embeddings=array([-0.0339059 ,  0.00772797, -0.016163  , ..., -0.0027726 ,\n",
      "       -0.01660998, -0.00915352])), Entity(name=data leakage when models are not run locally, label=Challenge, properties=embeddings=array([-0.04599591, -0.00727702, -0.01918665, ...,  0.00975389,\n",
      "        0.00198066,  0.00590902])), Entity(name=low level, fragmented threat intelligence, label=Challenge, properties=embeddings=array([-0.03412367, -0.01527198, -0.01254321, ..., -0.0026963 ,\n",
      "       -0.0036029 ,  0.00058851]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'significant manual effort for data annotation', 'label': 'Challenge'}, 'endNode': {'name': 'requirement for large amounts of labeled data', 'label': 'Challenge'}, 'name': 'exacerbates'}, {'startNode': {'name': 'bert crf', 'label': 'Model'}, 'endNode': {'name': 'limitations in accuracy of information extraction', 'label': 'Challenge'}, 'name': 'suffers from'}, {'startNode': {'name': 'ttpdrill', 'label': 'Tool'}, 'endNode': {'name': 'rule based or syntactic analysis', 'label': 'Methodology'}, 'name': 'employs'}, {'startNode': {'name': 'isolation of threat intelligence entities', 'label': 'Challenge'}, 'endNode': {'name': 'lack of high level information and relationships between intelligence entities', 'label': 'Challenge'}, 'name': 'leads to'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'biases and inaccuracies in entity extraction', 'label': 'Challenge'}, 'name': 'results in'}, {'startNode': {'name': 'lack of comprehensive approach to integrate low level and high level threat intelligence', 'label': 'Challenge'}, 'endNode': {'name': 'low level, fragmented threat intelligence', 'label': 'Challenge'}, 'name': 'contributes to'}, {'startNode': {'name': 'named entity recognition', 'label': 'Technique'}, 'endNode': {'name': 'traditional methods for entity and relationship extraction', 'label': 'Methodology'}, 'name': 'relies on'}, {'startNode': {'name': 'analysis of lengthy texts', 'label': 'Challenge'}, 'endNode': {'name': 'limitations in accuracy of information extraction', 'label': 'Challenge'}, 'name': 'contributes to'}, {'startNode': {'name': 'lack of authoritative open source annotated threat intelligence datasets', 'label': 'Challenge'}, 'endNode': {'name': 'requirement for large amounts of labeled data', 'label': 'Challenge'}, 'name': 'aggravates'}, {'startNode': {'name': 'absence of attack behaviors in textual information', 'label': 'Challenge'}, 'endNode': {'name': 'neglect to extract ttps from attack descriptions', 'label': 'Challenge'}, 'name': 'results in'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Challenge' name='biases and inaccuracies in entity extraction' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [biases and inaccuracies in entity extraction:Challenge] --merged--> [limitations in accuracy of information extraction:Challenge]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Methodology' name='traditional methods for entity and relationship extraction' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [traditional methods for entity and relationship extraction:Methodology] --merged--> [rule based or syntactic analysis:Methodology]\n",
      "[INFO][ISOLATED ENTITIES][TRY-3] Aie; there are some isolated entities without relations [Entity(name=misses deeper insights in threat intelligence analysis, label=Challenge, properties=embeddings=array([-0.02538196,  0.00469716, -0.01788239, ..., -0.00236396,\n",
      "        0.000703  , -0.0001705 ])), Entity(name=domain specific attributes of entities, label=Challenge, properties=embeddings=array([-0.02195742,  0.00671081, -0.0239307 , ...,  0.00169437,\n",
      "        0.00124357, -0.00296183])), Entity(name=knowledge graph, label=Data_Structure, properties=embeddings=array([ 0.00967516, -0.00251331, -0.02177497, ..., -0.00496721,\n",
      "       -0.01682499,  0.01573904])), Entity(name=data leakage when models are not run locally, label=Challenge, properties=embeddings=array([-0.04599591, -0.00727702, -0.01918665, ...,  0.00975389,\n",
      "        0.00198066,  0.00590902]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'misses deeper insights in threat intelligence analysis', 'label': 'Challenge'}, 'endNode': {'name': 'domain specific attributes of entities', 'label': 'Challenge'}, 'name': 'is exacerbated by'}, {'startNode': {'name': 'misses deeper insights in threat intelligence analysis', 'label': 'Challenge'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'is limited by'}, {'startNode': {'name': 'domain specific attributes of entities', 'label': 'Challenge'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'complicates construction of'}, {'startNode': {'name': 'data leakage when models are not run locally', 'label': 'Challenge'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'poses a risk to'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [is_exacerbated_by] --merged --> [exacerbates] \n",
      "[INFO] ------- Extracting Entities from the Document 4\n",
      "{'entities': [{'label': 'Solution', 'name': 'LLM-TIKG'}, {'label': 'Model', 'name': 'GPT-3.5'}, {'label': 'Model', 'name': 'Llama2-7B'}, {'label': 'Technique', 'name': 'Few-shot learning'}, {'label': 'Technique', 'name': 'Instruction-based Information Extraction'}, {'label': 'Technique', 'name': 'Low-Rank Adaptation (LoRA)'}, {'label': 'Database', 'name': 'Neo4j'}, {'label': 'Methodology', 'name': 'Data annotation and augmentation'}, {'label': 'Methodology', 'name': 'Entity and relationship extraction'}, {'label': 'Methodology', 'name': 'TTP classification'}, {'label': 'Methodology', 'name': 'Threat intelligence knowledge graph construction'}, {'label': 'Methodology', 'name': 'Similarity matching'}, {'label': 'Methodology', 'name': 'Link prediction'}, {'label': 'Tool', 'name': 'Docker'}, {'label': 'Algorithm', 'name': 'Logistic Regression'}, {'label': 'Algorithm', 'name': 'Naive Bayes'}]}\n",
      "[Entity(name=logistic regression, label=Algorithm, properties=embeddings=array([-0.032253  , -0.0044368 , -0.01019645, ...,  0.01478196,\n",
      "       -0.0032791 ,  0.02328226])), Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.0233    ,  0.01235305, -0.01369664, ..., -0.00109613,\n",
      "       -0.01064868,  0.00074564])), Entity(name=link prediction, label=Methodology, properties=embeddings=array([ 0.00068717, -0.01628907, -0.01671704, ..., -0.00875919,\n",
      "       -0.01384346, -0.00133373])), Entity(name=llm tikg, label=Solution, properties=embeddings=array([ 0.00039349, -0.02299757, -0.01205104, ...,  0.01500864,\n",
      "       -0.00224698,  0.0072205 ])), Entity(name=low rank adaptation (lora), label=Technique, properties=embeddings=array([-0.01221953, -0.01746495, -0.02165181, ...,  0.0144665 ,\n",
      "       -0.01420918,  0.02571036])), Entity(name=similarity matching, label=Methodology, properties=embeddings=array([-0.02299281, -0.02200627, -0.02660636, ...,  0.00462181,\n",
      "        0.00454835,  0.01350999])), Entity(name=docker, label=Tool, properties=embeddings=array([-0.05436936, -0.01850644, -0.00872477, ...,  0.01740875,\n",
      "        0.00778122,  0.01105773])), Entity(name=neo4j, label=Database, properties=embeddings=array([-0.00481371,  0.01033838, -0.01557506, ...,  0.00338041,\n",
      "       -0.00504572, -0.00161772])), Entity(name=entity and relationship extraction, label=Methodology, properties=embeddings=array([-0.00437859,  0.00178401, -0.01906987, ..., -0.00531792,\n",
      "       -0.00592523, -0.01095121])), Entity(name=naive bayes, label=Algorithm, properties=embeddings=array([-0.02363024, -0.01178105, -0.00500448, ...,  0.00862458,\n",
      "       -0.01489254,  0.01435162])), Entity(name=few shot learning, label=Technique, properties=embeddings=array([ 0.00662441,  0.02396213, -0.01718637, ..., -0.00018825,\n",
      "        0.0065904 ,  0.0082939 ])), Entity(name=ttp classification, label=Methodology, properties=embeddings=array([-0.00939821,  0.00161567, -0.02053983, ..., -0.00390094,\n",
      "       -0.00699803,  0.01610363])), Entity(name=threat intelligence knowledge graph construction, label=Methodology, properties=embeddings=array([ 0.00639563, -0.01667004, -0.01802387, ..., -0.01856618,\n",
      "       -0.00615528,  0.002171  ])), Entity(name=instruction based information extraction, label=Technique, properties=embeddings=array([ 0.01568687,  0.01288798, -0.0139314 , ..., -0.00819275,\n",
      "       -0.00531799,  0.00664551])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=data annotation and augmentation, label=Methodology, properties=embeddings=array([-0.02167405, -0.01294925, -0.01879562, ..., -0.01638829,\n",
      "       -0.00527837,  0.00420293]))]\n",
      "[INFO] Wohoo! Entity was matched --- [llm tikg:Solution] --merged--> [llm tikg:Methodology]\n",
      "[INFO] Wohoo! Entity was matched --- [entity and relationship extraction:Methodology] --merged--> [instruction based information extraction:Methodology]\n",
      "[INFO] Wohoo! Entity was matched --- [instruction based information extraction:Technique] --merged--> [instruction based information extraction:Methodology]\n",
      "[INFO] Wohoo! Entity was matched --- [data annotation and augmentation:Methodology] --merged--> [data annotation:Process]\n",
      "[INFO] ------- Extracting Relations from the Document 4\n",
      "{'relationships': [{'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'llama2 7b', 'label': 'Model'}, 'name': 'fine-tunes'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'name': 'uses for data annotation'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'threat intelligence knowledge graph construction', 'label': 'Methodology'}, 'name': 'enables'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'low rank adaptation (lora)', 'label': 'Technique'}, 'name': 'uses for fine-tuning'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'ttp classification', 'label': 'Methodology'}, 'name': 'improves'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'few shot learning', 'label': 'Technique'}, 'name': 'leverages'}, {'startNode': {'name': 'instruction based information extraction', 'label': 'Methodology'}, 'endNode': {'name': 'entity extraction accuracy', 'label': 'Process'}, 'name': 'improves'}, {'startNode': {'name': 'neo4j', 'label': 'Database'}, 'endNode': {'name': 'threat intelligence knowledge graph construction', 'label': 'Methodology'}, 'name': 'supports'}, {'startNode': {'name': 'docker', 'label': 'Tool'}, 'endNode': {'name': 'cti reports', 'label': 'Process'}, 'name': 'automates mapping of'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Process' name='entity extraction accuracy' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [entity extraction accuracy:Process] --merged--> [data annotation:Process]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Process' name='cti reports' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [cti reports:Process] --merged--> [ttp classification:Methodology]\n",
      "[INFO][ISOLATED ENTITIES][TRY-1] Aie; there are some isolated entities without relations [Entity(name=logistic regression, label=Algorithm, properties=embeddings=array([-0.032253  , -0.0044368 , -0.01019645, ...,  0.01478196,\n",
      "       -0.0032791 ,  0.02328226])), Entity(name=link prediction, label=Methodology, properties=embeddings=array([ 0.00068717, -0.01628907, -0.01671704, ..., -0.00875919,\n",
      "       -0.01384346, -0.00133373])), Entity(name=similarity matching, label=Methodology, properties=embeddings=array([-0.02299281, -0.02200627, -0.02660636, ...,  0.00462181,\n",
      "        0.00454835,  0.01350999])), Entity(name=naive bayes, label=Algorithm, properties=embeddings=array([-0.02363024, -0.01178105, -0.00500448, ...,  0.00862458,\n",
      "       -0.01489254,  0.01435162]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'logistic regression', 'label': 'Algorithm'}, 'endNode': {'name': 'naive bayes', 'label': 'Algorithm'}, 'name': 'complementary'}, {'startNode': {'name': 'link prediction', 'label': 'Methodology'}, 'endNode': {'name': 'similarity matching', 'label': 'Methodology'}, 'name': 'related'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-2] Aie; there are some isolated entities without relations [Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.0233    ,  0.01235305, -0.01369664, ..., -0.00109613,\n",
      "       -0.01064868,  0.00074564])), Entity(name=llm tikg, label=Methodology, properties=embeddings=array([ 0.00613084, -0.01793901, -0.02029365, ...,  0.00590774,\n",
      "       -0.00362842,  0.00754116])), Entity(name=low rank adaptation (lora), label=Technique, properties=embeddings=array([-0.01221953, -0.01746495, -0.02165181, ...,  0.0144665 ,\n",
      "       -0.01420918,  0.02571036])), Entity(name=docker, label=Tool, properties=embeddings=array([-0.05436936, -0.01850644, -0.00872477, ...,  0.01740875,\n",
      "        0.00778122,  0.01105773])), Entity(name=neo4j, label=Database, properties=embeddings=array([-0.00481371,  0.01033838, -0.01557506, ...,  0.00338041,\n",
      "       -0.00504572, -0.00161772])), Entity(name=instruction based information extraction, label=Methodology, properties=embeddings=array([ 0.00578922,  0.01256758, -0.01632442, ..., -0.01279191,\n",
      "       -0.0065698 ,  0.00066387])), Entity(name=few shot learning, label=Technique, properties=embeddings=array([ 0.00662441,  0.02396213, -0.01718637, ..., -0.00018825,\n",
      "        0.0065904 ,  0.0082939 ])), Entity(name=ttp classification, label=Methodology, properties=embeddings=array([-0.00939821,  0.00161567, -0.02053983, ..., -0.00390094,\n",
      "       -0.00699803,  0.01610363])), Entity(name=threat intelligence knowledge graph construction, label=Methodology, properties=embeddings=array([ 0.00639563, -0.01667004, -0.01802387, ..., -0.01856618,\n",
      "       -0.00615528,  0.002171  ])), Entity(name=instruction based information extraction, label=Methodology, properties=embeddings=array([ 0.00578922,  0.01256758, -0.01632442, ..., -0.01279191,\n",
      "       -0.0065698 ,  0.00066387])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=data annotation, label=Process, properties=embeddings=array([-0.01023736, -0.02583196, -0.01102544, ..., -0.00943764,\n",
      "       -0.00671153,  0.00328806]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'low rank adaptation (lora)', 'label': 'Technique'}, 'name': 'fine-tuned with'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'name': 'utilizes'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'threat intelligence knowledge graph construction', 'label': 'Methodology'}, 'name': 'enables'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'ttp classification', 'label': 'Methodology'}, 'name': 'includes'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'instruction based information extraction', 'label': 'Methodology'}, 'name': 'improves'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'data annotation', 'label': 'Process'}, 'name': 'used for'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'instruction based information extraction', 'label': 'Methodology'}, 'name': 'fine-tuned for'}, {'startNode': {'name': 'docker', 'label': 'Tool'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'supports'}, {'startNode': {'name': 'neo4j', 'label': 'Database'}, 'endNode': {'name': 'threat intelligence knowledge graph construction', 'label': 'Methodology'}, 'name': 'stores'}, {'startNode': {'name': 'few shot learning', 'label': 'Technique'}, 'endNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'name': 'leverages'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [fine_tuned_with] --merged --> [fine_tunes] \n",
      "[INFO] Wohoo! Relation was matched --- [fine_tuned_for] --merged --> [uses_for_fine_tuning] \n",
      "[INFO][ISOLATED ENTITIES][TRY-3] Aie; there are some isolated entities without relations [Entity(name=logistic regression, label=Algorithm, properties=embeddings=array([-0.032253  , -0.0044368 , -0.01019645, ...,  0.01478196,\n",
      "       -0.0032791 ,  0.02328226])), Entity(name=link prediction, label=Methodology, properties=embeddings=array([ 0.00068717, -0.01628907, -0.01671704, ..., -0.00875919,\n",
      "       -0.01384346, -0.00133373])), Entity(name=similarity matching, label=Methodology, properties=embeddings=array([-0.02299281, -0.02200627, -0.02660636, ...,  0.00462181,\n",
      "        0.00454835,  0.01350999])), Entity(name=naive bayes, label=Algorithm, properties=embeddings=array([-0.02363024, -0.01178105, -0.00500448, ...,  0.00862458,\n",
      "       -0.01489254,  0.01435162]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'logistic regression', 'label': 'Algorithm'}, 'endNode': {'name': 'naive bayes', 'label': 'Algorithm'}, 'name': 'is similar to'}, {'startNode': {'name': 'link prediction', 'label': 'Methodology'}, 'endNode': {'name': 'similarity matching', 'label': 'Methodology'}, 'name': 'is used with'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [is_used_with] --merged --> [used_for] \n",
      "[INFO] Wohoo! Relation was matched --- [uses_for_fine_tuning] --merged --> [fine_tuned_for] \n",
      "[INFO] Wohoo! Relation was matched --- [fine_tunes] --merged --> [fine_tunes_for] \n",
      "[INFO] Wohoo! Relation was matched --- [fine_tunes] --merged --> [fine_tunes_for] \n",
      "[INFO] Wohoo! Relation was matched --- [uses_for_fine_tuning] --merged --> [fine_tuned_for] \n",
      "[INFO] Wohoo! Relation was matched --- [is_similar_to] --merged --> [has_similarities_with] \n",
      "[INFO] ------- Extracting Entities from the Document 5\n",
      "{'entities': [{'name': 'LLMs', 'label': 'Methodology'}, {'name': 'real-time threat intelligence', 'label': 'Process'}, {'name': 'computational resources', 'label': 'Data Structure'}, {'name': 'knowledge graph', 'label': 'Data Structure'}, {'name': 'GPT-3.5', 'label': 'Model'}, {'name': 'TTP classification', 'label': 'Technique'}, {'name': 'manually labeled datasets', 'label': 'Data Structure'}, {'name': 'inter-entity relationships', 'label': 'Data Structure'}, {'name': 'malware', 'label': 'Entity'}, {'name': 'PowerShell', 'label': 'Tool'}, {'name': 'Llama2-7B', 'label': 'Model'}, {'name': 'entity extraction', 'label': 'Technique'}, {'name': 'epochs', 'label': 'Data Structure'}]}\n",
      "[Entity(name=malware, label=Entity, properties=embeddings=array([-0.03017815, -0.01199993, -0.00061923, ...,  0.00038222,\n",
      "       -0.01887033, -0.00437338])), Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.02329609,  0.01235883, -0.01369327, ..., -0.00109288,\n",
      "       -0.01063987,  0.00071635])), Entity(name=epochs, label=Data_Structure, properties=embeddings=array([-0.03965953,  0.00528157, -0.01367628, ...,  0.00754113,\n",
      "        0.0087681 ,  0.00978417])), Entity(name=real time threat intelligence, label=Process, properties=embeddings=array([ 0.0024493 , -0.02816956, -0.00345389, ..., -0.01195725,\n",
      "       -0.00210442,  0.00712263])), Entity(name=llms, label=Methodology, properties=embeddings=array([-0.01535019,  0.02322746, -0.02159394, ...,  0.00408897,\n",
      "        0.00071315,  0.00551295])), Entity(name=manually labeled datasets, label=Data_Structure, properties=embeddings=array([-0.00794322,  0.0043429 , -0.01943028, ..., -0.00838394,\n",
      "       -0.01523456,  0.00540519])), Entity(name=computational resources, label=Data_Structure, properties=embeddings=array([-0.02780924,  0.00102415, -0.02233749, ...,  0.00097857,\n",
      "       -0.01587218,  0.00278315])), Entity(name=entity extraction, label=Technique, properties=embeddings=array([ 0.01751037, -0.00416903, -0.01141112, ..., -0.00211178,\n",
      "       -0.00604253,  0.00075427])), Entity(name=inter entity relationships, label=Data_Structure, properties=embeddings=array([-0.04229755, -0.00577634, -0.02132191, ..., -0.0022291 ,\n",
      "       -0.00447056, -0.00417207])), Entity(name=powershell, label=Tool, properties=embeddings=array([-0.01994661, -0.00193481, -0.00637242, ..., -0.00707391,\n",
      "       -0.00590627,  0.00107447])), Entity(name=ttp classification, label=Technique, properties=embeddings=array([ 0.00050722,  0.00194431, -0.01813745, ...,  0.00069212,\n",
      "       -0.00575077,  0.02207098])), Entity(name=knowledge graph, label=Data_Structure, properties=embeddings=array([ 0.00967516, -0.00251331, -0.02177497, ..., -0.00496721,\n",
      "       -0.01682499,  0.01573904])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.0367532 , -0.00356611, -0.01605994, ...,  0.00153844,\n",
      "       -0.01379024,  0.01271022]))]\n",
      "[INFO] Wohoo! Entity was matched --- [llms:Methodology] --merged--> [llm tikg:Methodology]\n",
      "[INFO] Wohoo! Entity was matched --- [entity extraction:Technique] --merged--> [entity extraction:Task]\n",
      "[INFO] Wohoo! Entity was matched --- [ttp classification:Technique] --merged--> [ttp classification:Methodology]\n",
      "[INFO] ------- Extracting Relations from the Document 5\n",
      "{'relationships': [{'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'ttp classification', 'label': 'Methodology'}, 'name': 'used for'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'entity extraction', 'label': 'Task'}, 'name': 'performs'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'entity extraction', 'label': 'Task'}, 'name': 'performs'}, {'startNode': {'name': 'manually labeled datasets', 'label': 'Data_Structure'}, 'endNode': {'name': 'ttp classification', 'label': 'Methodology'}, 'name': 'improves'}, {'startNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'endNode': {'name': 'real time threat intelligence', 'label': 'Process'}, 'name': 'supports'}, {'startNode': {'name': 'inter entity relationships', 'label': 'Data_Structure'}, 'endNode': {'name': 'entity extraction', 'label': 'Task'}, 'name': 'complicates'}, {'startNode': {'name': 'powershell', 'label': 'Tool'}, 'endNode': {'name': 'malware', 'label': 'Entity'}, 'name': 'used by'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-1] Aie; there are some isolated entities without relations [Entity(name=epochs, label=Data_Structure, properties=embeddings=array([-0.03965953,  0.00528157, -0.01367628, ...,  0.00754113,\n",
      "        0.0087681 ,  0.00978417])), Entity(name=llm tikg, label=Methodology, properties=embeddings=array([ 0.00613084, -0.01793901, -0.02029365, ...,  0.00590774,\n",
      "       -0.00362842,  0.00754116])), Entity(name=computational resources, label=Data_Structure, properties=embeddings=array([-0.02780924,  0.00102415, -0.02233749, ...,  0.00097857,\n",
      "       -0.01587218,  0.00278315]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'epochs', 'label': 'Data_Structure'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'influences'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'computational resources', 'label': 'Data_Structure'}, 'name': 'requires'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-2] Aie; there are some isolated entities without relations [Entity(name=malware, label=Entity, properties=embeddings=array([-0.03017815, -0.01199993, -0.00061923, ...,  0.00038222,\n",
      "       -0.01887033, -0.00437338])), Entity(name=llama2 7b, label=Model, properties=embeddings=array([-0.02329609,  0.01235883, -0.01369327, ..., -0.00109288,\n",
      "       -0.01063987,  0.00071635])), Entity(name=real time threat intelligence, label=Process, properties=embeddings=array([ 0.0024493 , -0.02816956, -0.00345389, ..., -0.01195725,\n",
      "       -0.00210442,  0.00712263])), Entity(name=manually labeled datasets, label=Data_Structure, properties=embeddings=array([-0.00794322,  0.0043429 , -0.01943028, ..., -0.00838394,\n",
      "       -0.01523456,  0.00540519])), Entity(name=entity extraction, label=Task, properties=embeddings=array([-0.00535627, -0.00183553, -0.00696843, ...,  0.00140995,\n",
      "       -0.01277258,  0.00705865])), Entity(name=inter entity relationships, label=Data_Structure, properties=embeddings=array([-0.04229755, -0.00577634, -0.02132191, ..., -0.0022291 ,\n",
      "       -0.00447056, -0.00417207])), Entity(name=powershell, label=Tool, properties=embeddings=array([-0.01994661, -0.00193481, -0.00637242, ..., -0.00707391,\n",
      "       -0.00590627,  0.00107447])), Entity(name=ttp classification, label=Methodology, properties=embeddings=array([-0.00939821,  0.00161567, -0.02053983, ..., -0.00390094,\n",
      "       -0.00699803,  0.01610363])), Entity(name=knowledge graph, label=Data_Structure, properties=embeddings=array([ 0.00967516, -0.00251331, -0.02177497, ..., -0.00496721,\n",
      "       -0.01682499,  0.01573904])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.0367532 , -0.00356611, -0.01605994, ...,  0.00153844,\n",
      "       -0.01379024,  0.01271022]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'ttp classification', 'label': 'Methodology'}, 'name': 'used for'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'entity extraction', 'label': 'Task'}, 'name': 'performs'}, {'startNode': {'name': 'llama2 7b', 'label': 'Model'}, 'endNode': {'name': 'entity extraction', 'label': 'Task'}, 'name': 'performs'}, {'startNode': {'name': 'entity extraction', 'label': 'Task'}, 'endNode': {'name': 'inter entity relationships', 'label': 'Data_Structure'}, 'name': 'identifies'}, {'startNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'endNode': {'name': 'inter entity relationships', 'label': 'Data_Structure'}, 'name': 'contains'}, {'startNode': {'name': 'manually labeled datasets', 'label': 'Data_Structure'}, 'endNode': {'name': 'ttp classification', 'label': 'Methodology'}, 'name': 'improves'}, {'startNode': {'name': 'malware', 'label': 'Entity'}, 'endNode': {'name': 'powershell', 'label': 'Tool'}, 'name': 'uses'}, {'startNode': {'name': 'real time threat intelligence', 'label': 'Process'}, 'endNode': {'name': 'knowledge graph', 'label': 'Data_Structure'}, 'name': 'updates'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-3] Aie; there are some isolated entities without relations [Entity(name=epochs, label=Data_Structure, properties=embeddings=array([-0.03965953,  0.00528157, -0.01367628, ...,  0.00754113,\n",
      "        0.0087681 ,  0.00978417])), Entity(name=llm tikg, label=Methodology, properties=embeddings=array([ 0.00613084, -0.01793901, -0.02029365, ...,  0.00590774,\n",
      "       -0.00362842,  0.00754116])), Entity(name=computational resources, label=Data_Structure, properties=embeddings=array([-0.02780924,  0.00102415, -0.02233749, ...,  0.00097857,\n",
      "       -0.01587218,  0.00278315]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'epochs', 'label': 'Data_Structure'}, 'endNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'name': 'influences'}, {'startNode': {'name': 'llm tikg', 'label': 'Methodology'}, 'endNode': {'name': 'computational resources', 'label': 'Data_Structure'}, 'name': 'requires'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [used_by] --merged --> [used_for] \n"
     ]
    }
   ],
   "source": [
    "kg = itext2kg.build_graph(sections=distilled_docs[0], ent_threshold=0.7, rel_threshold=0.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We construct the second graph, noting that we already have an existing knowledge graph (for the first article)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] ------- Extracting Entities from the Document 1\n",
      "{'entities': [{'name': 'Cyber Threat Intelligence', 'label': 'Data Structure'}, {'name': 'Large Language Models', 'label': 'Methodology'}, {'name': 'Knowledge Graphs', 'label': 'Data Structure'}, {'name': 'Llama 2 7B chat', 'label': 'Model'}, {'name': 'Llama 70B chat', 'label': 'Model'}, {'name': 'Mistral', 'label': 'Model'}, {'name': 'Zephyr', 'label': 'Model'}, {'name': 'Prompt Engineering', 'label': 'Technique'}, {'name': 'Link Prediction', 'label': 'Technique'}]}\n",
      "[Entity(name=large language models, label=Methodology, properties=embeddings=array([-0.00911226,  0.00577835, -0.02530644, ...,  0.00196522,\n",
      "       -0.01056079,  0.0010359 ])), Entity(name=llama 70b chat, label=Model, properties=embeddings=array([-0.02101934, -0.0112489 , -0.0145149 , ..., -0.00411553,\n",
      "       -0.00812611,  0.00033667])), Entity(name=zephyr, label=Model, properties=embeddings=array([-0.02932358, -0.00352876, -0.00856502, ...,  0.00673707,\n",
      "       -0.00706773,  0.00075633])), Entity(name=mistral, label=Model, properties=embeddings=array([-1.57699929e-02, -5.84310694e-03, -1.23287983e-02, ...,\n",
      "        9.44069470e-05,  1.01220282e-03,  1.76801051e-02])), Entity(name=cyber threat intelligence, label=Data_Structure, properties=embeddings=array([-0.02215733, -0.02427665, -0.01456764, ..., -0.01640413,\n",
      "       -0.01011125, -0.00492441])), Entity(name=knowledge graphs, label=Data_Structure, properties=embeddings=array([ 4.75755669e-03, -2.23485753e-05, -2.47529168e-02, ...,\n",
      "       -1.17194327e-03, -1.14935327e-02,  1.09686761e-02])), Entity(name=llama 2 7b chat, label=Model, properties=embeddings=array([-0.02112724,  0.00317111, -0.01430733, ..., -0.00080403,\n",
      "       -0.00572659, -0.00309422])), Entity(name=prompt engineering, label=Technique, properties=embeddings=array([-0.00532502, -0.00012634, -0.0167981 , ..., -0.01088803,\n",
      "        0.00067679, -0.0025839 ])), Entity(name=link prediction, label=Technique, properties=embeddings=array([ 0.01055078, -0.01596783, -0.01434843, ..., -0.00415012,\n",
      "       -0.01257745,  0.00462144]))]\n",
      "[INFO] ------- Extracting Relations from the Document 1\n",
      "{'relationships': [{'startNode': {'name': 'large language models', 'label': 'Methodology'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'extracts information from'}, {'startNode': {'name': 'knowledge graphs', 'label': 'Data_Structure'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'organizes'}, {'startNode': {'name': 'llama 2 7b chat', 'label': 'Model'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'processes'}, {'startNode': {'name': 'llama 70b chat', 'label': 'Model'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'processes'}, {'startNode': {'name': 'prompt engineering', 'label': 'Technique'}, 'endNode': {'name': 'large language models', 'label': 'Methodology'}, 'name': 'enhances performance of'}, {'startNode': {'name': 'link prediction', 'label': 'Technique'}, 'endNode': {'name': 'knowledge graphs', 'label': 'Data_Structure'}, 'name': 'improves'}, {'startNode': {'name': 'mistral', 'label': 'Model'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'extracts triples from'}, {'startNode': {'name': 'zephyr', 'label': 'Model'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'extracts triples from'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] ------- Extracting Entities from the Document 2\n",
      "{'entities': [{'name': 'LLMs', 'label': 'Technology'}, {'name': 'CTI', 'label': 'Data Type'}, {'name': 'KG', 'label': 'Data Structure'}, {'name': 'GPT-3.5', 'label': 'Model'}, {'name': 'Mistral 7B Instruct', 'label': 'Model'}, {'name': 'Zephyr', 'label': 'Model'}, {'name': 'Llama 2', 'label': 'Model'}, {'name': 'ROUGE', 'label': 'Metric'}, {'name': 'TuckER', 'label': 'Model'}]}\n",
      "[Entity(name=kg, label=Data_Structure, properties=embeddings=array([-0.01725112, -0.01277054, -0.0067052 , ..., -0.00622153,\n",
      "        0.01098439,  0.01942785])), Entity(name=llama 2, label=Model, properties=embeddings=array([-0.0244907 ,  0.01406018, -0.02019313, ...,  0.00519124,\n",
      "       -0.00972902, -0.00215599])), Entity(name=zephyr, label=Model, properties=embeddings=array([-0.02932358, -0.00352876, -0.00856502, ...,  0.00673707,\n",
      "       -0.00706773,  0.00075633])), Entity(name=tucker, label=Model, properties=embeddings=array([-0.02200798, -0.01699331, -0.01177883, ...,  0.01368316,\n",
      "       -0.01175551,  0.01075253])), Entity(name=llms, label=Technology, properties=embeddings=array([-0.02011256,  0.03222934, -0.01485786, ...,  0.00736692,\n",
      "       -0.00728493,  0.0082642 ])), Entity(name=cti, label=Data_Type, properties=embeddings=array([-0.02064756, -0.01789579, -0.01118364, ...,  0.0021346 ,\n",
      "       -0.01453918, -0.0102459 ])), Entity(name=rouge, label=Metric, properties=embeddings=array([-0.02839964, -0.00981262, -0.01279165, ...,  0.01082069,\n",
      "        0.00144731,  0.00847762])), Entity(name=gpt 3.5, label=Model, properties=embeddings=array([-0.03675887, -0.00354618, -0.01606479, ...,  0.00152203,\n",
      "       -0.01378699,  0.01270167])), Entity(name=mistral 7b instruct, label=Model, properties=embeddings=array([-0.01136199, -0.00040766, -0.01676288, ..., -0.01045963,\n",
      "       -0.00268704,  0.00773313]))]\n",
      "[INFO] Wohoo! Entity was matched --- [llama 2:Model] --merged--> [llama 2 7b chat:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [mistral 7b instruct:Model] --merged--> [mistral:Model]\n",
      "[INFO] ------- Extracting Relations from the Document 2\n",
      "{'relationships': [{'startNode': {'name': 'llama 2 7b chat', 'label': 'Model'}, 'endNode': {'name': 'rouge', 'label': 'Metric'}, 'name': 'evaluated by'}, {'startNode': {'name': 'zephyr', 'label': 'Model'}, 'endNode': {'name': 'llama 2 7b chat', 'label': 'Model'}, 'name': 'outperforms'}, {'startNode': {'name': 'mistral', 'label': 'Model'}, 'endNode': {'name': 'llama 2 7b chat', 'label': 'Model'}, 'name': 'outperforms'}, {'startNode': {'name': 'gpt 3.5', 'label': 'Model'}, 'endNode': {'name': 'cti', 'label': 'Data_Type'}, 'name': 'extracts'}, {'startNode': {'name': 'tucker', 'label': 'Model'}, 'endNode': {'name': 'kg', 'label': 'Data_Structure'}, 'name': 'trained on'}, {'startNode': {'name': 'llms', 'label': 'Technology'}, 'endNode': {'name': 'cti', 'label': 'Data_Type'}, 'name': 'extracts'}, {'startNode': {'name': 'llms', 'label': 'Technology'}, 'endNode': {'name': 'kg', 'label': 'Data_Structure'}, 'name': 'constructs'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [extracts] --merged --> [extracts_information_from] \n",
      "[INFO] Wohoo! Relation was matched --- [extracts] --merged --> [extracts_information_from] \n",
      "[INFO] ------- Extracting Entities from the Document 3\n",
      "{'entities': [{'label': 'Methodology', 'name': 'TTPHunter'}, {'label': 'Methodology', 'name': 'LADDER'}, {'label': 'Concept', 'name': 'False Positive Rate'}, {'label': 'Concept', 'name': 'Scalability Challenges'}, {'label': 'Concept', 'name': 'Critical Information'}, {'label': 'Concept', 'name': 'Language Constraints'}, {'label': 'Concept', 'name': 'Predefined Schemas'}, {'label': 'Technique', 'name': 'Machine Learning'}, {'label': 'Technique', 'name': 'Information Extraction'}, {'label': 'Field', 'name': 'Cyber Threat Intelligence (CTI)'}, {'label': 'Methodology', 'name': 'TuckER'}, {'label': 'Concept', 'name': 'Transductive Link Prediction'}, {'label': 'Metric', 'name': 'Accuracy'}, {'label': 'Metric', 'name': 'F1-score'}, {'label': 'Metric', 'name': 'ROUGE'}, {'label': 'Model', 'name': 'Llama 2'}, {'label': 'Model', 'name': '13B Chat Model'}, {'label': 'Model', 'name': '70B Chat Model'}, {'label': 'Model', 'name': 'Mistral'}, {'label': 'Model', 'name': 'Zephyr'}, {'label': 'Model', 'name': 'Guidance Model'}, {'label': 'Framework', 'name': 'MITRE ATT&CK'}]}\n",
      "[Entity(name=ladder, label=Methodology, properties=embeddings=array([-0.03455975, -0.02132156, -0.00742427, ..., -0.00362885,\n",
      "       -0.0011657 , -0.00102891])), Entity(name=machine learning, label=Technique, properties=embeddings=array([-1.32371530e-02, -5.11907972e-05, -9.21922098e-03, ...,\n",
      "       -5.05077890e-03, -3.76505493e-03,  1.66565729e-02])), Entity(name=false positive rate, label=Concept, properties=embeddings=array([-0.02356641,  0.03739743, -0.00965432, ..., -0.01021348,\n",
      "       -0.00027601,  0.0032909 ])), Entity(name=accuracy, label=Metric, properties=embeddings=array([-1.16166366e-02,  1.81923289e-02, -1.71347221e-02, ...,\n",
      "        8.32151156e-05, -1.27281742e-03,  1.08790607e-02])), Entity(name=llama 2, label=Model, properties=embeddings=array([-0.02448273,  0.01403198, -0.02019799, ...,  0.00519061,\n",
      "       -0.00973372, -0.00214867])), Entity(name=guidance model, label=Model, properties=embeddings=array([-1.09800696e-02, -8.41970127e-03, -3.15387487e-02, ...,\n",
      "        7.61346542e-05, -3.38130253e-03,  7.43006160e-03])), Entity(name=tucker, label=Methodology, properties=embeddings=array([-0.00698515, -0.01838914, -0.00884513, ...,  0.00528944,\n",
      "       -0.01137234,  0.00529156])), Entity(name=cyber threat intelligence (cti), label=Field, properties=embeddings=array([-0.01296555, -0.01344177, -0.01582312, ..., -0.01718489,\n",
      "       -0.00388727, -0.00173754])), Entity(name=scalability challenges, label=Concept, properties=embeddings=array([-0.04935353,  0.01266451, -0.02354014, ...,  0.00931582,\n",
      "        0.00040022,  0.00070142])), Entity(name=predefined schemas, label=Concept, properties=embeddings=array([-0.01277242,  0.01663791, -0.02849419, ...,  0.00064219,\n",
      "       -0.00355879,  0.01092038])), Entity(name=zephyr, label=Model, properties=embeddings=array([-0.02931561, -0.00355695, -0.00856988, ...,  0.00673644,\n",
      "       -0.00707243,  0.00076366])), Entity(name=language constraints, label=Concept, properties=embeddings=array([-0.01603037,  0.02606252, -0.02238239, ...,  0.00025579,\n",
      "       -0.00573103,  0.00284212])), Entity(name=critical information, label=Concept, properties=embeddings=array([-0.01351113,  0.02212599, -0.01732925, ..., -0.00261399,\n",
      "        0.00309257, -0.01725544])), Entity(name=information extraction, label=Technique, properties=embeddings=array([ 0.01872601, -0.01047386, -0.01483992, ..., -0.00356148,\n",
      "       -0.01184594, -0.00089204])), Entity(name=13b chat model, label=Model, properties=embeddings=array([-0.01985062, -0.01937975, -0.0239487 , ..., -0.00042774,\n",
      "       -0.00506563,  0.00516712])), Entity(name=mistral, label=Model, properties=embeddings=array([-1.57620282e-02, -5.87130012e-03, -1.23336546e-02, ...,\n",
      "        9.37749981e-05,  1.00750318e-03,  1.76874318e-02])), Entity(name=f1 score, label=Metric, properties=embeddings=array([-0.02396641,  0.01382159, -0.00958496, ...,  0.00282305,\n",
      "       -0.0039212 ,  0.0087448 ])), Entity(name=ttphunter, label=Methodology, properties=embeddings=array([-0.01795455, -0.02058818, -0.0182592 , ..., -0.00087633,\n",
      "       -0.01843523,  0.00566419])), Entity(name=mitre att&ck, label=Framework, properties=embeddings=array([-0.01412911,  0.01215322, -0.01875818, ...,  0.00301392,\n",
      "       -0.02216506, -0.00868665])), Entity(name=70b chat model, label=Model, properties=embeddings=array([-0.02048206, -0.02366426, -0.02294483, ..., -0.00213903,\n",
      "       -0.00450236,  0.00424724])), Entity(name=rouge, label=Metric, properties=embeddings=array([-0.02839133, -0.00981639, -0.0127964 , ...,  0.01081628,\n",
      "        0.00143581,  0.00847334])), Entity(name=transductive link prediction, label=Concept, properties=embeddings=array([-0.00221729,  0.00680336, -0.02434708, ..., -0.00036857,\n",
      "       -0.00600015, -0.00188512]))]\n",
      "[INFO] Wohoo! Entity was matched --- [llama 2:Model] --merged--> [llama 2 7b chat:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [tucker:Methodology] --merged--> [tucker:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [cyber threat intelligence (cti):Field] --merged--> [cyber threat intelligence:Data_Structure]\n",
      "[INFO] Wohoo! Entity was matched --- [13b chat model:Model] --merged--> [llama 2 7b chat:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [70b chat model:Model] --merged--> [llama 70b chat:Model]\n",
      "[INFO] ------- Extracting Relations from the Document 3\n",
      "{'relationships': [{'startNode': {'name': 'TTPHunter', 'label': 'Methodology'}, 'endNode': {'name': 'false positive rate', 'label': 'Concept'}, 'name': 'faces limitations with'}, {'startNode': {'name': 'LADDER', 'label': 'Methodology'}, 'endNode': {'name': 'false positive rate', 'label': 'Concept'}, 'name': 'faces limitations with'}, {'startNode': {'name': 'TTPHunter', 'label': 'Methodology'}, 'endNode': {'name': 'scalability challenges', 'label': 'Concept'}, 'name': 'faces limitations with'}, {'startNode': {'name': 'LADDER', 'label': 'Methodology'}, 'endNode': {'name': 'scalability challenges', 'label': 'Concept'}, 'name': 'faces limitations with'}, {'startNode': {'name': 'machine learning', 'label': 'Technique'}, 'endNode': {'name': 'information extraction', 'label': 'Technique'}, 'name': 'used for'}, {'startNode': {'name': 'TuckER', 'label': 'Model'}, 'endNode': {'name': 'transductive link prediction', 'label': 'Concept'}, 'name': 'designed for'}, {'startNode': {'name': 'accuracy', 'label': 'Metric'}, 'endNode': {'name': 'generative models', 'label': 'Concept'}, 'name': 'inadequate for assessing'}, {'startNode': {'name': 'F1 score', 'label': 'Metric'}, 'endNode': {'name': 'generative models', 'label': 'Concept'}, 'name': 'inadequate for assessing'}, {'startNode': {'name': 'ROUGE', 'label': 'Metric'}, 'endNode': {'name': 'reference text', 'label': 'Concept'}, 'name': 'dependent on quality of'}, {'startNode': {'name': 'Llama 2 7B chat', 'label': 'Model'}, 'endNode': {'name': 'triples', 'label': 'Concept'}, 'name': 'struggles with generating'}, {'startNode': {'name': 'Llama 70B chat', 'label': 'Model'}, 'endNode': {'name': 'triples', 'label': 'Concept'}, 'name': 'struggles with generating'}, {'startNode': {'name': 'Mistral', 'label': 'Model'}, 'endNode': {'name': 'incorrect relationships', 'label': 'Concept'}, 'name': 'introduces'}, {'startNode': {'name': 'Zephyr', 'label': 'Model'}, 'endNode': {'name': 'incorrect relationships', 'label': 'Concept'}, 'name': 'introduces'}, {'startNode': {'name': 'guidance model', 'label': 'Model'}, 'endNode': {'name': 'incorrect triples', 'label': 'Concept'}, 'name': 'extracts'}, {'startNode': {'name': 'MITRE ATT&CK', 'label': 'Framework'}, 'endNode': {'name': 'unique entities', 'label': 'Concept'}, 'name': 'reduces number of'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='generative models' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [generative models:Concept] --merged--> [transductive link prediction:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='generative models' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [generative models:Concept] --merged--> [transductive link prediction:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='reference text' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [reference text:Concept] --merged--> [predefined schemas:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='triples' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [triples:Concept] --merged--> [predefined schemas:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='triples' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [triples:Concept] --merged--> [predefined schemas:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='incorrect relationships' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [incorrect relationships:Concept] --merged--> [language constraints:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='incorrect relationships' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [incorrect relationships:Concept] --merged--> [language constraints:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='incorrect triples' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [incorrect triples:Concept] --merged--> [predefined schemas:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='unique entities' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [unique entities:Concept] --merged--> [predefined schemas:Concept]\n",
      "[INFO][ISOLATED ENTITIES][TRY-1] Aie; there are some isolated entities without relations [Entity(name=cyber threat intelligence, label=Data_Structure, properties=embeddings=array([-0.02215733, -0.02427665, -0.01456764, ..., -0.01640413,\n",
      "       -0.01011125, -0.00492441])), Entity(name=critical information, label=Concept, properties=embeddings=array([-0.01351113,  0.02212599, -0.01732925, ..., -0.00261399,\n",
      "        0.00309257, -0.01725544]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'endNode': {'name': 'critical information', 'label': 'Concept'}, 'name': 'contains'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-2] Aie; there are some isolated entities without relations [Entity(name=ladder, label=Methodology, properties=embeddings=array([-0.03455975, -0.02132156, -0.00742427, ..., -0.00362885,\n",
      "       -0.0011657 , -0.00102891])), Entity(name=machine learning, label=Technique, properties=embeddings=array([-1.32371530e-02, -5.11907972e-05, -9.21922098e-03, ...,\n",
      "       -5.05077890e-03, -3.76505493e-03,  1.66565729e-02])), Entity(name=false positive rate, label=Concept, properties=embeddings=array([-0.02356641,  0.03739743, -0.00965432, ..., -0.01021348,\n",
      "       -0.00027601,  0.0032909 ])), Entity(name=accuracy, label=Metric, properties=embeddings=array([-1.16166366e-02,  1.81923289e-02, -1.71347221e-02, ...,\n",
      "        8.32151156e-05, -1.27281742e-03,  1.08790607e-02])), Entity(name=llama 2 7b chat, label=Model, properties=embeddings=array([-0.02112724,  0.00317111, -0.01430733, ..., -0.00080403,\n",
      "       -0.00572659, -0.00309422])), Entity(name=guidance model, label=Model, properties=embeddings=array([-1.09800696e-02, -8.41970127e-03, -3.15387487e-02, ...,\n",
      "        7.61346542e-05, -3.38130253e-03,  7.43006160e-03])), Entity(name=tucker, label=Model, properties=embeddings=array([-0.02200798, -0.01699331, -0.01177883, ...,  0.01368316,\n",
      "       -0.01175551,  0.01075253])), Entity(name=scalability challenges, label=Concept, properties=embeddings=array([-0.04935353,  0.01266451, -0.02354014, ...,  0.00931582,\n",
      "        0.00040022,  0.00070142])), Entity(name=predefined schemas, label=Concept, properties=embeddings=array([-0.01277242,  0.01663791, -0.02849419, ...,  0.00064219,\n",
      "       -0.00355879,  0.01092038])), Entity(name=zephyr, label=Model, properties=embeddings=array([-0.02931561, -0.00355695, -0.00856988, ...,  0.00673644,\n",
      "       -0.00707243,  0.00076366])), Entity(name=language constraints, label=Concept, properties=embeddings=array([-0.01603037,  0.02606252, -0.02238239, ...,  0.00025579,\n",
      "       -0.00573103,  0.00284212])), Entity(name=information extraction, label=Technique, properties=embeddings=array([ 0.01872601, -0.01047386, -0.01483992, ..., -0.00356148,\n",
      "       -0.01184594, -0.00089204])), Entity(name=llama 2 7b chat, label=Model, properties=embeddings=array([-0.02112724,  0.00317111, -0.01430733, ..., -0.00080403,\n",
      "       -0.00572659, -0.00309422])), Entity(name=mistral, label=Model, properties=embeddings=array([-1.57620282e-02, -5.87130012e-03, -1.23336546e-02, ...,\n",
      "        9.37749981e-05,  1.00750318e-03,  1.76874318e-02])), Entity(name=f1 score, label=Metric, properties=embeddings=array([-0.02396641,  0.01382159, -0.00958496, ...,  0.00282305,\n",
      "       -0.0039212 ,  0.0087448 ])), Entity(name=ttphunter, label=Methodology, properties=embeddings=array([-0.01795455, -0.02058818, -0.0182592 , ..., -0.00087633,\n",
      "       -0.01843523,  0.00566419])), Entity(name=mitre att&ck, label=Framework, properties=embeddings=array([-0.01412911,  0.01215322, -0.01875818, ...,  0.00301392,\n",
      "       -0.02216506, -0.00868665])), Entity(name=llama 70b chat, label=Model, properties=embeddings=array([-0.02101934, -0.0112489 , -0.0145149 , ..., -0.00411553,\n",
      "       -0.00812611,  0.00033667])), Entity(name=rouge, label=Metric, properties=embeddings=array([-0.02839133, -0.00981639, -0.0127964 , ...,  0.01081628,\n",
      "        0.00143581,  0.00847334])), Entity(name=transductive link prediction, label=Concept, properties=embeddings=array([-0.00221729,  0.00680336, -0.02434708, ..., -0.00036857,\n",
      "       -0.00600015, -0.00188512]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'TTPHunter', 'label': 'Methodology'}, 'endNode': {'name': 'false positive rate', 'label': 'Concept'}, 'name': 'has limitation'}, {'startNode': {'name': 'LADDER', 'label': 'Methodology'}, 'endNode': {'name': 'scalability challenges', 'label': 'Concept'}, 'name': 'faces'}, {'startNode': {'name': 'machine learning', 'label': 'Technique'}, 'endNode': {'name': 'information extraction', 'label': 'Technique'}, 'name': 'used for'}, {'startNode': {'name': 'TuckER', 'label': 'Model'}, 'endNode': {'name': 'transductive link prediction', 'label': 'Concept'}, 'name': 'designed for'}, {'startNode': {'name': 'Llama 2 7B chat', 'label': 'Model'}, 'endNode': {'name': 'inconsistencies in output formatting', 'label': 'Concept'}, 'name': 'exhibits'}, {'startNode': {'name': 'Llama 2 7B chat', 'label': 'Model'}, 'endNode': {'name': 'complex syntactical structures', 'label': 'Concept'}, 'name': 'struggles with'}, {'startNode': {'name': 'guidance model', 'label': 'Model'}, 'endNode': {'name': 'incorrect triples', 'label': 'Concept'}, 'name': 'extracts'}, {'startNode': {'name': 'MITRE ATT&CK', 'label': 'Framework'}, 'endNode': {'name': 'unique entities', 'label': 'Concept'}, 'name': 'reduces'}, {'startNode': {'name': 'ROUGE', 'label': 'Metric'}, 'endNode': {'name': 'quality of the reference text', 'label': 'Concept'}, 'name': 'depends on'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='inconsistencies in output formatting' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [inconsistencies in output formatting:Concept] --merged--> [language constraints:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='complex syntactical structures' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [complex syntactical structures:Concept] --merged--> [language constraints:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='incorrect triples' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [incorrect triples:Concept] --merged--> [predefined schemas:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='unique entities' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [unique entities:Concept] --merged--> [predefined schemas:Concept]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Concept' name='quality of the reference text' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [quality of the reference text:Concept] --merged--> [language constraints:Concept]\n",
      "[INFO] Wohoo! Relation was matched --- [struggles_with] --merged --> [struggles_with_generating] \n",
      "[INFO][ISOLATED ENTITIES][TRY-3] Aie; there are some isolated entities without relations [Entity(name=accuracy, label=Metric, properties=embeddings=array([-1.16166366e-02,  1.81923289e-02, -1.71347221e-02, ...,\n",
      "        8.32151156e-05, -1.27281742e-03,  1.08790607e-02])), Entity(name=cyber threat intelligence, label=Data_Structure, properties=embeddings=array([-0.02215733, -0.02427665, -0.01456764, ..., -0.01640413,\n",
      "       -0.01011125, -0.00492441])), Entity(name=zephyr, label=Model, properties=embeddings=array([-0.02931561, -0.00355695, -0.00856988, ...,  0.00673644,\n",
      "       -0.00707243,  0.00076366])), Entity(name=critical information, label=Concept, properties=embeddings=array([-0.01351113,  0.02212599, -0.01732925, ..., -0.00261399,\n",
      "        0.00309257, -0.01725544])), Entity(name=mistral, label=Model, properties=embeddings=array([-1.57620282e-02, -5.87130012e-03, -1.23336546e-02, ...,\n",
      "        9.37749981e-05,  1.00750318e-03,  1.76874318e-02])), Entity(name=f1 score, label=Metric, properties=embeddings=array([-0.02396641,  0.01382159, -0.00958496, ...,  0.00282305,\n",
      "       -0.0039212 ,  0.0087448 ])), Entity(name=llama 70b chat, label=Model, properties=embeddings=array([-0.02101934, -0.0112489 , -0.0145149 , ..., -0.00411553,\n",
      "       -0.00812611,  0.00033667]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'accuracy', 'label': 'Metric'}, 'endNode': {'name': 'f1 score', 'label': 'Metric'}, 'name': 'complementary_to'}, {'startNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'endNode': {'name': 'critical information', 'label': 'Concept'}, 'name': 'contains'}, {'startNode': {'name': 'zephyr', 'label': 'Model'}, 'endNode': {'name': 'mistral', 'label': 'Model'}, 'name': 'similar_to'}, {'startNode': {'name': 'llama 70b chat', 'label': 'Model'}, 'endNode': {'name': 'zephyr', 'label': 'Model'}, 'name': 'compared_to'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [extracts] --merged --> [extracts_information_from] \n",
      "[INFO] Wohoo! Relation was matched --- [extracts] --merged --> [extracts_information_from] \n",
      "[INFO] ------- Extracting Entities from the Document 4\n",
      "{'entities': [{'name': 'Llama 2', 'label': 'Open-Source LLM'}, {'name': 'Mistral 7B Instruct', 'label': 'Open-Source LLM'}, {'name': 'Zephyr', 'label': 'Open-Source LLM'}, {'name': 'Cyber Threat Intelligence', 'label': 'Domain'}, {'name': 'Knowledge Graph', 'label': 'Data Structure'}, {'name': 'Few-Shot Prompting', 'label': 'Technique'}, {'name': 'Fine-tuning', 'label': 'Technique'}, {'name': 'Prompt Engineering', 'label': 'Technique'}, {'name': 'Guidance Framework', 'label': 'Framework'}, {'name': 'Link Prediction', 'label': 'Task'}, {'name': 'ROUGE Metrics', 'label': 'Evaluation Metric'}, {'name': 'Ontology', 'label': 'Methodology'}, {'name': 'LoRA', 'label': 'Technique'}, {'name': 'Entity Type Extraction', 'label': 'Task'}]}\n",
      "[Entity(name=zephyr, label=Open_Source_LLM, properties=embeddings=array([-0.02971372,  0.00939889, -0.0060637 , ...,  0.00668537,\n",
      "       -0.01281737,  0.00850186])), Entity(name=fine tuning, label=Technique, properties=embeddings=array([ 0.00864315,  0.00359974, -0.01301005, ...,  0.0030607 ,\n",
      "        0.0104878 ,  0.00334119])), Entity(name=ontology, label=Methodology, properties=embeddings=array([-0.01114686,  0.00914564, -0.02327262, ..., -0.00573371,\n",
      "        0.00458136, -0.00319414])), Entity(name=guidance framework, label=Framework, properties=embeddings=array([-0.01567577,  0.00087424, -0.02547587, ..., -0.01495871,\n",
      "       -0.00601469,  0.0137455 ])), Entity(name=entity type extraction, label=Task, properties=embeddings=array([-0.01291279, -0.00426842, -0.01331237, ...,  0.00308756,\n",
      "       -0.00746352,  0.00672233])), Entity(name=lora, label=Technique, properties=embeddings=array([-0.01741025, -0.00138455, -0.00362205, ...,  0.0177188 ,\n",
      "       -0.01259704,  0.00522134])), Entity(name=llama 2, label=Open_Source_LLM, properties=embeddings=array([-0.02488734,  0.02699292, -0.0176782 , ...,  0.00513715,\n",
      "       -0.01547272,  0.00559289])), Entity(name=link prediction, label=Task, properties=embeddings=array([-0.012263  , -0.01363038, -0.0099204 , ..., -0.00064015,\n",
      "       -0.01932213,  0.01092823])), Entity(name=mistral 7b instruct, label=Open_Source_LLM, properties=embeddings=array([-0.01176441,  0.01250762, -0.01427598, ..., -0.0105019 ,\n",
      "       -0.00842559,  0.0154825 ])), Entity(name=prompt engineering, label=Technique, properties=embeddings=array([-0.00531937, -0.00012568, -0.0168041 , ..., -0.01088746,\n",
      "        0.00067579, -0.00258056])), Entity(name=cyber threat intelligence, label=Domain, properties=embeddings=array([-0.00636701, -0.01745356, -0.01651648, ..., -0.01415167,\n",
      "       -0.00266932, -0.00939322])), Entity(name=rouge metrics, label=Evaluation_Metric, properties=embeddings=array([-0.02943653, -0.01439829, -0.02460144, ...,  0.01527836,\n",
      "       -0.00406591, -0.00053031])), Entity(name=few shot prompting, label=Technique, properties=embeddings=array([ 0.02017757,  0.00231135, -0.01639281, ..., -0.00493846,\n",
      "        0.00215699, -0.00034044])), Entity(name=knowledge graph, label=Data_Structure, properties=embeddings=array([ 0.0096933 , -0.00251766, -0.02176321, ..., -0.00498565,\n",
      "       -0.01681987,  0.01573279]))]\n",
      "[INFO] Wohoo! Entity was matched --- [zephyr:Open_Source_LLM] --merged--> [zephyr:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [guidance framework:Framework] --merged--> [guidance model:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [link prediction:Task] --merged--> [link prediction:Technique]\n",
      "[INFO] Wohoo! Entity was matched --- [cyber threat intelligence:Domain] --merged--> [cyber threat intelligence:Data_Structure]\n",
      "[INFO] Wohoo! Entity was matched --- [knowledge graph:Data_Structure] --merged--> [knowledge graphs:Data_Structure]\n",
      "[INFO] ------- Extracting Relations from the Document 4\n",
      "{'relationships': [{'startNode': {'name': 'llama 2', 'label': 'Open_Source_LLM'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'used for extracting'}, {'startNode': {'name': 'mistral 7b instruct', 'label': 'Open_Source_LLM'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'used for extracting'}, {'startNode': {'name': 'zephyr', 'label': 'Model'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'used for extracting'}, {'startNode': {'name': 'fine tuning', 'label': 'Technique'}, 'endNode': {'name': 'llama 2', 'label': 'Open_Source_LLM'}, 'name': 'applied to'}, {'startNode': {'name': 'fine tuning', 'label': 'Technique'}, 'endNode': {'name': 'mistral 7b instruct', 'label': 'Open_Source_LLM'}, 'name': 'applied to'}, {'startNode': {'name': 'fine tuning', 'label': 'Technique'}, 'endNode': {'name': 'zephyr', 'label': 'Model'}, 'name': 'applied to'}, {'startNode': {'name': 'prompt engineering', 'label': 'Technique'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'used for extracting'}, {'startNode': {'name': 'guidance model', 'label': 'Model'}, 'endNode': {'name': 'entity type extraction', 'label': 'Task'}, 'name': 'equipped with'}, {'startNode': {'name': 'knowledge graphs', 'label': 'Data_Structure'}, 'endNode': {'name': 'link prediction', 'label': 'Technique'}, 'name': 'used for'}, {'startNode': {'name': 'rouge metrics', 'label': 'Evaluation_Metric'}, 'endNode': {'name': 'triples', 'label': 'Data_Structure'}, 'name': 'used to evaluate'}, {'startNode': {'name': 'ontology', 'label': 'Methodology'}, 'endNode': {'name': 'triples', 'label': 'Data_Structure'}, 'name': 'used to enforce'}, {'startNode': {'name': 'lora', 'label': 'Technique'}, 'endNode': {'name': 'llama 2', 'label': 'Open_Source_LLM'}, 'name': 'applied to'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Data_Structure' name='triples' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [triples:Data_Structure] --merged--> [knowledge graphs:Data_Structure]\n",
      "[INFO][INVENTED ENTITIES] Aie; the entities label='Data_Structure' name='triples' properties=EntityProperties(embeddings=None) is invented. Solving it ...\n",
      "[INFO] Wohoo! Entity was matched --- [triples:Data_Structure] --merged--> [knowledge graphs:Data_Structure]\n",
      "[INFO][ISOLATED ENTITIES][TRY-1] Aie; there are some isolated entities without relations [Entity(name=few shot prompting, label=Technique, properties=embeddings=array([ 0.02017757,  0.00231135, -0.01639281, ..., -0.00493846,\n",
      "        0.00215699, -0.00034044]))]. Solving them ...\n",
      "{'relationships': []}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-2] Aie; there are some isolated entities without relations [Entity(name=zephyr, label=Model, properties=embeddings=array([-0.02931561, -0.00355695, -0.00856988, ...,  0.00673644,\n",
      "       -0.00707243,  0.00076366])), Entity(name=fine tuning, label=Technique, properties=embeddings=array([ 0.00864315,  0.00359974, -0.01301005, ...,  0.0030607 ,\n",
      "        0.0104878 ,  0.00334119])), Entity(name=ontology, label=Methodology, properties=embeddings=array([-0.01114686,  0.00914564, -0.02327262, ..., -0.00573371,\n",
      "        0.00458136, -0.00319414])), Entity(name=guidance model, label=Model, properties=embeddings=array([-1.09800696e-02, -8.41970127e-03, -3.15387487e-02, ...,\n",
      "        7.61346542e-05, -3.38130253e-03,  7.43006160e-03])), Entity(name=entity type extraction, label=Task, properties=embeddings=array([-0.01291279, -0.00426842, -0.01331237, ...,  0.00308756,\n",
      "       -0.00746352,  0.00672233])), Entity(name=lora, label=Technique, properties=embeddings=array([-0.01741025, -0.00138455, -0.00362205, ...,  0.0177188 ,\n",
      "       -0.01259704,  0.00522134])), Entity(name=llama 2, label=Open_Source_LLM, properties=embeddings=array([-0.02488734,  0.02699292, -0.0176782 , ...,  0.00513715,\n",
      "       -0.01547272,  0.00559289])), Entity(name=link prediction, label=Technique, properties=embeddings=array([ 0.01055078, -0.01596783, -0.01434843, ..., -0.00415012,\n",
      "       -0.01257745,  0.00462144])), Entity(name=mistral 7b instruct, label=Open_Source_LLM, properties=embeddings=array([-0.01176441,  0.01250762, -0.01427598, ..., -0.0105019 ,\n",
      "       -0.00842559,  0.0154825 ])), Entity(name=prompt engineering, label=Technique, properties=embeddings=array([-0.00531937, -0.00012568, -0.0168041 , ..., -0.01088746,\n",
      "        0.00067579, -0.00258056])), Entity(name=cyber threat intelligence, label=Data_Structure, properties=embeddings=array([-0.02215733, -0.02427665, -0.01456764, ..., -0.01640413,\n",
      "       -0.01011125, -0.00492441])), Entity(name=rouge metrics, label=Evaluation_Metric, properties=embeddings=array([-0.02943653, -0.01439829, -0.02460144, ...,  0.01527836,\n",
      "       -0.00406591, -0.00053031])), Entity(name=few shot prompting, label=Technique, properties=embeddings=array([ 0.02017757,  0.00231135, -0.01639281, ..., -0.00493846,\n",
      "        0.00215699, -0.00034044])), Entity(name=knowledge graphs, label=Data_Structure, properties=embeddings=array([ 4.75755669e-03, -2.23485753e-05, -2.47529168e-02, ...,\n",
      "       -1.17194327e-03, -1.14935327e-02,  1.09686761e-02]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'zephyr', 'label': 'Model'}, 'endNode': {'name': 'entity type extraction', 'label': 'Task'}, 'name': 'supports'}, {'startNode': {'name': 'fine tuning', 'label': 'Technique'}, 'endNode': {'name': 'llama 2', 'label': 'Open_Source_LLM'}, 'name': 'enhances'}, {'startNode': {'name': 'ontology', 'label': 'Methodology'}, 'endNode': {'name': 'knowledge graphs', 'label': 'Data_Structure'}, 'name': 'structures'}, {'startNode': {'name': 'guidance model', 'label': 'Model'}, 'endNode': {'name': 'prompt engineering', 'label': 'Technique'}, 'name': 'utilizes'}, {'startNode': {'name': 'entity type extraction', 'label': 'Task'}, 'endNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'name': 'applies to'}, {'startNode': {'name': 'lora', 'label': 'Technique'}, 'endNode': {'name': 'fine tuning', 'label': 'Technique'}, 'name': 'complements'}, {'startNode': {'name': 'llama 2', 'label': 'Open_Source_LLM'}, 'endNode': {'name': 'link prediction', 'label': 'Technique'}, 'name': 'facilitates'}, {'startNode': {'name': 'mistral 7b instruct', 'label': 'Open_Source_LLM'}, 'endNode': {'name': 'few shot prompting', 'label': 'Technique'}, 'name': 'supports'}, {'startNode': {'name': 'prompt engineering', 'label': 'Technique'}, 'endNode': {'name': 'rouge metrics', 'label': 'Evaluation_Metric'}, 'name': 'optimizes'}, {'startNode': {'name': 'cyber threat intelligence', 'label': 'Data_Structure'}, 'endNode': {'name': 'knowledge graphs', 'label': 'Data_Structure'}, 'name': 'transforms into'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] Wohoo! Relation was matched --- [applies_to] --merged --> [applied_to] \n",
      "[INFO] Wohoo! Relation was matched --- [used_to_evaluate] --merged --> [evaluated_by] \n",
      "[INFO] ------- Extracting Entities from the Document 5\n",
      "{'entities': [{'name': 'LLMs', 'label': 'Methodology'}, {'name': 'KG construction', 'label': 'Process'}, {'name': 'Link Prediction', 'label': 'Process'}, {'name': 'ROUGE', 'label': 'Technique'}, {'name': 'Llama 2 models', 'label': 'Model'}, {'name': 'Fine-tuning', 'label': 'Technique'}, {'name': 'MITRE ATT&CK techniques', 'label': 'Framework'}]}\n",
      "[Entity(name=kg construction, label=Process, properties=embeddings=array([ 0.02338652, -0.01293936, -0.00763063, ..., -0.01072381,\n",
      "        0.00703863,  0.00107699])), Entity(name=fine tuning, label=Technique, properties=embeddings=array([ 0.00862002,  0.00360247, -0.01301157, ...,  0.00305719,\n",
      "        0.01050509,  0.00334156])), Entity(name=llms, label=Methodology, properties=embeddings=array([-0.01535955,  0.02323928, -0.02159254, ...,  0.00409005,\n",
      "        0.0007081 ,  0.00550224])), Entity(name=mitre att&ck techniques, label=Framework, properties=embeddings=array([-0.01837375,  0.00609435, -0.02109183, ..., -0.00367701,\n",
      "       -0.01859248, -0.00504423])), Entity(name=link prediction, label=Process, properties=embeddings=array([ 0.00867769, -0.0205633 , -0.00807419, ..., -0.00504197,\n",
      "       -0.01587836,  0.00258438])), Entity(name=llama 2 models, label=Model, properties=embeddings=array([-0.03331149,  0.01746199, -0.02564896, ...,  0.01029292,\n",
      "       -0.00477298,  0.00191681])), Entity(name=rouge, label=Technique, properties=embeddings=array([-0.00630716, -0.01727989, -0.01069901, ...,  0.00975289,\n",
      "       -0.00081157,  0.0108768 ]))]\n",
      "[INFO] Wohoo! Entity was matched --- [llms:Methodology] --merged--> [llms:Technology]\n",
      "[INFO] Wohoo! Entity was matched --- [mitre att&ck techniques:Framework] --merged--> [mitre att&ck:Framework]\n",
      "[INFO] Wohoo! Entity was matched --- [link prediction:Process] --merged--> [link prediction:Technique]\n",
      "[INFO] Wohoo! Entity was matched --- [llama 2 models:Model] --merged--> [llama 2 7b chat:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [rouge:Technique] --merged--> [rouge:Metric]\n",
      "[INFO] ------- Extracting Relations from the Document 5\n",
      "{'relationships': [{'startNode': {'name': 'llms', 'label': 'Technology'}, 'endNode': {'name': 'kg construction', 'label': 'Process'}, 'name': 'present challenges for'}, {'startNode': {'name': 'llms', 'label': 'Technology'}, 'endNode': {'name': 'link prediction', 'label': 'Technique'}, 'name': 'present challenges for'}, {'startNode': {'name': 'fine tuning', 'label': 'Technique'}, 'endNode': {'name': 'llama 2 7b chat', 'label': 'Model'}, 'name': 'resulted in repetitive output for'}, {'startNode': {'name': 'rouge', 'label': 'Metric'}, 'endNode': {'name': 'kg construction', 'label': 'Process'}, 'name': 'produced lower scores for'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-1] Aie; there are some isolated entities without relations [Entity(name=mitre att&ck, label=Framework, properties=embeddings=array([-0.01412911,  0.01215322, -0.01875818, ...,  0.00301392,\n",
      "       -0.02216506, -0.00868665]))]. Solving them ...\n",
      "{'relationships': []}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO][ISOLATED ENTITIES][TRY-2] Aie; there are some isolated entities without relations [Entity(name=kg construction, label=Process, properties=embeddings=array([ 0.02338652, -0.01293936, -0.00763063, ..., -0.01072381,\n",
      "        0.00703863,  0.00107699])), Entity(name=fine tuning, label=Technique, properties=embeddings=array([ 0.00862002,  0.00360247, -0.01301157, ...,  0.00305719,\n",
      "        0.01050509,  0.00334156])), Entity(name=llms, label=Technology, properties=embeddings=array([-0.02011256,  0.03222934, -0.01485786, ...,  0.00736692,\n",
      "       -0.00728493,  0.0082642 ])), Entity(name=mitre att&ck, label=Framework, properties=embeddings=array([-0.01412911,  0.01215322, -0.01875818, ...,  0.00301392,\n",
      "       -0.02216506, -0.00868665])), Entity(name=link prediction, label=Technique, properties=embeddings=array([ 0.01055078, -0.01596783, -0.01434843, ..., -0.00415012,\n",
      "       -0.01257745,  0.00462144])), Entity(name=llama 2 7b chat, label=Model, properties=embeddings=array([-0.02112724,  0.00317111, -0.01430733, ..., -0.00080403,\n",
      "       -0.00572659, -0.00309422])), Entity(name=rouge, label=Metric, properties=embeddings=array([-0.02839133, -0.00981639, -0.0127964 , ...,  0.01081628,\n",
      "        0.00143581,  0.00847334]))]. Solving them ...\n",
      "{'relationships': [{'startNode': {'name': 'kg construction', 'label': 'Process'}, 'endNode': {'name': 'llms', 'label': 'Technology'}, 'name': 'utilizes'}, {'startNode': {'name': 'fine tuning', 'label': 'Technique'}, 'endNode': {'name': 'llama 2 7b chat', 'label': 'Model'}, 'name': 'applied to'}, {'startNode': {'name': 'link prediction', 'label': 'Technique'}, 'endNode': {'name': 'llms', 'label': 'Technology'}, 'name': 'utilizes'}, {'startNode': {'name': 'rouge', 'label': 'Metric'}, 'endNode': {'name': 'fine tuning', 'label': 'Technique'}, 'name': 'evaluates'}, {'startNode': {'name': 'mitre att&ck', 'label': 'Framework'}, 'endNode': {'name': 'kg construction', 'label': 'Process'}, 'name': 'lacks mapping for'}]}\n",
      "[INFO] Verification of invented entities\n",
      "[INFO] ------- Matching the Document 1 Entities and Relationships with the Existing Global Entities/Relations\n",
      "[INFO] Wohoo! Entity was matched --- [ttphunter:Methodology] --merged--> [ttp classification:Methodology]\n",
      "[INFO] Wohoo! Entity was matched --- [llama 70b chat:Model] --merged--> [llama2 7b:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [llama 2:Open_Source_LLM] --merged--> [llama2 7b:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [entity type extraction:Task] --merged--> [entity extraction:Task]\n",
      "[INFO] Wohoo! Entity was matched --- [llama 2 7b chat:Model] --merged--> [llama2 7b:Model]\n",
      "[INFO] Wohoo! Entity was matched --- [few shot prompting:Technique] --merged--> [few shot learning:Technique]\n",
      "[INFO] Wohoo! Entity was matched --- [link prediction:Technique] --merged--> [link prediction:Methodology]\n",
      "[INFO] Wohoo! Entity was matched --- [information extraction:Technique] --merged--> [named entity recognition:Technique]\n",
      "[INFO] Wohoo! Entity was matched --- [knowledge graphs:Data_Structure] --merged--> [knowledge graph:Data_Structure]\n",
      "[INFO] Wohoo! Entity was matched --- [lora:Technique] --merged--> [low rank adaptation (lora):Technique]\n",
      "[INFO] Wohoo! Entity was matched --- [large language models:Methodology] --merged--> [large language model:Technology]\n",
      "[INFO] Wohoo! Relation was matched --- [outperforms] --merged --> [performs_better_than] \n",
      "[INFO] Wohoo! Relation was matched --- [outperforms] --merged --> [performs_better_than] \n",
      "[INFO] Wohoo! Relation was matched --- [depends_on] --merged --> [relies_on] \n",
      "[INFO] Wohoo! Relation was matched --- [similar_to] --merged --> [has_similarities_with] \n",
      "[INFO] Wohoo! Relation was matched --- [complementary_to] --merged --> [complementary] \n",
      "[INFO] Wohoo! Relation was matched --- [complements] --merged --> [complementary] \n"
     ]
    }
   ],
   "source": [
    "kg2 = itext2kg.build_graph(sections=distilled_docs[1], existing_knowledge_graph=kg, rel_threshold=0.7, ent_threshold=0.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Draw the graph\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final section involves visualizing the constructed knowledge graph using GraphIntegrator. The graph database Neo4j is accessed using specified credentials, and the resulting graph is visualized to provide a visual representation of the relationships and entities extracted from the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itext2kg.graph_integration import GraphIntegrator\n",
    "\n",
    "\n",
    "URI = \"bolt://localhost:7687\"\n",
    "USERNAME = \"neo4j\"\n",
    "PASSWORD = \"##\"\n",
    "\n",
    "GraphIntegrator(uri=URI, username=USERNAME, password=PASSWORD).visualize_graph(knowledge_graph=kg2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}