topic_vector_search_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2ab1eb85",
   "metadata": {},
   "source": [
    "\n",
    "# Topic Vector Search Demonstration\n",
    "\n",
    "This notebook demonstrates how to set up and retrieve documents from a vector store by searching for a topic model embedding. \n",
    "\n",
    "For ease, the BERTopic package is used to create the topic model. However, you can use any topic model pipeline that meets your needs. The last step in the pipeline should generate an embedding which represents the \"centroid\" of the topic. This is the embedding that will be used to retrieve search results. \n",
    "\n",
    "## Steps Overview\n",
    "1. Load the sample posts and store the `PostDocuments` on OpenSearch.\n",
    "2. Train a topic model using the methods defined in `topic_model.py`.\n",
    "3. Explore the results of the topic model, including coherence score and topic diversity.\n",
    "4. Search OpenSearch for posts matching the topic embeddings and evaluate the search performance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ad90900d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import pandas as pd\n",
    "pd.set_option(\"display.max_rows\", 500)\n",
    "pd.set_option(\"display.max_columns\", 100)\n",
    "pd.set_option('display.max_colwidth', 9999)\n",
    "pd.options.display.float_format = '{:.2f}'.format\n",
    "\n",
    "import sys\n",
    "import os\n",
    "sys.path.append(os.path.abspath(os.path.join('src')))\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "env_loaded = load_dotenv()\n",
    "if not env_loaded: \n",
    "    logger.error(\"Environment variables did not load. Did you create .env file in the root of the project?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f3632a5",
   "metadata": {},
   "source": [
    "\n",
    "## Step 1: Load Sample Posts and Store in OpenSearch\n",
    "\n",
    "In this step, we load the sample posts and store them as a `PostDocuments`\n",
    "in OpenSearch.  We're using a pydantic model to structure the data for ease. The\n",
    "model includes some convenience methods for pre-processing the text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "84751ac4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kas/Library/Caches/pypoetry/virtualenvs/topic-vector-search-_YaP3tWX-py3.12/lib/python3.12/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py:13: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from tqdm.autonotebook import tqdm, trange\n",
      "INFO:opensearch:HEAD http://localhost:9200/post_docs [status:200 request:0.012s]\n",
      "INFO:src.source:Loaded 145 posts from sample_posts.json.\n",
      "INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: mps\n",
      "INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: all-MiniLM-L6-v2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index 'post_docs' already exists.\n",
      "145\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kas/Library/Caches/pypoetry/virtualenvs/topic-vector-search-_YaP3tWX-py3.12/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
      "  warnings.warn(\n",
      "INFO:src.source:Completed creating embeddings for posts.\n",
      "INFO:opensearch:POST http://localhost:9200/_bulk [status:200 request:0.098s]\n",
      "INFO:opensearch:POST http://localhost:9200/_bulk [status:200 request:0.064s]\n",
      "INFO:opensearch:POST http://localhost:9200/_bulk [status:200 request:0.058s]\n",
      "INFO:src.source:Successfully stored 145 posts in OpenSearch.\n"
     ]
    }
   ],
   "source": [
    "# Step 1: Load the sample posts and store in OpenSearch\n",
    "from opensearchpy import OpenSearch\n",
    "import time\n",
    "# Import the main function to run the initial setup\n",
    "from src.source import create_index, load_sample_posts, convert_to_pydantic, create_embeddings_for_posts, store_posts_in_opensearch\n",
    "\n",
    "# Initialize OpenSearch client\n",
    "opensearch_client = OpenSearch(\n",
    "    hosts=[{\"host\": \"localhost\", \"port\": 9200}], http_compress=True\n",
    ")\n",
    "\n",
    "# Create the index\n",
    "create_index(opensearch_client)\n",
    "\n",
    "# Load sample posts from the JSON file\n",
    "sample_posts = load_sample_posts(\"sample_posts.json\")\n",
    "print(len(sample_posts))\n",
    "\n",
    "# Convert posts to Pydantic models\n",
    "posts = convert_to_pydantic(sample_posts)\n",
    "\n",
    "# Pre-process text and create embeddings for each post\n",
    "await create_embeddings_for_posts(posts)\n",
    "\n",
    "# Store the posts in OpenSearch using batch upload\n",
    "store_posts_in_opensearch(opensearch_client, posts)\n",
    "\n",
    "# Allow OpenSearch to index the documents\n",
    "time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e7b46cb6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.798s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of posts in OpenSearch: 145\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PostDocument(post_id='185bb492-e993-4f69-9b88-e55b59da7567', post_author='user_95', created_at=datetime.datetime(2023, 10, 24, 12, 55, 53, 722524, tzinfo=TzInfo(UTC)), modified_at=datetime.datetime(2024, 2, 28, 17, 9, 42, 572027, tzinfo=TzInfo(UTC)), post_text=\"Let's paws for a moment to appreciate the majesty of cats 🐱 Their grace and agility never fail to amaze me! 😻 #CatLove #FelineFun\", doc_embedding=[-0.024599889293313026, 0.0070482720620930195, 0.08802028745412827, -0.005882162135094404, -0.004295976832509041, 0.021553533151745796, 0.12377623468637466, -0.05751150846481323, -0.03581945225596428, -0.029644496738910675, 0.005496165249496698, -0.09166023880243301, 0.012848381884396076, 0.04113560542464256, -0.07764243334531784, 0.06196322292089462, -0.07074573636054993, -0.0001087921264115721, -0.03267202898859978, 0.10064049810171127, -0.02325579896569252, -0.009864791296422482, -0.0099931126460433, 0.02681971900165081, -0.07322884351015091, 0.034660059958696365, -0.042337533086538315, 0.022140417248010635, -0.02358122356235981, -0.05172241851687431, -0.052625250071287155, 0.04496075212955475, 0.11396876722574234, 0.07354581356048584, 0.02119922824203968, 0.0072319395840168, -0.01030302420258522, -0.007813145406544209, 0.03498144447803497, 0.02223760262131691, -0.057467054575681686, -0.10162170231342316, 0.007369971834123135, 0.026073245331645012, 0.016008548438549042, -0.012085233815014362, 0.03656909614801407, -0.09855293482542038, -0.014132659882307053, -0.0018015668028965592, -0.11707133799791336, -0.04205458611249924, -0.06335435807704926, -0.011704377830028534, 0.010498384945094585, 0.016543271020054817, 0.018574703484773636, -0.08664130419492722, -0.059017375111579895, 0.030000273138284683, -0.0013356690760701895, 0.030096568167209625, -0.004785099066793919, 0.05248057469725609, -0.022933730855584145, -0.08755138516426086, -0.000784078030847013, 0.032188523560762405, -0.06429247558116913, 0.09765392541885376, 0.014085638336837292, 0.017605258151888847, -0.02360733225941658, -0.024323005229234695, -0.0467795766890049, 0.05599888414144516, 0.01371487695723772, -0.03967215120792389, 0.03354613482952118, -0.02649025432765484, -0.04281022027134895, -0.04559795930981636, -0.03813014179468155, 0.006174392066895962, -0.06780565530061722, -0.02584773674607277, -0.06658078730106354, -0.022351115942001343, -0.05757549777626991, -0.02485044300556183, -0.0701572373509407, -0.018631741404533386, 0.002697000280022621, -0.014608016237616539, -0.03202774003148079, 0.02071215771138668, -0.007852599956095219, -0.10425926744937897, -0.10023140162229538, 0.0703030452132225, 0.02620924822986126, 0.01926392875611782, -0.06959329545497894, 0.05324315279722214, 0.01907297410070896, -0.0073760636150836945, -0.06620147824287415, 0.022681541740894318, 0.031691666692495346, 0.010623631998896599, 0.0456610731780529, -0.03131148964166641, -0.051038358360528946, 0.0030372508335858583, 0.039186615496873856, 0.03093954361975193, -0.08372660726308823, -0.06417951732873917, 0.16554544866085052, 0.00856502540409565, 0.05318517982959747, 0.039650529623031616, 0.014416464604437351, 0.007299323566257954, -0.05317870154976845, -0.05098700523376465, -0.008355838246643543, 6.519780234499967e-34, 0.007432403042912483, -0.014879297465085983, 0.022028792649507523, 0.05837918445467949, 0.09971961379051208, -0.022253410890698433, -0.02927274815738201, 0.01981610618531704, -0.05856028571724892, -0.020673250779509544, -0.08813849091529846, 0.11742503196001053, 0.014447402209043503, -0.07402215152978897, -0.061622798442840576, -0.04018164426088333, 0.013515313155949116, -0.03943150117993355, 0.028114264830946922, 0.025365682318806648, 0.014710787683725357, 0.04331647977232933, 0.056255046278238297, 0.04320286214351654, -0.05547463893890381, -0.10614423453807831, -0.017641525715589523, -0.07450202107429504, 0.01996508240699768, 0.016013160347938538, 0.06337232142686844, -0.022604526951909065, 0.04374002665281296, -0.03415128216147423, -0.04404408484697342, -0.057555027306079865, -0.010107235983014107, -0.04675257205963135, 0.03351867198944092, 0.07204382866621017, -0.0599110908806324, -0.018166759982705116, 0.01582135260105133, -0.005354699678719044, 0.03799109905958176, 0.03750110790133476, 0.023367784917354584, 0.11989369988441467, -0.030820650979876518, 0.048455990850925446, 0.05626466125249863, -0.006183104123920202, 0.042660925537347794, -0.014483899809420109, -0.01630295068025589, 0.009285887703299522, -0.03374519571661949, 0.07664931565523148, -0.0411737859249115, 0.03938508406281471, 0.02500942349433899, -0.03280682489275932, 0.12174105644226074, -0.1539536863565445, 0.0843697115778923, -0.030426211655139923, 0.046099960803985596, 0.002536521991714835, 0.015177971683442593, 0.1068485900759697, -0.055277060717344284, 0.04023302346467972, 0.030010486021637917, -0.05341443046927452, 0.05705957114696503, -0.030083972960710526, 0.02636677958071232, -0.0025990235153585672, -0.010836598463356495, 0.035377632826566696, -0.0164891816675663, 0.0636940449476242, -0.05877629667520523, 0.03020162135362625, -0.009759707376360893, 0.022906702011823654, 0.038523945957422256, -0.11529038846492767, -0.05759803205728531, 0.07195910811424255, -0.08933898061513901, 0.05066018924117088, 0.03981681168079376, -0.08064380288124084, -0.030314233154058456, -1.7252816912385102e-33, 0.043233707547187805, -0.028301391750574112, -0.03037763200700283, 0.04688814654946327, -0.034668199717998505, 0.04084585979580879, 0.034091975539922714, 0.03456336259841919, -0.024262020364403725, 0.07942719757556915, 0.00876648724079132, 0.04774397611618042, 0.04470161348581314, -0.11528671532869339, -0.04100993275642395, -0.03184107318520546, -0.011816938407719135, -0.018155043944716454, 0.03554685413837433, 0.05100030079483986, -0.027583863586187363, 0.0785209909081459, -0.07124818861484528, 0.05053560435771942, 0.04576461762189865, 0.021547870710492134, 0.04409143701195717, -0.05236084759235382, 0.021187856793403625, -0.06640943884849548, 0.08025963604450226, -0.05221976712346077, -0.09426309913396835, 0.02104569599032402, 0.015823431313037872, 0.03480175510048866, -0.029382972046732903, -0.04118369147181511, -0.08451767265796661, 0.06970202922821045, -0.032400622963905334, 0.008731800131499767, 0.010015349835157394, 0.04580215364694595, -0.000780916481744498, -0.04496494680643082, -0.04552668705582619, -0.023466283455491066, 0.007507318630814552, 0.027932850643992424, -0.02670133113861084, -0.10648511350154877, -0.021349670365452766, 0.027102060616016388, 0.006962382234632969, 0.0149019043892622, 0.08111727237701416, 0.013038773089647293, -0.06020506098866463, 0.01401776634156704, -0.031928230077028275, -0.01489749550819397, -0.02745467610657215, 0.05793100968003273, -0.018769994378089905, -0.03761854022741318, -0.00821333471685648, -0.003223527455702424, -0.06401820480823517, -0.027312561869621277, 0.03614276647567749, 0.013636388815939426, -0.10497308522462845, -0.05040654167532921, 0.016535187140107155, 0.10655232518911362, 0.07743583619594574, -0.04203048348426819, -0.07394625246524811, -0.1024227887392044, -0.009054954163730145, -0.008821515366435051, -0.004452801775187254, 0.028087193146348, 0.0049917977303266525, -0.0042714946903288364, 0.08382894098758698, 0.096869096159935, 0.027426498010754585, 0.0025937361642718315, 0.042017385363578796, 0.08015833795070648, 0.06973142921924591, -0.06669355183839798, 0.041519105434417725, -2.8646052641079223e-08, -0.05336882174015045, 0.027407752349972725, -0.09456636011600494, -0.02468417026102543, 0.037656597793102264, 0.04226170480251312, -0.01224735751748085, -0.10172165930271149, -0.06590767204761505, -0.02154485136270523, 0.08294504135847092, 0.05574310943484306, -0.027512328699231148, 0.03956316411495209, 0.05932727828621864, 0.012690281495451927, -0.027564307674765587, 0.020545927807688713, 0.023251788690686226, 0.010386865586042404, -0.04947938770055771, 0.0328882560133934, -0.05584833398461342, -0.07024192065000534, -0.09749562293291092, -0.029429899528622627, -0.07426785677671432, -0.050283364951610565, -0.04063636437058449, 0.02211422100663185, 0.08414135873317719, -0.0023195212706923485, 0.03584245219826698, -0.037728071212768555, 0.07290459424257278, -0.027060428634285927, 0.019227702170610428, -0.10790220648050308, 0.01873636804521084, -0.010626021772623062, 0.007380794268101454, 0.04464840888977051, 0.06320735067129135, -0.02275392785668373, -0.08433559536933899, -0.017879808321595192, 0.08941005915403366, 0.022096460685133934, 0.055281344801187515, 0.033261314034461975, 0.00953180342912674, 0.026851320639252663, 0.005269827786833048, 0.007586441468447447, 0.027951637282967567, 0.05251583084464073, 4.591816832544282e-06, 0.060311008244752884, 0.04121537134051323, 0.04048497974872589, 0.0976957306265831, 0.06748633086681366, 0.03453221917152405, -0.012602622620761395])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from src.index import INDEX_NAME\n",
    "from models import PostDocument\n",
    "\n",
    "# Ensure that the data is loaded correctly\n",
    "response = opensearch_client.search(index=INDEX_NAME, body={\"size\": 1000, \"query\": {\"match_all\": {}}})\n",
    "results = response[\"hits\"][\"hits\"]\n",
    "print(f\"Total number of posts in OpenSearch: {len(results)}\")\n",
    "\n",
    "# Display the first post document\n",
    "PostDocument(**results[0]['_source'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72c0f048-9dc8-42e4-896f-d2f3fed05fa8",
   "metadata": {},
   "source": [
    "Above is an example of a structured document. The pydantic model includes methods for pre-processing the post to output nlp-ready text. The embedding is stored on the document as a first-order object for indexing by OpenSearch."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b2d9fa7",
   "metadata": {},
   "source": [
    "\n",
    "## Step 2: Train a Topic Model\n",
    "\n",
    "We use the `TopicModeler` class defined in `topic_model.py` to train a BERTopic model on the sample posts stored in OpenSearch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3efa63bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: mps\n",
      "INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: all-MiniLM-L6-v2\n",
      "INFO:src.topic_model:Output directory already exists: output\n",
      "INFO:src.topic_model:Retrieving embeddings from OpenSearch.\n",
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.092s]\n",
      "INFO:src.topic_model:Retrieved 145 post documents.\n",
      "INFO:src.topic_model:Preprocessing text.\n",
      "INFO:src.topic_model:Extracting embeddings.\n",
      "INFO:src.topic_model:Training BERTopic model.\n",
      "OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
      "INFO:src.topic_model:Finished training BERTopic model with OpenAI representations.\n",
      "INFO:src.topic_model:Generating visualizations.\n",
      "INFO:src.topic_model:Visualizations generated and saved.\n",
      "INFO:src.topic_model:Storing model data.\n",
      "INFO:src.topic_model:Model saved to output/bertopic_model\n",
      "INFO:src.topic_model:Topics and probabilities saved.\n",
      "INFO:src.topic_model:Text with topics saved to output/topic_assignments.csv\n"
     ]
    }
   ],
   "source": [
    "# Step 2: Train a topic model\n",
    "\n",
    "# Import the TopicModeler class\n",
    "from src.topic_model import TopicModeler\n",
    "\n",
    "# Initialize the TopicModeler\n",
    "topic_modeler = TopicModeler(index_name=\"post_docs\")\n",
    "\n",
    "# Run the topic model training process\n",
    "topic_modeler.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83394d0a",
   "metadata": {},
   "source": [
    "\n",
    "## Step 3: Explore the Results of the Topic Model\n",
    "\n",
    "Explore the results of the trained topic model by calculating the coherence score and topic diversity.\n",
    "We also preview the reference posts, BERTopic keywords, and top 5 reference posts for each topic.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "406db764",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: mps\n",
      "INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: all-MiniLM-L6-v2\n",
      "INFO:src.topic_model:Output directory already exists: output\n",
      "INFO:src.topic_model:Loading BERTopic model.\n",
      "2024-09-20 12:52:55,624 - BERTopic - WARNING: You are loading a BERTopic model without explicitly defining an embedding model. If you want to also load in an embedding model, make sure to use `BERTopic.load(my_model, embedding_model=my_embedding_model)`.\n",
      "INFO:src.topic_model:Model loaded from output/bertopic_model\n",
      "INFO:gensim.corpora.dictionary:adding document #0 to Dictionary<0 unique tokens: []>\n",
      "INFO:gensim.corpora.dictionary:built Dictionary<1373 unique tokens: ['#CatLove', '#FelineFun', \"Let's\", 'Their', 'a']...> from 145 documents (total 3137 corpus positions)\n",
      "INFO:gensim.utils:Dictionary lifecycle event {'msg': 'built Dictionary<1373 unique tokens: [\\'#CatLove\\', \\'#FelineFun\\', \"Let\\'s\", \\'Their\\', \\'a\\']...> from 145 documents (total 3137 corpus positions)', 'datetime': '2024-09-20T12:52:55.629716', 'gensim': '4.3.3', 'python': '3.12.0 (main, Mar 18 2024, 22:21:23) [Clang 15.0.0 (clang-1500.0.40.1)]', 'platform': 'macOS-14.0-arm64-arm-64bit', 'event': 'created'}\n",
      "INFO:gensim.topic_coherence.probability_estimation:using ParallelWordOccurrenceAccumulator<processes=9, batch_size=64> to estimate probabilities from sliding windows\n",
      "INFO:gensim.topic_coherence.text_analysis:9 accumulators retrieved from output queue\n",
      "INFO:gensim.topic_coherence.text_analysis:accumulated word occurrence stats for 145 virtual documents\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coherence Score: 0.4449198280131731\n",
      "Topic Diversity: 1.0\n"
     ]
    }
   ],
   "source": [
    "os.environ['TOKENIZERS_PARALLELISM'] = 'true'\n",
    "# Step 3: Explore the results of the topic model\n",
    "\n",
    "from src.topic_model import TopicModeler\n",
    "from gensim.corpora import Dictionary\n",
    "from gensim.models.coherencemodel import CoherenceModel\n",
    "import pandas as pd\n",
    "import json\n",
    "\n",
    "# Load the trained topic model\n",
    "topic_modeler = TopicModeler(index_name=\"post_docs\")\n",
    "topic_model = topic_modeler.load_topic_model()\n",
    "\n",
    "# Load the sample posts\n",
    "with open(\"sample_posts.json\", \"r\") as file:\n",
    "    sample_posts = json.load(file)\n",
    "docs = [post['post_text'] for post in sample_posts]\n",
    "\n",
    "# Calculate coherence score\n",
    "def calculate_coherence_score(topic_model, docs):\n",
    "    try:\n",
    "        topics = topic_model.get_topics()\n",
    "        texts = [doc.split() for doc in docs]\n",
    "        dictionary = Dictionary(texts)\n",
    "        topics_words = [[word for word, _ in topic_model.get_topic(topic)] for topic in topics]\n",
    "        coherence_model = CoherenceModel(topics=topics_words, texts=texts, dictionary=dictionary, coherence=\"c_v\")\n",
    "        coherence_score = coherence_model.get_coherence()\n",
    "        return coherence_score\n",
    "    except Exception as error:\n",
    "        raise RuntimeError(f\"Error calculating coherence score: {error}\") from error\n",
    "\n",
    "# Calculate topic diversity\n",
    "def calculate_topic_diversity(topic_model):\n",
    "    topics = topic_model.get_topics()\n",
    "    topic_ids = list(topics.keys())\n",
    "    topic_diversity = len(set(topic_ids)) / len(topic_ids)\n",
    "    return topic_diversity\n",
    "\n",
    "# Calculate coherence score\n",
    "coherence_score = calculate_coherence_score(topic_model, docs)\n",
    "print(f\"Coherence Score: {coherence_score}\")\n",
    "\n",
    "# Calculate topic diversity\n",
    "topic_diversity = calculate_topic_diversity(topic_model)\n",
    "print(f\"Topic Diversity: {topic_diversity}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29fee2c6",
   "metadata": {},
   "source": [
    "## Review the topics and their top words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7c7b62ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "For this simplified example, 20 posts were generated for each topic. An additional 30 posts\n",
       "were generated on random subjects by OpenAI. The prompt was given a set of\n",
       "keywords for each topic. The goal was to generate a set of diverse and coherent\n",
       "samples for the purposes of this demonstration. If the model were perfectly\n",
       "tuned you'd expect to see 30 posts categorized as noise, and 20 posts in each\n",
       "of the other topics.\n",
       "\n",
       "Cluster parameters (HDBScan):\n",
       "- min_cluster_size=12  # Ensures clusters need at least n-points to form a distinct group\n",
       "- min_samples=5\n",
       "- metric=\"euclidean\"  # Does not support cosine distance with the standard backend, so we use Euclidean\n",
       "- cluster_selection_method=\"eom\",\n",
       "- cluster_selection_epsilon=0.001,  # Making cluster selection more\n",
       "conservative\n",
       "- prediction_data=True\n",
       "\n",
       "The topic model correctly identified the topics in the sample posts with a\n",
       "coherence score of 0.44 and a topic diversity of 1.0.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Markdown\n",
    "\n",
    "text = f\"\"\"\n",
    "For this simplified example, 20 posts were generated for each topic. An additional 30 posts\n",
    "were generated on random subjects by OpenAI. The prompt was given a set of\n",
    "keywords for each topic. The goal was to generate a set of diverse and coherent\n",
    "samples for the purposes of this demonstration. If the model were perfectly\n",
    "tuned you'd expect to see 30 posts categorized as noise, and 20 posts in each\n",
    "of the other topics.\n",
    "\n",
    "Cluster parameters (HDBScan):\n",
    "- min_cluster_size=12  # Ensures clusters need at least n-points to form a distinct group\n",
    "- min_samples=5\n",
    "- metric=\"euclidean\"  # Does not support cosine distance with the standard backend, so we use Euclidean\n",
    "- cluster_selection_method=\"eom\",\n",
    "- cluster_selection_epsilon=0.001,  # Making cluster selection more\n",
    "conservative\n",
    "- prediction_data=True\n",
    "\n",
    "The topic model correctly identified the topics in the sample posts with a\n",
    "coherence score of {round(coherence_score, 2)} and a topic diversity of {topic_diversity}.\n",
    "\"\"\"\n",
    "\n",
    "display(Markdown(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ea474d99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic</th>\n",
       "      <th>Count</th>\n",
       "      <th>Name</th>\n",
       "      <th>Representation</th>\n",
       "      <th>KeyBERT</th>\n",
       "      <th>OpenAI</th>\n",
       "      <th>Representative_Docs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>-1_characters_universe_therapy_walk</td>\n",
       "      <td>[characters, universe, therapy, walk, means, marvel, leaf_fluttering_in_wind, grown, crown, creators]</td>\n",
       "      <td>[marvel, therapy, comic, leaf_fluttering_in_wind, nature, walk, money, grown, creators, characters]</td>\n",
       "      <td>[Marvel Therapy Walk]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>0_cats_cat_love_truly</td>\n",
       "      <td>[cats, cat, love, truly, cat_face, eyes, paw_prints, smiling_cat_with_heart, grinning_cat, glowing_star]</td>\n",
       "      <td>[smiling_cat_with_heart, grinning_cat_with_smiling_eyes, catlove, cat, cats, cat_face, grinning_cat, feline, playfulfelines, smiling_face_with_smiling_eyes]</td>\n",
       "      <td>[Feline Love Bond]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>1_high_speed_rail_highspeedrail</td>\n",
       "      <td>[high, speed, rail, highspeedrail, project, california, future, transportation, progress, speed_trainbridge_at_night]</td>\n",
       "      <td>[highspeedrail, rail, speed_trainbridge_at_night, speed_trainrailway_track, transportation, infrastructure, speed_train, transportationit, railway_tracktrain, speed]</td>\n",
       "      <td>[Future Rail Connectivity]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "      <td>2_music_playlist_song_self</td>\n",
       "      <td>[music, playlist, song, self, time, headphone, share, let, world, feel]</td>\n",
       "      <td>[musical_notessparkling_heart, musical_note, musical_notes, music, musical_notesmiling_face_with_smiling_eyes, rainbowmusical_note, milky_waymusical_note, musical, melodies, musicrecommendation]</td>\n",
       "      <td>[Musical Discovery Journey]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>23</td>\n",
       "      <td>3_water_open_water_wave_swim</td>\n",
       "      <td>[water, open, water_wave, swim, man_swimming, challenge, nature, sunrise, alcatraz, sports_medal]</td>\n",
       "      <td>[man_swimming, water_wave, swim, swimming, waters, swimmers, surfing, dive, openwater, tides]</td>\n",
       "      <td>[Open Water Swimmer]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>21</td>\n",
       "      <td>4_let_community_equality_justice</td>\n",
       "      <td>[let, community, equality, justice, change, solidarity, vote, diversity, activism, africa]</td>\n",
       "      <td>[solidarity, activism, injustices, unite, community, africa, social, socialactivism, kindness, communities]</td>\n",
       "      <td>[Social Justice Activism]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>5</td>\n",
       "      <td>20</td>\n",
       "      <td>5_fog_city_francisco_san</td>\n",
       "      <td>[fog, city, francisco, san, mist, embrace, foggy, like, misty, karl]</td>\n",
       "      <td>[fog, foggy, francisco, mist, mistymornings, waterfront, night_with_starssparkles, san, misty, air]</td>\n",
       "      <td>[Foggy City Vibes]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Topic  Count                                 Name  \\\n",
       "0     -1      2  -1_characters_universe_therapy_walk   \n",
       "1      0     30                0_cats_cat_love_truly   \n",
       "2      1     25      1_high_speed_rail_highspeedrail   \n",
       "3      2     24           2_music_playlist_song_self   \n",
       "4      3     23         3_water_open_water_wave_swim   \n",
       "5      4     21     4_let_community_equality_justice   \n",
       "6      5     20             5_fog_city_francisco_san   \n",
       "\n",
       "                                                                                                          Representation  \\\n",
       "0                  [characters, universe, therapy, walk, means, marvel, leaf_fluttering_in_wind, grown, crown, creators]   \n",
       "1               [cats, cat, love, truly, cat_face, eyes, paw_prints, smiling_cat_with_heart, grinning_cat, glowing_star]   \n",
       "2  [high, speed, rail, highspeedrail, project, california, future, transportation, progress, speed_trainbridge_at_night]   \n",
       "3                                                [music, playlist, song, self, time, headphone, share, let, world, feel]   \n",
       "4                      [water, open, water_wave, swim, man_swimming, challenge, nature, sunrise, alcatraz, sports_medal]   \n",
       "5                             [let, community, equality, justice, change, solidarity, vote, diversity, activism, africa]   \n",
       "6                                                   [fog, city, francisco, san, mist, embrace, foggy, like, misty, karl]   \n",
       "\n",
       "                                                                                                                                                                                              KeyBERT  \\\n",
       "0                                                                                                 [marvel, therapy, comic, leaf_fluttering_in_wind, nature, walk, money, grown, creators, characters]   \n",
       "1                                        [smiling_cat_with_heart, grinning_cat_with_smiling_eyes, catlove, cat, cats, cat_face, grinning_cat, feline, playfulfelines, smiling_face_with_smiling_eyes]   \n",
       "2                               [highspeedrail, rail, speed_trainbridge_at_night, speed_trainrailway_track, transportation, infrastructure, speed_train, transportationit, railway_tracktrain, speed]   \n",
       "3  [musical_notessparkling_heart, musical_note, musical_notes, music, musical_notesmiling_face_with_smiling_eyes, rainbowmusical_note, milky_waymusical_note, musical, melodies, musicrecommendation]   \n",
       "4                                                                                                       [man_swimming, water_wave, swim, swimming, waters, swimmers, surfing, dive, openwater, tides]   \n",
       "5                                                                                         [solidarity, activism, injustices, unite, community, africa, social, socialactivism, kindness, communities]   \n",
       "6                                                                                                 [fog, foggy, francisco, mist, mistymornings, waterfront, night_with_starssparkles, san, misty, air]   \n",
       "\n",
       "                        OpenAI  Representative_Docs  \n",
       "0        [Marvel Therapy Walk]                  NaN  \n",
       "1           [Feline Love Bond]                  NaN  \n",
       "2   [Future Rail Connectivity]                  NaN  \n",
       "3  [Musical Discovery Journey]                  NaN  \n",
       "4         [Open Water Swimmer]                  NaN  \n",
       "5    [Social Justice Activism]                  NaN  \n",
       "6           [Foggy City Vibes]                  NaN  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_modeler.topic_model.get_topic_info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1ef8a275",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{-1: [['characters', 0.37347458805864675],\n",
       "  ['universe', 0.37347458805864675],\n",
       "  ['therapy', 0.37347458805864675],\n",
       "  ['walk', 0.37347458805864675],\n",
       "  ['means', 0.37347458805864675],\n",
       "  ['marvel', 0.37347458805864675],\n",
       "  ['leaf_fluttering_in_wind', 0.37347458805864675],\n",
       "  ['grown', 0.37347458805864675],\n",
       "  ['crown', 0.37347458805864675],\n",
       "  ['creators', 0.37347458805864675]],\n",
       " 0: [['cats', 0.10069224113528003],\n",
       "  ['cat', 0.09413007090890406],\n",
       "  ['love', 0.07134464228587827],\n",
       "  ['truly', 0.056600751203848464],\n",
       "  ['cat_face', 0.056600751203848464],\n",
       "  ['eyes', 0.056600751203848464],\n",
       "  ['paw_prints', 0.056600751203848464],\n",
       "  ['smiling_cat_with_heart', 0.04776083327986644],\n",
       "  ['grinning_cat', 0.04776083327986644],\n",
       "  ['glowing_star', 0.04528060096307877]],\n",
       " 1: [['high', 0.1827207227593195],\n",
       "  ['speed', 0.1322705024807515],\n",
       "  ['rail', 0.1322705024807515],\n",
       "  ['highspeedrail', 0.1228812552914524],\n",
       "  ['project', 0.11295801710357188],\n",
       "  ['california', 0.09689003850077074],\n",
       "  ['future', 0.06817809739488909],\n",
       "  ['transportation', 0.06584846930926899],\n",
       "  ['progress', 0.05124466988788326],\n",
       "  ['speed_trainbridge_at_night', 0.05124466988788326]],\n",
       " 2: [['music', 0.11450874605413072],\n",
       "  ['playlist', 0.06885452208303215],\n",
       "  ['song', 0.058100807495095254],\n",
       "  ['self', 0.052627373147616424],\n",
       "  ['time', 0.048772227793962904],\n",
       "  ['headphone', 0.04650370625275103],\n",
       "  ['share', 0.04650370625275103],\n",
       "  ['let', 0.0451683953977332],\n",
       "  ['world', 0.0445423972032354],\n",
       "  ['feel', 0.04131271324981929]],\n",
       " 3: [['water', 0.18348013603057375],\n",
       "  ['open', 0.169611946387063],\n",
       "  ['water_wave', 0.09465717463207417],\n",
       "  ['swim', 0.08445532492255357],\n",
       "  ['man_swimming', 0.07366421296383219],\n",
       "  ['challenge', 0.07037943743546131],\n",
       "  ['nature', 0.06963971517242898],\n",
       "  ['sunrise', 0.06215931978335558],\n",
       "  ['alcatraz', 0.06215931978335558],\n",
       "  ['sports_medal', 0.06215931978335558]],\n",
       " 4: [['let', 0.11208064625437518],\n",
       "  ['community', 0.10264485647810302],\n",
       "  ['equality', 0.09319379500540631],\n",
       "  ['justice', 0.09319379500540631],\n",
       "  ['change', 0.08553738039841918],\n",
       "  ['solidarity', 0.07863876735382659],\n",
       "  ['vote', 0.06294222567232814],\n",
       "  ['diversity', 0.06294222567232814],\n",
       "  ['activism', 0.06294222567232814],\n",
       "  ['africa', 0.06294222567232814]],\n",
       " 5: [['fog', 0.170831313885732],\n",
       "  ['city', 0.13390343260817125],\n",
       "  ['francisco', 0.11710836848957433],\n",
       "  ['san', 0.11710836848957433],\n",
       "  ['mist', 0.08780080920558613],\n",
       "  ['embrace', 0.07175000957159348],\n",
       "  ['foggy', 0.06526093289470047],\n",
       "  ['like', 0.0627812583751443],\n",
       "  ['misty', 0.05692234638114305],\n",
       "  ['karl', 0.05692234638114305]]}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics = topic_modeler.topic_model.get_topics()\n",
    "topics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21eb0dbb",
   "metadata": {},
   "source": [
    "HDBSCAN is a clustering algorithm that does not force documents into a cluster.\n",
    "Instead, it allows for outliers. As such, topic id [-1] is associated\n",
    "with documents that did not fit well into any cluster.  \n",
    "\n",
    "We could spend time hyper-tuning the model parameters. However, we're not using\n",
    "this model to classify posts. Our goal is to identify topics and associate those\n",
    "topics with keywords/embeddings. So, in evaluating how well the model performs,\n",
    "we want to focus on the keywords. These words will be used to define the 'topic\n",
    "embedding'. which will then be used to retrieve posts on the given topic. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bc2a892",
   "metadata": {},
   "source": [
    "Notes:\n",
    "- The coherence score is a measure of how interpretable the topics are. A higher coherence score indicates that the topics are more coherent and interpretable. This score is relatively low in this case, which suggests that the topics may not be well-defined. But, we're going to see what happens when we search for posts related to a specific topic anyway....\n",
    "- The topic diversity is a measure of how diverse the topics are. A higher topic diversity indicates that the topics are more distinct from each other. In this case, the topic diversity is `1.0` indicating 'perfect' separation. This is because we generated the documents, a real-world dataset would have a lower topic diversity.\n",
    "- If you are planning to use a model like this in production, you should pay\n",
    "close attention to the algorthom that identifies the key words and potentially\n",
    "write your own custom algorthim. c-TF-IDF has a tendency to score words that are\n",
    "\"rare' in the corpus more highly. This can result in these words being returned as\n",
    "keywords for the topic even though they are not included in many documents. One\n",
    "way to handle this is to set the `min-df` and `max-df` parameters in the\n",
    "CountVectorizer model that is passed to the the c-TF-IDF model.\n",
    "- It is often useful to have human-readable labels. So, as part of the topic\n",
    "model pipeline, this model calls OpenAI to generate a 3-word summary of the\n",
    "topic. The prompt passes the set of keywords identified by c-TF-IDF matrix and a set of\n",
    "representative posts. The result is a human-readable label. \n",
    "- It's important to note that the AI-generated labels strongly depend on the\n",
    "  keywords associated with each topic. Notice that OpenAI tries to label the set of\n",
    "  random posts assigned to [-1] based on keywords associated with this outlier\n",
    "  topic. It doesn't mean that the topic is a coherent cluster, however. So, this\n",
    "  topic would normally be ignored. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d33c795",
   "metadata": {},
   "source": [
    "\n",
    "## Step 4: Convert Keywords to Topic Embeddings and Search OpenSearch\n",
    "\n",
    "In this step, we convert the keywords from the BERTopic model into embeddings using the same embedding model (`all-MiniLM-L6-v2`). \n",
    "We then use these embeddings to search OpenSearch for matching posts.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3612a43d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'characters universe therapy walk means marvel leaf_fluttering_in_wind grown crown creators'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywords = topic_model.get_topic(-1)\n",
    "text = \" \".join([word for word, _ in keywords])\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "70501844",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.09it/s]\n",
      "Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.59it/s]\n",
      "Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.67it/s]\n",
      "Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.52it/s]\n",
      "Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.51it/s]\n",
      "Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.48it/s]\n",
      "Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.50it/s]\n"
     ]
    }
   ],
   "source": [
    "# Function to convert keywords to embeddings using the SentenceTransformer model\n",
    "def convert_keywords_to_embeddings(keywords, embedding_model):\n",
    "    text = \" \".join([word for word, _ in keywords])\n",
    "    return embedding_model.encode(text, convert_to_numpy=True)\n",
    "\n",
    "# Convert all topics' keywords to embeddings\n",
    "topic_embeddings = {topic_id: convert_keywords_to_embeddings(topic_model.get_topic(topic_id), topic_modeler.embedding_model) for topic_id in topics}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "82107577",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: mps\n",
      "INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: all-MiniLM-L6-v2\n",
      "INFO:src.search:Searching for similar documents in OpenSearch.\n",
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.029s]\n",
      "INFO:src.search:Found 20 similar documents.\n",
      "INFO:src.search:Searching for similar documents in OpenSearch.\n",
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.014s]\n",
      "INFO:src.search:Found 20 similar documents.\n",
      "INFO:src.search:Searching for similar documents in OpenSearch.\n",
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.017s]\n",
      "INFO:src.search:Found 20 similar documents.\n",
      "INFO:src.search:Searching for similar documents in OpenSearch.\n",
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.015s]\n",
      "INFO:src.search:Found 20 similar documents.\n",
      "INFO:src.search:Searching for similar documents in OpenSearch.\n",
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.014s]\n",
      "INFO:src.search:Found 20 similar documents.\n",
      "INFO:src.search:Searching for similar documents in OpenSearch.\n",
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.013s]\n",
      "INFO:src.search:Found 20 similar documents.\n",
      "INFO:src.search:Searching for similar documents in OpenSearch.\n",
      "INFO:opensearch:POST http://localhost:9200/post_docs/_search [status:200 request:0.012s]\n",
      "INFO:src.search:Found 20 similar documents.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Step 4: Convert Keywords to Topic Embeddings and Search OpenSearch\n",
    "\n",
    "from src.search import Searcher\n",
    "import numpy as np\n",
    "\n",
    "# Initialize the Searcher class\n",
    "searcher = Searcher(index_name=\"post_docs\")\n",
    "\n",
    "# Search OpenSearch for posts matching each topic embedding\n",
    "search_results = {}\n",
    "for topic_id, embedding in topic_embeddings.items():\n",
    "    # Search for similar posts\n",
    "    # Limit the search to the same number of posts generated for each topic\n",
    "    # Compare the top 20 similar documents to posts assigned to the topic by the\n",
    "    # topic model.\n",
    "    top_k = 20\n",
    "    results = searcher.search_similar_documents(embedding, top_k=top_k)\n",
    "    for result in results:\n",
    "        # Add keywords and topic id to results for visualization\n",
    "        result['keywords'] = topic_model.get_topic(topic_id)[0]\n",
    "    search_results[topic_id] = results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b19eb610",
   "metadata": {},
   "source": [
    "\n",
    "## Step 5: Evaluate the Search Results\n",
    "\n",
    "This step evaluates the performance of the search by comparing the posts retrieved from OpenSearch using topic embeddings with the posts assigned to each topic by the BERTopic model.\n",
    "We calculate the number of matches and the match ratio to assess how well the search results align with the topic assignments.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bfd5702d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>text</th>\n",
       "      <th>topic_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>185bb492-e993-4f69-9b88-e55b59da7567</td>\n",
       "      <td>let's paws for a moment to appreciate the majesty of cats cat_face their grace and agility never fail to amaze me smiling_cat_with_heart-eyes catlove felinefun</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25df52d2-1c88-4bf5-9330-57a8ec70252e</td>\n",
       "      <td>did you know that cats spend about 70 of their lives sleeping grinning_cat that's the dream life person_in_bed catnap lazycat</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ea28e66b-66bb-438e-90d1-935a694c8309</td>\n",
       "      <td>whiskers are not just cute accessories for cats they are essential tools for their sensory perception cat let's hear it for whisker power paw_prints catfacts</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                post_id  \\\n",
       "0  185bb492-e993-4f69-9b88-e55b59da7567   \n",
       "1  25df52d2-1c88-4bf5-9330-57a8ec70252e   \n",
       "2  ea28e66b-66bb-438e-90d1-935a694c8309   \n",
       "\n",
       "                                                                                                                                                              text  \\\n",
       "0  let's paws for a moment to appreciate the majesty of cats cat_face their grace and agility never fail to amaze me smiling_cat_with_heart-eyes catlove felinefun   \n",
       "1                                    did you know that cats spend about 70 of their lives sleeping grinning_cat that's the dream life person_in_bed catnap lazycat   \n",
       "2    whiskers are not just cute accessories for cats they are essential tools for their sensory perception cat let's hear it for whisker power paw_prints catfacts   \n",
       "\n",
       "   topic_id  \n",
       "0         0  \n",
       "1         0  \n",
       "2         0  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_assignments = pd.read_csv(\"./output/topic_assignments.csv\")\n",
    "topic_assignments.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "47f80bff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['7b8d7e06-0a25-455a-846b-ba10e6bfe3e0',\n",
       " '98465010-0583-4d93-b071-89a0f8d1a21b']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Group by 'topic' and aggregate 'post_ids'\n",
    "grouped = topic_assignments.groupby('topic_id')['post_id'].apply(list)\n",
    "post_assignments = dict(zip(grouped.index, grouped))\n",
    "post_assignments[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "323514ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Step 5: Evaluate the Search Results\n",
    "\n",
    "# Function to evaluate search results based on topic assignments\n",
    "evaluation = {}\n",
    "def evaluate_search_results(search_results, post_assignments, topic_ids):\n",
    "    for topic_id in list(topic_ids.keys())[0:]: # Only evaluate the topics that were generated\n",
    "        topic_search_result = search_results.get(topic_id, [])\n",
    "\n",
    "        # Extract post ids returned in search results for each topic\n",
    "        search_post_ids = [result['post_id'] for result in topic_search_result]\n",
    "\n",
    "        # Extract post_ids assigned to each topic\n",
    "        assigned_post_ids = post_assignments.get(topic_id, [])\n",
    "\n",
    "    # Compare retrieved documents with actual topic assignments\n",
    "        match_count = sum(1 for post in search_post_ids if post in assigned_post_ids)\n",
    "        evaluation[topic_id] = {\n",
    "            'retrieved_count': len(search_post_ids),\n",
    "            'assigned_count': len(assigned_post_ids),\n",
    "            'matches': match_count,\n",
    "            'match_ratio': match_count / len(search_post_ids) if search_post_ids else 0\n",
    "        }\n",
    "    return evaluation\n",
    "\n",
    "# Evaluate how well the search results match the posts assigned to each topic\n",
    "evaluation_results = evaluate_search_results(search_results, post_assignments, topics)\n",
    "evaluation_df = pd.DataFrame(evaluation_results).T\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "15dd4da2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>retrieved_count</th>\n",
       "      <th>assigned_count</th>\n",
       "      <th>matches</th>\n",
       "      <th>match_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>-1</th>\n",
       "      <td>20.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20.00</td>\n",
       "      <td>30.00</td>\n",
       "      <td>19.00</td>\n",
       "      <td>0.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.00</td>\n",
       "      <td>25.00</td>\n",
       "      <td>19.00</td>\n",
       "      <td>0.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20.00</td>\n",
       "      <td>24.00</td>\n",
       "      <td>19.00</td>\n",
       "      <td>0.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>20.00</td>\n",
       "      <td>23.00</td>\n",
       "      <td>18.00</td>\n",
       "      <td>0.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20.00</td>\n",
       "      <td>21.00</td>\n",
       "      <td>18.00</td>\n",
       "      <td>0.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>20.00</td>\n",
       "      <td>20.00</td>\n",
       "      <td>19.00</td>\n",
       "      <td>0.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    retrieved_count  assigned_count  matches  match_ratio\n",
       "-1            20.00            2.00     2.00         0.10\n",
       " 0            20.00           30.00    19.00         0.95\n",
       " 1            20.00           25.00    19.00         0.95\n",
       " 2            20.00           24.00    19.00         0.95\n",
       " 3            20.00           23.00    18.00         0.90\n",
       " 4            20.00           21.00    18.00         0.90\n",
       " 5            20.00           20.00    19.00         0.95"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluation_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b50d8d1",
   "metadata": {},
   "source": [
    "The matched ratio represents the number of posts assigned to a topic that were returned within the search results for the topic. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c0711c42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.81"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(evaluation_df['match_ratio'].mean(), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f043f32b",
   "metadata": {},
   "source": [
    "Now let's qualitatively eyeball the search results returned for each topic\n",
    "embedding. Do they align with the topics originally used to generate the posts? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c12a7143",
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.generate_posts import topics as generated_topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "164f7a32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Cats, cats, cats',\n",
       " 'Music recommendations',\n",
       " 'Social Activism',\n",
       " 'San Francisco Fog',\n",
       " 'California High Speed Rail',\n",
       " 'Open Water Swimming']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generated_topics = list(generated_topics.keys())\n",
    "generated_topics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1bd4b3d",
   "metadata": {},
   "source": [
    "To get a feel for the search results let's display: \n",
    "- The top five search results when searching for documents using the topic\n",
    "  embedding. \n",
    "- The mean search similarity score for the documents returned. \n",
    "- And the descriptive AI label assigned to each topic by OpenAI as part of the BERTopic pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a601a768",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic AI Label: ['Feline Love Bond']\n",
      "Search Results for Topic Embedding: 0\n",
      "Mean Similarity Score: 0.63\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>post_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.78</td>\n",
       "      <td>Let's paws for a moment to appreciate the majesty of cats 🐱 Their grace and agility never fail to amaze me! 😻 #CatLove #FelineFun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.77</td>\n",
       "      <td>The bond between a cat and its human is truly special and unique 🌟 It's a relationship built on trust, love, and mutual understanding 😻 #CatHumanBond</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.75</td>\n",
       "      <td>Fluffy cats are like living clouds of softness and love 💕 Who can resist their charm and irresistible cuddles? 😻 #FluffyLove</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.75</td>\n",
       "      <td>Every cat has its own unique purr-sonality 😺 Some are adventurous, others are cuddly, but all are special in their own way 🌟 #CatPurrsonality</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.73</td>\n",
       "      <td>The way cats effortlessly navigate their surroundings with grace and agility is truly mesmerizing 🐾 They are the epitome of elegance in motion! 😻 #GracefulCats</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   score  \\\n",
       "0   0.78   \n",
       "1   0.77   \n",
       "2   0.75   \n",
       "3   0.75   \n",
       "4   0.73   \n",
       "\n",
       "                                                                                                                                                         post_text  \n",
       "0                                Let's paws for a moment to appreciate the majesty of cats 🐱 Their grace and agility never fail to amaze me! 😻 #CatLove #FelineFun  \n",
       "1            The bond between a cat and its human is truly special and unique 🌟 It's a relationship built on trust, love, and mutual understanding 😻 #CatHumanBond  \n",
       "2                                     Fluffy cats are like living clouds of softness and love 💕 Who can resist their charm and irresistible cuddles? 😻 #FluffyLove  \n",
       "3                    Every cat has its own unique purr-sonality 😺 Some are adventurous, others are cuddly, but all are special in their own way 🌟 #CatPurrsonality  \n",
       "4  The way cats effortlessly navigate their surroundings with grace and agility is truly mesmerizing 🐾 They are the epitome of elegance in motion! 😻 #GracefulCats  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Topic AI Label: ['Future Rail Connectivity']\n",
      "Search Results for Topic Embedding: 1\n",
      "Mean Similarity Score: 0.7\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>post_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.87</td>\n",
       "      <td>Excited to see the progress made on the High Speed Rail project! This innovative infrastructure will redefine how we travel across California. 🚄🌉 #HighSpeedRail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.83</td>\n",
       "      <td>California's commitment to the High Speed Rail project demonstrates bold leadership in advancing modern transportation solutions. Let's keep the momentum going! 🚄🌉 #HighSpeedRail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.82</td>\n",
       "      <td>Exciting news for California! The High Speed Rail project is making great progress, connecting major cities like never before. 🚄🌉 #HighSpeedRail #Infrastructure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.81</td>\n",
       "      <td>As challenges arise, so does the determination to see the High Speed Rail project through to completion. Together, we can build a better future for California's transportation. 🚄🌉 #HighSpeedRail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.80</td>\n",
       "      <td>Construction of the High Speed Rail is underway, shaping the future of public transportation in California. Stay tuned for updates on this transformative project! 🚄🚧 #HighSpeedRail</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   score  \\\n",
       "0   0.87   \n",
       "1   0.83   \n",
       "2   0.82   \n",
       "3   0.81   \n",
       "4   0.80   \n",
       "\n",
       "                                                                                                                                                                                            post_text  \n",
       "0                                    Excited to see the progress made on the High Speed Rail project! This innovative infrastructure will redefine how we travel across California. 🚄🌉 #HighSpeedRail  \n",
       "1                  California's commitment to the High Speed Rail project demonstrates bold leadership in advancing modern transportation solutions. Let's keep the momentum going! 🚄🌉 #HighSpeedRail  \n",
       "2                                    Exciting news for California! The High Speed Rail project is making great progress, connecting major cities like never before. 🚄🌉 #HighSpeedRail #Infrastructure  \n",
       "3  As challenges arise, so does the determination to see the High Speed Rail project through to completion. Together, we can build a better future for California's transportation. 🚄🌉 #HighSpeedRail  \n",
       "4                Construction of the High Speed Rail is underway, shaping the future of public transportation in California. Stay tuned for updates on this transformative project! 🚄🚧 #HighSpeedRail  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Topic AI Label: ['Musical Discovery Journey']\n",
      "Search Results for Topic Embedding: 2\n",
      "Mean Similarity Score: 0.43\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>post_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.62</td>\n",
       "      <td>🎧 Let the music be your guide as you embark on a journey of self-discovery and emotional exploration through the power of melodies and lyrics. 🌌🎵</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.58</td>\n",
       "      <td>🎧 Dive into a world of soulful melodies with this Must-Listen album that will uplift your spirits and soothe your soul. 🎵 #MusicRecommendation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.56</td>\n",
       "      <td>🎧 Unwind after a long day with this chill playlist that will transport you to a state of relaxation and tranquility. 🌿🎵</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.52</td>\n",
       "      <td>⭐ Explore a hidden gem in the world of music with this Top Pick recommendation that deserves to be heard by music enthusiasts everywhere. 🎧🎶</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.52</td>\n",
       "      <td>👌 Need a mood booster? Look no further than this feel-good playlist that will brighten even the gloomiest of days. 🌈🎵</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   score  \\\n",
       "0   0.62   \n",
       "1   0.58   \n",
       "2   0.56   \n",
       "3   0.52   \n",
       "4   0.52   \n",
       "\n",
       "                                                                                                                                           post_text  \n",
       "0  🎧 Let the music be your guide as you embark on a journey of self-discovery and emotional exploration through the power of melodies and lyrics. 🌌🎵  \n",
       "1     🎧 Dive into a world of soulful melodies with this Must-Listen album that will uplift your spirits and soothe your soul. 🎵 #MusicRecommendation  \n",
       "2                            🎧 Unwind after a long day with this chill playlist that will transport you to a state of relaxation and tranquility. 🌿🎵  \n",
       "3       ⭐ Explore a hidden gem in the world of music with this Top Pick recommendation that deserves to be heard by music enthusiasts everywhere. 🎧🎶  \n",
       "4                              👌 Need a mood booster? Look no further than this feel-good playlist that will brighten even the gloomiest of days. 🌈🎵  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Topic AI Label: ['Open Water Swimmer']\n",
      "Search Results for Topic Embedding: 3\n",
      "Mean Similarity Score: 0.54\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>post_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.86</td>\n",
       "      <td>🏅 Conquer the iconic Alcatraz swim and write your name in the history of open water endurance challenges. 🌊 #Alcatraz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.81</td>\n",
       "      <td>🏅 Alcatraz awaits those brave enough to swim its open waters, a true test of endurance and determination. 🌊 #Alcatraz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.72</td>\n",
       "      <td>🌊 Dive into the open water and let the waves carry you to new adventures! 🏊‍♂️ #OpenWater #Swim</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.71</td>\n",
       "      <td>🏊‍♂️ Triathletes thrive in the open water, combining swimming with cycling and running for the ultimate challenge. 🏅 #Triathlon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.63</td>\n",
       "      <td>🚩 Buoy markers guide the way, marking the path for swimmers braving the open water challenge. 🏅 #Endurance</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   score  \\\n",
       "0   0.86   \n",
       "1   0.81   \n",
       "2   0.72   \n",
       "3   0.71   \n",
       "4   0.63   \n",
       "\n",
       "                                                                                                                         post_text  \n",
       "0            🏅 Conquer the iconic Alcatraz swim and write your name in the history of open water endurance challenges. 🌊 #Alcatraz  \n",
       "1            🏅 Alcatraz awaits those brave enough to swim its open waters, a true test of endurance and determination. 🌊 #Alcatraz  \n",
       "2                                  🌊 Dive into the open water and let the waves carry you to new adventures! 🏊‍♂️ #OpenWater #Swim  \n",
       "3  🏊‍♂️ Triathletes thrive in the open water, combining swimming with cycling and running for the ultimate challenge. 🏅 #Triathlon  \n",
       "4                       🚩 Buoy markers guide the way, marking the path for swimmers braving the open water challenge. 🏅 #Endurance  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Topic AI Label: ['Social Justice Activism']\n",
      "Search Results for Topic Embedding: 4\n",
      "Mean Similarity Score: 0.42\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>post_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.56</td>\n",
       "      <td>🤝 Building a strong community starts with understanding and respecting each other's differences. #Community #Diversity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.53</td>\n",
       "      <td>🌈 Transgender rights are human rights. Let's support and uplift our transgender community. #TransGenderRights #Equality</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.53</td>\n",
       "      <td>🚫 Say no to discrimination in all its forms. Embrace diversity and celebrate uniqueness. #NoHate #Diversity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.52</td>\n",
       "      <td>✊🏿 Black lives matter. Let's work towards ending systemic racism and inequality. #BlackLivesMatter #Equality</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.51</td>\n",
       "      <td>🌍 Let's join hands in solidarity to create a better world for all. #SocialActivism #Change #Solidarity</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   score  \\\n",
       "0   0.56   \n",
       "1   0.53   \n",
       "2   0.53   \n",
       "3   0.52   \n",
       "4   0.51   \n",
       "\n",
       "                                                                                                                 post_text  \n",
       "0   🤝 Building a strong community starts with understanding and respecting each other's differences. #Community #Diversity  \n",
       "1  🌈 Transgender rights are human rights. Let's support and uplift our transgender community. #TransGenderRights #Equality  \n",
       "2              🚫 Say no to discrimination in all its forms. Embrace diversity and celebrate uniqueness. #NoHate #Diversity  \n",
       "3             ✊🏿 Black lives matter. Let's work towards ending systemic racism and inequality. #BlackLivesMatter #Equality  \n",
       "4                   🌍 Let's join hands in solidarity to create a better world for all. #SocialActivism #Change #Solidarity  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Topic AI Label: ['Foggy City Vibes']\n",
      "Search Results for Topic Embedding: 5\n",
      "Mean Similarity Score: 0.63\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>post_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.74</td>\n",
       "      <td>When the fog rolls in, it's like the city takes on a whole new persona. 🌁🎭 San Francisco becomes a stage where mist and light dance in harmony. #FoggyMagic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.73</td>\n",
       "      <td>Get ready to cozy up in San Francisco's iconic fog blanket! 🌁 Embrace the chilly embrace of Karl the Fog as he weaves his misty magic over the Golden Gate. 🌫️ #SFWeather</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.72</td>\n",
       "      <td>San Francisco's fog is a reminder that beauty can be found even in the gloomiest of days. 🌧️🌁 Let's appreciate the artistry of Karl the Fog as he paints the city in shades of gray. #BeautyInFog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.72</td>\n",
       "      <td>When the fog blankets the city, it's like a veil of anonymity that allows San Francisco to reinvent itself with each passing day. 🌁💭 Embrace the ever-changing nature of the city under Karl the Fog's watchful eye. #Reinvention</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.72</td>\n",
       "      <td>In the embrace of the fog, San Francisco takes on a timeless quality. 🌁⏳ Let's savor the moment and appreciate the ephemeral beauty of a misty day in the city. #TimelessSF</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   score  \\\n",
       "0   0.74   \n",
       "1   0.73   \n",
       "2   0.72   \n",
       "3   0.72   \n",
       "4   0.72   \n",
       "\n",
       "                                                                                                                                                                                                                           post_text  \n",
       "0                                                                        When the fog rolls in, it's like the city takes on a whole new persona. 🌁🎭 San Francisco becomes a stage where mist and light dance in harmony. #FoggyMagic  \n",
       "1                                                          Get ready to cozy up in San Francisco's iconic fog blanket! 🌁 Embrace the chilly embrace of Karl the Fog as he weaves his misty magic over the Golden Gate. 🌫️ #SFWeather  \n",
       "2                                  San Francisco's fog is a reminder that beauty can be found even in the gloomiest of days. 🌧️🌁 Let's appreciate the artistry of Karl the Fog as he paints the city in shades of gray. #BeautyInFog  \n",
       "3  When the fog blankets the city, it's like a veil of anonymity that allows San Francisco to reinvent itself with each passing day. 🌁💭 Embrace the ever-changing nature of the city under Karl the Fog's watchful eye. #Reinvention  \n",
       "4                                                        In the embrace of the fog, San Francisco takes on a timeless quality. 🌁⏳ Let's savor the moment and appreciate the ephemeral beauty of a misty day in the city. #TimelessSF  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Display the top search results returned for each topic embedding.\n",
    "topic_ids = list(topics.keys())\n",
    "for topic_id in topic_ids[1:]: # Skip the outlier cluster\n",
    "     topic_model_ai_label = topic_modeler.topic_model.get_topic_info(topic_id)['OpenAI'].values[0]\n",
    "     print(f\"Topic AI Label: {topic_model_ai_label}\")\n",
    "     print(f\"Search Results for Topic Embedding: {topic_id}\")\n",
    "\n",
    "     mean_similarity_search_score = pd.DataFrame(search_results[topic_id])['score'].mean()\n",
    "     # Adjust the mean search score to be between -1 and 1\n",
    "     # Cosine similarity returns a number between -1 and 1, but because OpenSearch relevance scores can’t be below 0, the k-NN plugin adds 1 to get the final score.\n",
    "     print(f\"Mean Similarity Score: {round(mean_similarity_search_score - 1, 2)}\")\n",
    "\n",
    "     # Convert the DataFrame column to a string with left alignment\n",
    "     df = pd.DataFrame(search_results[topic_id])[['score','post_text']].head(5)\n",
    "     df['score'] = df['score'].apply(lambda x: round(x - 1, 2))\n",
    "     display(df)\n",
    "     print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24b69149",
   "metadata": {},
   "source": [
    "# Conclusion "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e05cdf29",
   "metadata": {},
   "source": [
    "- The topic model was able to identify the topics in the sample posts. Given the size of the dataset, this is not a strong model. Many outliers were classed into a topic adding noise to the topics. However, it performs well enough to identify keywords and generate embeddings based on those keywords for each topic.\n",
    "- The search results for each topic embedding were evaluated based on the number of posts assigned to each topic by the topic model that were also returned by the search query. The search results were able to retrieve posts that matched the assigned topics with an average match ratio > 0.90 (not including the outlier topic.)\n",
    "  \n",
    "- When configuring the pipeline for the topic model, consider the following:\n",
    "    - Pre-processing posts to ensure that the unstructured text is handled in a way that the text passed to the embedding has a high level of document parity. This may mean using headings, sub-headings and other contextual data to chunk your documents. Or, text processing such as removing numbers, converting emojis to text, image capturing, handling urls, etc.  \n",
    "    - Keywords and representative posts matter more than overall fit when evaluating the topic model. In order to perform well in search, the topic embedding must represent the centroid of the topic, not the overall word distribution. The more informative the keywords are, the more precise the search results. Generally, you want to use the smallest set of keywords that represent the topic well. This may mean writing a custom algorithm to select representative features and generating an embedding.\n",
    "    - When handling the represenatation step in the pipeline consider pruning more common words and infrequent words so that these do not skew the labeling of the topic. If it would improve the topic model, you may want to do this at an early stage in the pipeline. However, if you do it earlier, remember those words (which may become more frequent over time) will no longer be incorporated in the embedding for the document for search purposes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f58e6738-c2ac-48c8-b8e7-03e638d6fce5",
   "metadata": {},
   "source": [
    "# Teardown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "db308380-7d89-4c59-8496-4a2fdf8ea697",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:opensearch:POST http://localhost:9200/post_docs/_delete_by_query [status:200 request:0.040s]\n",
      "INFO:opensearch:HEAD http://localhost:9200/post_docs [status:200 request:0.003s]\n",
      "INFO:opensearch:DELETE http://localhost:9200/post_docs [status:200 request:0.051s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted 145 documents from index 'post_docs'.\n",
      "Index 'post_docs' deleted successfully.\n"
     ]
    }
   ],
   "source": [
    "from opensearchpy import OpenSearch \n",
    "from src.source import delete_all_documents\n",
    "from src.index import delete_index\n",
    "\n",
    "# Initialize OpenSearch client\n",
    "opensearch_client = OpenSearch(\n",
    "    hosts=[{\"host\": \"localhost\", \"port\": 9200}], http_compress=True\n",
    ")\n",
    "\n",
    "# Delete any existing documents \n",
    "delete_all_documents(opensearch_client)\n",
    "\n",
    "#Delete index \n",
    "delete_index(opensearch_client)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}