qa_engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c515de6b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![Roche logo](images/roche-logo-blue-aligned-right.png)\n",
    "\n",
    "<p>&nbsp;</p>\n",
    "<p>&nbsp;</p>\n",
    "\n",
    "# Building a Q&A engine with LangChain and open-source LLMs\n",
    "## Marek Grzenkowicz\n",
    "\n",
    "#### December 2023\n",
    "\n",
    "<p>&nbsp;</p>\n",
    "<p>&nbsp;</p>\n",
    "\n",
    "![DevAI logo](images/devai-logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a698d5f5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## About Roche\n",
    "\n",
    "![Key Roche Informatics hubs](images/roche-about.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea2265f8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Large Language Models did not appear out of nowhere\n",
    "\n",
    "### Natural language processing (NLP)\n",
    "\n",
    "- Standardized tasks (question answering, summarization, sentiment analysis, ...)\n",
    "- Evaluation benchmarks\n",
    "- **Leaderboards**\n",
    "- Word and sentence **embeddings** (word2vec, GloVe, fastText, ELMo, BERT, ...)\n",
    "\n",
    "### Machine learning\n",
    "\n",
    "- Neural networks\n",
    "- Deep learning\n",
    "- CUDA\n",
    "- Transformer architecture\n",
    "- Attention (ML technique)\n",
    "- Reinforced learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3068c119",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Objectives\n",
    "\n",
    "1. Build an LLM application **without extensive NLP expertise**\n",
    "2. Enable **local development and execution** (easy to replace with cloud-hosted inference APIs later)\n",
    "3. Utilize **open-source models**\n",
    "4. Prototype a solution for a **real business challenge**\n",
    "    1. Use semantic search for proprietary company documents\n",
    "    2. Query the index in natural language"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0def2e66",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tools\n",
    "\n",
    "- [LangChain](https://python.langchain.com/docs/get_started/introduction.html) - framework for developing applications powered by LLMs\n",
    "- [Hugging Face Hub](https://huggingface.co/models) - repository of pre-trained language models\n",
    "  - [Transformers](https://huggingface.co/docs/transformers/index) - downloading the models\n",
    "- [lmsys/fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5-3b-v1.0) - compact, open-source LLM from [lmsys.org](https://lmsys.org) \n",
    "- [ChromaDB](https://www.trychroma.com) - embedding database (vector store)\n",
    "- [Jupyter Notebook](https://jupyter.org/) with the [RISE](https://github.com/damianavila/RISE) extension - IDE with a slideshow feature"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7f7334c",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## It's full of LLMs!\n",
    "\n",
    "![LLM Evolutionary Tree](https://raw.githubusercontent.com/Mooler0410/LLMsPracticalGuide/main/imgs/tree.jpg)\n",
    "\n",
    "Source: [github.com/Mooler0410/LLMsPracticalGuide/](https://github.com/Mooler0410/LLMsPracticalGuide/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8afe1963",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why the `lmsys/fastchat-t5-3b-v1.0` model?\n",
    "\n",
    "- GPU + 4GB memory - 1B parameters at best\n",
    "- CPU + 32GB RAM - 3B parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c7e20a",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- [Open LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) ➡ [`CobraMamba/mamba-gpt-3b-v3`](https://huggingface.co/CobraMamba/mamba-gpt-3b-v3) claims to surpass some 12B models\n",
    "  - but it is slow ➡ [Open LLM performace leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d957d6e",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "- [lmsys/fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5-3b-v1.0)\n",
    "  - _A commercial-friendly, compact, yet powerful chat assistant_\n",
    "  - The first model to actually generate any response on my laptop in reasonable time 🎉"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ee1ac86",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some duct tape first 🙈"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d2e4dd33",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:46:12.732800Z",
     "start_time": "2023-12-08T13:46:09.093080Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-17T21:41:02.423021Z",
     "iopub.status.busy": "2023-08-17T21:41:02.421305Z",
     "iopub.status.idle": "2023-08-17T21:41:02.437203Z",
     "shell.execute_reply": "2023-08-17T21:41:02.431892Z",
     "shell.execute_reply.started": "2023-08-17T21:41:02.422912Z"
    },
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# careful! important warnings may get hidden\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import logging\n",
    "from transformers.utils import logging\n",
    "logging.set_verbosity(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bbd4edbe",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:46:12.746540Z",
     "start_time": "2023-12-08T13:46:12.737016Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:06:50.604289Z",
     "iopub.status.busy": "2023-08-16T22:06:50.601671Z",
     "iopub.status.idle": "2023-08-16T22:06:50.619156Z",
     "shell.execute_reply": "2023-08-16T22:06:50.616908Z",
     "shell.execute_reply.started": "2023-08-16T22:06:50.604128Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# https://github.com/chroma-core/chroma/blob/main/chromadb/__init__.py#L57\n",
    "\n",
    "import sys\n",
    "__import__(\"pysqlite3\")\n",
    "sys.modules[\"sqlite3\"] = sys.modules.pop(\"pysqlite3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "566e9d28",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:46:12.768053Z",
     "start_time": "2023-12-08T13:46:12.749731Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "table, td, tr, th { border: none!important; }\n",
       "th { background: #FFFFFF; padding: 50px; text-align: center!important; }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    "table, td, tr, th { border: none!important; }\n",
    "th { background: #FFFFFF; padding: 50px; text-align: center!important; }\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "517f00e4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Load a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5e02bdd6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:46:13.329499Z",
     "start_time": "2023-12-08T13:46:12.773825Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:06:50.623104Z",
     "iopub.status.busy": "2023-08-16T22:06:50.621944Z",
     "iopub.status.idle": "2023-08-16T22:06:50.640917Z",
     "shell.execute_reply": "2023-08-16T22:06:50.637588Z",
     "shell.execute_reply.started": "2023-08-16T22:06:50.623017Z"
    },
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.llms import HuggingFacePipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "035dd36a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:46:50.833105Z",
     "start_time": "2023-12-08T13:46:13.332908Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:06:50.645364Z",
     "iopub.status.busy": "2023-08-16T22:06:50.644624Z",
     "iopub.status.idle": "2023-08-16T22:07:33.764232Z",
     "shell.execute_reply": "2023-08-16T22:07:33.714360Z",
     "shell.execute_reply.started": "2023-08-16T22:06:50.645309Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id, task = \"lmsys/fastchat-t5-3b-v1.0\", \"text2text-generation\"\n",
    "\n",
    "# model will be downloaded on first use and cached in ~/.cache/huggingface/hub/\n",
    "\n",
    "model = HuggingFacePipeline.from_model_id(\n",
    "    model_id=model_id,\n",
    "    task=task,\n",
    "    model_kwargs={\n",
    "        \"temperature\": 0,\n",
    "        \"max_length\": 1000\n",
    "    },\n",
    "    device=-1,  # CPU\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83aecfba",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Initialize an LLM chain and start asking questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "356e652b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:46:52.229762Z",
     "start_time": "2023-12-08T13:46:50.841943Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:07:33.872300Z",
     "iopub.status.busy": "2023-08-16T22:07:33.858476Z",
     "iopub.status.idle": "2023-08-16T22:07:34.086456Z",
     "shell.execute_reply": "2023-08-16T22:07:34.082581Z",
     "shell.execute_reply.started": "2023-08-16T22:07:33.864984Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from langchain import PromptTemplate, LLMChain\n",
    "\n",
    "template_text = \"\"\"\n",
    "{question}\n",
    "\"\"\"\n",
    "template = PromptTemplate(template=template_text, input_variables=[\"question\"])\n",
    "\n",
    "llm_chain = LLMChain(llm=model, prompt=template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ca4756f2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:36.683436Z",
     "start_time": "2023-12-08T13:46:52.232687Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:07:34.093984Z",
     "iopub.status.busy": "2023-08-16T22:07:34.093054Z",
     "iopub.status.idle": "2023-08-16T22:08:23.115323Z",
     "shell.execute_reply": "2023-08-16T22:08:23.114319Z",
     "shell.execute_reply.started": "2023-08-16T22:07:34.093904Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pad> Sheryl  Crow  is  an  American  singer,  songwriter,  and  actress.  She  is  best  known  for  her  role  as  the  lead  singer  and  lead  guitarist  of  the  rock  band  The  Band wagon,  and  for  her  role  as  the  lead  singer  and  lead  guitarist  of  the  alternative  rock  band  The  Mamas  and  the  Papas.  Crow  has  also  been  a  member  of  the  band  The  Mamas  and  the  Papas  since  its  formation  in  1995.\\n'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_chain(\"Who is Sheryl Crow?\")[\"text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45e5b74f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## ↪️ 2019: Language models cannot augment knowledge automatically\n",
    "\n",
    "!['Run, Baby, Run' and deaths of Huxley and Kennedy](./images/sheryl-crow-huxley-kennedy.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a9190d2",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## ↪️ 2023: ChatGPT 3.5 still fails at that! 😮 (for this particular example)\n",
    "\n",
    "!['Run, Baby, Run' confuses ChatGPT](./images/sheryl-crow-chatgpt.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26aafabb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Prompt engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3863d1ce",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:36.701380Z",
     "start_time": "2023-12-08T13:47:36.693754Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:08:23.120133Z",
     "iopub.status.busy": "2023-08-16T22:08:23.119666Z",
     "iopub.status.idle": "2023-08-16T22:08:23.127261Z",
     "shell.execute_reply": "2023-08-16T22:08:23.125549Z",
     "shell.execute_reply.started": "2023-08-16T22:08:23.120104Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "template_text = \"\"\"\n",
    "Provide brief answers, use 10 words or less.\n",
    "{question}\n",
    "\"\"\"\n",
    "template = PromptTemplate(template=template_text, input_variables=[\"question\"])\n",
    "\n",
    "llm_chain = LLMChain(llm=model, prompt=template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e1be847d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:39.240148Z",
     "start_time": "2023-12-08T13:47:36.707930Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:08:23.144759Z",
     "iopub.status.busy": "2023-08-16T22:08:23.139118Z",
     "iopub.status.idle": "2023-08-16T22:08:26.141136Z",
     "shell.execute_reply": "2023-08-16T22:08:26.139836Z",
     "shell.execute_reply.started": "2023-08-16T22:08:23.144710Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pad> Singer-songwriter'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_chain(\"Who is Sheryl Crow?\")[\"text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2743b4e8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Easy questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d8c8ce7f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:41.189439Z",
     "start_time": "2023-12-08T13:47:39.248366Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:08:26.146251Z",
     "iopub.status.busy": "2023-08-16T22:08:26.144220Z",
     "iopub.status.idle": "2023-08-16T22:08:28.416205Z",
     "shell.execute_reply": "2023-08-16T22:08:28.414969Z",
     "shell.execute_reply.started": "2023-08-16T22:08:26.146142Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pad> Europe'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_chain(\"Where is Poland located?\")[\"text\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5f13f29e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:46.791522Z",
     "start_time": "2023-12-08T13:47:41.193109Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:08:28.441585Z",
     "iopub.status.busy": "2023-08-16T22:08:28.440090Z",
     "iopub.status.idle": "2023-08-16T22:08:35.085948Z",
     "shell.execute_reply": "2023-08-16T22:08:35.084869Z",
     "shell.execute_reply.started": "2023-08-16T22:08:28.441507Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pad> Bialowieza Forest is a protected forest in Poland.'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_chain(\"What is the Bialowieza Forest?\")[\"text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca50ed8e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Harder questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5f28f639",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:50.670021Z",
     "start_time": "2023-12-08T13:47:46.794796Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:08:35.093745Z",
     "iopub.status.busy": "2023-08-16T22:08:35.092959Z",
     "iopub.status.idle": "2023-08-16T22:08:40.537052Z",
     "shell.execute_reply": "2023-08-16T22:08:40.535729Z",
     "shell.execute_reply.started": "2023-08-16T22:08:35.093666Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pad> \"Birch Tree\"'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_chain(\"What does the name 'Bialowieza' mean in English?\")[\"text\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6066e8c9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:53.719463Z",
     "start_time": "2023-12-08T13:47:50.674669Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:08:40.539606Z",
     "iopub.status.busy": "2023-08-16T22:08:40.538716Z",
     "iopub.status.idle": "2023-08-16T22:08:50.661029Z",
     "shell.execute_reply": "2023-08-16T22:08:50.659823Z",
     "shell.execute_reply.started": "2023-08-16T22:08:40.539565Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pad> Yellow.\\n'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_chain(\"\"\"Bialowieza Forest trails are marked with colors.\n",
    "   What's the color of the Wolf’s Trail?\"\"\")[\"text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5045560d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What now?\n",
    "\n",
    "# Should I fine-tune the base model? 🤔"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15671b76",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ↪️ Embeddings as a vector representation of text\n",
    "\n",
    "![cosine similarity in 2D](./images/vectors-cos-sim-500.png)\n",
    "\n",
    "The actual embedding spaces have **100s or even 1000s of dimensions**! 🤯\n",
    "\n",
    "An embedding vector can represent: a **word**, an entire **sentence** or a longer **chunk of text**.\n",
    "\n",
    "Source: [github.com/grzenkom/do-androids-read/](https://github.com/grzenkom/do-androids-read/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6efd3c3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ↪️ Vector similarity measures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d49c5278",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:57.143492Z",
     "start_time": "2023-12-08T13:47:53.722976Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  1.233  ,   4.2963 ,  -7.9738 , -10.121  ,   1.8207 ,   1.4098 ,\n",
       "        -4.518  ,  -5.2261 ,  -0.29157,   0.95234], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "vector_dog = nlp.vocab[u\"dog\"].vector\n",
    "vector_dog[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "cec6a4fb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:47:57.166957Z",
     "start_time": "2023-12-08T13:47:57.149773Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cos(dog, husky) =  0.811\n",
      "cos(dog, cows ) =  0.352\n",
      "cos(dog, stone) =  0.071\n",
      "cos(dog, Xerox) = -0.134\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "for word in [\"husky\", \"cows\", \"stone\", \"Xerox\"]:\n",
    "    print(f\"cos(dog, {word:<5}) = {cosine_similarity([vector_dog], [nlp.vocab[word].vector])[0][0]:>6.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35145c01",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "❗Cosine similarity is one of many vector similarity and vector distance measures."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5321677",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## ↪️ Arithmetic of word vectors\n",
    "\n",
    "\\begin{equation}\n",
    "\\LARGE{\\mathit{ v_{parent} + v_{woman} \\approx v_{x} }}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "565a48e4",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "![sum of \"parent\" and \"woman\" vectors](./images/vector-mother.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af7a39a6",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\\begin{equation}\n",
    "\\LARGE{\\mathit{ v_{seawater} - v_{salt} \\approx v_{x} }}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c20367a",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "![difference of \"seawater\" and \"salt\" vectors](./images/vector-water.png)\n",
    "\n",
    "Source: [github.com/grzenkom/do-androids-read/](https://github.com/grzenkom/do-androids-read/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10d63fce",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Retrieval Augmented Generation (RAG)\n",
    "\n",
    "With RAG, documents can be stored as embeddings in a vector database, queried\n",
    "for based on semantic meaning, and then these relevant splits are passed into\n",
    "**model prompt via the context window**. LLM uses these text chunks from\n",
    "original documents to generate an answer.\n",
    "\n",
    "![Question Answering flow](images/langchain-qa-flow.jpg)\n",
    "\n",
    "Source: [python.langchain.com/docs/use_cases/question_answering/](https://python.langchain.com/docs/use_cases/question_answering/#overview)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f97e1686",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## LangChain integrations\n",
    "\n",
    "1. [Catalog](https://integrations.langchain.com/llms)\n",
    "2. [Documentation](https://python.langchain.com/docs/integrations)\n",
    "\n",
    "![LangChain integrations](./images/langchain-integrations.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ce7c79d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-29T18:39:47.648775Z",
     "start_time": "2023-08-29T18:39:47.644757Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RAG step 1 - load documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1e971853",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:48:36.775816Z",
     "start_time": "2023-12-08T13:47:57.172230Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:08:50.663435Z",
     "iopub.status.busy": "2023-08-16T22:08:50.662895Z",
     "iopub.status.idle": "2023-08-16T22:09:09.419282Z",
     "shell.execute_reply": "2023-08-16T22:09:09.417508Z",
     "shell.execute_reply.started": "2023-08-16T22:08:50.663380Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import WikipediaLoader\n",
    "\n",
    "loader = WikipediaLoader(query=\"Białowieża Forest\", lang=\"en\")\n",
    "bf_docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "040ddb4d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:48:38.079230Z",
     "start_time": "2023-12-08T13:48:36.780800Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:09:09.424667Z",
     "iopub.status.busy": "2023-08-16T22:09:09.422729Z",
     "iopub.status.idle": "2023-08-16T22:09:10.085782Z",
     "shell.execute_reply": "2023-08-16T22:09:10.084546Z",
     "shell.execute_reply.started": "2023-08-16T22:09:09.424571Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import WebBaseLoader\n",
    "\n",
    "urls = [\n",
    "    \"https://bpn.com.pl/index.php?option=com_content&task=view&id=651&Itemid=297&lang=en\",\n",
    "    \"https://bpn.com.pl/index.php?option=com_content&task=view&id=27&Itemid=211&lang=en\"\n",
    "]\n",
    "\n",
    "loader = WebBaseLoader(web_paths=urls)\n",
    "bpn_page = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e47197f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-29T18:39:47.648775Z",
     "start_time": "2023-08-29T18:39:47.644757Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RAG step 2 - split documents into chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ac9b0f97",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:48:38.134643Z",
     "start_time": "2023-12-08T13:48:38.081756Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:09:10.089999Z",
     "iopub.status.busy": "2023-08-16T22:09:10.087364Z",
     "iopub.status.idle": "2023-08-16T22:09:10.228097Z",
     "shell.execute_reply": "2023-08-16T22:09:10.225802Z",
     "shell.execute_reply.started": "2023-08-16T22:09:10.089875Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "# RecursiveCharacterTextSplitter splits text recursively on characters\n",
    "# [\"\\n\\n\", \"\\n\", \" \", \"\"] and stops as soon as the chunks are small enough\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=500, chunk_overlap=0\n",
    ")\n",
    "all_splits = text_splitter.split_documents(bf_docs + bpn_page)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7740590d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-29T18:39:47.648775Z",
     "start_time": "2023-08-29T18:39:47.644757Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RAG step 3 - initialize `sentence_transformers` embedding model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2e24add8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:48:41.293897Z",
     "start_time": "2023-12-08T13:48:38.137813Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:09:10.231405Z",
     "iopub.status.busy": "2023-08-16T22:09:10.230699Z",
     "iopub.status.idle": "2023-08-16T22:09:33.259577Z",
     "shell.execute_reply": "2023-08-16T22:09:33.258129Z",
     "shell.execute_reply.started": "2023-08-16T22:09:10.231350Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from langchain.embeddings import HuggingFaceEmbeddings\n",
    "\n",
    "hf_embeddings = HuggingFaceEmbeddings(\n",
    "    model_name=\"sentence-transformers/all-mpnet-base-v2\",\n",
    "    model_kwargs={\n",
    "        \"device\": \"cpu\"\n",
    "    },\n",
    "    encode_kwargs={\n",
    "        \"normalize_embeddings\": False\n",
    "    }\n",
    ")\n",
    "\n",
    "# `all-mpnet-base-v2` maps sentences & paragraphs to a 768 dimensional\n",
    "# dense vector space and can be used for tasks like clustering or semantic search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbeabe23",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-29T18:39:47.648775Z",
     "start_time": "2023-08-29T18:39:47.644757Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RAG step 4 - calculate embeddings for text chunks and store them in a database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "dc27f47d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:49:33.582076Z",
     "start_time": "2023-12-08T13:48:41.296949Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:09:10.231405Z",
     "iopub.status.busy": "2023-08-16T22:09:10.230699Z",
     "iopub.status.idle": "2023-08-16T22:09:33.259577Z",
     "shell.execute_reply": "2023-08-16T22:09:33.258129Z",
     "shell.execute_reply.started": "2023-08-16T22:09:10.231350Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from langchain.vectorstores import Chroma\n",
    "\n",
    "vector_store = Chroma.from_documents(\n",
    "    documents=all_splits, embedding=hf_embeddings\n",
    ")\n",
    "\n",
    "# Chroma's default similarity measure is `l2`, not `cosine`\n",
    "#  - https://docs.trychroma.com/usage-guide#changing-the-distance-function\n",
    "#  - https://github.com/nmslib/hnswlib/tree/master#python-bindings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ab34f81",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-29T18:39:47.648775Z",
     "start_time": "2023-08-29T18:39:47.644757Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## ↪️ Similarity search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "39b07025",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:49:33.673070Z",
     "start_time": "2023-12-08T13:49:33.586863Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " score  | size | document chunk\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"0.570 |  245 | Białowieża [bʲawɔˈvʲɛʐa] in Poland's Podlasie Province, in t ...\",\n",
       " '0.624 |  450 | == Location ==\\nBiałowieża is in eastern Poland, in Podlasie  ...',\n",
       " '0.716 |   95 | village of Białowieża lies within the forest. Białowieża mea ...',\n",
       " '0.748 |  498 | The Białowieża Forest takes its name from the Polish village ...',\n",
       " '0.768 |  500 | Białowieża National Park (Polish: Białowieski Park Narodowy) ...']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# similarity_search_with_score - lower score represents more similarity\n",
    "\n",
    "# examples: Białowieża, bison, nature protection, pancake recipe\n",
    "relevant_splits = vector_store.similarity_search_with_score(\"Białowieża\", k=5)\n",
    "print(\" score  | size | document chunk\")\n",
    "[\n",
    "    f'{score:.3f} | {len(chunk.page_content):>4} | {chunk.page_content[:60]} ...'\n",
    "    for chunk, score in relevant_splits\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89ee07d5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-29T18:39:47.648775Z",
     "start_time": "2023-08-29T18:39:47.644757Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RAG step 5 - initialize a Q&A chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4df87ac3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:49:33.683159Z",
     "start_time": "2023-12-08T13:49:33.677965Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:09:33.262533Z",
     "iopub.status.busy": "2023-08-16T22:09:33.261903Z",
     "iopub.status.idle": "2023-08-16T22:09:34.035364Z",
     "shell.execute_reply": "2023-08-16T22:09:34.034420Z",
     "shell.execute_reply.started": "2023-08-16T22:09:33.262483Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "# the QA chain is constructed with the LLM model (loaded previously)\n",
    "# and the embedding database\n",
    "qa_chain = RetrievalQA.from_chain_type(\n",
    "    llm=model, retriever=vector_store.as_retriever()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8da2a814",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### `RetrievalQA` chain - processing steps\n",
    "\n",
    "1. fetch document splits relevant to the question from the vector database,\n",
    "2. inject the retrieved splits into the model prompt,\n",
    "3. have the LLM generate an answer based on original documents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07b9f19b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-29T18:39:47.648775Z",
     "start_time": "2023-08-29T18:39:47.644757Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RAG step 6 - generate answers with the Q&A chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "edbc4533",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:50:02.508705Z",
     "start_time": "2023-12-08T13:49:33.687323Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:09:34.038393Z",
     "iopub.status.busy": "2023-08-16T22:09:34.037898Z",
     "iopub.status.idle": "2023-08-16T22:10:09.259053Z",
     "shell.execute_reply": "2023-08-16T22:10:09.257612Z",
     "shell.execute_reply.started": "2023-08-16T22:09:34.038357Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"<pad> The  name  'Bialowieza'  means  'White  Tower'  in  English.\\n\""
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"What does the name 'Bialowieza' mean in English?\"\n",
    "\n",
    "qa_chain(f\"Provide brief answers, use 10 words or less. {question}\")[\"result\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2e644748",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:50:24.046914Z",
     "start_time": "2023-12-08T13:50:02.512336Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:10:09.261416Z",
     "iopub.status.busy": "2023-08-16T22:10:09.260834Z",
     "iopub.status.idle": "2023-08-16T22:10:46.003982Z",
     "shell.execute_reply": "2023-08-16T22:10:45.998835Z",
     "shell.execute_reply.started": "2023-08-16T22:10:09.261366Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"<pad> The  Wolf's  Trail  is  green.\\n\""
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"\"\"Bialowieza Forest trails are marked with colors.\n",
    "   What's the color of the Wolf’s Trail?\"\"\"\n",
    "\n",
    "qa_chain(f\"Provide brief answers, use 10 words or less. {question}\")[\"result\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "796e380a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## No more hallucinations then?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f7547af8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:50:45.504073Z",
     "start_time": "2023-12-08T13:50:24.050570Z"
    },
    "execution": {
     "iopub.execute_input": "2023-08-16T22:10:46.006264Z",
     "iopub.status.busy": "2023-08-16T22:10:46.005570Z",
     "iopub.status.idle": "2023-08-16T22:11:22.084775Z",
     "shell.execute_reply": "2023-08-16T22:11:22.082796Z",
     "shell.execute_reply.started": "2023-08-16T22:10:46.006221Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pad> No,  only  scientists  can  navigate  freely.\\n'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"Are the Bialowieza Forest walking trails available to the public?\"\n",
    "\n",
    "qa_chain(f\"Provide brief answers, use 10 words or less. {question}\")[\"result\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27739140",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Wait! But why?! 🕵\n",
    "\n",
    "It is possible to configure the QA chain to return the source chunks used to generate the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1690637d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:51:07.169710Z",
     "start_time": "2023-12-08T13:50:45.509588Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "explainable_qa_chain = RetrievalQA.from_chain_type(\n",
    "    llm=model, retriever=vector_store.as_retriever(),\n",
    "    return_source_documents=True\n",
    ")\n",
    "\n",
    "explainable_qa_output = explainable_qa_chain(f\"Provide brief answers, use 10 words or less. {question}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac720d13",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Source documents - metadata 🕵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1224447a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:51:07.186645Z",
     "start_time": "2023-12-08T13:51:07.173096Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Bialowieski Park Narodowy - Walking | https://bpn.com.pl/index.php?option=com_content&task=view&id=651&Itemi',\n",
       " 'Białowieża National Park            | https://en.wikipedia.org/wiki/Bia%C5%82owie%C5%BCa_National_Park',\n",
       " 'Białowieża Forest                   | https://en.wikipedia.org/wiki/Bia%C5%82owie%C5%BCa_Forest',\n",
       " 'Białowieża Forest                   | https://en.wikipedia.org/wiki/Bia%C5%82owie%C5%BCa_Forest']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[\n",
    "    f\"{chunk.metadata['title'][:35]:<35} | {chunk.metadata['source'][:70]}\"\n",
    "    for chunk in explainable_qa_output['source_documents']\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78aaf88b",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Source documents - content 🕵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f13f37b8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-08T13:51:07.199300Z",
     "start_time": "2023-12-08T13:51:07.193409Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tourists about undergrowth plants, animal trails and processes occuring in the natural forest untainted by manmade activity. Remaining trails are made available to specialised groups, on every such occasion permission from the Park’s Directorate is required.',\n",
       " 'Often called the \"last untouched wilderness of Europe\", the Białowieża National Park is the only one of its kind among the 23 national parks of Poland. Its inner zone belongs to the realm of old-growth forest which has been living without much human intervention for almost 800 years. Only scientists can navigate the strictly protected area freely. Each group of tourists is limited to no more than 20 people, and the presence of an official guide is mandatory. A classic road tour leads through',\n",
       " '== Nature protection ==\\n\\n\\n=== Białowieża National Park, Poland ===',\n",
       " 'On the Polish side, part of the Białowieża Forest is protected as the Białowieża National Park (Polish: Białowieski Park Narodowy), with an area of about 105 km2 (41 sq mi). There is also the Białowieża Glade (Polish: Polana Białowieska), with a complex of buildings once owned by the tsars of Russia during the Partitions of Poland. At present, a hotel and restaurant with a car park is located there. Guided tours into the strictly protected areas of the park can be arranged on foot, bike or by']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[\n",
    "    f\"{chunk.page_content}\"\n",
    "    for chunk in explainable_qa_output['source_documents']\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b9a066a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Next steps\n",
    "\n",
    "1. Other data loaders\n",
    "    - PDF files\n",
    "    - slide decks\n",
    "    - Google Sites\n",
    "    - files in Google Drive\n",
    "2. Conversational user experience\n",
    "    - chat user interface\n",
    "    - memory of the conversation\n",
    "4. Larger model\n",
    "5. GPU processing\n",
    "6. Other chunking strategies\n",
    "    - find optimal chunk size\n",
    "    - ensure chunks preserve the context of the source documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25c569ce",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Links\n",
    "\n",
    "### Overview\n",
    "\n",
    "- [How OpenAI trained ChatGPT](https://blog.quastor.org/p/openai-trained-chatgpt)\n",
    "    - 🎥 [State of GPT | Andrej Karpathy](https://www.youtube.com/watch?v=s6zNXZaIiiI)\n",
    "- [Generative AI exists because of the transformer](https://ig.ft.com/generative-ai)\n",
    "- [Catching up on the weird world of LLMs](https://simonwillison.net/2023/Aug/3/weird-world-of-llms)\n",
    "- [What We Know About LLMs  (Primer)](https://willthompson.name/what-we-know-about-llms-primer)\n",
    "- [The history, timeline, and future of LLMs](https://toloka.ai/blog/history-of-llms)\n",
    "- [The Practical Guides for Large Language Models](https://github.com/Mooler0410/LLMsPracticalGuide)\n",
    "- [The Many Ways that Digital Minds Can Know](https://moultano.wordpress.com/2023/06/28/the-many-ways-that-digital-minds-can-know)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24431409",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Links (cont.)\n",
    "\n",
    "### Courses, tutorials\n",
    "\n",
    "- 👩‍🎓 [Large Language Models with Semantic Search](https://learn.deeplearning.ai/large-language-models-semantic-search) (great explanation of **embeddings** + sandbox to experiment with provided code samples)\n",
    "- [Running a Hugging Face Large Language Model (LLM) locally on my laptop](https://www.markhneedham.com/blog/2023/06/23/hugging-face-run-llm-model-locally-laptop)\n",
    "- [Why You (Probably) Don’t Need to Fine-tune an LLM](https://www.tidepool.so/2023/08/17/why-you-probably-dont-need-to-fine-tune-an-llm/)\n",
    "- [Building RAG-based LLM Applications for Production](https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7940b3df",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Links (cont.)\n",
    "\n",
    "### Challenges and future research\n",
    "\n",
    "- [Open challenges in LLM research](https://huyenchip.com/2023/08/16/llm-research-open-challenges.html) \n",
    "- [Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc](https://arxiv.org/abs/2308.04445)\n",
    "- [OWASP Top 10 for Large Language Model Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications) \n",
    "\n",
    "### Skepticism\n",
    "\n",
    "- 🎥 [Why AI Is Incredibly Smart and Shockingly Stupid | Yejin Choi](https://www.youtube.com/watch?v=SvBR0OGT5VI)\n",
    "- [Anti-hype LLM reading list](https://gist.github.com/veekaybee/be375ab33085102f9027853128dc5f0e)\n",
    "- [What if Generative AI turned out to be a Dud?](https://garymarcus.substack.com/p/what-if-generative-ai-turned-out)\n",
    "    - And [Marcus on AI](https://garymarcus.substack.com) in general"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8d74cf2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![Roche logo](images/roche-logo-blue-aligned-right.png)\n",
    "\n",
    "<h1 style=\"color: #0341D6; font-style: italic; text-align: center;\">\n",
    "    Doing now what patients need next\n",
    "</h1>\n",
    "\n",
    "&nbsp;\n",
    "&nbsp;\n",
    "&nbsp;\n",
    "&nbsp;\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <th><h2>Rate the talk 💯</h2></th>\n",
    "        <th><h2>Get the slides ⏬</h2></th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><img src=\"images/devai-rate-qr.png\" style=\"max-height: 350px\" /></td>\n",
    "        <td>\n",
    "            <h2><a href=\"https://go.roche.com/devai\">go.roche.com/devai</a></h2>\n",
    "            <h3>(with additional content!)</h3>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}