internet_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b03f7e84-c7c6-4895-8231-e651c93ab039",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Enhancing Internet Search with RAG and Open Large Language Models\n",
    "\n",
    "This article introduces an internet search tool based on [Retrieval-Augmented Generation (RAG)](https://qdrant.tech/articles/what-is-rag-in-ai/), designed to answer questions posed in natural language. It searches the internet for relevant information and generates concise responses based on the search results.\n",
    "The tool is designed for local deployment, utilizing smaller open LLMs (specifically [Llama3-8B](https://huggingface.co/krasserm/Meta-Llama-3-8B-Instruct-GGUF)) and a local metasearch engine. This setup eliminates the need for external search-API keys.\n",
    "\n",
    "To dive straight into using the tool, skip ahead to the [Usage and examples](internet_search.ipynb#Usage-and-examples) section.\n",
    "\n",
    "### Improving Search Accuracy and Relevance with RAG\n",
    "\n",
    "Answering questions with large language models (LLMs) alone faces two major challenges. First, these models can produce hallucinations, presenting fabricated or inaccurate information as factual. Second, the answers provided by LLMs are constrained by their knowledge cutoff date, meaning they cannot provide information more current than their last update. This results in responses that may not only be outdated but also incorrect. The RAG-based internet search tool is designed to overcome these specific shortcomings. It addresses the issue of outdated knowledge by fetching current and relevant information directly from the internet. Additionally, it mitigates the problem of hallucinations by providing the LLM with factual information, instructing it to extract the most relevant information from the prompt and generate a precise and comprehensive answer. \n",
    "\n",
    "The search tool utilizes the open-source [SearXNG](https://github.com/searxng/searxng) metasearch engine to collect information from multiple internet sources. It uses a [reranking model](https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v1) to rank the returned results by relevance, and employs a locally deployed [Llama3-8B](https://huggingface.co/krasserm/Meta-Llama-3-8B-Instruct-GGUF) model to process search results and generate a response. The tool was specifically designed to work with smaller models like [Llama3-8B](https://huggingface.co/krasserm/Meta-Llama-3-8B-Instruct-GGUF), which has a limited context size of 8192 tokens and fewer capabilities compared to larger open-source models such as Llama3-70B, or commercial models like GPT-4.\n",
    "Instead of passing all the information from an internet search directly to the model for response generation - a strategy suited to larger models - the tool implements multiple pre-processing steps for each search result. It extracts the most relevant information and compiles a single summary for each webpage. This process allows the response generation LLM to focus on the information most relevant to the query and ensures that the context size limit is not exceeded.\n",
    "\n",
    "The internet search tool is a component of the [bot-with-plan](README.md) project. It can be used either as a standalone tool or by an agent equipped with tool-handling capabilities. This is illustrated in [Planner fine-tuning on synthetic agent trajectories](planner_finetuned.ipynb) where the internet search tool is integrated into an agentic workflow that uses tools, implementing an agentic RAG approach."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca708ee7-af35-4c75-85e4-93063389d9a5",
   "metadata": {},
   "source": [
    "### Exploring the Internet Search RAG Pipeline\n",
    "\n",
    "In this section, we will explore the RAG pipeline that powers the internet search, using a practical example to illustrate the process. Let's consider the following query: *\"What is the name of the latest NASA rover sent to Mars, when was it launched, and from which location?\"*\n",
    "\n",
    "The image below outlines the different stages of the internet search RAG pipeline involved in processing this query:\n",
    "\n",
    "<img src=\"docs/img/llm_internet_search_pipeline.jpg\" width=\"1200\" alt=\"Internet Search RAG Pipeline\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abff3a92-e34d-45c9-8856-81b72fd91c49",
   "metadata": {},
   "source": [
    "#### Stage 1: Query Transformation\n",
    "\n",
    "Queries are typically expressed in natural language and often include extraneous details that might degrade the effectiveness of a search engine. To adress this issue, queries are processed by a [LLM-based query rewriter](./gba/tools/search/query.py) ([Llama3-8B](https://huggingface.co/krasserm/Meta-Llama-3-8B-Instruct-GGUF)). This component filters out irrelevant information and refines the input query into an optimized search query that is better suited for internet search engines.\n",
    "\n",
    "The query from our example *\"What is the name of the latest NASA rover sent to Mars, when was it launched, and from which location?\"* is transformed into the following search query:\n",
    "\n",
    "```\n",
    "'latest NASA rover to Mars launch location'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "940d1264-cb4b-49b7-8d09-91d2b8e2c1b5",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### Stage 2: Information Retrieval\n",
    "\n",
    "After the input query is converted into a search query, it is passed to a local [SearXNG](https://github.com/searxng/searxng) meta-search engine. The engine is configured to query a list of search providers, sending the search query to all providers simultaneously. It then collects and aggregates the results into a single ranked list, which includes the URL, title, and a short snippet for each result. This approach allows the system to collect information from a wide range of diverse internet sources.\n",
    "\n",
    "*Note: the current configuration uses a static list of search engines for better reproducibility of search results. This configuration can easily be modified to incorporate additional search providers.*\n",
    "\n",
    "Applying the search query from the previous stage, we retrieve the following search results (only the top 2 results are shown):\n",
    "\n",
    "```\n",
    "https://en.wikipedia.org/wiki/Perseverance_(rover) \n",
    "Title: Perseverance (rover)\n",
    "Snippet: Perseverance, nicknamed Percy, [2] is a car -sized Mars rover designed to explore the Jezero crater on Mars as part of NASA 's Mars 2020 mission ...\n",
    "\n",
    "https://science.nasa.gov/mission/mars-2020-perseverance/ \n",
    "Title: Mars 2020: Perseverance Rover\n",
    "Snippet: Science. The Mars 2020 Perseverance Rover searches for signs of ancient microbial life, to advance NASA's quest to explore the past habitability of Mars ...\n",
    "```\n",
    "\n",
    "The search results returned by SearXNG are ranked and the *top-k* search results are selected for further processing. The content for these *top-k* webpages is loaded using the [partition_html](https://docs.unstructured.io/open-source/core-functionality/partitioning#partition-html) method of the [Unstructured](https://docs.unstructured.io/welcome) library. This method retrieves the content from a webpage and returns a list of text nodes. Nodes are filtered by type, preferring node types that are more likely to contain relevant information.\n",
    "\n",
    "The following output shows a sample the nodes (after filtering) from the first URL `https://en.wikipedia.org/wiki/Perseverance_(rover)`. As we can see, not all of these nodes contain information that is relevant to the user request:\n",
    "\n",
    "```\n",
    "(1) NASA Mars rover deployed in 2021\n",
    "(2) Mars rover designed to explore the\n",
    "(3) Mars 2020 mission. It was manufactured by the\n",
    "(4) Jet Propulsion Laboratory and launched on July 30, 2020, at 11:50\n",
    "(5) Earth days, or 3 years, 3 months and 17 days) since its landing. Following the rover's arrival, NASA named the landing site\n",
    "...\n",
    "(40) Associate Administrator of NASA's Science Mission Directorate, Thomas Zurbuchen selected the name Perseverance following a nationwide K-12 student \"name the rover\" contest that attracted more than 28,000 proposals. A seventh-grade student, Alexander Mather from Lake Braddock Secondary School in Burke, Virginia, submitted the winning entry at the Jet Propulsion Laboratory. In addition to the honor of naming the rover, Mather and his family were invited to NASA's Kennedy Space Center to watch the rover's July 2020 launch from Cape Canaveral Air Force Station (CCAFS) in Florida.\n",
    "...\n",
    "(46) The Perseverance rover lifted off successfully on July 30, 2020, at 11:50:00 UTC aboard a United Launch Alliance Atlas V launch vehicle from Space Launch Complex 41, at Cape Canaveral Air Force Station (CCAFS) in Florida.\n",
    "```\n",
    "\n",
    "After completing this stage, we obtain a list of *top-k* webpages. Each webpage includes a URL, a title, a snippet, and a list of content nodes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "393c414c-9789-43ee-b3e6-85f1898ac5d5",
   "metadata": {},
   "source": [
    "#### Stage 3: Ranking of Search Results\n",
    "\n",
    "It is not effective to include all content nodes from the webpages in the response generation step. Many of these nodes may not hold information relevant to the query and could introduce noise that potentially distracts the LLM during response generation. Additionally, using all nodes would increase the overall processing time, as the LLM needs to process a larger number of input tokens. To address these issues, this stage ranks the content nodes of each webpage based on their relevance to the query using a [reranking model](https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v1). This model assigns a relevance score to each content node, reorders the nodes by this score and returns the *top-k* nodes.\n",
    "\n",
    "After reranking the content nodes from the previous stage, we identify the the following most relevant nodes:\n",
    "\n",
    "```\n",
    "Rank: 1 / Score: 0.9879\n",
    "Associate Administrator of NASA's Science Mission Directorate, Thomas Zurbuchen selected the name Perseverance following a nationwide K-12 student \"name the rover\" contest that attracted more than 28,000 proposals. A seventh-grade student, Alexander Mather from Lake Braddock Secondary School in Burke, Virginia, submitted the winning entry at the Jet Propulsion Laboratory. In addition to the honor of naming the rover, Mather and his family were invited to NASA's Kennedy Space Center to watch the rover's July 2020 launch from Cape Canaveral Air Force Station (CCAFS) in Florida.\n",
    "\n",
    "Rank: 2 / Score: 0.9872\n",
    "The Perseverance rover lifted off successfully on July 30, 2020, at 11:50:00 UTC aboard a United Launch Alliance Atlas V launch vehicle from Space Launch Complex 41, at Cape Canaveral Air Force Station (CCAFS) in Florida.\n",
    "```\n",
    "\n",
    "The reranking process returned content nodes that are highly relevant to the query. These nodes, which include detailed information about the Mars rover's name, launch location, and date, were initially ranked lower, at positions `40` and `46`, in the previous stage.\n",
    "\n",
    "After reranking the content nodes, the *top-k* nodes are selected for every webpage for further processing. These nodes are concatenated with the webpage snippet to form a webpage document used in the next stage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0040f00e-cbd0-4151-b15a-33c895070757",
   "metadata": {},
   "source": [
    "#### Stage 4: Information extraction\n",
    "\n",
    "In this stage, each webpage document is processed by a [LLM-based content extractor](./gba/tools/search/extract.py) that uses a [Llama3-8B](https://huggingface.co/krasserm/Meta-Llama-3-8B-Instruct-GGUF) model. The model is instructed to extract key information from a webpage document and create a concise summary. This removes unnecessary details, enabling the LLM in the subsequent response generation stage to focus on the essential information relevant to answer the query.\n",
    "\n",
    "The following output shows the information extracted from the snippet and content nodes retrieved from `https://en.wikipedia.org/wiki/Perseverance_(rover)` in the previous stages:\n",
    "\n",
    "```\n",
    "Perseverance, nicknamed Percy, is a car-sized Mars rover designed to explore the Jezero crater on Mars as part of NASA's Mars 2020 mission. It was manufactured by the Jet Propulsion Laboratory and launched on July 30, 2020, at 11:50 UTC from Cape Canaveral Air Force Station (CCAFS) in Florida.\n",
    "```\n",
    "\n",
    "The name and launch date of the Mars rover is extracted from the webpage snippet, whereas the launch location is extracted from the rank-1 and rank-2 nodes retrieved in the previous stage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815c6876-833a-4885-bf95-1025450374bf",
   "metadata": {},
   "source": [
    "#### Stage 5: Response generation\n",
    "\n",
    "In the final stage of the pipeline a LLM ([Llama3-8B](https://huggingface.co/krasserm/Meta-Llama-3-8B-Instruct-GGUF)) is [instructed](./gba/tools/search/search_internet.py#L25) to generate a response to the query using only the webpage documents prepared in the previous stage. This approach is crucial as it restricts the LLM to the provided context, reducing tendencies to produce responses based on outdated or unsupported information (i.e. hallucinations).\n",
    "\n",
    "The following shows the response of the search tool for our example query *\"What is the name of the latest NASA rover sent to Mars, when was it launched, and from which location?\"*\n",
    "\n",
    "```\n",
    "The latest NASA rover sent to Mars is the Perseverance rover, which was launched on July 30, 2020, at 11:50 UTC from Cape Canaveral Air Force Station (CCAFS) in Florida.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b55ae638-8d1a-4e29-bc0b-22cb0b9b0208",
   "metadata": {},
   "source": [
    "### Usage and examples\n",
    "\n",
    "The internet search tool requires a locally deployed instance of [SearXNG](https://github.com/searxng/searxng) and a [Llama3-8B](https://huggingface.co/krasserm/Meta-Llama-3-8B-Instruct-GGUF) model served using [llama.cpp](https://github.com/ggerganov/llama.cpp).\n",
    "To get started, follow the setup instructions outlined [here](./gba/tools/search/README.md#search-internet-tool) to setup SearXNG and [here](README.md#getting-started) to install the required dependencies and serve the model (skip downloads for other mentioned models).\n",
    "\n",
    "The `SearchInternetTool` is instantiated with the specified SearXNG endpoint, alongside the LLM and the reranker:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "446bd4eb-21c0-42ab-9390-4eac5af2673e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers import CrossEncoder\n",
    "from gba.client import Llama3Instruct, LlamaCppClient\n",
    "from gba.tools.search.search_internet import SearchInternetTool\n",
    "from gba.utils import Scratchpad\n",
    "\n",
    "# locally deployed SearXNG instance\n",
    "searxng_endpoint=\"http://localhost:8080\"\n",
    "\n",
    "# Proxy for 8-bit quantized Llama-3-8B-Instruct\n",
    "llama3 = Llama3Instruct(llm=LlamaCppClient(url=\"http://localhost:8084/completion\", temperature=-1))\n",
    "\n",
    "# mxbai-rerank-large-v1 reranker\n",
    "rerank_model = CrossEncoder(\"mixedbread-ai/mxbai-rerank-large-v1\", device=\"cuda\")\n",
    "\n",
    "search_internet_tool = SearchInternetTool(\n",
    "    llm=llama3,\n",
    "    rerank_model=rerank_model,\n",
    "    searxng_endpoint=searxng_endpoint,\n",
    "    fetch_webpage_timeout=5.0,\n",
    "    top_k_documents=3,\n",
    "    top_k_nodes_per_document=5,    \n",
    "    top_k_snippets=None,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4355d3a-838d-4b6e-9e58-b8e02e8dd559",
   "metadata": {},
   "source": [
    "To run the example query from the section [Exploring the Internet Search RAG Pipeline](internet_search.ipynb#Exploring-the-Internet-Search-RAG-Pipeline) use the `internet_search_tool` as illustrated below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "970f3e71-65dc-4501-9512-94481a125525",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Searching the internet for query 'latest NASA rover to Mars launch location'\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The latest NASA rover sent to Mars is the Perseverance rover, which was launched on July 30, 2020, at 11:50 UTC from Cape Canaveral Air Force Station (CCAFS) in Florida.'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_internet_tool.search(\n",
    "    query=\"What is the name of the latest NASA rover sent to Mars, when was it launched, and from which location?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85e4ffa1-5545-465e-946f-96dbb58cc4e2",
   "metadata": {},
   "source": [
    "Now that we know the name of the Mars rover let's get some more information about it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2b6e5cc8-f2fd-424b-be3f-b34910df4db5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Searching the internet for query 'Perseverance rover power system energy sources'\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"The Perseverance rover's power system works by converting heat generated from the natural decay of plutonium-238 into electricity using a multi-mission radioisotope thermoelectric generator (MMRTG), which produces about 110 watts of power, similar to a light bulb.\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_internet_tool.search(\n",
    "    query=\"How does the Perseverance rover's power system work, and what are its energy sources?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e6a5da1-2a39-4fa3-b8ae-5209bf1089cc",
   "metadata": {},
   "source": [
    "The search tool cannot only retrieve historical information but also current updates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "31b47c88-a76d-4851-a81e-3d7bf68ff9bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Searching the internet for query 'current weather Vienna Celsius'\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The current weather in Vienna is 26°C.'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_internet_tool.search(\n",
    "    query=\"What is the current weather in vienna in celcius\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5f9a505-ea8b-4ab4-a4e6-e98e9b434a0d",
   "metadata": {},
   "source": [
    "The tool is also capable of processing more complex user requests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8309d5ff-c638-4ddf-b44a-f65d8c38aa2e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Searching the internet for query '2020 Best Picture Oscar winner number of Oscars categories'\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The movie \"Parasite\" won the Best Picture Oscar in 2020 and it won four awards, including Best Picture, Best Director, Best Original Screenplay, and Best International Feature Film.'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_internet_tool.search(\n",
    "    query=\"Search the internet for the movie that won the Best Picture Oscar in 2020 to find information about the number of Oscars it won and the categories of those awards.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a87dc482-bb25-49c1-a56a-fb1fccc2ba3b",
   "metadata": {},
   "source": [
    "### Improvements\n",
    "\n",
    "While the search tool generally performs well, there are several areas that could be improved in future work:\n",
    "\n",
    "* Support for dynamic web pages: currently, the search tool is limited to loading static pages.  Supporting dynamic web content that utilizes JavaScript would provide access to a broader spectrum of information.\n",
    "* SearXNG search provider configuration: at present, the configuration of search providers is restricted to ensure reliable outputs. Expanding the number of search providers and sources would increase the diversity of the search results.\n",
    "* Optimization of pipeline steps: the accuracy of individual search pipeline steps (query rewriting, content extraction, and response generation) could be improved by further refinement of the LLM prompts.\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "In this article, we discussed the capabilities and design of a RAG-based internet search tool that leverages the power of smaller, open-source Large Language Models like Llama3-8B. By integrating these models with the SearXNG metasearch engine, this tool addresses significant challenges in information retrieval using large language models - such as the issues of outdated information and hallucinations. The practical setup described not only ensures that the information retrieved is current and relevant to the intput query but also tailored to fit within the constraints of smaller models. Importantly, the entire system is designed to run locally, providing access to diverse internet sources without the complexities of external APIs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "grammar-based-agents",
   "language": "python",
   "name": "grammar-based-agents"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}