[](./examples/Youtube-Search-QA-Bot/main.py) [](./examples/Youtube-Search-QA-Bot/index.js) [](#) [](#)||
-| [Langchain: Code Docs QA bot](/examples/Code-Documentation-QA-Bot/) | [](./examples/Code-Documentation-QA-Bot/main.py) [](./examples/Code-Documentation-QA-Bot/index.js) [](#) [](#)||
-| [Databricks DBRX Website Bot](./examples/databricks_DBRX_website_bot/) | [](./examples/databricks_DBRX_website_bot/main.py) [](#) [](#)|
-| [CLI-based SDK Manual Chatbot with Phidata](/examples/CLI-SDK-Manual-Chatbot-Locally/) | [](./examples/CLI-SDK-Manual-Chatbot-Locally/assistant.py) [](#) [](#)|
-| [TransformersJS Embedding example](./examples/js-transformers/) |[](./examples/js-transformers/index.js) [](#) [](#)| |
-| [Inbuilt Hybrid Search](/examples/Inbuilt-Hybrid-Search) | [](#) [](#)||
-| [Audio Search](./examples/audio_search/) | [](./examples/audio_search/main.py) [](#) [](#)| |
-| [Multi-lingual search](/examples/multi-lingual-wiki-qa) | [](./examples/multi-lingual-wiki-qa/main.py) [](#) [](#)| |
-| [Hybrid search BM25 & lancedb ](./examples/Hybrid_search_bm25_lancedb/) | [](#) [](#)|[](https://blog.lancedb.com/hybrid-search-combining-bm25-and-semantic-search-for-better-results-with-lan-1358038fe7e6)|
-| [Search Within Images](/examples/search-within-images-with-sam-and-clip/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/search-within-images-with-sam-and-clip/main.ipynb) [](#) [](#)| [](https://blog.lancedb.com/search-within-an-image-331b54e4285e)|
-| [Accelerate Vector Search Applications Using OpenVINO](/examples/Accelerate-Vector-Search-Applications-Using-OpenVINO/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Accelerate-Vector-Search-Applications-Using-OpenVINO/clip_text_image_search.ipynb) [](#) [](#)| [](https://blog.lancedb.com/accelerate-vector-search-applications-using-openvino-lancedb/)|
+### Build from Scratch
+
+Build applications/examples using LanceDB for efficient vector-based document retrieval.
+
+| Build from Scratch | Interactive Notebook & Scripts |
+|-------- | -------------: |
+|||
+| [Build RAG from Scratch](./tutorials/RAG-from-Scratch) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/RAG-from-Scratch/RAG_from_Scratch.ipynb) [](#) [](#)| |
+| [Local RAG from Scratch with Llama3](./tutorials/Local-RAG-from-Scratch) | [](./tutorials/Local-RAG-from-Scratch/rag.py) [](#) [](#)| |
+||||
+
+### MultiModal
+
+Create a multimodal search application using LanceDB for efficient vector-based retrieval of text and image data. Input text or image queries to find the most relevant documents and images from your corpus.
+
+| Multimodal | Interactive Notebook & Scripts | Blog |
+| --------- | -------------------------- | ----------- |
+||||
| [Multimodal CLIP: DiffusionDB](/examples/multimodal_clip_diffusiondb/) | [](./examples/multimodal_clip_diffusiondb/main.py) [](#) [](#)| [](https://blog.lancedb.com/multi-modal-ai-made-easy-with-lancedb-clip-5aaf8801c939/)|
| [Multimodal CLIP: Youtube videos](/examples/multimodal_video_search/) | [](./examples/multimodal_video_search/main.py) [](#) [](#)|[](https://blog.lancedb.com/multi-modal-ai-made-easy-with-lancedb-clip-5aaf8801c939/)|
| [Multimodal Image + Text Search](/examples/multimodal_search/) | [](./examples/multimodal_search/main.py) [](#) [](#)| [](https://blog.lancedb.com/multi-modal-ai-made-easy-with-lancedb-clip-5aaf8801c939/)|
-| [Movie Recommender](/examples/movie-recommender/) | [](./examples/movie-recommender/main.py) [](#)| |
-| [Product Recommender](./examples/product-recommender/) | [](./examples/product-recommender/main.py)[](#)| |
-| [Arxiv paper recommender](/examples/arxiv-recommender) | [](./examples/arxiv-recommender/main.py) [](#) [](#)| |
+||||
+
+### RAG
+
+Develop a Retrieval-Augmented Generation (RAG) application using LanceDB for efficient vector-based information retrieval. Input text queries to retrieve relevant documents and generate comprehensive answers by combining retrieved information.
+
+| RAG | Interactive Notebook & Scripts | Blog |
+| --------- | -------------------------- | ----------- |
+||||
| [Improve RAG with Re-ranking](/examples/RAG_Reranking/) | [](#) [](#)|[](https://blog.lancedb.com/simplest-method-to-improve-rag-pipeline-re-ranking-cf6eaec6d544)|
-| [Improve RAG with FLARE](/examples/Advanced-RAG-with-FLARE) | [](./examples/Advanced-RAG-with-FLARE/app.py) [](#) [](#)| [](https://blog.lancedb.com/better-rag-with-active-retrieval-augmented-generation-flare-3b66646e2a9f)|
+| [Instruct-Multitask](./examples/instruct-multitask) | [](./examples/instruct-multitask/main.py) [](#) [](#)|[](https://blog.lancedb.com/multitask-embedding-with-lancedb-be18ec397543)|
| [Improve RAG with HyDE](/examples/Advance-RAG-with-HyDE/) | [](#) [](#)|[](https://blog.lancedb.com/advanced-rag-precise-zero-shot-dense-retrieval-with-hyde-0946c54dfdcb)|
| [Improve RAG with LOTR ](/examples/Advance_RAG_LOTR/) | [](#) [](#)|[](https://blog.lancedb.com/better-rag-with-lotr-lord-of-retriever-23c8336b9a35)|
| [Advanced RAG: Parent Document Retriever](/examples/parent_document_retriever/) | [](#) [](#)|[](https://blog.lancedb.com/modified-rag-parent-document-bigger-chunk-retriever-62b3d1e79bc6)|
+| [Corrective RAG with Langgraph](./tutorials/Corrective-RAG-with_Langgraph/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Corrective-RAG-with_Langgraph/CRAG_with_Langgraph.ipynb) [](#) [](#)| [](https://blog.lancedb.com/implementing-corrective-rag-in-the-easiest-way-2/)|
+| [Contextual-Compression-with-RAG](/examples/Contextual-Compression-with-RAG/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Contextual-Compression-with-RAG/main.ipynb) [](#) [](#)|[](https://blog.lancedb.com/enhance-rag-integrate-contextual-compression-and-filtering-for-precision-a29d4a810301/) |
+| [Improve RAG with FLARE](./examples/better-rag-FLAIR) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/better-rag-FLAIR/main.ipynb) [](#) [](#) [](#)|[](https://blog.lancedb.com/better-rag-with-active-retrieval-augmented-generation-flare-3b66646e2a9f/) |
| [Query Expansion and Reranker ](/examples/QueryExpansion&Reranker/) | [](#) [](#)|[](https://blog.lancedb.com/improving-rag-with-query-expansion-reranking-models/)|
| [RAG Fusion](/examples/RAG_Fusion/) | [](#) [](#)|
-| [Contextual-Compression-with-RAG](/examples/Contextual-Compression-with-RAG/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Contextual-Compression-with-RAG/main.ipynb) [](#) [](#)|[](https://blog.lancedb.com/enhance-rag-integrate-contextual-compression-and-filtering-for-precision-a29d4a810301/) |
-| [Instruct-Multitask](./examples/instruct-multitask) | [](./examples/instruct-multitask/main.py) [](#) [](#)|[](https://blog.lancedb.com/multitask-embedding-with-lancedb-be18ec397543)|
+| [Agentic RAG ](/tutorials/Agentic_RAG/) | [](#) [](#)|
+||||
+
+### Vector Search
+
+Build a vector search application using LanceDB for efficient vector-based document retrieval. Input text queries to find the most relevant documents from your corpus.
+
+| Vector Search | Interactive Notebook & Scripts | Blog |
+| --------- | -------------------------- | ----------- |
+||||
+| [Inbuilt Hybrid Search](/examples/Inbuilt-Hybrid-Search) | [](#) [](#)||
+| [Hybrid search BM25 & lancedb ](./examples/Hybrid_search_bm25_lancedb/) | [](#) [](#) |[](https://blog.lancedb.com/hybrid-search-combining-bm25-and-semantic-search-for-better-results-with-lan-1358038fe7e6)|
+| [NER powered Semantic Search](./tutorials/NER-powered-Semantic-Search) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/NER-powered-Semantic-Search/NER_powered_Semantic_Search_with_LanceDB.ipynb) [](#) [](#)| [](https://blog.lancedb.com/ner-powered-semantic-search-using-lancedb-51051dc3e493) |
+| [Audio Search](./examples/audio_search/) | [](./examples/audio_search/main.py) [](#) [](#)| |
+| [Multi-lingual search](/examples/multi-lingual-wiki-qa) | [](./examples/multi-lingual-wiki-qa/main.py) [](#) [](#)| |
+| [Facial Recognition](./examples/facial_recognition) | [](#)|
+[Sentiment Analysis : Analysing Hotel Reviews](/examples/Sentiment-Analysis-Analyse-Hotel-Reviews/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Sentiment-Analysis-Analyse-Hotel-Reviews/Sentiment_Analysis_using_LanceDB.ipynb) [](#) [](#)| [](https://blog.lancedb.com/sentiment-analysis-using-lancedb-2da3cb1e3fa6)|
+| [Imagebind demo app](./examples/imagebind_demo/) | 
[](#)|
+| [Search Within Images](/examples/search-within-images-with-sam-and-clip/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/search-within-images-with-sam-and-clip/main.ipynb) [](#) [](#)| [](https://blog.lancedb.com/search-within-an-image-331b54e4285e)|
+| [Vector Search with TransformersJS](./examples/js-transformers/) |[](./examples/js-transformers/index.js) [](#) [](#)| |
+| [Accelerate Vector Search Applications Using OpenVINO](/examples/Accelerate-Vector-Search-Applications-Using-OpenVINO/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Accelerate-Vector-Search-Applications-Using-OpenVINO/clip_text_image_search.ipynb) [](#) [](#)| [](https://blog.lancedb.com/accelerate-vector-search-applications-using-openvino-lancedb/)|
+||||
+
+### Chatbot
+
+Create a chatbot application using LanceDB for efficient vector-based response generation. Input user queries to retrieve relevant context and generate coherent, context-aware replies.
+
+| Chatbot | Interactive Notebook & Scripts | Blog |
+| --------- | -------------------------- | ----------- |
+||||
+| [Databricks DBRX Website Bot](./examples/databricks_DBRX_website_bot/) | [](./examples/databricks_DBRX_website_bot/main.py) [](#) [](#)|
+| [CLI-based SDK Manual Chatbot with Phidata](/examples/CLI-SDK-Manual-Chatbot-Locally/) | [](./examples/CLI-SDK-Manual-Chatbot-Locally/assistant.py) [](#) [](#)|
+| [Youtube transcript search bot](/examples/Youtube-Search-QA-Bot/) | [](./examples/Youtube-Search-QA-Bot/main.py) [](./examples/Youtube-Search-QA-Bot/index.js) [](#) [](#)||
+| [Langchain: Code Docs QA bot](/examples/Code-Documentation-QA-Bot/) | [](./examples/Code-Documentation-QA-Bot/main.py) [](./examples/Code-Documentation-QA-Bot/index.js) [](#) [](#)||
+| [Context-Aware Chatbot using Llama 2 & LanceDB](./tutorials/chatbot_using_Llama2_&_lanceDB) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/chatbot_using_Llama2_&_lanceDB/main.ipynb) [](#) [](#)| [](https://blog.lancedb.com/context-aware-chatbot-using-llama-2-lancedb-as-vector-database-4d771d95c755) |
+||||
+
+
+### Evaluation
+
+Develop an evaluation application. Input reference and candidate texts to measure their performance on various metrics.
+
+| Evaluation | Interactive Notebook & Scripts | Blog |
+| --------- | -------------------------- | ----------- |
+||||
| [Evaluating Prompts with Prompttools](/examples/prompttools-eval-prompts/) | [](#) [](#) [](#)| |
+| [Evaluating RAG with RAGAs](./examples/Evaluating_RAG_with_RAGAs/) | [](#) [](#)| |
+||||
+
+### AI Agents
+
+Design an AI agents coordination application with LanceDB for efficient vector-based communication and collaboration. Input queries to enable AI agents to exchange information, coordinate tasks, and achieve shared goals effectively.
+
+| AI Agents | Interactive Notebook & Scripts | Blog |
+| --------- | -------------------------- | ----------- |
+||||
| [AI Agents: Reducing Hallucination](/examples/reducing_hallucinations_ai_agents/) | [](./examples/reducing_hallucinations_ai_agents/main.py) [](./examples/reducing_hallucinations_ai_agents/index.js) [](#) [](#) |[](https://blog.lancedb.com/how-to-reduce-hallucinations-from-llm-powered-agents-using-long-term-memory-72f262c3cc1f/)|
| [AI Trends Searcher with CrewAI](./examples/AI-Trends-with-CrewAI/) | [](#) [](#)|[](https://blog.lancedb.com/track-ai-trends-crewai-agents-rag/)|
| [SuperAgent Autogen](/examples/SuperAgent_Autogen) | [](#) [](#)||
-[Sentiment Analysis : Analysing Hotel Reviews](/examples/Sentiment-Analysis-Analyse-Hotel-Reviews/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Sentiment-Analysis-Analyse-Hotel-Reviews/Sentiment_Analysis_using_LanceDB.ipynb) [](#) [](#)| [](https://blog.lancedb.com/sentiment-analysis-using-lancedb-2da3cb1e3fa6)|
-| [Facial Recognition](./examples/facial_recognition) | [](#)|
-| [Imagebind demo app](/examples/imagebind_demo/) | [](#)|
+||||
+
+### Recommender Systems
+
+Create a recommender system application with LanceDB for efficient vector-based item recommendation. Input user preferences or item features to generate personalized recommendations and enhance user experience.
+
+| Recommender Systems | Interactive Notebook & Scripts | Blog |
+| --------- | -------------------------- | ----------- |
+||||
+| [Movie Recommender](/examples/movie-recommender/) | [](./examples/movie-recommender/main.py) [](#)| |
+| [Movie Recommender with Genre](./examples/movie-recommendation-with-genres/) | [](#)| [](https://blog.lancedb.com/movie-recommendation-system-using-lancedb-and-doc2vec/)|
+| [Product Recommender](./examples/product-recommender/) | [](./examples/product-recommender/main.py)[](#)| |
+| [Arxiv paper recommender](/examples/arxiv-recommender) | [](./examples/arxiv-recommender/main.py) [](#) [](#)| |
+||||
+### Concepts
+Checkout concepts of LLM applications pipeline to ensures accurate information retrieval.
+
+| Concepts | Interactive Notebook | Blog |
+| --------- | -------------------------- | ----------- |
+| | | |
+| [A Primer on Text Chunking and its Types](./tutorials/different-types-text-chunking-in-RAG) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/different-types-text-chunking-in-RAG/Text_Chunking_on_RAG_application_with_LanceDB.ipynb) [](#)| [](https://blog.lancedb.com/a-primer-on-text-chunking-and-its-types-a420efc96a13) |
+| [Langchain LlamaIndex Chunking](./tutorials/Langchain-LlamaIndex-Chunking) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Langchain-LlamaIndex-Chunking/Langchain_Llamaindex_chunking.ipynb) [](#)| [](https://blog.lancedb.com/chunking-techniques-with-langchain-and-llamaindex/) |
+| [Comparing Cohere Rerankers with LanceDB](./tutorials/cohere-reranker) | [](#)| [](https://blog.lancedb.com/benchmarking-cohere-reranker-with-lancedb/) |
+| [Product Quantization: Compress High Dimensional Vectors](https://blog.lancedb.com/benchmarking-lancedb-92b01032874a-2/) |[](#) | [](https://blog.lancedb.com/benchmarking-lancedb-92b01032874a-2/) |
+| [LLMs, RAG, & the missing storage layer for AI](https://blog.lancedb.com/llms-rag-the-missing-storage-layer-for-ai-28ded35fa984) | [](#)| [](https://blog.lancedb.com/llms-rag-the-missing-storage-layer-for-ai-28ded35fa984/) |
+| [Fine-Tuning LLM using PEFT & QLoRA](./tutorials/fine-tuning_LLM_with_PEFT_QLoRA) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb) [](#) [](#)| [](https://blog.lancedb.com/optimizing-llms-a-step-by-step-guide-to-fine-tuning-with-peft-and-qlora-22eddd13d25b) |
+| [Extracting Complex tables-text from PDFs using LlamaParse ](./tutorials/Advace_RAG_LlamaParser) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Advace_RAG_LlamaParser/main.ipynb) [](#) [](#) [](#)| |
+||||
## Projects & Applications
These are ready to use applications built using LanceDB serverless vector database. You can explore these open source projects, use parts of them in your projects or build your applications on top of these.
@@ -89,27 +180,7 @@ These are ready to use applications built using LanceDB serverless vector databa
| [ Fastapi RAG template ](https://github.com/lancedb/vectordb-recipes/tree/main/applications/Chatbot_RAG_with_FASTAPI) | FastAPI based RAG template with Websocket support | |
| [ GTE MLX RAG ](https://github.com/lancedb/vectordb-recipes/tree/main/applications/GTE_mlx_RAG) | mlx based RAG model using lancedb api support | |
| [ Healthcare Chatbot ](https://github.com/lancedb/vectordb-recipes/tree/main/applications/Healthcare_chatbot/) | Healthcare chatbot using domain specific LLM & Embedding model | |
-
-
-
-## Tutorials
-Looking to get started with LLMs, vectorDBs, and the world of Generative AI? These in-depth tutorials and courses cover these concepts with practical follow along colabs where possible.
-| Tutorial | Interactive Environment | Blog Link |
-| --------- | -------------------------- | ----------- |
-| | | |
-| [Build RAG from Scratch](./tutorials/RAG-from-Scratch) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/RAG-from-Scratch/RAG_from_Scratch.ipynb) [](#) [](#)| |
-| [Local RAG from Scratch with Llama3](./tutorials/Local-RAG-from-Scratch) | [](./tutorials/Local-RAG-from-Scratch/rag.py) [](#) [](#)| |
-| [A Primer on Text Chunking and its Types](./tutorials/different-types-text-chunking-in-RAG) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/different-types-text-chunking-in-RAG/Text_Chunking_on_RAG_application_with_LanceDB.ipynb) [](#)| [](https://blog.lancedb.com/a-primer-on-text-chunking-and-its-types-a420efc96a13) |
-| [Langchain LlamaIndex Chunking](./tutorials/Langchain-LlamaIndex-Chunking) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Langchain-LlamaIndex-Chunking/Langchain_Llamaindex_chunking.ipynb) [](#)| [](https://blog.lancedb.com/chunking-techniques-with-langchain-and-llamaindex/) |
-| [Comparing Cohere Rerankers with LanceDB](./tutorials/cohere-reranker) | [](#)| [](https://blog.lancedb.com/benchmarking-cohere-reranker-with-lancedb/) |
-| [NER powered Semantic Search](./tutorials/NER-powered-Semantic-Search) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/NER-powered-Semantic-Search/NER_powered_Semantic_Search_with_LanceDB.ipynb) [](#) [](#)| [](https://blog.lancedb.com/ner-powered-semantic-search-using-lancedb-51051dc3e493) |
-| [Product Quantization: Compress High Dimensional Vectors](https://blog.lancedb.com/benchmarking-lancedb-92b01032874a-2/) |[](#) | [](https://blog.lancedb.com/benchmarking-lancedb-92b01032874a-2/) |
-| [Corrective RAG with Langgraph](./tutorials/Corrective-RAG-with_Langgraph/) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Corrective-RAG-with_Langgraph/CRAG_with_Langgraph.ipynb) [](#) [](#)| [](https://blog.lancedb.com/implementing-corrective-rag-in-the-easiest-way-2/)|
-| [LLMs, RAG, & the missing storage layer for AI](https://blog.lancedb.com/llms-rag-the-missing-storage-layer-for-ai-28ded35fa984) | [](#)| [](https://blog.lancedb.com/llms-rag-the-missing-storage-layer-for-ai-28ded35fa984/) |
-| [Fine-Tuning LLM using PEFT & QLoRA](./tutorials/fine-tuning_LLM_with_PEFT_QLoRA) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb) [](#) [](#)| [](https://blog.lancedb.com/optimizing-llms-a-step-by-step-guide-to-fine-tuning-with-peft-and-qlora-22eddd13d25b) |
-| [Context-Aware Chatbot using Llama 2 & LanceDB](./tutorials/chatbot_using_Llama2_&_lanceDB) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/chatbot_using_Llama2_&_lanceDB/main.ipynb) [](#) [](#)| [](https://blog.lancedb.com/context-aware-chatbot-using-llama-2-lancedb-as-vector-database-4d771d95c755) |
-| [Better RAG with FLARE](./tutorials/better-rag-FLAIR) | [](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/better-rag-FLAIR/main.ipynb) [](#) [](#) [](#)|[](https://blog.lancedb.com/better-rag-with-active-retrieval-augmented-generation-flare-3b66646e2a9f/) |
-
+||||
**🌟 New! 🌟 Applied GenAI and VectorDB course on Udacity**
diff --git a/assets/imagebind-demo.png b/assets/imagebind-demo.png
new file mode 100644
index 00000000..c1e28ae1
Binary files /dev/null and b/assets/imagebind-demo.png differ
diff --git a/assets/movie-recommendation-with-genre.png b/assets/movie-recommendation-with-genre.png
new file mode 100644
index 00000000..6b26c968
Binary files /dev/null and b/assets/movie-recommendation-with-genre.png differ
diff --git a/assets/rag_evaluation_flow.png b/assets/rag_evaluation_flow.png
new file mode 100644
index 00000000..4b74b6cb
Binary files /dev/null and b/assets/rag_evaluation_flow.png differ
diff --git a/assets/superagent-autogen.png b/assets/superagent-autogen.png
index d41d2484..9908c624 100644
Binary files a/assets/superagent-autogen.png and b/assets/superagent-autogen.png differ
diff --git a/examples/Code-Documentation-QA-Bot/lancedb_cloud/README.md b/examples/Code-Documentation-QA-Bot/lancedb_cloud/README.md
new file mode 100644
index 00000000..88b6eca7
--- /dev/null
+++ b/examples/Code-Documentation-QA-Bot/lancedb_cloud/README.md
@@ -0,0 +1,33 @@
+# Code documentation Q&A bot example with LangChain
+
+
+
+
+This Q&A bot will allow you to query your own documentation easily using questions. We'll also demonstrate the use of LangChain and LanceDB Cloud using the OpenAI API. In this example we'll use **Numpy 1.26** documentation, but, this could be replaced for your own docs as well.
+Colab walkthrough -
+
+
+### Set credentials
+if you would like to set api key through an environment variable:
+```
+export LANCEDB_API_KEY="sk_..."
+```
+or
+```
+import os
+import getpass
+
+os.environ["LANCEDB_API_KEY"] = getpass.getpass("Enter Your LANCEDB API Key:")
+```
+
+replace the following lines in main.py with your project slug and api key"
+```
+db_url="db://your-project-slug-name"
+api_key="sk_..."
+region="us-east-1"
+```
+
+### Run the script
+```python
+OPENAI_API_KEY=... python main.py --query "what is a vectordb?"
+```
\ No newline at end of file
diff --git a/examples/Code-Documentation-QA-Bot/lancedb_cloud/main.ipynb b/examples/Code-Documentation-QA-Bot/lancedb_cloud/main.ipynb
new file mode 100644
index 00000000..5b40c699
--- /dev/null
+++ b/examples/Code-Documentation-QA-Bot/lancedb_cloud/main.ipynb
@@ -0,0 +1,484 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "13cb272e",
+ "metadata": {},
+ "source": [
+ "# Code documentation Q&A bot example with LangChain\n",
+ "\n",
+ "\n",
+ "This Q&A bot will allow you to query your own documentation easily using questions. We'll also demonstrate the use of LangChain and LanceDB using the OpenAI API.\n",
+ "\n",
+ "In this example we'll **Numpy 1.26** documentation, but, this could be replaced for your own docs as well"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9a0e829a",
+ "metadata": {
+ "id": "wgPbKbpumkhH"
+ },
+ "source": [
+ "### Credentials\n",
+ "\n",
+ "Copy and paste the project name and the api key from your project page.\n",
+ "These will be used later to [connect to LanceDB Cloud](#scroll-to=5q8m6GMD7sGu)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "6553603f",
+ "metadata": {
+ "id": "rqEXT5-fmofw"
+ },
+ "outputs": [],
+ "source": [
+ "project_slug = \"your-project-slug\" # @param {type:\"string\"}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "36ef9c45",
+ "metadata": {
+ "id": "5LYmBomPmswi"
+ },
+ "outputs": [],
+ "source": [
+ "api_key = \"sk_...\" # @param {type:\"string\"}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "33ba6af1",
+ "metadata": {
+ "id": "Xs6tr6CMnBrr"
+ },
+ "source": [
+ "You can also set the LANCEDB_API_KEY as an environment variable. More details can be found **here**."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Le27BWs2vDbB"
+ },
+ "source": [
+ "Since we will be using OPENAI API, let us set the OPENAI API KEY as well."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "-2-fyVPKu9fl"
+ },
+ "outputs": [],
+ "source": [
+ "openai_api_key = \"sk-...\" # @param {type:\"string\"}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1991331f-4316-417a-b693-e2f27cbe9ea7",
+ "metadata": {},
+ "source": [
+ "### Installing dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e8a49c31",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! pip install -U langchain langchain-openai langchain-community"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "66638d6c",
+ "metadata": {
+ "id": "QR9W53zStdlz"
+ },
+ "outputs": [],
+ "source": [
+ "! pip install -qq tiktoken unstructured pandas lancedb"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0QQL4lm8lTzg"
+ },
+ "source": [
+ "### Importing libraries"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "vP6d6JUShgqo"
+ },
+ "outputs": [],
+ "source": [
+ "import openai\n",
+ "import os\n",
+ "import re\n",
+ "import pickle\n",
+ "import requests\n",
+ "import zipfile\n",
+ "from pathlib import Path\n",
+ "\n",
+ "from langchain.document_loaders import UnstructuredHTMLLoader\n",
+ "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+ "from langchain.vectorstores import LanceDB\n",
+ "from langchain_openai import OpenAI, OpenAIEmbeddings\n",
+ "from langchain.chains import RetrievalQA\n",
+ "\n",
+ "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
+ "assert openai.models.list() is not None"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8eKRYd2F7v5n"
+ },
+ "source": [
+ "### Get the data\n",
+ "To make this easier, we've downloaded Numpy documentation and stored the raw HTML files for you to download. Once the docs are downloaded, we then use LangChain's HTML document readers to parse them and store them in LanceDB as a vector store, along with relevant metadata.\n",
+ "By default we use numpy docs, but you can replace this with your own docs as well."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "l0ezDr7suAf_"
+ },
+ "outputs": [],
+ "source": [
+ "numpy_docs = requests.get(\"https://numpy.org/doc/1.26/numpy-html.zip\")\n",
+ "with open(\"numpy-html.zip\", \"wb\") as f:\n",
+ " f.write(numpy_docs.content)\n",
+ "\n",
+ "file = zipfile.ZipFile(\"numpy-html.zip\")\n",
+ "file = file.extractall(path=\"numpy_docs\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HJf8xZmX8VJC"
+ },
+ "source": [
+ "We'll create a simple **helper function** that can help to extract metadata, so it can used later when querying with filters. In this case, we want to keep the lineage of the uri or path for each document that has been processed:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "5aljyqpUiViE"
+ },
+ "outputs": [],
+ "source": [
+ "# Pre-processing and loading the documentation\n",
+ "\n",
+ "# Next, let's pre-process and load the documentation. To make sure we don't need to do this repeatedly if we were updating code,\n",
+ "# we're caching it using pickle so we can retrieve it again (this could take a few minutes to run the first time you do it).\n",
+ "# We'll also add some more metadata to the docs here such as the title and version of the code:\n",
+ "\n",
+ "\n",
+ "def get_document_title(document_list):\n",
+ " titles = []\n",
+ " for doc in document_list:\n",
+ " if \"metadata\" in doc and \"source\" in doc[\"metadata\"]:\n",
+ " m = str(doc[\"metadata\"][\"source\"])\n",
+ " title = re.findall(\"numpy_docs(.*).html\", m)\n",
+ " print(title)\n",
+ " if title:\n",
+ " titles.append(title[0])\n",
+ " else:\n",
+ " titles.append(\"\")\n",
+ " else:\n",
+ " titles.append(\"\")\n",
+ " return titles"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "PCufm9Xr8eWp"
+ },
+ "source": [
+ "### Pre-processing and loading the documents\n",
+ "\n",
+ "Next, let's pre-process and load the documents. To make sure we don't need to do this repeatedly while updating code, we're caching it using pickle so it can be retrieved again (this could take a few minutes to run the first time you do it). We'll also add extra metadata to the docs here such as the title and version of the code:\n",
+ "\n",
+ "*Note*: This step might take up to 10 minutes to run!\n",
+ "*Note*: If there is some issue with nltk package, kindly try using\n",
+ "```\n",
+ "import nltk\n",
+ "nltk.download('punkt')\n",
+ "```\n",
+ "or try to manually install the [nltk_data](https://github.com/nltk/nltk_data/tree/gh-pages) package and unzip the **punkt tokenizer** zip and the **averaged_perceptron_tagger** zip file in the packages folder."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 443
+ },
+ "id": "964Z2sZA247g",
+ "outputId": "236df468-a630-4691-85a4-886835cfc02d"
+ },
+ "outputs": [],
+ "source": [
+ "from tqdm import tqdm\n",
+ "\n",
+ "docs = []\n",
+ "docs_path = Path(\"docs.pkl\")\n",
+ "for p in tqdm(Path(\"numpy_docs\").rglob(\"*.html\")):\n",
+ " if p.is_dir():\n",
+ " continue\n",
+ " loader = UnstructuredHTMLLoader(p)\n",
+ " raw_document = loader.load()\n",
+ " # docs.append(raw_document)\n",
+ " title = get_document_title(raw_document)\n",
+ " m = {\"title\": title}\n",
+ " if raw_document:\n",
+ " raw_document[0].metadata.update(m)\n",
+ " raw_document[0].metadata[\"source\"] = str(raw_document[0].metadata[\"source\"])\n",
+ " docs.extend(raw_document)\n",
+ "\n",
+ "\n",
+ "if docs:\n",
+ " with open(docs_path, \"wb\") as fh:\n",
+ " pickle.dump(docs, fh)\n",
+ "else:\n",
+ " with open(docs_path, \"rb\") as fh:\n",
+ " docs = pickle.load(fh)\n",
+ "\n",
+ "len(docs)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cntAuaUU_TER"
+ },
+ "source": [
+ "### Generating emebeddings from our docs\n",
+ "\n",
+ "Now that we have our raw documents loaded, we need to pre-process them to generate embeddings:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "dHw2DSAj3u9B"
+ },
+ "outputs": [],
+ "source": [
+ "text_splitter = RecursiveCharacterTextSplitter(\n",
+ " chunk_size=1000,\n",
+ " chunk_overlap=200,\n",
+ ")\n",
+ "documents = text_splitter.split_documents(docs)\n",
+ "embeddings = OpenAIEmbeddings()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IiM4DJvC_2dV"
+ },
+ "source": [
+ "### Store data in LanceDB Cloud\n",
+ "\n",
+ "Let's connect to LanceDB so we can store our documents, It requires 0 setup !"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GV77SSi-AK0v"
+ },
+ "outputs": [],
+ "source": [
+ "uri = \"db://\" + project_slug\n",
+ "table_name = \"langchain_vectorstore\"\n",
+ "\n",
+ "vectorstore = LanceDB(\n",
+ " embedding=embeddings,\n",
+ " uri=uri, # your remote database URI\n",
+ " api_key=api_key,\n",
+ " region=\"us-east-1\",\n",
+ " table_name=table_name, # Optional, defaults to \"vectors\"\n",
+ " mode=\"overwrite\", # Optional, defaults to \"overwrite\"\n",
+ ")\n",
+ "\n",
+ "doc_ids = vectorstore.add_documents(documents=documents)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sZOUxfqzXr1m"
+ },
+ "source": [
+ "Now let's create our RetrievalQA chain using the LanceDB vector store:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "4nDltKClAhhU"
+ },
+ "outputs": [],
+ "source": [
+ "qa = RetrievalQA.from_chain_type(\n",
+ " llm=OpenAI(), chain_type=\"stuff\", retriever=vectorstore.as_retriever()\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xoS-WKXMXvvR"
+ },
+ "source": [
+ "And thats it! We're all setup. The next step is to run some queries, let's try a few:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7SKSlyq2iwpK"
+ },
+ "source": [
+ "### Query"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "6aSZr8fCXx9s",
+ "outputId": "ac5b5663-d45f-48c0-9f0a-f272e1a3ec2d"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'query': 'tell me about the numpy library?',\n",
+ " 'result': ' The NumPy library is an open source Python library that is used for working with numerical data in Python. It contains multidimensional array and matrix data structures, and provides methods for efficient operations on these arrays. It is widely used in various fields of science and engineering and is a core component of the scientific Python and PyData ecosystems. It also offers a large library of high-level mathematical functions for working with arrays and matrices. '}"
+ ]
+ },
+ "execution_count": 14,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "query = \"tell me about the numpy library?\"\n",
+ "qa.invoke(query)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "EtBw5EH7lv9_",
+ "outputId": "1745f881-fa15-44b5-e692-b702babce734"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'query': \"What's the current version of numpy?\",\n",
+ " 'result': '\\nThe current version of numpy is 1.16.4.'}"
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "query = \"What's the current version of numpy?\"\n",
+ "qa.invoke(query)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "fR4CmF9ylvzw",
+ "outputId": "1b33bb78-4b3f-4dea-addd-75f56eb4e5e6"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'query': 'What kind of linear algebra related operations can be done in numpy?',\n",
+ " 'result': ' The numpy package provides various operations related to linear algebra, such as decompositions, matrix eigenvalues, norms, solving equations and inverting matrices, and performing linear algebra on several matrices at once. It also has support for logic functions, masked array operations, mathematical functions, matrix library, miscellaneous routines, padding arrays, polynomials, random sampling, set routines, sorting, searching, counting, statistics, and window functions.'}"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "query = \"What kind of linear algebra related operations can be done in numpy?\"\n",
+ "qa.invoke(query)"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.12.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/examples/Code-Documentation-QA-Bot/lancedb_cloud/main.py b/examples/Code-Documentation-QA-Bot/lancedb_cloud/main.py
new file mode 100644
index 00000000..e9eb9605
--- /dev/null
+++ b/examples/Code-Documentation-QA-Bot/lancedb_cloud/main.py
@@ -0,0 +1,136 @@
+# %% [markdown]
+# # Code documentation Q&A bot example with LangChain
+#
+# This Q&A bot will allow you to query your own documentation easily using questions. We'll also demonstrate the use of LangChain and LanceDB using the OpenAI API.
+#
+# In this example we'll use Pandas 2.0 documentation, but, this could be replaced for your own docs as well
+
+import argparse
+import os
+import pickle
+import re
+import zipfile
+from pathlib import Path
+
+import openai
+import requests
+from langchain.chains import RetrievalQA
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain_community.document_loaders import BSHTMLLoader
+from langchain_community.vectorstores import LanceDB
+from langchain_openai import OpenAI, OpenAIEmbeddings
+
+
+def get_document_title(document_list):
+ titles = []
+ for doc in document_list:
+ if "metadata" in doc and "source" in doc["metadata"]:
+ m = str(doc["metadata"]["source"])
+ title = re.findall("numpy_docs(.*).html", m)
+ if title:
+ titles.append(title[0])
+ else:
+ titles.append("")
+ else:
+ titles.append("")
+ return titles
+
+
+def arg_parse():
+ default_query = "tell me about the numpy library?"
+ # default_query = "What's the current version of numpy?"
+
+ parser = argparse.ArgumentParser(description="Code Documentation QA Bot")
+ parser.add_argument(
+ "--query", type=str, default=default_query, help="query to search"
+ )
+ parser.add_argument("--openai-key", type=str, help="OpenAI API Key")
+ args = parser.parse_args()
+
+ if not args.openai_key:
+ if "OPENAI_API_KEY" not in os.environ:
+ raise ValueError(
+ "OPENAI_API_KEY environment variable not set. Please set it or pass --openai_key"
+ )
+ else:
+ openai.api_key = args.openai_key
+
+ return args
+
+
+def pre_process():
+ from tqdm import tqdm
+
+ docs = []
+ docs_path = Path("docs.pkl")
+ for p in tqdm(Path("numpy_docs").rglob("*.html")):
+ if p.is_dir():
+ continue
+ # loader = UnstructuredHTMLLoader(p)
+ loader = BSHTMLLoader(p, open_encoding="utf8")
+ raw_document = loader.load()
+ # docs.append(raw_document)
+ title = get_document_title(raw_document)
+ m = {"title": title}
+ if raw_document:
+ raw_document[0].metadata.update(m)
+ raw_document[0].metadata["source"] = str(raw_document[0].metadata["source"])
+ docs.extend(raw_document)
+
+ if docs:
+ with open(docs_path, "wb") as fh:
+ pickle.dump(docs, fh)
+ else:
+ with open(docs_path, "rb") as fh:
+ docs = pickle.load(fh)
+
+ return docs
+
+
+if __name__ == "__main__":
+ args = arg_parse()
+
+ numpy_docs = requests.get("https://numpy.org/doc/1.26/numpy-html.zip")
+ with open("numpy-html.zip", "wb") as f:
+ f.write(numpy_docs.content)
+
+ file = zipfile.ZipFile("numpy-html.zip")
+ file = file.extractall(path="numpy_docs")
+
+ docs = pre_process()
+
+ print("Loaded {} documents".format(len(docs)))
+ text_splitter = RecursiveCharacterTextSplitter(
+ chunk_size=1000,
+ chunk_overlap=200,
+ )
+ documents = text_splitter.split_documents(docs)
+ embeddings = OpenAIEmbeddings()
+
+ db_url = "db://your-project-slug"
+ api_key = "sk_..."
+ region = "us-east-1"
+ table_name = "langchain_vectorstore"
+
+ vectorstore = LanceDB(
+ embedding=embeddings,
+ uri=db_url, # your remote database URI
+ api_key=api_key,
+ region="us-east-1",
+ table_name=table_name, # Optional, defaults to "vectors"
+ mode="overwrite", # Optional, defaults to "overwrite"
+ )
+
+ # insert documents in batches
+ batch_size = 10000
+ for i in range(0, len(documents), batch_size):
+ print(f"ingesting batch of {i} : {i+batch_size}")
+ batch = documents[i : i + batch_size]
+ vectorstore.add_documents(batch)
+
+ qa = RetrievalQA.from_chain_type(
+ llm=OpenAI(), chain_type="stuff", retriever=vectorstore.as_retriever()
+ )
+
+ result = qa.run(args.query)
+ print(result)
diff --git a/examples/Code-Documentation-QA-Bot/lancedb_cloud/requirements.txt b/examples/Code-Documentation-QA-Bot/lancedb_cloud/requirements.txt
new file mode 100644
index 00000000..79d2c50b
--- /dev/null
+++ b/examples/Code-Documentation-QA-Bot/lancedb_cloud/requirements.txt
@@ -0,0 +1,8 @@
+argparse
+openai
+langchain-community
+langchain-openai
+lancedb
+unstructured
+tiktoken
+polars
diff --git a/examples/Evaluating_RAG_with_RAGAs/Evaluating_RAG_with_RAGAs.ipynb b/examples/Evaluating_RAG_with_RAGAs/Evaluating_RAG_with_RAGAs.ipynb
new file mode 100644
index 00000000..f69e55d3
--- /dev/null
+++ b/examples/Evaluating_RAG_with_RAGAs/Evaluating_RAG_with_RAGAs.ipynb
@@ -0,0 +1,1158 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "language_info": {
+ "name": "python"
+ },
+ "widgets": {
+ "application/vnd.jupyter.widget-state+json": {
+ "91f4187ef74b4c0791fa9058899f7454": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "HBoxModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "HBoxModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "HBoxView",
+ "box_style": "",
+ "children": [
+ "IPY_MODEL_d9fb5f1092e24ba59cb768842ff8f828",
+ "IPY_MODEL_6441a4ce0f644c51aa6c140de43ed31d",
+ "IPY_MODEL_a6b459a38e5c4386b85ee7ebc0e302a4"
+ ],
+ "layout": "IPY_MODEL_cf96076499974020b541a541648028f4"
+ }
+ },
+ "d9fb5f1092e24ba59cb768842ff8f828": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "HTMLModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "HTMLModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "HTMLView",
+ "description": "",
+ "description_tooltip": null,
+ "layout": "IPY_MODEL_82cd48cdf6f144e19cb3ef0a0553b689",
+ "placeholder": "",
+ "style": "IPY_MODEL_8e537aa094004828b08a55a94cbd7dff",
+ "value": "Evaluating: 100%"
+ }
+ },
+ "6441a4ce0f644c51aa6c140de43ed31d": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "FloatProgressModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "FloatProgressModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "ProgressView",
+ "bar_style": "success",
+ "description": "",
+ "description_tooltip": null,
+ "layout": "IPY_MODEL_55a45baafaad4a4ca2cad720da0a80aa",
+ "max": 27,
+ "min": 0,
+ "orientation": "horizontal",
+ "style": "IPY_MODEL_7f2a940f7f114e439f63e5e900748511",
+ "value": 27
+ }
+ },
+ "a6b459a38e5c4386b85ee7ebc0e302a4": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "HTMLModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "HTMLModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "HTMLView",
+ "description": "",
+ "description_tooltip": null,
+ "layout": "IPY_MODEL_50282721d29447c89bc05559a99183dd",
+ "placeholder": "",
+ "style": "IPY_MODEL_23c5a724d0fa40efac45f1fe8fc0b9c5",
+ "value": " 27/27 [00:23<00:00, 4.90s/it]"
+ }
+ },
+ "cf96076499974020b541a541648028f4": {
+ "model_module": "@jupyter-widgets/base",
+ "model_name": "LayoutModel",
+ "model_module_version": "1.2.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/base",
+ "_model_module_version": "1.2.0",
+ "_model_name": "LayoutModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "LayoutView",
+ "align_content": null,
+ "align_items": null,
+ "align_self": null,
+ "border": null,
+ "bottom": null,
+ "display": null,
+ "flex": null,
+ "flex_flow": null,
+ "grid_area": null,
+ "grid_auto_columns": null,
+ "grid_auto_flow": null,
+ "grid_auto_rows": null,
+ "grid_column": null,
+ "grid_gap": null,
+ "grid_row": null,
+ "grid_template_areas": null,
+ "grid_template_columns": null,
+ "grid_template_rows": null,
+ "height": null,
+ "justify_content": null,
+ "justify_items": null,
+ "left": null,
+ "margin": null,
+ "max_height": null,
+ "max_width": null,
+ "min_height": null,
+ "min_width": null,
+ "object_fit": null,
+ "object_position": null,
+ "order": null,
+ "overflow": null,
+ "overflow_x": null,
+ "overflow_y": null,
+ "padding": null,
+ "right": null,
+ "top": null,
+ "visibility": null,
+ "width": null
+ }
+ },
+ "82cd48cdf6f144e19cb3ef0a0553b689": {
+ "model_module": "@jupyter-widgets/base",
+ "model_name": "LayoutModel",
+ "model_module_version": "1.2.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/base",
+ "_model_module_version": "1.2.0",
+ "_model_name": "LayoutModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "LayoutView",
+ "align_content": null,
+ "align_items": null,
+ "align_self": null,
+ "border": null,
+ "bottom": null,
+ "display": null,
+ "flex": null,
+ "flex_flow": null,
+ "grid_area": null,
+ "grid_auto_columns": null,
+ "grid_auto_flow": null,
+ "grid_auto_rows": null,
+ "grid_column": null,
+ "grid_gap": null,
+ "grid_row": null,
+ "grid_template_areas": null,
+ "grid_template_columns": null,
+ "grid_template_rows": null,
+ "height": null,
+ "justify_content": null,
+ "justify_items": null,
+ "left": null,
+ "margin": null,
+ "max_height": null,
+ "max_width": null,
+ "min_height": null,
+ "min_width": null,
+ "object_fit": null,
+ "object_position": null,
+ "order": null,
+ "overflow": null,
+ "overflow_x": null,
+ "overflow_y": null,
+ "padding": null,
+ "right": null,
+ "top": null,
+ "visibility": null,
+ "width": null
+ }
+ },
+ "8e537aa094004828b08a55a94cbd7dff": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "DescriptionStyleModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "DescriptionStyleModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "StyleView",
+ "description_width": ""
+ }
+ },
+ "55a45baafaad4a4ca2cad720da0a80aa": {
+ "model_module": "@jupyter-widgets/base",
+ "model_name": "LayoutModel",
+ "model_module_version": "1.2.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/base",
+ "_model_module_version": "1.2.0",
+ "_model_name": "LayoutModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "LayoutView",
+ "align_content": null,
+ "align_items": null,
+ "align_self": null,
+ "border": null,
+ "bottom": null,
+ "display": null,
+ "flex": null,
+ "flex_flow": null,
+ "grid_area": null,
+ "grid_auto_columns": null,
+ "grid_auto_flow": null,
+ "grid_auto_rows": null,
+ "grid_column": null,
+ "grid_gap": null,
+ "grid_row": null,
+ "grid_template_areas": null,
+ "grid_template_columns": null,
+ "grid_template_rows": null,
+ "height": null,
+ "justify_content": null,
+ "justify_items": null,
+ "left": null,
+ "margin": null,
+ "max_height": null,
+ "max_width": null,
+ "min_height": null,
+ "min_width": null,
+ "object_fit": null,
+ "object_position": null,
+ "order": null,
+ "overflow": null,
+ "overflow_x": null,
+ "overflow_y": null,
+ "padding": null,
+ "right": null,
+ "top": null,
+ "visibility": null,
+ "width": null
+ }
+ },
+ "7f2a940f7f114e439f63e5e900748511": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "ProgressStyleModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "ProgressStyleModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "StyleView",
+ "bar_color": null,
+ "description_width": ""
+ }
+ },
+ "50282721d29447c89bc05559a99183dd": {
+ "model_module": "@jupyter-widgets/base",
+ "model_name": "LayoutModel",
+ "model_module_version": "1.2.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/base",
+ "_model_module_version": "1.2.0",
+ "_model_name": "LayoutModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "LayoutView",
+ "align_content": null,
+ "align_items": null,
+ "align_self": null,
+ "border": null,
+ "bottom": null,
+ "display": null,
+ "flex": null,
+ "flex_flow": null,
+ "grid_area": null,
+ "grid_auto_columns": null,
+ "grid_auto_flow": null,
+ "grid_auto_rows": null,
+ "grid_column": null,
+ "grid_gap": null,
+ "grid_row": null,
+ "grid_template_areas": null,
+ "grid_template_columns": null,
+ "grid_template_rows": null,
+ "height": null,
+ "justify_content": null,
+ "justify_items": null,
+ "left": null,
+ "margin": null,
+ "max_height": null,
+ "max_width": null,
+ "min_height": null,
+ "min_width": null,
+ "object_fit": null,
+ "object_position": null,
+ "order": null,
+ "overflow": null,
+ "overflow_x": null,
+ "overflow_y": null,
+ "padding": null,
+ "right": null,
+ "top": null,
+ "visibility": null,
+ "width": null
+ }
+ },
+ "23c5a724d0fa40efac45f1fe8fc0b9c5": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "DescriptionStyleModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "DescriptionStyleModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "StyleView",
+ "description_width": ""
+ }
+ }
+ }
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Evaluating RAG with RAGAs using GPT-4o\n",
+ "\n",
+ "Ragas is a **framework for evaluating Retrieval Augmented Generation (RAG) pipelines**.\n",
+ "\n",
+ "Ragas provides you with the tools/metrics based on the latest research for evaluating LLM-generated text to give you insights about your RAG pipeline. Ragas can be integrated with your CI/CD to provide continuous checks to ensure performance.\n",
+ "\n",
+ "GPT4-o is used as an LLM to generate responses out of semantically close context chunks.\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "AwhxwHTf4VZp"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "1mr4fXemsYci",
+ "outputId": "dd51d890-7da2-45ec-b14f-d96169bb8bdf"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m973.5/973.5 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m320.6/320.6 kB\u001b[0m \u001b[31m38.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.9/18.9 MB\u001b[0m \u001b[31m71.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.1/84.1 kB\u001b[0m \u001b[31m11.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m308.5/308.5 kB\u001b[0m \u001b[31m38.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m122.8/122.8 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m22.8/22.8 MB\u001b[0m \u001b[31m58.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m542.0/542.0 kB\u001b[0m \u001b[31m46.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m51.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m48.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.1/71.1 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.5/142.5 kB\u001b[0m \u001b[31m20.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.7/98.7 kB\u001b[0m \u001b[31m13.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m18.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m22.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m21.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 kB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h"
+ ]
+ }
+ ],
+ "source": [
+ "!pip install langchain openai lancedb ragas -q"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Setup `OPENAI_API_KEY` as an environment variable"
+ ],
+ "metadata": {
+ "id": "z8hT0Jn74ZmT"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "\n",
+ "os.environ[\"OPENAI_API_KEY\"] = \"sk-proj-...\""
+ ],
+ "metadata": {
+ "id": "YHgQd_1rI04R"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Load .txt file and convert them into chunks"
+ ],
+ "metadata": {
+ "id": "mM0tf_vo6GbI"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import requests\n",
+ "from langchain.document_loaders import TextLoader\n",
+ "from langchain.text_splitter import CharacterTextSplitter\n",
+ "\n",
+ "url = \"https://raw.githubusercontent.com/hwchase17/chroma-langchain/master/state_of_the_union.txt\"\n",
+ "res = requests.get(url)\n",
+ "with open(\"state_of_the_union.txt\", \"w\") as f:\n",
+ " f.write(res.text)\n",
+ "\n",
+ "# Load the data\n",
+ "loader = TextLoader(\"./state_of_the_union.txt\")\n",
+ "documents = loader.load()\n",
+ "\n",
+ "# Chunk the data\n",
+ "text_splitter = CharacterTextSplitter(chunk_size=200, chunk_overlap=10)\n",
+ "chunks = text_splitter.split_documents(documents)"
+ ],
+ "metadata": {
+ "id": "IkLbg-_1I3Rt",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "4248c952-c719-4a30-ee7e-06d2f1b17449"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 215, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 232, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 242, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 219, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 304, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 205, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 332, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 215, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 203, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 281, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 201, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 250, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 325, which is longer than the specified 200\n",
+ "WARNING:langchain_text_splitters.base:Created a chunk of size 242, which is longer than the specified 200\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Setup Retriever\n",
+ "\n",
+ "Retriever utilizes **LanceDB** for scalable vector search and advanced retrieval in RAG, delivering blazing fast performance for searching large sets of embeddings."
+ ],
+ "metadata": {
+ "id": "pgetSLZXEJ2Q"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from langchain.embeddings import OpenAIEmbeddings\n",
+ "from langchain.vectorstores import LanceDB\n",
+ "import lancedb\n",
+ "\n",
+ "openai_embed = OpenAIEmbeddings()\n",
+ "\n",
+ "# Setup lancedb\n",
+ "db = lancedb.connect(\"/tmp/lancedb\")\n",
+ "table = db.create_table(\n",
+ " \"raga_eval\",\n",
+ " data=[{\"vector\": openai_embed.embed_query(\"Hello World\"), \"text\": \"Hello World\"}],\n",
+ " mode=\"overwrite\",\n",
+ ")\n",
+ "\n",
+ "# Populate vector database\n",
+ "vectorstore = LanceDB.from_documents(\n",
+ " client=table, documents=chunks, embedding=openai_embed, by_text=False\n",
+ ")\n",
+ "\n",
+ "# Define vectorstore as retriever to enable semantic search\n",
+ "retriever = vectorstore.as_retriever()"
+ ],
+ "metadata": {
+ "id": "2PYhU_vvJC0P"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Setup RAG Pipeline with Prompt template"
+ ],
+ "metadata": {
+ "id": "9CFnNEfuExj7"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from langchain.chat_models import ChatOpenAI\n",
+ "from langchain.prompts import ChatPromptTemplate\n",
+ "from langchain.schema.runnable import RunnablePassthrough\n",
+ "from langchain.schema.output_parser import StrOutputParser\n",
+ "\n",
+ "# Define LLM\n",
+ "llm = ChatOpenAI(model_name=\"gpt-4o\", temperature=0)\n",
+ "\n",
+ "# Define Prompt template\n",
+ "template = \"\"\"You are an assistant for question-answering tasks.\n",
+ "Use the following pieces of retrieved context to answer the question.\n",
+ "If you don't know the answer, just say that you don't know.\n",
+ "Use two sentences maximum and keep the answer concise.\n",
+ "Question: {question}\n",
+ "Context: {context}\n",
+ "Answer:\n",
+ "\"\"\"\n",
+ "\n",
+ "prompt = ChatPromptTemplate.from_template(template)\n",
+ "\n",
+ "# Setup RAG pipeline\n",
+ "rag_chain = (\n",
+ " {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
+ " | prompt\n",
+ " | llm\n",
+ " | StrOutputParser()\n",
+ ")"
+ ],
+ "metadata": {
+ "id": "-TiQhbNyLSKv"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Sample Questions with their Expected Answers\n",
+ "\n",
+ "Define a set of questions with their answers for creating dataset including ground truth, generated answers with their context using which they are generated."
+ ],
+ "metadata": {
+ "id": "Ge8JtkNXFXhI"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from datasets import Dataset\n",
+ "\n",
+ "questions = [\n",
+ " \"What did the president say about Justice Breyer?\",\n",
+ " \"What did the president say about Intel's CEO?\",\n",
+ " \"What did the president say about gun violence?\",\n",
+ "]\n",
+ "ground_truth = [\n",
+ " \"The president said that Justice Breyer has dedicated his life to serve the country and thanked him for his service.\",\n",
+ " \"The president said that Pat Gelsinger is ready to increase Intel's investment to $100 billion.\",\n",
+ " \"The president asked Congress to pass proven measures to reduce gun violence.\",\n",
+ "]\n",
+ "answers = []\n",
+ "contexts = []\n",
+ "\n",
+ "# Inference\n",
+ "for query in questions:\n",
+ " answers.append(rag_chain.invoke(query))\n",
+ " contexts.append(\n",
+ " [docs.page_content for docs in retriever.get_relevant_documents(query)]\n",
+ " )\n",
+ "\n",
+ "# To dict\n",
+ "data = {\n",
+ " \"question\": questions,\n",
+ " \"answer\": answers,\n",
+ " \"contexts\": contexts,\n",
+ " \"ground_truth\": ground_truth,\n",
+ "}\n",
+ "\n",
+ "# Convert dict to dataset\n",
+ "dataset = Dataset.from_dict(data)"
+ ],
+ "metadata": {
+ "id": "PGiU57QJMP0J"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### RAGA Evaluation Pipeline\n",
+ "\n",
+ "Simple pipeline of RAGA for evaluation with the listed metrics to understand and evaluate the RAG system.\n",
+ "\n",
+ "**Metrics** on which we will evaulate are answer_correctness,\n",
+ "faithfulness,\n",
+ "answer_similarity,\n",
+ "context_precision,\n",
+ "context_utilization,\n",
+ "context_recall,\n",
+ "context_relevancy,\n",
+ "answer_relevancy, and\n",
+ "context_entity_recall"
+ ],
+ "metadata": {
+ "id": "szBZ1nwkFruF"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from ragas import evaluate\n",
+ "from ragas.metrics import (\n",
+ " answer_correctness,\n",
+ " faithfulness,\n",
+ " answer_similarity,\n",
+ " context_precision,\n",
+ " context_utilization,\n",
+ " context_recall,\n",
+ " context_relevancy,\n",
+ " answer_relevancy,\n",
+ " context_entity_recall,\n",
+ ")\n",
+ "\n",
+ "\n",
+ "# evaluating dataest on listed metrics\n",
+ "result = evaluate(\n",
+ " dataset=dataset,\n",
+ " metrics=[\n",
+ " answer_correctness,\n",
+ " faithfulness,\n",
+ " answer_similarity,\n",
+ " context_precision,\n",
+ " context_utilization,\n",
+ " context_recall,\n",
+ " context_relevancy,\n",
+ " answer_relevancy,\n",
+ " context_entity_recall,\n",
+ " ],\n",
+ ")\n",
+ "\n",
+ "\n",
+ "df = result.to_pandas()"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 49,
+ "referenced_widgets": [
+ "91f4187ef74b4c0791fa9058899f7454",
+ "d9fb5f1092e24ba59cb768842ff8f828",
+ "6441a4ce0f644c51aa6c140de43ed31d",
+ "a6b459a38e5c4386b85ee7ebc0e302a4",
+ "cf96076499974020b541a541648028f4",
+ "82cd48cdf6f144e19cb3ef0a0553b689",
+ "8e537aa094004828b08a55a94cbd7dff",
+ "55a45baafaad4a4ca2cad720da0a80aa",
+ "7f2a940f7f114e439f63e5e900748511",
+ "50282721d29447c89bc05559a99183dd",
+ "23c5a724d0fa40efac45f1fe8fc0b9c5"
+ ]
+ },
+ "id": "Samkm2TnMUQA",
+ "outputId": "f50e72d4-55fd-4f74-f0f6-334af8353ae2"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "Evaluating: 0%| | 0/27 [00:00\n",
+ " | \n", + " | question | \n", + "answer | \n", + "contexts | \n", + "ground_truth | \n", + "answer_correctness | \n", + "faithfulness | \n", + "answer_similarity | \n", + "context_precision | \n", + "context_utilization | \n", + "context_recall | \n", + "context_relevancy | \n", + "answer_relevancy | \n", + "context_entity_recall | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "What did the president say about Justice Breyer? | \n", + "The president honored Justice Stephen Breyer a... | \n", + "[And I did that 4 days ago, when I nominated C... | \n", + "The president said that Justice Breyer has ded... | \n", + "0.415487 | \n", + "1.0 | \n", + "0.911948 | \n", + "1.0 | \n", + "1.0 | \n", + "1.0 | \n", + "0.200000 | \n", + "0.841589 | \n", + "0.500000 | \n", + "
| 1 | \n", + "What did the president say about Intel's CEO? | \n", + "The president said that Intel’s CEO, Pat Gelsi... | \n", + "[Intel’s CEO, Pat Gelsinger, who is here tonig... | \n", + "The president said that Pat Gelsinger is ready... | \n", + "0.619998 | \n", + "0.0 | \n", + "0.980103 | \n", + "1.0 | \n", + "1.0 | \n", + "1.0 | \n", + "0.090909 | \n", + "0.897084 | \n", + "0.750000 | \n", + "
| 2 | \n", + "What did the president say about gun violence? | \n", + "The president called on Congress to pass prove... | \n", + "[And I ask Congress to pass proven measures to... | \n", + "The president asked Congress to pass proven me... | \n", + "0.606230 | \n", + "1.0 | \n", + "0.924895 | \n", + "1.0 | \n", + "1.0 | \n", + "1.0 | \n", + "0.250000 | \n", + "0.914888 | \n", + "0.666667 | \n", + "
+
+
+Ragas is a **framework for evaluating Retrieval Augmented Generation (RAG) pipelines**.
+
+Ragas provides you with the tools/metrics based on the latest research for evaluating LLM-generated text to give you insights about your RAG pipeline. Ragas can be integrated with your CI/CD to provide continuous checks to ensure performance.
+
+GPT4-o is used as an LLM to generate responses out of semantically close context chunks.
+
+
+
+Try it out on Colab -
\ No newline at end of file
diff --git a/examples/LlamaIndex-demo/lancedb_cloud/README.md b/examples/LlamaIndex-demo/lancedb_cloud/README.md
new file mode 100644
index 00000000..636ab133
--- /dev/null
+++ b/examples/LlamaIndex-demo/lancedb_cloud/README.md
@@ -0,0 +1,29 @@
+# LlamaIndex and LanceDB Cloud Demo
+
+In this demo, we are going to show how to use LanceDB Cloud to perform vector searches in LlamaIndex
+
+
+### Set credentials
+if you would like to set api key through an environment variable:
+```
+export LANCEDB_API_KEY="sk_..."
+```
+or
+```
+import os
+import getpass
+
+os.environ["LANCEDB_API_KEY"] = getpass.getpass("Enter Your LANCEDB API Key:")
+```
+
+replace the following lines in main.py with your project slug and api key"
+```
+db_url="db://your-project-slug-name"
+api_key="sk_..."
+region="us-east-1"
+```
+
+### Run the script
+```python
+OPENAI_API_KEY=... python main.py
+```
\ No newline at end of file
diff --git a/examples/LlamaIndex-demo/lancedb_cloud/main.ipynb b/examples/LlamaIndex-demo/lancedb_cloud/main.ipynb
new file mode 100644
index 00000000..6a36bd18
--- /dev/null
+++ b/examples/LlamaIndex-demo/lancedb_cloud/main.ipynb
@@ -0,0 +1,391 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "13cb272e",
+ "metadata": {},
+ "source": [
+ "# Vector search with LanceDB Cloud and LlamaIndex \n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9a0e829a",
+ "metadata": {
+ "id": "wgPbKbpumkhH"
+ },
+ "source": [
+ "### Credentials\n",
+ "\n",
+ "Copy and paste the project name and the api key from your project page.\n",
+ "These will be used later to [connect to LanceDB Cloud](#scroll-to=5q8m6GMD7sGu)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "6553603f",
+ "metadata": {
+ "id": "rqEXT5-fmofw"
+ },
+ "outputs": [],
+ "source": [
+ "project_slug = \"your-project-slug\" # @param {type:\"string\"}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "36ef9c45",
+ "metadata": {
+ "id": "5LYmBomPmswi"
+ },
+ "outputs": [],
+ "source": [
+ "api_key = \"sk_...\" # @param {type:\"string\"}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "33ba6af1",
+ "metadata": {
+ "id": "Xs6tr6CMnBrr"
+ },
+ "source": [
+ "You can also set the LANCEDB_API_KEY as an environment variable. More details can be found **here**."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Le27BWs2vDbB"
+ },
+ "source": [
+ "Since we will be using OPENAI API, let us set the OPENAI API KEY as well."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "-2-fyVPKu9fl"
+ },
+ "outputs": [],
+ "source": [
+ "openai_api_key = \"sk-...\" # @param {type:\"string\"}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1991331f-4316-417a-b693-e2f27cbe9ea7",
+ "metadata": {},
+ "source": [
+ "### Installing dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e8a49c31",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! pip install llama-index-vector-stores-lancedb llama-index-readers-file llama-index-embeddings-openai llama-index-llms-openai"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0QQL4lm8lTzg"
+ },
+ "source": [
+ "### Importing libraries"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "vP6d6JUShgqo"
+ },
+ "outputs": [],
+ "source": [
+ "import openai\n",
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "# Uncomment to see debug logs\n",
+ "# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)\n",
+ "# logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
+ "\n",
+ "from llama_index.core import SimpleDirectoryReader, Document, StorageContext\n",
+ "from llama_index.core import VectorStoreIndex\n",
+ "from llama_index.vector_stores.lancedb import LanceDBVectorStore\n",
+ "import textwrap\n",
+ "\n",
+ "openai.api_key = openai_api_key\n",
+ "assert openai.models.list() is not None"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8eKRYd2F7v5n"
+ },
+ "source": [
+ "### Download the data\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "l0ezDr7suAf_"
+ },
+ "outputs": [],
+ "source": [
+ "! mkdir -p 'data/paul_graham/'\n",
+ "! wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'\n",
+ "! ls 'data/paul_graham/'"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HJf8xZmX8VJC"
+ },
+ "source": [
+ "Load the documents stored in the data/paul_graham/ using the SimpleDirectoryReader:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "5aljyqpUiViE"
+ },
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader(\"data/paul_graham/\").load_data()\n",
+ "print(\"Document ID:\", documents[0].doc_id, \"Document Hash:\", documents[0].hash)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IiM4DJvC_2dV"
+ },
+ "source": [
+ "### Store data in LanceDB Cloud\n",
+ "\n",
+ "Let's connect to LanceDB so we can store our documents, It requires 0 setup !"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GV77SSi-AK0v"
+ },
+ "outputs": [],
+ "source": [
+ "uri = \"db://\" + project_slug\n",
+ "table_name = \"llamaindex_vectorstore\" #optional, default table name is \"vectors\" \n",
+ "\n",
+ "vector_store = LanceDBVectorStore( \n",
+ " uri=uri, # your remote DB URI\n",
+ " api_key=\"sk_..\", # lancedb cloud api key\n",
+ " region=\"your-region\" # the region you configured\n",
+ " ...\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sZOUxfqzXr1m"
+ },
+ "source": [
+ "### Create an index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "4nDltKClAhhU"
+ },
+ "outputs": [],
+ "source": [
+ "storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
+ "\n",
+ "index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xoS-WKXMXvvR"
+ },
+ "source": [
+ "And thats it! We're all setup. The next step is to run some queries, let's try a few:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7SKSlyq2iwpK"
+ },
+ "source": [
+ "### Query the index\n",
+ "We can now ask questions using the created index. Filtering can be enabled via `MetadataFilters` or use native lance `where` clause."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5eb6419b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "from llama_index.core.vector_stores import (\n",
+ " MetadataFilters,\n",
+ " FilterOperator,\n",
+ " FilterCondition,\n",
+ " MetadataFilter,\n",
+ ")\n",
+ "\n",
+ "date = datetime.today().strftime(\"%Y-%m-%d\")\n",
+ "query_filters = MetadataFilters(\n",
+ " filters=[\n",
+ " MetadataFilter(\n",
+ " key=\"creation_date\",\n",
+ " operator=FilterOperator.EQ,\n",
+ " value=date, # using current date as the latest data is scraped\n",
+ " ),\n",
+ " MetadataFilter(key=\"file_size\", value=75040, operator=FilterOperator.GT),\n",
+ " ],\n",
+ " condition=FilterCondition.AND,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Viaweb charged $100 a month for a small store and $300 a month for a big one.\n",
+ "metadata - ..."
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "query_engine = index.as_query_engine(\n",
+ " filters=query_filters,\n",
+ ")\n",
+ "\n",
+ "response = query_engine.query(\"How much did Viaweb charge per month?\")\n",
+ "print(response)\n",
+ "print(\"metadata -\", response.metadata)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0c1c6c73",
+ "metadata": {},
+ "source": [
+ "Let's use LanceDB filters(SQL like) directly via the `where` clause :"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0a2bcc07",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "lance_filter = \"metadata.file_name = 'paul_graham_essay.txt' \"\n",
+ "retriever = index.as_retriever(vector_store_kwargs={\"where\": lance_filter})\n",
+ "response = retriever.retrieve(\"What did the author do growing up?\")\n",
+ "print(response[0].get_content())\n",
+ "print(\"metadata -\", response[0].metadata)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sZOUxfqzXr1m"
+ },
+ "source": [
+ "### Append data to the index \n",
+ "You can also add data to an existing index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "069fc099",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "del index\n",
+ "\n",
+ "index = VectorStoreIndex.from_documents(\n",
+ " [Document(text=\"The sky is purple in Portland, Maine\")],\n",
+ " uri=\"/tmp/new_dataset\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b5cffcfe",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Portland, Maine\n"
+ ]
+ }
+ ],
+ "source": [
+ "query_engine = index.as_query_engine()\n",
+ "response = query_engine.query(\"Where is the sky purple?\")\n",
+ "print(textwrap.fill(str(response), 100))"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.12.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/examples/LlamaIndex-demo/lancedb_cloud/main.py b/examples/LlamaIndex-demo/lancedb_cloud/main.py
new file mode 100644
index 00000000..ef4e168d
--- /dev/null
+++ b/examples/LlamaIndex-demo/lancedb_cloud/main.py
@@ -0,0 +1,89 @@
+import os
+import textwrap
+from datetime import datetime
+
+import openai
+import requests
+from llama_index.core import (
+ Document,
+ SimpleDirectoryReader,
+ StorageContext,
+ VectorStoreIndex,
+)
+from llama_index.vector_stores.lancedb import LanceDBVectorStore
+
+if __name__ == "__main__":
+ if "OPENAI_API_KEY" not in os.environ:
+ raise ValueError("OPENAI_API_KEY environment variable not set. Please set it")
+ else:
+ openai.api_key = os.environ["OPENAI_API_KEY"]
+
+ # Download the document
+ data_path = r"data/paul_graham/"
+ if not os.path.exists(data_path):
+ os.makedirs(data_path)
+ url = "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt"
+ r = requests.get(url)
+ with open(data_path + "/paul_graham_essay.txt", "wb") as f:
+ f.write(r.content)
+
+ # Load the document
+ documents = SimpleDirectoryReader(data_path).load_data()
+ print("Document ID:", documents[0].doc_id, "Document Hash:", documents[0].hash)
+
+ # Create a LanceDBVectorStore and create an index
+ vector_store = LanceDBVectorStore(
+ uri="db://your-project-slug", # your remote DB URI
+ api_key="sk_...", # lancedb cloud api key
+ region="us-east-1", # the region you configured
+ )
+
+ storage_context = StorageContext.from_defaults(vector_store=vector_store)
+
+ index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
+
+ # Query via MetadataFilters
+ from llama_index.core.vector_stores import (
+ FilterCondition,
+ FilterOperator,
+ MetadataFilter,
+ MetadataFilters,
+ )
+
+ date = datetime.today().strftime("%Y-%m-%d")
+ query_filters = MetadataFilters(
+ filters=[
+ MetadataFilter(key="creation_date", operator=FilterOperator.EQ, value=date),
+ MetadataFilter(key="file_size", value=75040, operator=FilterOperator.GT),
+ ],
+ condition=FilterCondition.AND,
+ )
+
+ query_engine = index.as_query_engine(
+ filters=query_filters,
+ )
+
+ response = query_engine.query("How much did Viaweb charge per month?")
+ print("==== query via MetadataFilters")
+ print(response)
+ print("metadata -", response.metadata)
+
+ # Query via LanceDB where clause
+ lance_filter = "metadata.file_name = 'paul_graham_essay.txt' "
+ retriever = index.as_retriever(vector_store_kwargs={"where": lance_filter})
+ response = retriever.retrieve("What did the author do growing up?")
+ print("==== query via LanceDB where clause")
+ print(response[0].get_content())
+ print("metadata -", response[0].metadata)
+
+ # add data to an existing index and query with the new data
+ del index
+
+ index = VectorStoreIndex.from_documents(
+ [Document(text="The sky is purple in Portland, Maine")],
+ uri="/tmp/new_dataset",
+ )
+ query_engine = index.as_query_engine()
+ response = query_engine.query("Where is the sky purple?")
+ print("==== query with new data")
+ print(textwrap.fill(str(response), 100))
diff --git a/examples/LlamaIndex-demo/lancedb_cloud/requirements.txt b/examples/LlamaIndex-demo/lancedb_cloud/requirements.txt
new file mode 100644
index 00000000..8272f875
--- /dev/null
+++ b/examples/LlamaIndex-demo/lancedb_cloud/requirements.txt
@@ -0,0 +1,5 @@
+llama-index-vector-stores-lancedb
+llama-index-readers-file
+llama-index-embeddings-openai
+llama-index-llms-openai
+lancedb
\ No newline at end of file
diff --git a/examples/QueryExpansion&Reranker/README.md b/examples/QueryExpansion&Reranker/README.md
index b89b35d8..3daf63c6 100644
--- a/examples/QueryExpansion&Reranker/README.md
+++ b/examples/QueryExpansion&Reranker/README.md
@@ -19,4 +19,4 @@ Our focus is on improving the precision and recall of document retrieval process
For a detailed exploration of the concepts and methodologies discussed in this project,
visit our blog
-[Read the Blog Post](https://blog.lancedb.com/improving-rag-with-query-expansion-reranking-models/)
+[Read the Blog Post](https://aksdesai1998.medium.com/improving-rag-with-query-expansion-reranking-models-31d252856580)
diff --git a/examples/SuperAgent_Autogen/main.ipynb b/examples/SuperAgent_Autogen/main.ipynb
index 9278654b..13047b03 100644
--- a/examples/SuperAgent_Autogen/main.ipynb
+++ b/examples/SuperAgent_Autogen/main.ipynb
@@ -11,55 +11,53 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3poVgyh-bZJ-",
- "outputId": "ad799a6e-7eec-4e14-dae3-f7e86c9e67cc"
+ "outputId": "7244e7cf-8eca-481d-fd05-d21a82460d47"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.8/88.8 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m811.8/811.8 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.7/226.7 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m14.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.9/111.9 kB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m284.0/284.0 kB\u001b[0m \u001b[31m13.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m295.2/295.2 kB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.0/77.0 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m21.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m239.4/239.4 kB\u001b[0m \u001b[31m23.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.7/55.7 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.6/21.6 MB\u001b[0m \u001b[31m34.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m38.3/38.3 MB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.4/55.4 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.7/98.7 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
- "llmx 0.0.15a0 requires cohere, which is not installed.\n",
- "ibis-framework 7.1.0 requires pyarrow<15,>=2, but you have pyarrow 15.0.0 which is incompatible.\u001b[0m\u001b[31m\n",
- "\u001b[0m"
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.8/88.8 kB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m973.5/973.5 kB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m12.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m320.6/320.6 kB\u001b[0m \u001b[31m14.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.9/18.9 MB\u001b[0m \u001b[31m36.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m290.4/290.4 kB\u001b[0m \u001b[31m22.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m296.7/296.7 kB\u001b[0m \u001b[31m20.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.0/77.0 kB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m308.5/308.5 kB\u001b[0m \u001b[31m24.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m122.8/122.8 kB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m22.8/22.8 MB\u001b[0m \u001b[31m30.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.5/142.5 kB\u001b[0m \u001b[31m1.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.7/98.7 kB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h"
]
}
],
"source": [
- "%pip install pyautogen~=0.1.0 langchain openai tiktoken lancedb pypdf -q -U"
+ "%pip install pyautogen~=0.1.0 langchain langchain_community openai tiktoken lancedb pypdf -q -U"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 2,
"metadata": {
"id": "0tLTTT9ucFEb"
},
"outputs": [],
"source": [
- "from langchain.embeddings import OpenAIEmbeddings\n",
+ "from langchain_community.embeddings import OpenAIEmbeddings\n",
"from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
"from langchain.document_loaders import PyPDFLoader\n",
"from langchain.memory import ConversationBufferMemory\n",
@@ -80,60 +78,88 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 3,
"metadata": {
- "id": "6RuVu12whCG0"
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "6RuVu12whCG0",
+ "outputId": "1300d37d-65ae-4e24-cbfb-d1cc701db28d"
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Requirement already satisfied: pyautogen in /usr/local/lib/python3.10/dist-packages (0.1.14)\n",
+ "Requirement already satisfied: diskcache in /usr/local/lib/python3.10/dist-packages (from pyautogen) (5.6.3)\n",
+ "Requirement already satisfied: flaml in /usr/local/lib/python3.10/dist-packages (from pyautogen) (2.1.2)\n",
+ "Requirement already satisfied: openai<1 in /usr/local/lib/python3.10/dist-packages (from pyautogen) (0.28.1)\n",
+ "Requirement already satisfied: python-dotenv in /usr/local/lib/python3.10/dist-packages (from pyautogen) (1.0.1)\n",
+ "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from pyautogen) (2.4.0)\n",
+ "Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.10/dist-packages (from openai<1->pyautogen) (2.31.0)\n",
+ "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from openai<1->pyautogen) (4.66.4)\n",
+ "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from openai<1->pyautogen) (3.9.5)\n",
+ "Requirement already satisfied: NumPy>=1.17 in /usr/local/lib/python3.10/dist-packages (from flaml->pyautogen) (1.25.2)\n",
+ "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (3.3.2)\n",
+ "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (3.7)\n",
+ "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (2.0.7)\n",
+ "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (2024.2.2)\n",
+ "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<1->pyautogen) (1.3.1)\n",
+ "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<1->pyautogen) (23.2.0)\n",
+ "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<1->pyautogen) (1.4.1)\n",
+ "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<1->pyautogen) (6.0.5)\n",
+ "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<1->pyautogen) (1.9.4)\n",
+ "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<1->pyautogen) (4.0.3)\n"
+ ]
+ }
+ ],
"source": [
"!pip install pyautogen"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 38,
"metadata": {
"id": "sUFdvTyVh8xF"
},
"outputs": [],
"source": [
"import lancedb\n",
+ "import os\n",
+ "\n",
+ "# setup OPENAI API KEY\n",
+ "os.environ[\"OPENAI_API_KEY\"] = \"sk-....\"\n",
"\n",
- "embeddings = OpenAIEmbeddings(openai_api_key=\"sk-yourapikey\")"
+ "embeddings = OpenAIEmbeddings()"
]
},
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "kztlyFIXU8m-"
- },
- "source": []
- },
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ODKg12trdhX-",
- "outputId": "a7041322-f633-496c-a8e8-126a81cbb5d9"
+ "outputId": "4736d14d-454c-4458-98c4-d2d0d2edceb9"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "--2024-02-11 04:40:16-- https://pdf.usaid.gov/pdf_docs/PA00TBCT.pdf\n",
- "Resolving pdf.usaid.gov (pdf.usaid.gov)... 23.7.61.67, 2600:1408:ec00:380::1923, 2600:1408:ec00:38f::1923\n",
- "Connecting to pdf.usaid.gov (pdf.usaid.gov)|23.7.61.67|:443... connected.\n",
+ "--2024-05-28 05:17:50-- https://pdf.usaid.gov/pdf_docs/PA00TBCT.pdf\n",
+ "Resolving pdf.usaid.gov (pdf.usaid.gov)... 23.4.180.157, 2600:1408:5400:197::1923, 2600:1408:5400:183::1923\n",
+ "Connecting to pdf.usaid.gov (pdf.usaid.gov)|23.4.180.157|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 6419525 (6.1M) [application/pdf]\n",
"Saving to: ‘food.pdf’\n",
"\n",
- "food.pdf 100%[===================>] 6.12M --.-KB/s in 0.1s \n",
+ "food.pdf 100%[===================>] 6.12M 27.0MB/s in 0.2s \n",
"\n",
- "2024-02-11 04:40:16 (42.7 MB/s) - ‘food.pdf’ saved [6419525/6419525]\n",
+ "2024-05-28 05:17:51 (27.0 MB/s) - ‘food.pdf’ saved [6419525/6419525]\n",
"\n"
]
}
@@ -143,14 +169,40 @@
"!wget -O food.pdf https://pdf.usaid.gov/pdf_docs/PA00TBCT.pdf"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yV0pNPiRPy8h"
+ },
+ "source": [
+ "# create file name with OAI_CONFIG_LIT.json"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 53,
+ "metadata": {
+ "id": "yWDhjTDMcFBi"
+ },
+ "outputs": [],
+ "source": [
+ "# create file name with OAI_CONFIG_LIT.\n",
+ "import json\n",
+ "\n",
+ "config = [{\"model\": \"gpt-4\", \"api_key\": os.environ[\"OPENAI_API_KEY\"]}]\n",
+ "\n",
+ "with open(\"OAI_CONFIG_LIT.json\", \"w\") as fp:\n",
+ " json.dump(config, fp)"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {
"id": "1oC3NAFyd4Kb"
},
"source": [
- "create OAI_CONFIG_LIST.json file in pwd & upload\n",
- "in it\n",
+ "**create OAI_CONFIG_LIST.json file in pwd & upload\n",
+ "in it**\n",
"\n",
"\n",
"[\n",
@@ -163,7 +215,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 41,
"metadata": {
"id": "H1bRXWu-cE_C"
},
@@ -179,30 +231,9 @@
")"
]
},
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "yV0pNPiRPy8h"
- },
- "source": [
- "# create file name with OAI_CONFIG_LIT.json & put below authentications code"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "yWDhjTDMcFBi"
- },
- "outputs": [],
- "source": [
- "# create file name with OAI_CONFIG_LIT.\n",
- "[{\"model\": \"gpt-4\", \"api_key\": \"sk-yourapikey\"}]"
- ]
- },
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 42,
"metadata": {
"id": "5gapqmsscFG-"
},
@@ -218,70 +249,30 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 43,
"metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "5dLkCqa0dLXV",
- "outputId": "28ab5984-c875-4281-95e5-d48bfdd12e99"
+ "id": "5dLkCqa0dLXV"
},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.embeddings.openai.OpenAIEmbeddings` was deprecated in langchain-community 0.1.0 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import OpenAIEmbeddings`.\n",
- " warn_deprecated(\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"import lancedb\n",
"\n",
- "embeddings = OpenAIEmbeddings(openai_api_key=\"sk-yourapikey\")\n",
- "\n",
- "db = lancedb.connect(\"/tmp/lancedb\")\n",
- "table = db.create_table(\n",
- " \"my_table\",\n",
- " data=[\n",
- " {\n",
- " \"vector\": embeddings.embed_query(\"Hello food\"),\n",
- " \"text\": \"Hello food\",\n",
- " \"id\": \"1\",\n",
- " }\n",
- " ],\n",
- " mode=\"overwrite\",\n",
- ")\n",
+ "embeddings = OpenAIEmbeddings()\n",
"\n",
- "vectorstore = LanceDB.from_documents(docs, embeddings, connection=table)"
+ "vectorstore = LanceDB.from_documents(documents=docs, embedding=embeddings)"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 45,
"metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "YMBoF5kucFMJ",
- "outputId": "13d7edab-5f3d-4698-fe6f-40f33dcd865a"
+ "id": "YMBoF5kucFMJ"
},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.llms.openai.OpenAI` was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import OpenAI`.\n",
- " warn_deprecated(\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"qa = ConversationalRetrievalChain.from_llm(\n",
" OpenAI(\n",
" temperature=0,\n",
- " openai_api_key=\"sk-yourapikey\",\n",
" ),\n",
" vectorstore.as_retriever(),\n",
" memory=ConversationBufferMemory(memory_key=\"chat_history\", return_messages=True),\n",
@@ -290,7 +281,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 46,
"metadata": {
"id": "HjSVygLIcSEX"
},
@@ -303,34 +294,26 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 47,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 160
+ "height": 90
},
"id": "XCqxSaQSepsW",
- "outputId": "c1fc1bdc-9f2e-467b-cb51-fdde3fc964ae"
+ "outputId": "309a74fe-05c6-4d78-ad3e-e720b77d6e3c"
},
"outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function `__call__` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.\n",
- " warn_deprecated(\n"
- ]
- },
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
- "' Good food is food that provides the recommended amounts of nutrients for the body to perform all its physiological activities. It is important to eat the right food, at the right time, in the right amounts, and prepared correctly in order to maintain a balanced diet and promote good nutrition. Good food is essential for physical and cognitive development and can improve overall health and quality of life.'"
+ "' Good food is any type of food that provides the recommended amounts of nutrients for the body to perform its physiological activities. It should be eaten at the right time, in the right amounts, and prepared correctly. Good food is important for physical and cognitive development, and can help prevent health problems. Foods can also be classified according to their functions in the body, such as energy-giving foods, body-building foods, and protective foods.'"
]
},
- "execution_count": 18,
+ "execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
@@ -342,13 +325,13 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "BY4Fz-l7cUCA",
- "outputId": "3a10b926-3659-46d3-d76b-7e083daf8fca"
+ "outputId": "e90c9b52-ab8e-4176-e249-fba0f20cae68"
},
"outputs": [
{
@@ -357,16 +340,16 @@
"text": [
"Requirement already satisfied: pyautogen in /usr/local/lib/python3.10/dist-packages (0.1.14)\n",
"Requirement already satisfied: diskcache in /usr/local/lib/python3.10/dist-packages (from pyautogen) (5.6.3)\n",
- "Requirement already satisfied: flaml in /usr/local/lib/python3.10/dist-packages (from pyautogen) (2.1.1)\n",
+ "Requirement already satisfied: flaml in /usr/local/lib/python3.10/dist-packages (from pyautogen) (2.1.2)\n",
"Requirement already satisfied: openai<1 in /usr/local/lib/python3.10/dist-packages (from pyautogen) (0.28.1)\n",
"Requirement already satisfied: python-dotenv in /usr/local/lib/python3.10/dist-packages (from pyautogen) (1.0.1)\n",
"Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from pyautogen) (2.4.0)\n",
"Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.10/dist-packages (from openai<1->pyautogen) (2.31.0)\n",
- "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from openai<1->pyautogen) (4.66.1)\n",
- "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from openai<1->pyautogen) (3.9.3)\n",
- "Requirement already satisfied: NumPy>=1.17.0rc1 in /usr/local/lib/python3.10/dist-packages (from flaml->pyautogen) (1.23.5)\n",
+ "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from openai<1->pyautogen) (4.66.4)\n",
+ "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from openai<1->pyautogen) (3.9.5)\n",
+ "Requirement already satisfied: NumPy>=1.17 in /usr/local/lib/python3.10/dist-packages (from flaml->pyautogen) (1.25.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (3.3.2)\n",
- "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (3.6)\n",
+ "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (3.7)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (2.0.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai<1->pyautogen) (2024.2.2)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai<1->pyautogen) (1.3.1)\n",
@@ -393,7 +376,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 48,
"metadata": {
"id": "Vca8Y_khcUID"
},
@@ -425,7 +408,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 49,
"metadata": {
"id": "1XHjzIYAcfE7"
},
@@ -451,13 +434,13 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 50,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "hOZKxakHchZ4",
- "outputId": "7ca36fe0-a211-409a-abb0-57e3cd05e429"
+ "outputId": "20693c38-773a-47d2-bfc4-8311c7ef4d1d"
},
"outputs": [
{
@@ -475,7 +458,6 @@
"\n",
"***** Suggested function Call: answer_food_question *****\n",
"Arguments: \n",
- "\n",
"{\n",
" \"question\": \"what is good food?\"\n",
"}\n",
@@ -487,13 +469,13 @@
"user_proxy (to assistant):\n",
"\n",
"***** Response from calling function \"answer_food_question\" *****\n",
- " Good food is food that is able to provide the recommended amounts of nutrients for the body to perform all its physiological activities. It is important for our health and well-being because it helps us maintain a balanced diet, promotes physical and cognitive development, and protects us from foodborne illnesses. Good food also ensures that we have enough energy for physical activity and basic body functions, and it helps us maintain a healthy weight. Additionally, good food can improve our overall quality of life and productivity.\n",
+ " Good food is food that is able to provide the recommended amounts of nutrients for the body to perform all its physiological activities. It is important because it helps with physical and cognitive development, promotes good health, and improves the quality of life. Good food should be eaten at the right time, in the right amounts, and prepared correctly. It can also be classified into different categories based on its function in the body, such as energy-giving foods, body-building foods, and protective foods.\n",
"*****************************************************************\n",
"\n",
"--------------------------------------------------------------------------------\n",
"assistant (to user_proxy):\n",
"\n",
- "Good food is food that is able to provide the recommended amounts of nutrients for the body to perform all its physiological activities. It is important for our health and well-being because it helps us maintain a balanced diet, promotes physical and cognitive development, and protects us from foodborne illnesses. Good food also ensures that we have enough energy for physical activity and basic body functions, and it helps us maintain a healthy weight. Additionally, good food can improve our overall quality of life and productivity.\n",
+ "Good food is food that provides the recommended amounts of nutrients for the body to perform all its physiological activities. It is important because it helps with physical and cognitive development, promotes good health, and improves the quality of life. Good food should be eaten at the right time, in the right amounts, and prepared correctly. It can also be classified into different categories based on its function in the body, such as energy-giving foods, body-building foods, and protective foods.\n",
"\n",
"TERMINATE\n",
"\n",
@@ -518,13 +500,13 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 51,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "UDXo2V06fNjz",
- "outputId": "37cf6766-9b68-4a81-e0c6-245d2af28a30"
+ "outputId": "057425b5-f0c1-4132-cdc6-fd03197e95c3"
},
"outputs": [
{
@@ -549,26 +531,26 @@
" - Deficiency Symptoms: Osteoporosis, rickets in children, muscle cramps, dental problems.\n",
"\n",
"2. Iron:\n",
- " - Sources: Red meat, poultry, eggs, fruits, green vegetables, fortified bread.\n",
- " - Functions: Essential for the production of red blood cells, helps in oxygen transport.\n",
- " - Deficiency Symptoms: Anemia, fatigue, weakness, immune system problems.\n",
+ " - Sources: Red meat, poultry, fish, legumes, fortified cereals.\n",
+ " - Functions: Essential for the production of red blood cells, carries oxygen in the blood.\n",
+ " - Deficiency Symptoms: Anemia, fatigue, weakness, pale skin, shortness of breath.\n",
"\n",
- "3. Magnesium:\n",
- " - Sources: Nuts, seeds, whole grains, green leafy vegetables, fish, beans, yogurt.\n",
- " - Functions: Helps in over 300 enzyme reactions, including regulation of blood pressure, supports immune system.\n",
- " - Deficiency Symptoms: Loss of appetite, nausea, fatigue, weakness, muscle cramps, numbness.\n",
+ "3. Potassium:\n",
+ " - Sources: Bananas, oranges, cantaloupes, raisins, nuts, fish, chicken, beef, and pork.\n",
+ " - Functions: Helps maintain fluid balance, nerve transmission, muscle contractions.\n",
+ " - Deficiency Symptoms: Weakness, fatigue, muscle cramps, constipation.\n",
"\n",
- "4. Potassium:\n",
- " - Sources: Bananas, oranges, cantaloupe, honeydew, apricots, grapefruit, cooked spinach, cooked broccoli, potatoes, sweet potatoes, mushrooms, peas, cucumbers, zucchini, eggplant, pumpkins, leafy greens.\n",
- " - Functions: Maintains fluid balance, helps in nerve transmission and muscle contraction.\n",
- " - Deficiency Symptoms: Fatigue, weakness, constipation, muscle cramps.\n",
+ "4. Magnesium:\n",
+ " - Sources: Green leafy vegetables, nuts, seeds, whole grains, fish.\n",
+ " - Functions: Involved in over 300 enzymatic reactions in the body including energy production, protein synthesis, muscle and nerve function.\n",
+ " - Deficiency Symptoms: Loss of appetite, nausea, fatigue, weakness, muscle cramps, numbness and tingling.\n",
"\n",
"5. Zinc:\n",
- " - Sources: Meat, shellfish, legumes, seeds, nuts, dairy, eggs, whole grains.\n",
- " - Functions: Necessary for immune function, protein synthesis, DNA synthesis, cell division, wound healing.\n",
- " - Deficiency Symptoms: Growth retardation, loss of appetite, impaired immune function, hair loss, diarrhea, delayed sexual maturation.\n",
+ " - Sources: Meat, shellfish, legumes, seeds, nuts, dairy, eggs.\n",
+ " - Functions: Supports immune function, protein synthesis, wound healing, DNA synthesis, and cell division.\n",
+ " - Deficiency Symptoms: Loss of appetite, impaired immune function, hair loss, diarrhea, delayed sexual maturation.\n",
"\n",
- "Please note that this is not an exhaustive list and there are other essential minerals as well. Also, the symptoms of deficiency can vary from person to person and can often be symptoms of other conditions as well. Always consult with a healthcare provider for accurate information.\n",
+ "Please note that this is not an exhaustive list and there are many other essential minerals that the body needs. It's also important to remember that while these minerals are essential for health, they should be consumed in moderation as too much can also lead to health problems. Always consult with a healthcare provider or a registered dietitian for personalized advice.\n",
"\n",
"--------------------------------------------------------------------------------\n",
"user_proxy (to assistant):\n",
@@ -592,13 +574,13 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 52,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "UrlFGYW0g0sJ",
- "outputId": "e82396e5-3dca-402c-889c-394557aeea0d"
+ "outputId": "0fbf3ed8-0103-4904-9737-2a41e6e1cf0d"
},
"outputs": [
{
@@ -628,22 +610,13 @@
"user_proxy (to assistant):\n",
"\n",
"***** Response from calling function \"answer_food_question\" *****\n",
- " Foods that are rich in Vitamin A, such as yellow/orange fruits and vegetables, dark green and deep yellow fruits and vegetables, liver, egg yolk, dairy products, and margarine can help maintain healthy eyes.\n",
+ " Fruits and vegetables, particularly dark green leafy vegetables and yellow fruits, are considered protective and can help keep eyes healthy.\n",
"*****************************************************************\n",
"\n",
"--------------------------------------------------------------------------------\n",
"assistant (to user_proxy):\n",
"\n",
- "Foods that are rich in Vitamin A can help maintain healthy eyes. These include:\n",
- "\n",
- "1. Yellow/orange fruits and vegetables: These include carrots, sweet potatoes, pumpkins, and apricots.\n",
- "2. Dark green and deep yellow fruits and vegetables: These include spinach, kale, and other leafy greens.\n",
- "3. Liver: This is a great source of Vitamin A.\n",
- "4. Egg yolk: This is another good source of Vitamin A.\n",
- "5. Dairy products: These include milk, cheese, and yogurt.\n",
- "6. Margarine: This is also a good source of Vitamin A.\n",
- "\n",
- "Including these foods in your diet can help keep your eyes healthy.\n",
+ "Fruits and vegetables, particularly dark green leafy vegetables and yellow fruits, are considered protective and can help keep eyes healthy. These foods are rich in vitamins A, C, E, and minerals like Copper and Zinc which are essential for eye health. Foods like carrots, sweet potatoes, spinach, kale, and other dark green leafy vegetables; and fish like salmon and tuna are good for eye health.\n",
"\n",
"TERMINATE\n",
"\n",
@@ -665,15 +638,6 @@
"\"\"\",\n",
")"
]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "En6-kvjcjaid"
- },
- "outputs": [],
- "source": []
}
],
"metadata": {
diff --git a/tutorials/better-rag-FLAIR/README.md b/examples/better-rag-FLAIR/README.md
similarity index 100%
rename from tutorials/better-rag-FLAIR/README.md
rename to examples/better-rag-FLAIR/README.md
diff --git a/tutorials/better-rag-FLAIR/app.py b/examples/better-rag-FLAIR/app.py
similarity index 100%
rename from tutorials/better-rag-FLAIR/app.py
rename to examples/better-rag-FLAIR/app.py
diff --git a/tutorials/better-rag-FLAIR/main.ipynb b/examples/better-rag-FLAIR/main.ipynb
similarity index 100%
rename from tutorials/better-rag-FLAIR/main.ipynb
rename to examples/better-rag-FLAIR/main.ipynb
diff --git a/tutorials/better-rag-FLAIR/requirements.txt b/examples/better-rag-FLAIR/requirements.txt
similarity index 100%
rename from tutorials/better-rag-FLAIR/requirements.txt
rename to examples/better-rag-FLAIR/requirements.txt
diff --git a/examples/databricks_DBRX_website_bot/README.md b/examples/databricks_DBRX_website_bot/README.md
index e99db29e..18d3883e 100644
--- a/examples/databricks_DBRX_website_bot/README.md
+++ b/examples/databricks_DBRX_website_bot/README.md
@@ -15,7 +15,7 @@ export DATABRICKS_TOKEN=
diff --git a/examples/movie-recommendation-with-genres/README.md b/examples/movie-recommendation-with-genres/README.md
new file mode 100644
index 00000000..f79e4e4a
--- /dev/null
+++ b/examples/movie-recommendation-with-genres/README.md
@@ -0,0 +1,9 @@
+# Movie Recommendation using Emebeddings and VectorDB
+
+
+
+This example provides a comprehensive guide on creating a movie recommendation system by leveraging the power of Embeddings and VectorDB. We'll explore how combining these two techniques can significantly enhance the recommendation experience, addressing key challenges faced by traditional systems.
+
+Colab walkthrough -
+
+[Read the Blog Post](https://blog.lancedb.com/movie-recommendation-system-using-lancedb-and-doc2vec/)
\ No newline at end of file
diff --git a/examples/movie-recommendation-with-genres/movie_recommendation_with_doc2vec_and_lancedb.ipynb b/examples/movie-recommendation-with-genres/movie_recommendation_with_doc2vec_and_lancedb.ipynb
new file mode 100644
index 00000000..2a75b19a
--- /dev/null
+++ b/examples/movie-recommendation-with-genres/movie_recommendation_with_doc2vec_and_lancedb.ipynb
@@ -0,0 +1,614 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "K45xhdPRsZJV"
+ },
+ "source": [
+ "# Movie Recommendation System using Doc2vec Embeddings and Vector DB"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XUj6NXD0sdgf"
+ },
+ "source": [
+ "This Colab notebook aims to illustrate the process of creating a recommendation system using embeddings and a Vector DB.\n",
+ "\n",
+ "This approach involves combining the various movie genres or characteristics of a movie to form Doc2Vec embeddings, which offer a comprehensive portrayal of the movie content.\n",
+ "\n",
+ "These embeddings serve dual purposes: they can either be directly inputted into a classification model for genre classification or stored in a VectorDB. By storing embeddings in a VectorDB, efficient retrieval and query search for recommendations become possible at a later stage.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qEa74a_Wtpc7"
+ },
+ "source": [
+ "### Installing the relevant dependencies\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "hyde90IntuFi"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install torch scikit-learn lancedb nltk gensim lancedb scipy==1.12 kaggle"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "shPjHTZbtxTh"
+ },
+ "source": [
+ "## Kaggle Configuration and Data Needs\n",
+ "\n",
+ "We are using a movies metadata data which is being uploaded on the Kaggle. To download the dataset and use it for our recommendation system, we will need a `kaggle.json` file containing our creds.\n",
+ "\n",
+ "You can download the `kaggle.json` file from your Kaggle account settings. Follow these steps and make your life easy.\n",
+ "\n",
+ "1. Go to Kaggle and log in to your account.\n",
+ "2. Navigate to Your Account Settings and click on your profile picture in the top right corner of the page, Now From the dropdown menu, select `Account`.\n",
+ "3. Scroll down to the `API` section, Click on `Create New API Token`. This will download a file named kaggle.json to your computer.\n",
+ "\n",
+ "Once you have the `kaggle.json` file, you need to upload it here on colab data space. After uploading the `kaggle.json` file, run the following code to set up the credentials and download the dataset in `data` directory"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "6Tl2qzgKsWtF"
+ },
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import os\n",
+ "\n",
+ "# Assuming kaggle.json is uploaded to the current directory\n",
+ "with open(\"kaggle.json\") as f:\n",
+ " kaggle_credentials = json.load(f)\n",
+ "\n",
+ "os.environ[\"KAGGLE_USERNAME\"] = kaggle_credentials[\"username\"]\n",
+ "os.environ[\"KAGGLE_KEY\"] = kaggle_credentials[\"key\"]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "8va-0of3sU0x"
+ },
+ "outputs": [],
+ "source": [
+ "from kaggle.api.kaggle_api_extended import KaggleApi\n",
+ "\n",
+ "# Initialize the Kaggle API\n",
+ "api = KaggleApi()\n",
+ "api.authenticate()\n",
+ "\n",
+ "# Specify the dataset you want to download\n",
+ "dataset = \"rounakbanik/the-movies-dataset\"\n",
+ "destination = \"data/\"\n",
+ "\n",
+ "# Create the destination directory if it doesn't exist\n",
+ "if not os.path.exists(destination):\n",
+ " os.makedirs(destination)\n",
+ "\n",
+ "# Download the dataset\n",
+ "api.dataset_download_files(dataset, path=destination, unzip=True)\n",
+ "\n",
+ "print(f\"Dataset {dataset} downloaded to {destination}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "hBYzad3lrY4e",
+ "outputId": "5a8f7983-80be-47e0-aa9c-ae4e10495c1e"
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [00:00<00:00, 5050.83it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5161.29it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5006.18it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5222.83it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5216.24it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5171.35it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5109.78it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5222.42it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5133.39it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5024.74it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5117.18it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 4963.78it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5405.55it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5369.51it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5349.33it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5374.53it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5194.32it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5296.75it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5204.32it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5309.43it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5333.12it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5289.35it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5317.42it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5322.46it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5378.43it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5488.32it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5546.43it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 2502.38it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5369.91it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 4354.99it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5193.60it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5536.27it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 3476.56it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 4819.07it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 4500.37it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5184.11it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5098.14it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5523.73it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 4655.12it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5113.63it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5336.63it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5564.83it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5310.91it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 5533.46it/s]\n",
+ "100%|██████████| 1000/1000 [00:00<00:00, 4255.41it/s]\n",
+ "100%|██████████| 466/466 [00:00<00:00, 5617.03it/s]\n",
+ "Building Vocabulary: 100%|██████████| 44506/44506 [00:00<00:00, 104121.48it/s]\n",
+ "Epoch 1: 100%|██████████| 44506/44506 [00:02<00:00, 20444.80it/s]\n",
+ "Epoch 2: 100%|██████████| 44506/44506 [00:02<00:00, 20700.43it/s]\n",
+ "Epoch 3: 100%|██████████| 44506/44506 [00:02<00:00, 20831.06it/s]\n",
+ "Epoch 4: 100%|██████████| 44506/44506 [00:02<00:00, 20885.78it/s]\n",
+ "Epoch 5: 100%|██████████| 44506/44506 [00:02<00:00, 19616.38it/s]\n",
+ "Epoch 6: 100%|██████████| 44506/44506 [00:02<00:00, 19634.24it/s]\n",
+ "Epoch 7: 100%|██████████| 44506/44506 [00:02<00:00, 20579.08it/s]\n",
+ "Epoch 8: 100%|██████████| 44506/44506 [00:02<00:00, 20727.00it/s]\n",
+ "Epoch 9: 100%|██████████| 44506/44506 [00:02<00:00, 21242.19it/s]\n",
+ "Epoch 10: 100%|██████████| 44506/44506 [00:02<00:00, 18476.39it/s]\n",
+ "Epoch 11: 100%|██████████| 44506/44506 [00:02<00:00, 21169.07it/s]\n",
+ "Epoch 12: 100%|██████████| 44506/44506 [00:02<00:00, 20967.64it/s]\n",
+ "Epoch 13: 100%|██████████| 44506/44506 [00:02<00:00, 20192.34it/s]\n",
+ "Epoch 14: 100%|██████████| 44506/44506 [00:02<00:00, 18910.62it/s]\n",
+ "Epoch 15: 100%|██████████| 44506/44506 [00:02<00:00, 20810.41it/s]\n",
+ "Epoch 16: 100%|██████████| 44506/44506 [00:02<00:00, 21361.88it/s]\n",
+ "Epoch 17: 100%|██████████| 44506/44506 [00:02<00:00, 18440.51it/s]\n",
+ "Epoch 18: 100%|██████████| 44506/44506 [00:02<00:00, 21206.01it/s]\n",
+ "Epoch 19: 100%|██████████| 44506/44506 [00:02<00:00, 20086.00it/s]\n",
+ "Epoch 20: 100%|██████████| 44506/44506 [00:02<00:00, 20943.08it/s]\n"
+ ]
+ }
+ ],
+ "source": [
+ "import pandas as pd\n",
+ "import numpy as np\n",
+ "import torch\n",
+ "import torch.nn as nn\n",
+ "import torch.optim as optim\n",
+ "from torch.utils.data import DataLoader, TensorDataset\n",
+ "from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n",
+ "from nltk.tokenize import word_tokenize\n",
+ "from sklearn.preprocessing import MultiLabelBinarizer\n",
+ "from sklearn.model_selection import train_test_split\n",
+ "from tqdm import tqdm\n",
+ "\n",
+ "# Read data from CSV file\n",
+ "movie_data = pd.read_csv(\n",
+ " \"/Users/vipul/Nova/Projects/genre_spectrum/movies_metadata.csv\", low_memory=False\n",
+ ")\n",
+ "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
+ "\n",
+ "\n",
+ "def preprocess_data(movie_data_chunk):\n",
+ " tagged_docs = []\n",
+ " valid_indices = []\n",
+ " movie_info = []\n",
+ "\n",
+ " # Wrap your loop with tqdm\n",
+ " for i, row in tqdm(movie_data_chunk.iterrows(), total=len(movie_data_chunk)):\n",
+ " try:\n",
+ " # Constructing movie text\n",
+ " movies_text = \"\"\n",
+ " movies_text += \"Overview: \" + row[\"overview\"] + \"\\n\"\n",
+ " genres = \", \".join([genre[\"name\"] for genre in eval(row[\"genres\"])])\n",
+ " movies_text += \"Overview: \" + row[\"overview\"] + \"\\n\"\n",
+ " movies_text += \"Genres: \" + genres + \"\\n\"\n",
+ " movies_text += \"Title: \" + row[\"title\"] + \"\\n\"\n",
+ " tagged_docs.append(\n",
+ " TaggedDocument(words=word_tokenize(movies_text.lower()), tags=[str(i)])\n",
+ " )\n",
+ " valid_indices.append(i)\n",
+ " movie_info.append((row[\"title\"], genres))\n",
+ " except Exception as e:\n",
+ " continue\n",
+ "\n",
+ " return tagged_docs, valid_indices, movie_info\n",
+ "\n",
+ "\n",
+ "def train_doc2vec_model(tagged_data, num_epochs=20):\n",
+ " # Initialize Doc2Vec model\n",
+ " doc2vec_model = Doc2Vec(vector_size=100, min_count=2, epochs=num_epochs)\n",
+ " doc2vec_model.build_vocab(tqdm(tagged_data, desc=\"Building Vocabulary\"))\n",
+ " for epoch in range(num_epochs):\n",
+ " doc2vec_model.train(\n",
+ " tqdm(tagged_data, desc=f\"Epoch {epoch+1}\"),\n",
+ " total_examples=doc2vec_model.corpus_count,\n",
+ " epochs=doc2vec_model.epochs,\n",
+ " )\n",
+ "\n",
+ " return doc2vec_model\n",
+ "\n",
+ "\n",
+ "# Preprocess data and extract genres for the first 1000 movies\n",
+ "chunk_size = 1000\n",
+ "tagged_data = []\n",
+ "valid_indices = []\n",
+ "movie_info = []\n",
+ "for chunk_start in range(0, len(movie_data), chunk_size):\n",
+ " movie_data_chunk = movie_data.iloc[chunk_start : chunk_start + chunk_size]\n",
+ " chunk_tagged_data, chunk_valid_indices, chunk_movie_info = preprocess_data(\n",
+ " movie_data_chunk\n",
+ " )\n",
+ " tagged_data.extend(chunk_tagged_data)\n",
+ " valid_indices.extend(chunk_valid_indices)\n",
+ " movie_info.extend(chunk_movie_info)\n",
+ "\n",
+ "doc2vec_model = train_doc2vec_model(tagged_data)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VryHT1zVuEp0"
+ },
+ "source": [
+ "### Training a Neural Network for the Genre Classification Task"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "3pVNy2UKt5lu"
+ },
+ "outputs": [],
+ "source": [
+ "# Extract genre labels for the valid indices\n",
+ "genres_list = []\n",
+ "for i in valid_indices:\n",
+ " row = movie_data.loc[i]\n",
+ " genres = [genre[\"name\"] for genre in eval(row[\"genres\"])]\n",
+ " genres_list.append(genres)\n",
+ "\n",
+ "mlb = MultiLabelBinarizer()\n",
+ "genre_labels = mlb.fit_transform(genres_list)\n",
+ "\n",
+ "embeddings = []\n",
+ "for i in valid_indices:\n",
+ " embeddings.append(doc2vec_model.dv[str(i)])\n",
+ "X_train, X_test, y_train, y_test = train_test_split(\n",
+ " embeddings, genre_labels, test_size=0.2, random_state=42\n",
+ ")\n",
+ "\n",
+ "X_train_np = np.array(X_train, dtype=np.float32)\n",
+ "y_train_np = np.array(y_train, dtype=np.float32)\n",
+ "X_test_np = np.array(X_test, dtype=np.float32)\n",
+ "y_test_np = np.array(y_test, dtype=np.float32)\n",
+ "\n",
+ "X_train_tensor = torch.tensor(X_train_np)\n",
+ "y_train_tensor = torch.tensor(y_train_np)\n",
+ "X_test_tensor = torch.tensor(X_test_np)\n",
+ "y_test_tensor = torch.tensor(y_test_np)\n",
+ "\n",
+ "\n",
+ "class GenreClassifier(nn.Module):\n",
+ " def __init__(self, input_size, output_size):\n",
+ " super(GenreClassifier, self).__init__()\n",
+ " self.fc1 = nn.Linear(input_size, 512)\n",
+ " self.bn1 = nn.BatchNorm1d(512)\n",
+ " self.fc2 = nn.Linear(512, 256)\n",
+ " self.bn2 = nn.BatchNorm1d(256)\n",
+ " self.fc3 = nn.Linear(256, 128)\n",
+ " self.bn3 = nn.BatchNorm1d(128)\n",
+ " self.fc4 = nn.Linear(128, output_size)\n",
+ " self.relu = nn.ReLU()\n",
+ " self.dropout = nn.Dropout(p=0.2) # Adjust the dropout rate as needed\n",
+ "\n",
+ " def forward(self, x):\n",
+ " x = self.fc1(x)\n",
+ " x = self.bn1(x)\n",
+ " x = self.relu(x)\n",
+ " x = self.dropout(x)\n",
+ " x = self.fc2(x)\n",
+ " x = self.bn2(x)\n",
+ " x = self.relu(x)\n",
+ " x = self.dropout(x)\n",
+ " x = self.fc3(x)\n",
+ " x = self.bn3(x)\n",
+ " x = self.relu(x)\n",
+ " x = self.dropout(x)\n",
+ " x = self.fc4(x)\n",
+ " return x\n",
+ "\n",
+ "\n",
+ "# Move model to the selected device\n",
+ "model = GenreClassifier(input_size=100, output_size=len(mlb.classes_)).to(device)\n",
+ "\n",
+ "# Define loss function and optimizer\n",
+ "criterion = nn.BCEWithLogitsLoss()\n",
+ "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
+ "\n",
+ "# Training loop\n",
+ "epochs = 50\n",
+ "batch_size = 64\n",
+ "\n",
+ "train_dataset = TensorDataset(X_train_tensor.to(device), y_train_tensor.to(device))\n",
+ "train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " model.train()\n",
+ " running_loss = 0.0\n",
+ " for inputs, labels in train_loader:\n",
+ " inputs, labels = inputs.to(device), labels.to(device) # Move data to device\n",
+ " optimizer.zero_grad()\n",
+ " outputs = model(inputs)\n",
+ " loss = criterion(outputs, labels)\n",
+ " loss.backward()\n",
+ " optimizer.step()\n",
+ " running_loss += loss.item() * inputs.size(0)\n",
+ " epoch_loss = running_loss / len(train_loader.dataset)\n",
+ " print(f\"Epoch [{epoch + 1}/{epochs}], Loss: {epoch_loss:.4f}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yV8lTDYIubEQ"
+ },
+ "source": [
+ "### Testing the `model` to see if our model is able to predict the genres for the movies from the test dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "73D3aqdJuct8"
+ },
+ "outputs": [],
+ "source": [
+ "from sklearn.metrics import f1_score\n",
+ "\n",
+ "model.eval()\n",
+ "with torch.no_grad():\n",
+ " X_test_tensor, y_test_tensor = X_test_tensor.to(device), y_test_tensor.to(\n",
+ " device\n",
+ " ) # Move test data to device\n",
+ " outputs = model(X_test_tensor)\n",
+ " test_loss = criterion(outputs, y_test_tensor)\n",
+ " print(f\"Test Loss: {test_loss.item():.4f}\")\n",
+ "\n",
+ "\n",
+ "thresholds = [0.1] * len(mlb.classes_)\n",
+ "thresholds_tensor = torch.tensor(thresholds, device=device).unsqueeze(0)\n",
+ "\n",
+ "# Convert the outputs to binary predictions using varying thresholds\n",
+ "predicted_labels = (outputs > thresholds_tensor).cpu().numpy()\n",
+ "\n",
+ "# Convert binary predictions and actual labels to multi-label format\n",
+ "predicted_multilabels = mlb.inverse_transform(predicted_labels)\n",
+ "actual_multilabels = mlb.inverse_transform(y_test_np)\n",
+ "\n",
+ "# Print the Predicted and Actual Labels for each movie\n",
+ "for i, (predicted, actual) in enumerate(zip(predicted_multilabels, actual_multilabels)):\n",
+ " print(f\"Movie {i+1}:\")\n",
+ " print(f\" Predicted Labels: {predicted}\")\n",
+ " print(f\" Actual Labels: {actual}\")\n",
+ "\n",
+ "\n",
+ "# Compute F1-score\n",
+ "f1 = f1_score(y_test_np, predicted_labels, average=\"micro\")\n",
+ "print(f\"F1-score: {f1:.4f}\")\n",
+ "\n",
+ "# Saving the trained model\n",
+ "torch.save(model.state_dict(), \"trained_model.pth\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kZrHpMm4un0G"
+ },
+ "source": [
+ "### Storing the Doc2Vec Embeddings into LanceDB VectorDatabase"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "BTTNb9irrY4h"
+ },
+ "outputs": [],
+ "source": [
+ "import lancedb\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "\n",
+ "data = []\n",
+ "\n",
+ "for i in valid_indices:\n",
+ " embedding = doc2vec_model.dv[str(i)]\n",
+ " title, genres = movie_info[valid_indices.index(i)]\n",
+ " data.append({\"title\": title, \"genres\": genres, \"vector\": embedding.tolist()})\n",
+ "\n",
+ "db = lancedb.connect(\".db\")\n",
+ "tbl = db.create_table(\"doc2vec_embeddings\", data, mode=\"Overwrite\")\n",
+ "db[\"doc2vec_embeddings\"].head()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ciUFn7uQrY4i"
+ },
+ "outputs": [],
+ "source": [
+ "def get_recommendations(title):\n",
+ " pd_data = pd.DataFrame(data)\n",
+ " result = (\n",
+ " tbl.search(pd_data[pd_data[\"title\"] == title][\"vector\"].values[0])\n",
+ " .metric(\"cosine\")\n",
+ " .limit(10)\n",
+ " .to_pandas()\n",
+ " )\n",
+ " return result[[\"title\"]]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8Kz-JGsTuwmk"
+ },
+ "source": [
+ "### D-Day : Let's generate some recommendations"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "uw_El12JrY4j",
+ "outputId": "c245bab5-7966-4fd1-ec72-37f708c3b570"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | title | \n", + "
|---|---|
| 0 | \n", + "Vertical Limit | \n", + "
| 1 | \n", + "Demons of War | \n", + "
| 2 | \n", + "Fear and Desire | \n", + "
| 3 | \n", + "Escape from Sobibor | \n", + "
| 4 | \n", + "Last Girl Standing | \n", + "
| 5 | \n", + "K2: Siren of the Himalayas | \n", + "
| 6 | \n", + "Ghost Ship | \n", + "
| 7 | \n", + "Camp Massacre | \n", + "
| 8 | \n", + "Captain Nemo and the Underwater City | \n", + "
| 9 | \n", + "Seas Beneath | \n", + "