t81_559_class_08_3_small_llm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "whjsJasuhstV"
   },
   "source": [
    "**Note: This notebook is not designed to run from CoLab**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "euOZxlIMhstX"
   },
   "source": [
    "# T81-559: Applications of Generative Artificial Intelligence\n",
    "**Module 8: Kaggle**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d4Yov72PhstY"
   },
   "source": [
    "# Module 8 Material\n",
    "\n",
    "* Part 8.1: Introduction to Kaggle [[Video]](https://www.youtube.com/watch?v=t0iz2zZ-jXU&ab_channel=JeffHeaton) [[Notebook]](t81_559_class_08_1_kaggle_intro.ipynb)\n",
    "* Part 8.2: Kaggle Notebooks [[Video]](https://www.youtube.com/watch?v=5Bv8rFm_cas&ab_channel=JeffHeaton) [[Notebook]](t81_559_class_08_2_kaggle_notebooks.ipynb)\n",
    "* **Part 8.3: Small Large Language Models** [[Video]](https://www.youtube.com/watch?v=1Hm337_vVCM&ab_channel=JeffHeaton) [[Notebook]](t81_559_class_08_3_small_llm.ipynb)\n",
    "* Part 8.4: Accessing Small LLM from Kaggle [[Video]](https://www.youtube.com/watch?v=o5PriYNQrqo&ab_channel=JeffHeaton) [[Notebook]](t81_559_class_08_4_kaggle_llm.ipynb)\n",
    "* Part 8.5: Current Semester's Kaggle [[Video]]() [[Notebook]](t81_559_class_08_5_kaggle_project.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AcAUP0c3hstY"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running and maps Google Drive if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xsI496h5hstZ",
    "outputId": "77bd0ac7-e5db-4fc6-b3b0-14a90108d416"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "try:\n",
    "    from google.colab import drive, userdata\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# OpenAI Secrets\n",
    "if COLAB:\n",
    "    raise Exception(\"This notebook is not designed for CoLab\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pC9A-LaYhsta"
   },
   "source": [
    "# 8.3: Small Large Language Models\n",
    "\n",
    "Large Language Models (LLMs) can be run on regular laptop and desktop computers. Many users successfully run 7 billion parameter LLMs on these computers, even without the need for a dedicated GPU. Three popular platforms for running these models are Ollama, LMStudio, and another that you can choose based on your preferences and requirements.\n",
    "\n",
    "In this course, we will focus on using Ollama and LMStudio. Both platforms are well-suited for running LLMs on local machines and provide user-friendly interfaces and comprehensive support.\n",
    "\n",
    "Please note that the examples in this section should be executed locally on your own computer, rather than using cloud-based solutions like Google Colab. This approach ensures you have full control over the setup and can experience the performance and capabilities of running LLMs on your personal hardware.\n",
    "\n",
    "The following are some options for running LLM's locally.\n",
    "\n",
    "* [Ollama](https://ollama.com/)\n",
    "* [LMStudio](https://lmstudio.ai/)\n",
    "* [GPT4All](https://www.nomic.ai/gpt4all)\n",
    "\n",
    "## LangChain with LMStudio\n",
    "\n",
    "We will now demonstrate LMStudio, a powerful platform for running large language models locally. LMStudio can operate as a server that emulates the OpenAI protocol, enabling the use of the OpenAI LangChain driver. This capability allows seamless integration with applications and workflows that rely on the OpenAI API. The following code snippet sends a \"Hello World\" message to a model running on LMStudio, showcasing its ability to process and respond to text inputs efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model response:\n",
      "Hello!\n",
      "\n",
      "-----------\n",
      "{'token_usage': {'completion_tokens': 3, 'prompt_tokens': 7, 'total_tokens': 10}, 'model_name': 'C:\\\\Users\\\\jeffh\\\\.cache\\\\lm-studio\\\\models\\\\TheBloke\\\\OpenHermes-2.5-Mistral-7B-GGUF\\\\openhermes-2.5-mistral-7b.Q2_K.gguf', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}\n"
     ]
    }
   ],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "# We use the OpenAI langchain driver to communicate with LMStudio\n",
    "llm = ChatOpenAI(\n",
    "  temperature= 0.0,\n",
    "  openai_api_key=\"na\",\n",
    "  base_url=\"http://localhost:1234/v1/\")\n",
    "\n",
    "print(\"Model response:\")\n",
    "output = llm.invoke(\"Hello world\")\n",
    "print(output.content)\n",
    "print(\"-----------\")\n",
    "print(output.response_metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LangChain with Ollama\n",
    "\n",
    "You can also use Ollama for Mac, which, similar to LMStudio, establishes a local server to run large language models. However, unlike LMStudio, Ollama has a LangChain driver specifically created for it, providing a more streamlined and optimized integration for users. This driver facilitates easier setup and enhanced performance, making Ollama a compelling choice for running LLMs locally on Mac systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\" Hello there! How can I help you today? Is this your first time interacting with me? If you have any questions or topics you'd like to discuss, feel free to ask! I am here to assist you.\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_community.llms import Ollama\n",
    "\n",
    "llm2 = Ollama(model=\"mistral\")\n",
    "llm2.invoke(\"Hello world\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.11 (genai)",
   "language": "python",
   "name": "genai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}