assignments/assignment_yourname_class9.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XKzF6dMaiLyP"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_generative_ai/blob/main/assignments/assignment_yourname_class9.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YDTXd8-Lmp8Q"
   },
   "source": [
    "# T81-559: Applications of Generative AI\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).\n",
    "\n",
    "**Module 9 Assignment: MultiModal Models**\n",
    "\n",
    "**Student Name: Your Name**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ncNrAEpzmp8S"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "If you are using Google CoLab, it will be necessary to mount your GDrive so that you can send your notebook during the submit process. Running the following code will map your GDrive to ```/content/drive```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fU9UhAxTmp8S",
    "outputId": "dc743818-98f7-4cae-ce4d-f7255fd0e3fe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n",
      "Note: using Google CoLab\n",
      "Collecting langchain\n",
      "  Downloading langchain-0.3.0-py3-none-any.whl.metadata (7.1 kB)\n",
      "Collecting openai\n",
      "  Downloading openai-1.45.0-py3-none-any.whl.metadata (22 kB)\n",
      "Collecting langchain_openai\n",
      "  Downloading langchain_openai-0.2.0-py3-none-any.whl.metadata (2.6 kB)\n",
      "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (6.0.2)\n",
      "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.34)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.10.5)\n",
      "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n",
      "Collecting langchain-core<0.4.0,>=0.3.0 (from langchain)\n",
      "  Downloading langchain_core-0.3.0-py3-none-any.whl.metadata (6.2 kB)\n",
      "Collecting langchain-text-splitters<0.4.0,>=0.3.0 (from langchain)\n",
      "  Downloading langchain_text_splitters-0.3.0-py3-none-any.whl.metadata (2.3 kB)\n",
      "Collecting langsmith<0.2.0,>=0.1.17 (from langchain)\n",
      "  Downloading langsmith-0.1.120-py3-none-any.whl.metadata (13 kB)\n",
      "Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.26.4)\n",
      "Requirement already satisfied: pydantic<3.0.0,>=2.7.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.9.1)\n",
      "Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.32.3)\n",
      "Collecting tenacity!=8.4.0,<9.0.0,>=8.1.0 (from langchain)\n",
      "  Downloading tenacity-8.5.0-py3-none-any.whl.metadata (1.2 kB)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)\n",
      "Collecting httpx<1,>=0.23.0 (from openai)\n",
      "  Downloading httpx-0.27.2-py3-none-any.whl.metadata (7.1 kB)\n",
      "Collecting jiter<1,>=0.4.0 (from openai)\n",
      "  Downloading jiter-0.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB)\n",
      "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai) (1.3.1)\n",
      "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai) (4.66.5)\n",
      "Requirement already satisfied: typing-extensions<5,>=4.11 in /usr/local/lib/python3.10/dist-packages (from openai) (4.12.2)\n",
      "Collecting tiktoken<1,>=0.7 (from langchain_openai)\n",
      "  Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n",
      "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (2.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (24.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.1.0)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.11.1)\n",
      "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (3.8)\n",
      "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (1.2.2)\n",
      "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2024.8.30)\n",
      "Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai)\n",
      "  Downloading httpcore-1.0.5-py3-none-any.whl.metadata (20 kB)\n",
      "Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx<1,>=0.23.0->openai)\n",
      "  Downloading h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)\n",
      "Collecting jsonpatch<2.0,>=1.33 (from langchain-core<0.4.0,>=0.3.0->langchain)\n",
      "  Downloading jsonpatch-1.33-py2.py3-none-any.whl.metadata (3.0 kB)\n",
      "Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.4.0,>=0.3.0->langchain) (24.1)\n",
      "Collecting orjson<4.0.0,>=3.9.14 (from langsmith<0.2.0,>=0.1.17->langchain)\n",
      "  Downloading orjson-3.10.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (50 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.4/50.4 kB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.23.3 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (2.23.3)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2.0.7)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.1.0)\n",
      "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken<1,>=0.7->langchain_openai) (2024.5.15)\n",
      "Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain-core<0.4.0,>=0.3.0->langchain)\n",
      "  Downloading jsonpointer-3.0.0-py2.py3-none-any.whl.metadata (2.3 kB)\n",
      "Downloading langchain-0.3.0-py3-none-any.whl (1.0 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading openai-1.45.0-py3-none-any.whl (374 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m374.1/374.1 kB\u001b[0m \u001b[31m20.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading langchain_openai-0.2.0-py3-none-any.whl (51 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m51.5/51.5 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading httpx-0.27.2-py3-none-any.whl (76 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.4/76.4 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading httpcore-1.0.5-py3-none-any.whl (77 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading jiter-0.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (318 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m318.9/318.9 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading langchain_core-0.3.0-py3-none-any.whl (405 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m405.1/405.1 kB\u001b[0m \u001b[31m23.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading langchain_text_splitters-0.3.0-py3-none-any.whl (25 kB)\n",
      "Downloading langsmith-0.1.120-py3-none-any.whl (289 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m289.8/289.8 kB\u001b[0m \u001b[31m21.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading tenacity-8.5.0-py3-none-any.whl (28 kB)\n",
      "Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m36.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading jsonpatch-1.33-py2.py3-none-any.whl (12 kB)\n",
      "Downloading orjson-3.10.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m141.9/141.9 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading jsonpointer-3.0.0-py2.py3-none-any.whl (7.6 kB)\n",
      "Installing collected packages: tenacity, orjson, jsonpointer, jiter, h11, tiktoken, jsonpatch, httpcore, httpx, openai, langsmith, langchain-core, langchain-text-splitters, langchain_openai, langchain\n",
      "  Attempting uninstall: tenacity\n",
      "    Found existing installation: tenacity 9.0.0\n",
      "    Uninstalling tenacity-9.0.0:\n",
      "      Successfully uninstalled tenacity-9.0.0\n",
      "Successfully installed h11-0.14.0 httpcore-1.0.5 httpx-0.27.2 jiter-0.5.0 jsonpatch-1.33 jsonpointer-3.0.0 langchain-0.3.0 langchain-core-0.3.0 langchain-text-splitters-0.3.0 langchain_openai-0.2.0 langsmith-0.1.120 openai-1.45.0 orjson-3.10.7 tenacity-8.5.0 tiktoken-0.7.0\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "try:\n",
    "    from google.colab import drive, userdata\n",
    "    drive.mount('/content/drive', force_remount=True)\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# OpenAI Secrets\n",
    "if COLAB:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')\n",
    "\n",
    "# Install needed libraries in CoLab\n",
    "if COLAB:\n",
    "    !pip install langchain openai langchain_openai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QSKZqD1Mmp-C"
   },
   "source": [
    "# Assignment Submit Function\n",
    "\n",
    "You will submit the 10 programming assignments electronically.  The following submit function can be used to do this.  My server will perform a basic check of each assignment and let you know if it sees any basic problems.\n",
    "\n",
    "**It is unlikely that should need to modify this function.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "7F2MhA7bjag8"
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "import PIL\n",
    "import PIL.Image\n",
    "import io\n",
    "\n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - List of pandas dataframes or images.\n",
    "# key - Your student key that was emailed to you.\n",
    "# course - The course that you are in, currently t81-558 or t81-559.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.\n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,course,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    payload = []\n",
    "    for item in data:\n",
    "        if type(item) is PIL.Image.Image:\n",
    "            buffered = BytesIO()\n",
    "            item.save(buffered, format=\"PNG\")\n",
    "            payload.append({'PNG':base64.b64encode(buffered.getvalue()).decode('ascii')})\n",
    "        elif type(item) is pd.core.frame.DataFrame:\n",
    "            payload.append({'CSV':base64.b64encode(item.to_csv(index=False).encode('ascii')).decode(\"ascii\")})\n",
    "    r= requests.post(\"https://api.heatonresearch.com/wu/submit\",\n",
    "        headers={'x-api-key':key}, json={ 'payload': payload,'assignment': no, 'course':course, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code==200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8fJKkSenqklH"
   },
   "source": [
    "# Assignment Instructions\n",
    "\n",
    "For this assignment you are provided with 10 image files that contain 10 different webcam pictures taken at the [Venice Sidewalk Cafe](https://www.westland.net/beachcam/) a WebCam that has been in opration since 1996.  You can find the 10 images here:\n",
    "\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk1.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk2.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk3.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk4.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk5.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk6.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk7.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk8.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk9.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk10.jpg\n",
    "\n",
    "You can see a sample of the WebCam here:\n",
    "\n",
    "![alt text](https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk1.jpg)\n",
    "\n",
    "\n",
    "* image - The image number, 1 through 10.\n",
    "* crowded - Is this image crowded with people? (1=yes, 0=no)\n",
    "* cars - Are there cars in this image? (1=yes, 0=no)\n",
    "* bikes - Are there bikes in this image? (1=yes, 0=no)\n",
    "\n",
    "Your submitted data frame should also contain a column that identifies which image generated each row.  This column should be named **image** and contain integer numbers between 1 and 10.  There should be 10 rows in total.  The complete data frame should look something like this (not necessarily exactly these numbers).\n",
    "\n",
    "|image|crowded|cars|bikes|\n",
    "|-|-|-|-|\n",
    "|1|0|0|1\n",
    "|2|0|1|1\n",
    "|3|1|0|0\n",
    "|...|...|...|...|\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qYOvD3M7ofQl"
   },
   "source": [
    "### Example MultiModal Code\n",
    "\n",
    "You should use a MultiModal model to obtain the data for each of the 10 images. You should be able to construct a single prompt that gets you the three needed values for each item. I suggest you use the \"gpt-4o-mini\" model with a temperature of 0.1. You will need to develop a prompt that looks for each of the requested values.\n",
    "\n",
    "The following code shows an example of running a MultiModal model with a prompt on the first image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MY3gVyidmp-K",
    "outputId": "c8e1b8f4-ecb9-48a3-de62-2cb130caf4c7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The image depicts a beach scene, likely at a coastal location. In the\n",
      "foreground, there are palm trees and a grassy area leading to the beach. People\n",
      "can be seen walking along a path, some carrying bags or rolling carts. There are\n",
      "benches and a few tents set up, suggesting a market or food stand. The beach is\n",
      "sandy and extends toward the ocean, which is visible in the background. The sky\n",
      "is clear, indicating a sunny day, and there are minimal clouds. Overall, it\n",
      "appears to be a lively and vibrant beach environment.\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.messages import HumanMessage\n",
    "from langchain_openai import ChatOpenAI\n",
    "import base64\n",
    "import httpx\n",
    "import textwrap\n",
    "\n",
    "MODEL = \"gpt-4o-mini\"\n",
    "image_url = 'https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk1.jpg'\n",
    "prompt = \"Describe this image.\"\n",
    "\n",
    "# Initialize the GPT model\n",
    "model = ChatOpenAI(model=\"gpt-4o-mini\")\n",
    "\n",
    "# Fetch image data and encode it in base64\n",
    "image_data = base64.b64encode(httpx.get(image_url).content).decode(\"utf-8\")\n",
    "\n",
    "# Create a message with both text and the image\n",
    "message = HumanMessage(\n",
    "    content=[\n",
    "        {\"type\": \"text\", \"text\": prompt},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": f\"data:image/jpeg;base64,{image_data}\"}},\n",
    "    ],\n",
    ")\n",
    "\n",
    "# Get response with a modified prompt from GPT\n",
    "response = model.invoke([message])\n",
    "\n",
    "# Wrap the text output to avoid scrolling off the screen in Colab\n",
    "wrapped_output = textwrap.fill(response.content, width=80)\n",
    "print(wrapped_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lCy_pvDXqYv4"
   },
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SFmqHkLbYQVm",
    "outputId": "4acbcda6-b4e5-48e1-bc5d-10d61ed7b793"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: Submitted Assignment 9 (t81-559) for jtheaton:\n",
      "You have submitted this assignment 4 times. (this is fine)\n",
      "No errors, warnings, or notes on your data. Rock on! You will probably do well, but no guarantee. :-)\n"
     ]
    }
   ],
   "source": [
    "# Add your solution here\n",
    "\n",
    "# This is your student key that I emailed to you at the beginnning of the semester.\n",
    "key = \"5iuwhudihwiao6dsfw7dE2ml08iNfVOg6l0O3M06\"  # This is an example key and will not work.\n",
    "\n",
    "# You must also identify your source file.  (modify for your local setup)\n",
    "file='/content/drive/MyDrive/Colab Notebooks/assignment_solution_class9.ipynb'  # Google CoLab\n",
    "\n",
    "submit(source_file=file,data=[df],key=key,course='t81-559',no=9)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "anaconda-cloud": {},
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.11 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}