Models/Llama3_2_Vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fbc587a1-80ad-4e5f-8631-1d9bc25dbbec",
   "metadata": {
    "collapsed": false,
    "name": "intro_notebook"
   },
   "source": [
    "# Run Llama 3.2 Vision Model on Snowflake's ML Container runtime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25bb963d-f2a6-4706-9487-3ccb56908992",
   "metadata": {
    "collapsed": false,
    "name": "cell1"
   },
   "source": [
    "Let's use Meta's Llama-3.2-11B-Vision-Instruct model with Hugging Face transformers!  \n",
    "\n",
    "- Turn an image into text\n",
    "- Turn an image of a table into a JSON representation.\n",
    "- Understand an invoice document\n",
    "\n",
    "This multimodal model is optimized for visual recognition, image reasoning, captioning, and answering general questions about an image. It can take both text and images as input, check out the model card for more information [Llama 3.2 11B](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct).\n",
    "\n",
    "You do need GPUs for this model. The 11B version requires greater than 30GB of GPU Memory (for AWS customers you want to use at least the GPU_NV_M compute pool."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee7d466e-8e92-4442-9d6f-f54d13635a61",
   "metadata": {
    "collapsed": false,
    "name": "cell12"
   },
   "source": [
    "#### Meta's Models\n",
    "\n",
    "You will need to fill out a form for access to the Meta Models.\n",
    "\n",
    "Meta Llama 3.2 is licensed under the LLAMA 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Customers are responsible for ensuring their compliance with the terms of this license and the Llama 3.2 Acceptable Use Policy.\n",
    "\n",
    "Note: Meta does not grant rights for the multimodal models in the Llama 3.2 license to users domiciled in the European Union, or companies with principle place of business in the European Union. See the Llama 3.2 Acceptable Use Policy for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b00b9479-e00e-4f12-8062-cb9a4fd1cf9e",
   "metadata": {
    "collapsed": false,
    "name": "intro_standard_imports"
   },
   "source": [
    "### Standard Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3775908f-ca36-4846-8f38-5adca39217f2",
   "metadata": {
    "codeCollapsed": false,
    "collapsed": false,
    "language": "python",
    "name": "standard_imports"
   },
   "outputs": [],
   "source": [
    "# Import python packages\n",
    "# We can also use Snowpark for our analyses!\n",
    "from snowflake.snowpark.context import get_active_session\n",
    "session = get_active_session()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be98fdc8-192a-47ee-b12a-da7f42b29204",
   "metadata": {
    "collapsed": false,
    "name": "intro_upgrade_transformers"
   },
   "source": [
    "### Upgrade transformers\n",
    "If you get an error here, make sure you have an external integration to install packages. You will also need the external integration for connecting to the Hugging Face hub to download the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caed9318-68c7-4449-a17a-f6ff0f074135",
   "metadata": {
    "codeCollapsed": false,
    "collapsed": false,
    "language": "python",
    "name": "cell4"
   },
   "outputs": [],
   "source": [
    "!pip install \"transformers>=4.45.0\" --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a942101-5cfb-4199-a3fb-284b920d44c7",
   "metadata": {
    "collapsed": false,
    "name": "intro_import_task_libs"
   },
   "source": [
    "### Import task specific libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d50cbf4-0c8d-4950-86cb-114990437ac9",
   "metadata": {
    "codeCollapsed": false,
    "collapsed": false,
    "language": "python",
    "name": "import_task_libs"
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import torch\n",
    "from PIL import Image\n",
    "from huggingface_hub import login\n",
    "from transformers import MllamaForConditionalGeneration, AutoProcessor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cd33259-39a4-4ab4-973e-ee7c0f45b3ca",
   "metadata": {
    "collapsed": false,
    "name": "intro_set_params"
   },
   "source": [
    "### Set params and log-in into Huggingface hub\n",
    "\n",
    "Add your [Hugging Face token](https://huggingface.co/settings/tokens) for downloading Llama. Please note, the [Llama vision models](https://huggingface.co/collections/meta-llama/llama-32-66f448ffc8c32f949b04c8cf) are gated. Meta requires you submit a form on Hugging Face for access to the model. You will need to do that before you can download the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2bb7d32-8d6d-48cc-81e5-da8e77734b05",
   "metadata": {
    "codeCollapsed": false,
    "collapsed": false,
    "language": "python",
    "name": "set_params"
   },
   "outputs": [],
   "source": [
    "hf_token = \"hf_XXX\"\n",
    "model_id = \"meta-llama/Llama-3.2-11B-Vision-Instruct\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87cdb1d9-2547-4e59-8414-0c79582ed56e",
   "metadata": {
    "collapsed": false,
    "name": "cell2"
   },
   "source": [
    "Let's use transformers which makes it easy to use the new Llama models. Transformers has added support for [Llama vision models](https://huggingface.co/docs/transformers/en/model_doc/mllama)  through the `MllamaForConditionalGeneration`. Other transformer models will require using and AutoClass or a specific class for the model, see more in the [transformers docs](https://huggingface.co/docs/transformers/en/model_doc/auto)\n",
    "\n",
    "The `device_map` option set to `auto` means for multi-GPU systems, the model will automatically use [Big Model Inference](https://huggingface.co/docs/accelerate/concept_guides/big_model_inference) from Hugging Face. This distributes the large vision language model across multiple GPUs!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c695373e-ac74-4b62-a1f1-08206cbd5c81",
   "metadata": {
    "codeCollapsed": false,
    "collapsed": false,
    "language": "python",
    "name": "download_model"
   },
   "outputs": [],
   "source": [
    "login(hf_token)\n",
    "model = MllamaForConditionalGeneration.from_pretrained(\n",
    "    model_id,\n",
    "    torch_dtype=torch.bfloat16,\n",
    "    device_map=\"auto\",\n",
    ")\n",
    "processor = AutoProcessor.from_pretrained(model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a616df2-4dda-431b-8e20-d1fffa4af219",
   "metadata": {
    "collapsed": false,
    "name": "cell3"
   },
   "source": [
    "### Image to Text -- Let's use this image and convert it to text.\n",
    "\n",
    "![image](https://huggingface.co/spaces/rajistics/llamavision/resolve/main/llama.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebd20a4a-eede-4142-8b42-67b11e3ee9a2",
   "metadata": {
    "collapsed": false,
    "name": "cell13"
   },
   "source": [
    "Let's makes this fun by asking us to convert the text description into a haiku"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11765634-9e48-4171-a417-7424bfd6d615",
   "metadata": {
    "codeCollapsed": false,
    "collapsed": false,
    "language": "python",
    "name": "open_image"
   },
   "outputs": [],
   "source": [
    "url = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg\"\n",
    "url = \"https://huggingface.co/spaces/rajistics/llamavision/resolve/main/llama.png\"\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": [\n",
    "        {\"type\": \"image\"},\n",
    "        {\"type\": \"text\", \"text\": \"If I had to write a haiku for this one, it would be: \"}\n",
    "    ]}\n",
    "]\n",
    "\n",
    "image = Image.open(requests.get(url, stream=True).raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38e94137-835d-4f75-8145-8779dc812261",
   "metadata": {
    "collapsed": false,
    "language": "python",
    "name": "transform_input"
   },
   "outputs": [],
   "source": [
    "input_text = processor.apply_chat_template(messages, add_generation_prompt=True)\n",
    "inputs = processor(image, input_text, return_tensors=\"pt\").to(model.device)\n",
    "\n",
    "output = model.generate(**inputs, max_new_tokens=300)\n",
    "print(processor.decode(output[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2131cf7-b2c2-4ed5-9734-666420e94cd9",
   "metadata": {
    "collapsed": false,
    "name": "cell11"
   },
   "source": [
    "#### Results\n",
    "\n",
    "<|begin_of_text|><|begin_of_text|><|start_header_id|>user<|end_header_id|>  \n",
    "<|image|>If I had to write a haiku for this one, it would be: <|eot_id|><|start_header_id|>assistant<|end_header_id|>  \n",
    "**Haiku: Cool Llama in Shades**  \n",
    "In sunglasses, llama's gaze  \n",
    "Chillin' with a smile, so bright  \n",
    "Winter's chill, no fear<|eot_id|>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bc3cf19-31fb-4857-b591-5a1ec7fff2bd",
   "metadata": {
    "collapsed": false,
    "name": "cell7"
   },
   "source": [
    "### Table to Text/JSON\n",
    "\n",
    "![image](https://huggingface.co/spaces/rajistics/llamavision/resolve/main/data-table-example1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e641282-5638-4b55-84c9-5a887c724683",
   "metadata": {
    "collapsed": false,
    "language": "python",
    "name": "cell6"
   },
   "outputs": [],
   "source": [
    "url = \"https://huggingface.co/spaces/rajistics/llamavision/resolve/main/data-table-example1.png\"\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": [\n",
    "        {\"type\": \"image\"},\n",
    "        {\"type\": \"text\", \"text\": \"Parse the table into a JSON representation where the methods are keys and the datasets are subkeys. \"}\n",
    "    ]}\n",
    "]\n",
    "\n",
    "image = Image.open(requests.get(url, stream=True).raw)\n",
    "\n",
    "input_text = processor.apply_chat_template(messages, add_generation_prompt=True)\n",
    "inputs = processor(image, input_text, return_tensors=\"pt\").to(model.device)\n",
    "\n",
    "output = model.generate(**inputs, max_new_tokens=300)\n",
    "print(processor.decode(output[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "632f3138-d7d5-46d8-95cb-93063e0fb105",
   "metadata": {
    "collapsed": false,
    "name": "cell10"
   },
   "source": [
    "#### Results\n",
    "\n",
    "<|begin_of_text|><|start_header_id|>user<|end_header_id|>  \n",
    "<|image|>Parse the table into a JSON representation where the methods are keys and the datasets are subkeys. <|eot_id|><|start_header_id|>assistant<|end_header_id|>  \n",
    "Sure, here is the table data in a JSON representation:  \n",
    "```  \n",
    "{  \n",
    "  \"Salt Concentration (%)\": {  \n",
    "    \"0\": {  \n",
    "      \"Trial #1\": 77.23,  \n",
    "      \"Trial #2\": 74.50,  \n",
    "      \"Trial #3\": 64.88,  \n",
    "      \"Trial #4\": 75.27,  \n",
    "      \"Trial #5\": 54.66  \n",
    "    },  \n",
    "    \"3\": {  \n",
    "      \"Trial #1\": 85.23,  \n",
    "      \"Trial #2\": 92.82,  \n",
    "      \"Trial #3\": 78.91,  \n",
    "      \"Trial #4\": 60.71,  \n",
    "      \"Trial #5\": 57.96  \n",
    "    },  \n",
    "    \"6\": {  \n",
    "      \"Trial #1\": 88.39,  \n",
    "      \"Trial #2\": 100.05,  \n",
    "      \"Trial #3\": 73.66,  \n",
    "      \"Trial #4\": 66.51,  \n",
    "      \"Trial #5\": 64.54  \n",
    "    },  \n",
    "    \"9\": {  \n",
    "      \"Trial #1\": 80.71,  \n",
    "      \"Trial #2\": 100.05,  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1d7c8d5-dee2-4797-ae0e-d05024a5c7cb",
   "metadata": {
    "collapsed": false,
    "name": "cell9"
   },
   "source": [
    "### Document Understanding\n",
    "\n",
    "![image](https://huggingface.co/spaces/huggingface-projects/llama-3.2-vision-11B/resolve/main/examples/invoice.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afe4ae41-0ab9-4a50-b9d0-412bfe181006",
   "metadata": {
    "collapsed": false,
    "language": "python",
    "name": "cell8"
   },
   "outputs": [],
   "source": [
    "url = \"https://huggingface.co/spaces/huggingface-projects/llama-3.2-vision-11B/resolve/main/examples/invoice.png\"\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": [\n",
    "        {\"type\": \"image\"},\n",
    "        {\"type\": \"text\", \"text\": \"How long does it take from invoice date to due date? Be short and concise.\"}\n",
    "    ]}\n",
    "]\n",
    "\n",
    "image = Image.open(requests.get(url, stream=True).raw)\n",
    "\n",
    "input_text = processor.apply_chat_template(messages, add_generation_prompt=True)\n",
    "inputs = processor(image, input_text, return_tensors=\"pt\").to(model.device)\n",
    "\n",
    "output = model.generate(**inputs, max_new_tokens=300)\n",
    "print(processor.decode(output[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be310276-96ba-496f-aae1-1e50fd70ea34",
   "metadata": {
    "collapsed": false,
    "name": "cell5"
   },
   "source": [
    "#### Results \n",
    "\n",
    "<|begin_of_text|><|start_header_id|>user<|end_header_id|>  \n",
    "<|image|>How long does it take from invoice date to due date? Be short and concise.<|eot_id|><|start_header_id|>assistant<|end_header_id|>  \n",
    "To calculate the time difference between the invoice date and the due date, we need to subtract the invoice date from the due date.  \n",
    "**Invoice Date:** 11/02/2019  \n",
    "**Due Date:** 26/02/2019  \n",
    "**Calculation:**  \n",
    "*   Difference in days = Due Date - Invoice Date  \n",
    "*   Difference in days = 26/02/2019 - 11/02/2019  \n",
    "*   Difference in days = 15 days  \n",
    "The time difference between the invoice date and the due date is **15 days**.<|eot_id|>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Streamlit Notebook",
   "name": "streamlit"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}